KVM: optimize for MMIO handled
cover KVM: optimize for MMIO handled
The idea of this patchset is from Avi:
| We could cache the result of a miss in an spte by using a reserved bit, and | checking the page fault error code (or seeing if we get an ept violation or | ept misconfiguration), so if we get repeated mmio on a page, we don't need to | search the slot list/tree. | (https://lkml.org/lkml/2011/2/22/221)
The aim of this patchset is to support fast mmio emulate, it reduce searching mmio gfn from memslots which is very expensive since we need to walk all slots for mmio gfn, and the other advantage is: we can reduce guest page table walking for soft mmu.
Lockless walk shadow page table is introduced in this patchset, it is the light way to check the page fault is the real mmio page fault or something is running out of our mind.
And, if shadow_notrap_nonpresent_pte is enabled(bypass_guest_pf=1), mmio page fault and normal page fault is mixed(the reserved is set for all page fault), it has little regression, if the box can generate lots of mmio access, for example, the network server, it can disable shadow_notrap_nonpresent_pte and enable mmio pf, after all, we can enable/disable mmio pf at the runtime.
The performance test result:
Netperf (TCP_RR):
===========================
ept is enabled:
Before After 1st 709.58 734.60 2nd 715.40 723.75 3rd 713.45 724.22
ept=0 bypass_guest_pf=0:
Before After 1st 706.10 709.63 2nd 709.38 715.80 3rd 695.90 710.70
Kernbech (do not redirect output to /dev/null)
==========================
ept is enabled:
Before After 1st 2m34.749s 2m33.482s 2nd 2m34.651s 2m33.161s 3rd 2m34.543s 2m34.271s
ept=0 bypass_guest_pf=0:
Before After 1st 4m43.467s 4m41.873s 2nd 4m45.225s 4m41.668s 3rd 4m47.029s 4m40.128s
PATCH 1/15 KVM: MMU: fix walking shadow page table
Properly check the last mapping, and do not walk to the next level if last spte is met
--- arch/x86/kvm/mmu.c | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu.c +5 -4
@@ -1517,10 +1517,6 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)if (iterator->level <PT_PAGE_TABLE_LEVEL )return false; -if (iterator->level ==PT_PAGE_TABLE_LEVEL ) -if (is_large_pte (*iterator->sptep)) -return false; - iterator->index =SHADOW_PT_INDEX (iterator->addr, iterator->level); iterator->sptep = ((u64 *)__va (iterator->shadow_addr)) + iterator->index;return true;
@@ -1528,6 +1524,11 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)static void shadow_walk_next (struct kvm_shadow_walk_iterator *iterator) { +if (is_last_spte (*iterator->sptep, iterator->level)) { + iterator->level =0 ; +return ; + } + iterator->shadow_addr = *iterator->sptep &PT64_BASE_ADDR_MASK ; --iterator->level; }
Signature · 1 line
1.7.4.4
PATCH 2/15 KVM: MMU: do not update slot bitmap if spte is nonpresent
In reply to Xiao GuangrongSet slot bitmap only if the spte is present
--- arch/x86/kvm/mmu.c | 15 +++++++-------- 1 files changed, 7 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu.c +7 -8
@@ -743,9 +743,6 @@ static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)struct kvm_mmu_page *sp;unsigned long *rmapp; -if (!is_rmap_spte (*spte)) -return 0 ; - sp =page_header (__pa (spte));kvm_mmu_page_set_gfn (sp, spte - sp->spt, gfn); rmapp =gfn_to_rmap (vcpu->kvm, gfn, sp->role.level);
@@ -2078,11 +2075,13 @@ static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep,if (!was_rmapped &&is_large_pte (*sptep)) ++vcpu->kvm->stat.lpages; -page_header_update_slot (vcpu->kvm, sptep, gfn); -if (!was_rmapped) { - rmap_count =rmap_add (vcpu, sptep, gfn); -if (rmap_count >RMAP_RECYCLE_THRESHOLD ) -rmap_recycle (vcpu, sptep, gfn); +if (is_shadow_present_pte (*sptep)) { +page_header_update_slot (vcpu->kvm, sptep, gfn); +if (!was_rmapped) { + rmap_count =rmap_add (vcpu, sptep, gfn); +if (rmap_count >RMAP_RECYCLE_THRESHOLD ) +rmap_recycle (vcpu, sptep, gfn); + } }kvm_release_pfn_clean (pfn);if (speculative) {
Signature · 1 line
1.7.4.4
PATCH 3/15 KVM: x86: avoid unnecessarily guest page table walking
In reply to Xiao GuangrongWe already get the guest physical address, so use it to read guest data directly to avoid walking guest page table again
--- arch/x86/kvm/x86.c | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +1 -2
@@ -3930,8 +3930,7 @@ static int emulator_read_emulated(struct x86_emulate_ctxt *ctxt,if ((gpa &PAGE_MASK ) ==APIC_DEFAULT_PHYS_BASE ) goto mmio; -if (kvm_read_guest_virt (ctxt, addr, val, bytes, exception) - ==X86EMUL_CONTINUE ) +if (!kvm_read_guest (vcpu->kvm, gpa, val, bytes))return X86EMUL_CONTINUE ; mmio:
Signature · 1 line
1.7.4.4
PATCH 4/15 KVM: MMU: cache mmio info on page fault path
In reply to Xiao GuangrongIf the page fault is caused by mmio, we can cache the mmio info, later, we do not need to walk guest page table and quickly know it is a mmio fault while we emulate the mmio instruction
--- arch/x86/include/asm/kvm_host.h | 5 +++ arch/x86/kvm/mmu.c | 21 +++++---------- arch/x86/kvm/mmu.h | 23 +++++++++++++++++ arch/x86/kvm/paging_tmpl.h | 21 ++++++++++----- arch/x86/kvm/x86.c | 52 ++++++++++++++++++++++++++++++-------- arch/x86/kvm/x86.h | 36 +++++++++++++++++++++++++++ 6 files changed, 126 insertions(+), 32 deletions(-)
arch/x86/include/asm/kvm_host.h +5 -0
@@ -414,6 +414,11 @@ struct kvm_vcpu_arch {u64 mcg_ctl;u64 *mce_banks; +/* Cache MMIO info */ +u64 mmio_gva; +unsigned access; +gfn_t mmio_gfn; +/* used for guest single stepping over the given code position */ unsigned long singlestep_rip;
arch/x86/kvm/mmu.c +7 -14
@@ -217,11 +217,6 @@ void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask, }EXPORT_SYMBOL_GPL (kvm_mmu_set_mask_ptes); -static bool is_write_protection (struct kvm_vcpu *vcpu) -{ -return kvm_read_cr0_bits (vcpu,X86_CR0_WP ); -} -static int is_cpuid_PSE36 (void ) {return 1 ;
@@ -243,11 +238,6 @@ static int is_large_pte(u64 pte)return pte &PT_PAGE_SIZE_MASK ; } -static int is_writable_pte (unsigned long pte) -{ -return pte &PT_WRITABLE_MASK ; -} -static int is_dirty_gpte (unsigned long pte) {return pte &PT_DIRTY_MASK ;
@@ -2238,15 +2228,17 @@ static void kvm_send_hwpoison_signal(unsigned long address, struct task_struct *send_sig_info (SIGBUS , &info, tsk); } -static int kvm_handle_bad_page (struct kvm *kvm,gfn_t gfn,pfn_t pfn) +static int kvm_handle_bad_page (struct kvm_vcpu *vcpu,gva_t gva, +unsigned access,gfn_t gfn,pfn_t pfn) {kvm_release_pfn_clean (pfn);if (is_hwpoison_pfn (pfn)) { -kvm_send_hwpoison_signal (gfn_to_hva (kvm, gfn), current); +kvm_send_hwpoison_signal (gfn_to_hva (vcpu->kvm, gfn), current);return 0 ; }else if (is_fault_pfn (pfn))return -EFAULT ; +vcpu_cache_mmio_info (vcpu, gva, gfn, access);return 1 ; }
@@ -2328,7 +2320,7 @@ static int nonpaging_map(struct kvm_vcpu *vcpu, gva_t v, int write, gfn_t gfn,/* mmio */ if (is_error_pfn (pfn)) -return kvm_handle_bad_page (vcpu->kvm, gfn, pfn); +return kvm_handle_bad_page (vcpu, v,ACC_ALL , gfn, pfn);spin_lock (&vcpu->kvm->mmu_lock);if (mmu_notifier_retry (vcpu, mmu_seq))
@@ -2555,6 +2547,7 @@ static void mmu_sync_roots(struct kvm_vcpu *vcpu)if (!VALID_PAGE (vcpu->arch.mmu.root_hpa))return ; +vcpu_clear_mmio_info (vcpu, ~0ull );trace_kvm_mmu_audit (vcpu,AUDIT_PRE_SYNC );if (vcpu->arch.mmu.root_level ==PT64_ROOT_LEVEL ) {hpa_t root = vcpu->arch.mmu.root_hpa;
@@ -2701,7 +2694,7 @@ static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa, u32 error_code,/* mmio */ if (is_error_pfn (pfn)) -return kvm_handle_bad_page (vcpu->kvm, gfn, pfn); +return kvm_handle_bad_page (vcpu,0 ,0 , gfn, pfn);spin_lock (&vcpu->kvm->mmu_lock);if (mmu_notifier_retry (vcpu, mmu_seq)) goto out_unlock;
arch/x86/kvm/mmu.h +23 -0
@@ -76,4 +76,27 @@ static inline int is_present_gpte(unsigned long pte) return pte & PT_PRESENT_MASK; } +static inline int is_writable_pte (unsigned long pte) +{ +return pte &PT_WRITABLE_MASK ; +} + +static inline bool is_write_protection (struct kvm_vcpu *vcpu) +{ +return kvm_read_cr0_bits (vcpu,X86_CR0_WP ); +} + +static inline bool check_write_user_access (struct kvm_vcpu *vcpu, +bool write_fault,bool user_fault, +unsigned long pte) +{ +if (unlikely (write_fault && !is_writable_pte (pte) + && (user_fault ||is_write_protection (vcpu)))) +return false; + +if (unlikely (user_fault && !(pte &PT_USER_MASK ))) +return false; + +return true; +}#endif
arch/x86/kvm/paging_tmpl.h +14 -7
@@ -201,11 +201,8 @@ walk:break ; } -if (unlikely (write_fault && !is_writable_pte (pte) - && (user_fault ||is_write_protection (vcpu)))) - eperm = true; - -if (unlikely (user_fault && !(pte &PT_USER_MASK ))) +if (!check_write_user_access (vcpu, write_fault, user_fault, + pte)) eperm = true;#if PTTYPE ==64
@@ -624,8 +621,16 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gva_t addr, u32 error_code,return 0 ;/* mmio */ -if (is_error_pfn (pfn)) -return kvm_handle_bad_page (vcpu->kvm, walker.gfn, pfn); +if (is_error_pfn (pfn)) { +unsigned access = walker.pte_access; +bool dirty =is_dirty_gpte (walker.ptes[walker.level -1 ]); + +if (dirty) + access &= ~ACC_WRITE_MASK ; + +return kvm_handle_bad_page (vcpu,mmu_is_nested (vcpu) ?0 : + addr, access, walker.gfn, pfn); + }spin_lock (&vcpu->kvm->mmu_lock);if (mmu_notifier_retry (vcpu, mmu_seq))
@@ -665,6 +670,8 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva)u64 *sptep;int need_flush =0 ; +vcpu_clear_mmio_info (vcpu, gva); +spin_lock (&vcpu->kvm->mmu_lock);for_each_shadow_entry (vcpu, gva, iterator) {
arch/x86/kvm/x86.c +41 -11
83 lines not on this page · raw message
arch/x86/kvm/x86.h +36 -0
@@ -75,6 +75,42 @@ static inline u32 bit(int bitno)return 1 << (bitno &31 ); } +static inline void vcpu_cache_mmio_info (struct kvm_vcpu *vcpu, +gva_t gva,gfn_t gfn,unsigned access) +{ + vcpu->arch.mmio_gva = gva &PAGE_MASK ; + vcpu->arch.access = access; + vcpu->arch.mmio_gfn = gfn; +} + +/* +* Clear the mmio cache info for the given gva, +* specially, if gva is ~0ul, we clear all mmio cache info. +*/ +static inline void vcpu_clear_mmio_info (struct kvm_vcpu *vcpu,gva_t gva) +{ +if (gva != (~0ul ) && vcpu->arch.mmio_gva != (gva &PAGE_MASK )) +return ; + + vcpu->arch.mmio_gva =0 ; +} + +static inline bool vcpu_match_mmio_gva (struct kvm_vcpu *vcpu,unsigned long gva) +{ +if (vcpu->arch.mmio_gva && vcpu->arch.mmio_gva == (gva &PAGE_MASK )) +return true; + +return false; +} + +static inline bool vcpu_match_mmio_gpa (struct kvm_vcpu *vcpu,gpa_t gpa) +{ +if (vcpu->arch.mmio_gfn && vcpu->arch.mmio_gfn == gpa >>PAGE_SHIFT ) +return true; + +return false; +} +void kvm_before_handle_nmi (struct kvm_vcpu *vcpu);void kvm_after_handle_nmi (struct kvm_vcpu *vcpu);int kvm_inject_realmode_interrupt (struct kvm_vcpu *vcpu,int irq,int inc_eip);
Signature · 1 line
1.7.4.4
PATCH 5/15 KVM: MMU: optimize to handle dirty bit
In reply to Xiao GuangrongIf dirty bit is not set, we can make the pte access read-only to avoid handing dirty bit everywhere
--- arch/x86/kvm/mmu.c | 13 ++++++------- arch/x86/kvm/paging_tmpl.h | 30 ++++++++++-------------------- 2 files changed, 16 insertions(+), 27 deletions(-)
arch/x86/kvm/mmu.c +6 -7
@@ -1923,7 +1923,7 @@ static int mmu_need_write_protect(struct kvm_vcpu *vcpu, gfn_t gfn,static int set_spte (struct kvm_vcpu *vcpu,u64 *sptep,unsigned pte_access,int user_fault, -int write_fault,int dirty,int level, +int write_fault,int level,gfn_t gfn,pfn_t pfn,bool speculative,bool can_unsync,bool host_writable) {
@@ -1938,8 +1938,7 @@ static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep, spte =PT_PRESENT_MASK ;if (!speculative) spte |= shadow_accessed_mask; -if (!dirty) - pte_access &= ~ACC_WRITE_MASK ; +if (pte_access &ACC_EXEC_MASK ) spte |= shadow_x_mask;else
@@ -2014,7 +2013,7 @@ done:static void mmu_set_spte (struct kvm_vcpu *vcpu,u64 *sptep,unsigned pt_access,unsigned pte_access, -int user_fault,int write_fault,int dirty, +int user_fault,int write_fault,int *ptwrite,int level,gfn_t gfn,pfn_t pfn,bool speculative,bool host_writable)
@@ -2050,7 +2049,7 @@ static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep, }if (set_spte (vcpu, sptep, pte_access, user_fault, write_fault, - dirty, level, gfn, pfn, speculative, true, + level, gfn, pfn, speculative, true, host_writable)) {if (write_fault) *ptwrite =1 ;
@@ -2120,7 +2119,7 @@ static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu,for (i =0 ; i < ret; i++, gfn++, start++)mmu_set_spte (vcpu, start,ACC_ALL , - access,0 ,0 ,1 ,NULL , + access,0 ,0 ,NULL , sp->role.level, gfn,page_to_pfn (pages[i]), true, true);
@@ -2184,7 +2183,7 @@ static int __direct_map(struct kvm_vcpu *vcpu, gpa_t v, int write,unsigned pte_access =ACC_ALL ;mmu_set_spte (vcpu, iterator.sptep,ACC_ALL , pte_access, -0 , write,1 , &pt_write, +0 , write, &pt_write, level, gfn, pfn, prefault, map_writable);direct_pte_prefetch (vcpu, iterator.sptep); ++vcpu->stat.pf_fixed;
arch/x86/kvm/paging_tmpl.h +10 -20
@@ -106,6 +106,9 @@ static unsigned FNAME(gpte_access)(struct kvm_vcpu *vcpu, pt_element_t gpte)unsigned access; access = (gpte & (PT_WRITABLE_MASK |PT_USER_MASK )) |ACC_EXEC_MASK ; +if (!is_dirty_gpte (gpte)) + access &= ~ACC_WRITE_MASK ; +#if PTTYPE ==64 if (vcpu->arch.mmu.nx) access &= ~(gpte >>PT64_NX_SHIFT );
@@ -378,7 +381,7 @@ static void FNAME(update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,* vcpu->arch.update_pte.pfn was fetched from get_user_pages(write = 1). */ mmu_set_spte (vcpu, spte, sp->role.access, pte_access,0 ,0 , -is_dirty_gpte (gpte),NULL ,PT_PAGE_TABLE_LEVEL , +NULL ,PT_PAGE_TABLE_LEVEL ,gpte_to_gfn (gpte), pfn, true, true); }
@@ -429,7 +432,6 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,unsigned pte_access;gfn_t gfn;pfn_t pfn; -bool dirty;if (spte == sptep)continue ;
@@ -444,16 +446,15 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw, pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte); gfn =gpte_to_gfn (gpte); - dirty =is_dirty_gpte (gpte); pfn =pte_prefetch_gfn_to_pfn (vcpu, gfn, - (pte_access &ACC_WRITE_MASK ) && dirty); + pte_access &ACC_WRITE_MASK );if (is_error_pfn (pfn)) {kvm_release_pfn_clean (pfn);break ; }mmu_set_spte (vcpu, spte, sp->role.access, pte_access,0 ,0 , - dirty,NULL ,PT_PAGE_TABLE_LEVEL , gfn, +NULL ,PT_PAGE_TABLE_LEVEL , gfn, pfn, true, true); } }
@@ -469,7 +470,6 @@ static u64 *FNAME(fetch)(struct kvm_vcpu *vcpu, gva_t addr, {unsigned access = gw->pt_access;struct kvm_mmu_page *sp =NULL ; -bool dirty =is_dirty_gpte (gw->ptes[gw->level -1 ]);int top_level;unsigned direct_access;struct kvm_shadow_walk_iterator it;
@@ -478,8 +478,6 @@ static u64 *FNAME(fetch)(struct kvm_vcpu *vcpu, gva_t addr,return NULL ; direct_access = gw->pt_access & gw->pte_access; -if (!dirty) - direct_access &= ~ACC_WRITE_MASK ; top_level = vcpu->arch.mmu.root_level;if (top_level ==PT32E_ROOT_LEVEL )
@@ -538,7 +536,7 @@ static u64 *FNAME(fetch)(struct kvm_vcpu *vcpu, gva_t addr, }mmu_set_spte (vcpu, it.sptep, access, gw->pte_access & access, - user_fault, write_fault, dirty, ptwrite, it.level, + user_fault, write_fault, ptwrite, it.level, gw->gfn, pfn, prefault, map_writable);FNAME (pte_prefetch)(vcpu, gw, it.sptep);
@@ -621,17 +619,9 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gva_t addr, u32 error_code,return 0 ;/* mmio */ -if (is_error_pfn (pfn)) { -unsigned access = walker.pte_access; -bool dirty =is_dirty_gpte (walker.ptes[walker.level -1 ]); - -if (dirty) - access &= ~ACC_WRITE_MASK ; - +if (is_error_pfn (pfn))return kvm_handle_bad_page (vcpu,mmu_is_nested (vcpu) ?0 : - addr, access, walker.gfn, pfn); - } - + addr, walker.pte_access, walker.gfn, pfn);spin_lock (&vcpu->kvm->mmu_lock);if (mmu_notifier_retry (vcpu, mmu_seq)) goto out_unlock;
@@ -852,7 +842,7 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) host_writable = sp->spt[i] &SPTE_HOST_WRITEABLE ;set_spte (vcpu, &sp->spt[i], pte_access,0 ,0 , -is_dirty_gpte (gpte),PT_PAGE_TABLE_LEVEL , gfn, +PT_PAGE_TABLE_LEVEL , gfn,spte_to_pfn (sp->spt[i]), true, false, host_writable); }
Signature · 1 line
1.7.4.4
PATCH 6/15 KVM: MMU: cleanup for FNAME(fetch)
In reply to Xiao Guangronggw->pte_access is the final access permission, since it is unified with gw->pt_access when we walked guest page table:
FNAME(walk_addr_generic):
pte_access = pt_access & FNAME(gpte_access)(vcpu, pte);
--- arch/x86/kvm/paging_tmpl.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/paging_tmpl.h +2 -2
@@ -477,7 +477,7 @@ static u64 *FNAME(fetch)(struct kvm_vcpu *vcpu, gva_t addr,if (!is_present_gpte (gw->ptes[gw->level -1 ]))return NULL ; - direct_access = gw->pt_access & gw->pte_access; + direct_access = gw->pte_access; top_level = vcpu->arch.mmu.root_level;if (top_level ==PT32E_ROOT_LEVEL )
@@ -535,7 +535,7 @@ static u64 *FNAME(fetch)(struct kvm_vcpu *vcpu, gva_t addr,link_shadow_page (it.sptep, sp); } -mmu_set_spte (vcpu, it.sptep, access, gw->pte_access & access, +mmu_set_spte (vcpu, it.sptep, access, gw->pte_access, user_fault, write_fault, ptwrite, it.level, gw->gfn, pfn, prefault, map_writable);FNAME (pte_prefetch)(vcpu, gw, it.sptep);
Signature · 1 line
1.7.4.4
PATCH 7/15 KVM: MMU: rename 'pt_write' to 'emulate'
In reply to Xiao GuangrongIf 'pt_write' is true, we need to emulate the fault. And in later patch, we need to emulate the fault even though it is not a pt_write event, so rename it to better fit the meaning
--- arch/x86/kvm/mmu.c | 10 +++++----- arch/x86/kvm/paging_tmpl.h | 16 ++++++++-------- 2 files changed, 13 insertions(+), 13 deletions(-)
arch/x86/kvm/mmu.c +5 -5
@@ -2014,7 +2014,7 @@ done:static void mmu_set_spte (struct kvm_vcpu *vcpu,u64 *sptep,unsigned pt_access,unsigned pte_access,int user_fault,int write_fault, -int *ptwrite,int level,gfn_t gfn, +int *emulate,int level,gfn_t gfn,pfn_t pfn,bool speculative,bool host_writable) {
@@ -2052,7 +2052,7 @@ static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep, level, gfn, pfn, speculative, true, host_writable)) {if (write_fault) - *ptwrite =1 ; + *emulate =1 ;kvm_mmu_flush_tlb (vcpu); }
@@ -2175,7 +2175,7 @@ static int __direct_map(struct kvm_vcpu *vcpu, gpa_t v, int write, {struct kvm_shadow_walk_iterator iterator;struct kvm_mmu_page *sp; -int pt_write =0 ; +int emulate =0 ;gfn_t pseudo_gfn;for_each_shadow_entry (vcpu, (u64 )gfn <<PAGE_SHIFT , iterator) {
@@ -2183,7 +2183,7 @@ static int __direct_map(struct kvm_vcpu *vcpu, gpa_t v, int write,unsigned pte_access =ACC_ALL ;mmu_set_spte (vcpu, iterator.sptep,ACC_ALL , pte_access, -0 , write, &pt_write, +0 , write, &emulate, level, gfn, pfn, prefault, map_writable);direct_pte_prefetch (vcpu, iterator.sptep); ++vcpu->stat.pf_fixed;
@@ -2211,7 +2211,7 @@ static int __direct_map(struct kvm_vcpu *vcpu, gpa_t v, int write, | shadow_accessed_mask); } } -return pt_write; +return emulate; }static void kvm_send_hwpoison_signal (unsigned long address,struct task_struct *tsk)
arch/x86/kvm/paging_tmpl.h +8 -8
@@ -465,7 +465,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,static u64 *FNAME (fetch )(struct kvm_vcpu *vcpu,gva_t addr,struct guest_walker *gw,int user_fault,int write_fault,int hlevel, -int *ptwrite,pfn_t pfn,bool map_writable, +int *emulate,pfn_t pfn,bool map_writable,bool prefault) {unsigned access = gw->pt_access;
@@ -536,7 +536,7 @@ static u64 *FNAME(fetch)(struct kvm_vcpu *vcpu, gva_t addr, }mmu_set_spte (vcpu, it.sptep, access, gw->pte_access, - user_fault, write_fault, ptwrite, it.level, + user_fault, write_fault, emulate, it.level, gw->gfn, pfn, prefault, map_writable);FNAME (pte_prefetch)(vcpu, gw, it.sptep);
@@ -570,7 +570,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gva_t addr, u32 error_code,int user_fault = error_code &PFERR_USER_MASK ;struct guest_walker walker;u64 *sptep; -int write_pt =0 ; +int emulate =0 ;int r;pfn_t pfn;int level =PT_PAGE_TABLE_LEVEL ;
@@ -631,19 +631,19 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gva_t addr, u32 error_code,if (!force_pt_level)transparent_hugepage_adjust (vcpu, &walker.gfn, &pfn, &level); sptep =FNAME (fetch)(vcpu, addr, &walker, user_fault, write_fault, - level, &write_pt, pfn, map_writable, prefault); + level, &emulate, pfn, map_writable, prefault); (void )sptep; -pgprintk ("%s: shadow pte %p %llx ptwrite %d\n" , __func__, - sptep, *sptep, write_pt); +pgprintk ("%s: shadow pte %p %llx emulate %d\n" , __func__, + sptep, *sptep, emulate); -if (!write_pt) +if (!emulate) vcpu->arch.last_pt_write_count =0 ;/* reset fork detector */ ++vcpu->stat.pf_fixed;trace_kvm_mmu_audit (vcpu,AUDIT_POST_PAGE_FAULT );spin_unlock (&vcpu->kvm->mmu_lock); -return write_pt; +return emulate; out_unlock:spin_unlock (&vcpu->kvm->mmu_lock);
Signature · 1 line
1.7.4.4
PATCH 8/15 KVM: MMU: count used shadow pages on preparing path
In reply to Xiao GuangrongMove counting used shadow pages from committing path to preparing path to reduce tlb flush on some paths
--- arch/x86/kvm/mmu.c | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu.c +5 -5
@@ -1039,7 +1039,7 @@ static inline void kvm_mod_used_mmu_pages(struct kvm *kvm, int nr)percpu_counter_add (&kvm_total_used_mmu_pages, nr); } -static void kvm_mmu_free_page (struct kvm *kvm,struct kvm_mmu_page *sp) +static void kvm_mmu_free_page (struct kvm_mmu_page *sp) {ASSERT (is_empty_shadow_page (sp->spt));hlist_del (&sp->hash_link);
@@ -1048,7 +1048,6 @@ static void kvm_mmu_free_page(struct kvm *kvm, struct kvm_mmu_page *sp)if (!sp->role.direct)free_page ((unsigned long )sp->gfns);kmem_cache_free (mmu_page_header_cache, sp); -kvm_mod_used_mmu_pages (kvm,-1 ); }static unsigned kvm_page_table_hashfn (gfn_t gfn)
@@ -1655,6 +1654,7 @@ static int kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,/* Count self */ ret++;list_move (&sp->link, invalid_list); +kvm_mod_used_mmu_pages (kvm,-1 ); }else {list_move (&sp->link, &kvm->arch.active_mmu_pages);kvm_reload_remote_mmus (kvm);
@@ -1678,7 +1678,7 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm,do { sp = list_first_entry(invalid_list, struct kvm_mmu_page, link);WARN_ON (!sp->role.invalid || sp->root_count); -kvm_mmu_free_page (kvm, sp); +kvm_mmu_free_page (sp); }while (!list_empty (invalid_list)); }
@@ -1704,8 +1704,8 @@ void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned int goal_nr_mmu_pages) page =container_of (kvm->arch.active_mmu_pages.prev, struct kvm_mmu_page, link);kvm_mmu_prepare_zap_page (kvm, page, &invalid_list); -kvm_mmu_commit_zap_page (kvm, &invalid_list); } +kvm_mmu_commit_zap_page (kvm, &invalid_list); goal_nr_mmu_pages = kvm->arch.n_used_mmu_pages; }
@@ -3290,9 +3290,9 @@ void __kvm_mmu_free_some_pages(struct kvm_vcpu *vcpu) sp =container_of (vcpu->kvm->arch.active_mmu_pages.prev, struct kvm_mmu_page, link);kvm_mmu_prepare_zap_page (vcpu->kvm, sp, &invalid_list); -kvm_mmu_commit_zap_page (vcpu->kvm, &invalid_list); ++vcpu->kvm->stat.mmu_recycled; } +kvm_mmu_commit_zap_page (vcpu->kvm, &invalid_list); } int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t cr2, u32 error_code,
Signature · 1 line
1.7.4.4
PATCH 9/15 KVM: MMU: split kvm_mmu_free_page
In reply to Xiao GuangrongSplit kvm_mmu_free_page to kvm_mmu_free_lock_parts and kvm_mmu_free_unlock_parts
One is used to free the parts which is under mmu lock and the other is used to free the parts which can allow be freed out of mmu lock
It is used by later patch
--- arch/x86/kvm/mmu.c | 16 +++++++++++++--- 1 files changed, 13 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu.c +13 -3
@@ -1039,17 +1039,27 @@ static inline void kvm_mod_used_mmu_pages(struct kvm *kvm, int nr)percpu_counter_add (&kvm_total_used_mmu_pages, nr); } -static void kvm_mmu_free_page (struct kvm_mmu_page *sp) +static void kvm_mmu_free_lock_parts (struct kvm_mmu_page *sp) {ASSERT (is_empty_shadow_page (sp->spt));hlist_del (&sp->hash_link); -list_del (&sp->link); -free_page ((unsigned long )sp->spt);if (!sp->role.direct)free_page ((unsigned long )sp->gfns); +} + +static void kvm_mmu_free_unlock_parts (struct kvm_mmu_page *sp) +{ +list_del (&sp->link); +free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); } +static void kvm_mmu_free_page (struct kvm_mmu_page *sp) +{ +kvm_mmu_free_lock_parts (sp); +kvm_mmu_free_unlock_parts (sp); +} +static unsigned kvm_page_table_hashfn (gfn_t gfn) {return gfn & ((1 <<KVM_MMU_HASH_SHIFT ) -1 );
Signature · 1 line
1.7.4.4
PATCH 10/15 KVM: MMU: lockless walking shadow page table
In reply to Xiao GuangrongUsing rcu to protect shadow pages table to be freed, so we can safely walk it, it should run fast and is needed by mmio page fault
--- arch/x86/include/asm/kvm_host.h | 4 ++ arch/x86/kvm/mmu.c | 79 ++++++++++++++++++++++++++++++--------- arch/x86/kvm/mmu.h | 4 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 69 insertions(+), 20 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -232,6 +232,8 @@ struct kvm_mmu_page {unsigned int unsync_children;unsigned long parent_ptes;/* Reverse mapping for parent_pte */ DECLARE_BITMAP (unsync_child_bitmap,512 ); + +struct rcu_head rcu; };struct kvm_pv_mmu_op_buffer {
@@ -478,6 +480,8 @@ struct kvm_arch {u64 hv_guest_os_id;u64 hv_hypercall; +atomic_t reader_counter; +#ifdef CONFIG_KVM_MMU_AUDIT int audit_point;#endif
arch/x86/kvm/mmu.c +61 -18
103 lines not on this page · raw message
arch/x86/kvm/mmu.h +3 -1
@@ -48,7 +48,9 @@#define PFERR_RSVD_MASK (1U << 3)#define PFERR_FETCH_MASK (1U << 4) -int kvm_mmu_get_spte_hierarchy (struct kvm_vcpu *vcpu,u64 addr,u64 sptes[4 ]); +int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr, +u64 sptes[4 ]); +int kvm_init_shadow_mmu (struct kvm_vcpu *vcpu,struct kvm_mmu *context);static inline unsigned int kvm_mmu_available_pages (struct kvm *kvm)
arch/x86/kvm/vmx.c +1 -1
@@ -4681,7 +4681,7 @@ static int handle_ept_misconfig(struct kvm_vcpu *vcpu)printk (KERN_ERR "EPT: Misconfiguration.\n" );printk (KERN_ERR "EPT: GPA: 0x%llx\n" , gpa); - nr_sptes =kvm_mmu_get_spte_hierarchy (vcpu, gpa, sptes); + nr_sptes =kvm_mmu_walk_shadow_page_lockless (vcpu, gpa, sptes);for (i =PT64_ROOT_LEVEL ; i >PT64_ROOT_LEVEL - nr_sptes; --i)ept_misconfig_inspect_spte (vcpu, sptes[i-1 ], i);
Signature · 1 line
1.7.4.4
PATCH 11/15 KVM: MMU: filter out the mmio pfn from the fault pfn
In reply to Xiao GuangrongIf the page fault is caused by mmio, the gfn can not be found in memslots, and 'bad_pfn' is returned on gfn_to_hva path, so we can use 'bad_pfn' to identify the mmio page fault.
And, to clarify the meaning of mmio pfn, we return fault page instead of bad page when the gfn is not allowed to prefetch
--- arch/x86/kvm/mmu.c | 4 ++-- include/linux/kvm_host.h | 5 +++++ virt/kvm/kvm_main.c | 16 ++++++++++++++-- 3 files changed, 21 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu.c +2 -2
@@ -2133,8 +2133,8 @@ static pfn_t pte_prefetch_gfn_to_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, slot =gfn_to_memslot_dirty_bitmap (vcpu, gfn, no_dirty_log);if (!slot) { -get_page (bad_page); -return page_to_pfn (bad_page); +get_page (fault_page); +return page_to_pfn (fault_page); } hva =gfn_to_hva_memslot (slot, gfn);
include/linux/kvm_host.h +5 -0
@@ -326,12 +326,17 @@ static inline struct kvm_memslots *kvm_memslots(struct kvm *kvm)static inline int is_error_hpa (hpa_t hpa) {return hpa >>HPA_MSB ; }extern struct page *bad_page; +extern struct page *fault_page; +extern pfn_t bad_pfn; +extern pfn_t fault_pfn;int is_error_page (struct page *page);int is_error_pfn (pfn_t pfn);int is_hwpoison_pfn (pfn_t pfn);int is_fault_pfn (pfn_t pfn); +int is_mmio_pfn (pfn_t pfn); +int is_invalid_pfn (pfn_t pfn);int kvm_is_error_hva (unsigned long addr); int kvm_set_memory_region(struct kvm *kvm, struct kvm_userspace_memory_region *mem,
virt/kvm/kvm_main.c +14 -2
@@ -97,8 +97,8 @@ static bool largepages_enabled = true;static struct page *hwpoison_page;static pfn_t hwpoison_pfn; -static struct page *fault_page; -static pfn_t fault_pfn; +struct page *fault_page; +pfn_t fault_pfn;inline int kvm_is_mmio_pfn (pfn_t pfn) {
@@ -926,6 +926,18 @@ int is_fault_pfn(pfn_t pfn) }EXPORT_SYMBOL_GPL (is_fault_pfn); +int is_mmio_pfn (pfn_t pfn) +{ +return pfn == bad_pfn; +} +EXPORT_SYMBOL_GPL (is_mmio_pfn); + +int is_invalid_pfn (pfn_t pfn) +{ +return pfn == hwpoison_pfn || pfn == fault_pfn; +} +EXPORT_SYMBOL_GPL (is_invalid_pfn); +static inline unsigned long bad_hva (void ) {return PAGE_OFFSET ;
Signature · 1 line
1.7.4.4
PATCH 12/15 KVM: MMU: abstract some functions to handle fault pfn
In reply to Xiao GuangrongIntroduce handle_abnormal_pfn to handle fault pfn on page fault path, introduce mmu_invalid_pfn to handle fault pfn on prefetch path
It is the preparing work for mmio page fault support
--- arch/x86/kvm/mmu.c | 47 ++++++++++++++++++++++++++++++++----------- arch/x86/kvm/paging_tmpl.h | 12 +++++----- 2 files changed, 41 insertions(+), 18 deletions(-)
arch/x86/kvm/mmu.c +35 -12
@@ -2269,18 +2269,15 @@ static void kvm_send_hwpoison_signal(unsigned long address, struct task_struct *send_sig_info (SIGBUS , &info, tsk); } -static int kvm_handle_bad_page (struct kvm_vcpu *vcpu,gva_t gva, -unsigned access,gfn_t gfn,pfn_t pfn) +static int kvm_handle_bad_page (struct kvm_vcpu *vcpu,gfn_t gfn,pfn_t pfn) {kvm_release_pfn_clean (pfn);if (is_hwpoison_pfn (pfn)) {kvm_send_hwpoison_signal (gfn_to_hva (vcpu->kvm, gfn), current);return 0 ; - }else if (is_fault_pfn (pfn)) -return -EFAULT ; + } -vcpu_cache_mmio_info (vcpu, gva, gfn, access); -return 1 ; +return -EFAULT ; } static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
@@ -2325,6 +2322,33 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, } } +static bool mmu_invalid_pfn (pfn_t pfn) +{ +return unlikely (is_invalid_pfn (pfn) ||is_mmio_pfn (pfn)); +} + +static bool handle_abnormal_pfn (struct kvm_vcpu *vcpu,gva_t gva,gfn_t gfn, +pfn_t pfn,unsigned access,int *ret_val) +{ +bool ret = true; + +/* The pfn is invalid, report the error! */ +if (unlikely (is_invalid_pfn (pfn))) { + *ret_val =kvm_handle_bad_page (vcpu, gfn, pfn); + goto exit; + } + +if (unlikely (is_mmio_pfn (pfn))) { +vcpu_cache_mmio_info (vcpu, gva, gfn,ACC_ALL ); + *ret_val =1 ; + goto exit; + } + + ret = false; +exit: +return ret; +} +static bool try_async_pf (struct kvm_vcpu *vcpu,bool prefault,gfn_t gfn,gva_t gva,pfn_t *pfn,bool write,bool *writable);
@@ -2359,9 +2383,8 @@ static int nonpaging_map(struct kvm_vcpu *vcpu, gva_t v, int write, gfn_t gfn,if (try_async_pf (vcpu, prefault, gfn, v, &pfn, write, &map_writable))return 0 ; -/* mmio */ -if (is_error_pfn (pfn)) -return kvm_handle_bad_page (vcpu, v,ACC_ALL , gfn, pfn); +if (handle_abnormal_pfn (vcpu, v, gfn, pfn,ACC_ALL , &r)) +return r;spin_lock (&vcpu->kvm->mmu_lock);if (mmu_notifier_retry (vcpu, mmu_seq))
@@ -2762,9 +2785,9 @@ static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa, u32 error_code,if (try_async_pf (vcpu, prefault, gfn, gpa, &pfn, write, &map_writable))return 0 ; -/* mmio */ -if (is_error_pfn (pfn)) -return kvm_handle_bad_page (vcpu,0 ,0 , gfn, pfn); +if (handle_abnormal_pfn (vcpu,0 , gfn, pfn,ACC_ALL , &r)) +return r; +spin_lock (&vcpu->kvm->mmu_lock);if (mmu_notifier_retry (vcpu, mmu_seq)) goto out_unlock;
arch/x86/kvm/paging_tmpl.h +6 -6
@@ -371,7 +371,7 @@ static void FNAME(update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,pgprintk ("%s: gpte %llx spte %p\n" , __func__, (u64 )gpte, spte); pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte); pfn =gfn_to_pfn_atomic (vcpu->kvm,gpte_to_gfn (gpte)); -if (is_error_pfn (pfn)) { +if (mmu_invalid_pfn (pfn)) {kvm_release_pfn_clean (pfn);return ; }
@@ -448,7 +448,7 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw, gfn =gpte_to_gfn (gpte); pfn =pte_prefetch_gfn_to_pfn (vcpu, gfn, pte_access &ACC_WRITE_MASK ); -if (is_error_pfn (pfn)) { +if (mmu_invalid_pfn (pfn)) {kvm_release_pfn_clean (pfn);break ; }
@@ -618,10 +618,10 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gva_t addr, u32 error_code, &map_writable))return 0 ; -/* mmio */ -if (is_error_pfn (pfn)) -return kvm_handle_bad_page (vcpu,mmu_is_nested (vcpu) ?0 : - addr, walker.pte_access, walker.gfn, pfn); +if (handle_abnormal_pfn (vcpu,mmu_is_nested (vcpu) ?0 : addr, + walker.gfn, pfn, walker.pte_access, &r)) +return r; +spin_lock (&vcpu->kvm->mmu_lock);if (mmu_notifier_retry (vcpu, mmu_seq)) goto out_unlock;
Signature · 1 line
1.7.4.4
PATCH 13/15 KVM: VMX: modify the default value of nontrap shadow pte
In reply to Xiao GuangrongModify the default value to identify nontrap shadow pte and mmio shadow pte whill will be introduced in later patch
--- arch/x86/kvm/vmx.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kvm/vmx.c +1 -1
@@ -7110,7 +7110,7 @@ static int __init vmx_init(void)kvm_disable_tdp ();if (bypass_guest_pf) -kvm_mmu_set_nonpresent_ptes (~0xffeull ,0ull ); +kvm_mmu_set_nonpresent_ptes (0xfull <<49 |1ull ,0ull );return 0 ;
Signature · 1 line
1.7.4.4
PATCH 14/15 KVM: MMU: mmio page fault support
In reply to Xiao GuangrongThe idea is from Avi:
| We could cache the result of a miss in an spte by using a reserved bit, and | checking the page fault error code (or seeing if we get an ept violation or | ept misconfiguration), so if we get repeated mmio on a page, we don't need to | search the slot list/tree. | (https://lkml.org/lkml/2011/2/22/221)
When the page fault is caused by mmio, we cache the info in the shadow page table, and also set the reserved bits in the shadow page table, so if the mmio is caused again, we can quickly identify it and emulate it directly
Searching mmio gfn in memslots is heavy since we need to walk all memeslots, it can be reduced by this feature, and also avoid walking guest page table for soft mmu.
This feature can be disabled/enabled at the runtime, if shadow_notrap_nonpresent_pte is enabled, the PFER.RSVD is always set, we need to walk shadow page table for all page fault, so disable this feature if shadow_notrap_nonpresent is enabled.
--- arch/x86/kvm/mmu.c | 149 ++++++++++++++++++++++++++++++++++++++++--- arch/x86/kvm/mmu.h | 4 +- arch/x86/kvm/paging_tmpl.h | 32 +++++++++- arch/x86/kvm/vmx.c | 12 +++- 4 files changed, 180 insertions(+), 17 deletions(-)
arch/x86/kvm/mmu.c +138 -11
@@ -91,6 +91,9 @@ module_param(dbg, bool, 0644);static int oos_shadow =1 ;module_param (oos_shadow, bool,0644 ); +static int __read_mostly mmio_pf =1 ; +module_param (mmio_pf, bool,0644 ); +#ifndef MMU_DEBUG #define ASSERT (x) do { } while (0)#else
@@ -193,6 +196,44 @@ static u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */static u64 __read_mostly shadow_user_mask;static u64 __read_mostly shadow_accessed_mask;static u64 __read_mostly shadow_dirty_mask; +static u64 __read_mostly shadow_mmio_mask = (0xffull <<49 |1ULL ); + +static void __set_spte (u64 *sptep,u64 spte) +{ +set_64bit (sptep, spte); +} + +static void mark_mmio_spte (u64 *sptep,u64 gfn,unsigned access) +{ + access &=ACC_WRITE_MASK |ACC_USER_MASK ; + +__set_spte (sptep, shadow_mmio_mask | access | gfn <<PAGE_SHIFT ); +} + +static bool is_mmio_spte (u64 spte) +{ +return (spte & shadow_mmio_mask) == shadow_mmio_mask; +} + +static gfn_t get_mmio_spte_gfn (u64 spte) +{ +return (spte & ~shadow_mmio_mask) >>PAGE_SHIFT ; +} + +static unsigned get_mmio_spte_access (u64 spte) +{ +return (spte & ~shadow_mmio_mask) & ~PAGE_MASK ; +} + +static bool set_mmio_spte (u64 *sptep,gfn_t gfn,pfn_t pfn,unsigned access) +{ +if (unlikely (is_mmio_pfn (pfn))) { +mark_mmio_spte (sptep, gfn, access); +return true; + } + +return false; +}static inline u64 rsvd_bits (int s,int e) {
@@ -203,6 +244,8 @@ void kvm_mmu_set_nonpresent_ptes(u64 trap_pte, u64 notrap_pte) { shadow_trap_nonpresent_pte = trap_pte; shadow_notrap_nonpresent_pte = notrap_pte; +if (trap_pte != notrap_pte) + mmio_pf =0 ; }EXPORT_SYMBOL_GPL (kvm_mmu_set_nonpresent_ptes);
@@ -230,7 +273,8 @@ static int is_nx(struct kvm_vcpu *vcpu)static int is_shadow_present_pte (u64 pte) {return pte != shadow_trap_nonpresent_pte - && pte != shadow_notrap_nonpresent_pte; + && pte != shadow_notrap_nonpresent_pte + && !is_mmio_spte (pte); }static int is_large_pte (u64 pte)
@@ -269,11 +313,6 @@ static gfn_t pse36_gfn_delta(u32 gpte)return (gpte &PT32_DIR_PSE36_MASK ) << shift; } -static void __set_spte (u64 *sptep,u64 spte) -{ -set_64bit (sptep, spte); -} -static u64 __xchg_spte (u64 *sptep,u64 new_spte) {#ifdef CONFIG_X86_64
@@ -1972,6 +2011,9 @@ static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep,u64 spte, entry = *sptep;int ret =0 ; +if (set_mmio_spte (sptep, gfn, pfn, pte_access)) +return 0 ; +/* * We don't set the accessed bit, since we sometimes want to see * whether the guest actually used the pte (in order to detect
@@ -2098,6 +2140,9 @@ static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep,kvm_mmu_flush_tlb (vcpu); } +if (unlikely (is_mmio_spte (*sptep) && emulate)) + *emulate =1 ; +pgprintk ("%s: setting spte %llx\n" , __func__, *sptep); pgprintk("instantiating %s PTE (%s) at %llx (%llx) addr %p\n", is_large_pte(*sptep)? "2MB" : "4kB",
@@ -2324,7 +2369,10 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,static bool mmu_invalid_pfn (pfn_t pfn) { -return unlikely (is_invalid_pfn (pfn) ||is_mmio_pfn (pfn)); +if (unlikely (!mmio_pf &&is_mmio_pfn (pfn))) +return true; + +return unlikely (is_invalid_pfn (pfn)); } static bool handle_abnormal_pfn(struct kvm_vcpu *vcpu, gva_t gva, gfn_t gfn,
@@ -2340,8 +2388,10 @@ static bool handle_abnormal_pfn(struct kvm_vcpu *vcpu, gva_t gva, gfn_t gfn,if (unlikely (is_mmio_pfn (pfn))) {vcpu_cache_mmio_info (vcpu, gva, gfn,ACC_ALL ); - *ret_val =1 ; - goto exit; +if (!mmio_pf) { + *ret_val =1 ; + goto exit; + } } ret = false;
@@ -2656,7 +2706,7 @@ static gpa_t nonpaging_gva_to_gpa_nested(struct kvm_vcpu *vcpu, gva_t vaddr,return vcpu->arch.nested_mmu.translate_gpa (vcpu, vaddr, access); } -int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr, +static int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr,u64 sptes[4 ]) {struct kvm_shadow_walk_iterator iterator;
@@ -2683,7 +2733,75 @@ int kvm_mmu_walk_shadow_page_lockless(struct kvm_vcpu *vcpu, u64 addr,return nr_sptes; } -EXPORT_SYMBOL_GPL (kvm_mmu_walk_shadow_page_lockless); + +static bool quickly_check_mmio_pf (struct kvm_vcpu *vcpu,u64 addr,bool direct) +{ +if (direct &&vcpu_match_mmio_gpa (vcpu, addr)) +return true; + +if (vcpu_match_mmio_gva (vcpu, addr)) +return true; + +return false; +} + +/* +* If it is a real mmio page fault, return 1 and emulat the instruction +* directly, return 0 if it needs page fault path to fix it, -1 is +* returned if bug is detected. +*/ +int handle_mmio_page_fault_common (struct kvm_vcpu *vcpu,u64 addr, +u64 sptes[4 ],int *nr_sptes,bool direct) +{ +if (quickly_check_mmio_pf (vcpu, addr, direct)) +return 1 ; + + sptes[0 ] = shadow_trap_nonpresent_pte; + *nr_sptes =kvm_mmu_walk_shadow_page_lockless (vcpu, addr, sptes); + +if (is_mmio_spte (sptes[0 ])) { +gfn_t gfn =get_mmio_spte_gfn (sptes[0 ]); +unsigned access =get_mmio_spte_access (sptes[0 ]); + +if (direct) + addr =0 ; +vcpu_cache_mmio_info (vcpu, addr, gfn, access); +return 1 ; + } + +/* +* It's ok if the gva is remapped by other cpus on shadow guest, +* it's a BUG if the gfn is not a mmio page. +*/ +if (direct &&is_shadow_present_pte (sptes[0 ])) +return -1 ; + +/* +* It's ok if the page table is zapped by other cpus or the page +* fault is caused by shadow_trap_nonpresent_pte, let the page +* fault path to fix it. +*/ +return 0 ; +} +EXPORT_SYMBOL_GPL (handle_mmio_page_fault_common); + +static int handle_mmio_page_fault (struct kvm_vcpu *vcpu,u64 addr, +u32 error_code,bool direct) +{ +u64 sptes[4 ]; +int nr_sptes, ret; + +if (!mmio_pf) +return 0 ; + +if (!(error_code &PFERR_RSVD_MASK )) +return 0 ; + + ret =handle_mmio_page_fault_common (vcpu, addr, sptes, &nr_sptes, + direct); +WARN_ON (ret <0 ); +return ret; +}static int nonpaging_page_fault (struct kvm_vcpu *vcpu,gva_t gva,u32 error_code,bool prefault)
@@ -2692,6 +2810,11 @@ static int nonpaging_page_fault(struct kvm_vcpu *vcpu, gva_t gva,int r;pgprintk ("%s: gva %lx error %x\n" , __func__, gva, error_code); + + r =handle_mmio_page_fault (vcpu, gva, error_code, true); +if (r) +return r; + r =mmu_topup_memory_caches (vcpu);if (r)return r;
@@ -2768,6 +2891,10 @@ static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa, u32 error_code,ASSERT (vcpu);ASSERT (VALID_PAGE (vcpu->arch.mmu.root_hpa)); + r =handle_mmio_page_fault (vcpu, gpa, error_code, true); +if (r) +return r; + r =mmu_topup_memory_caches (vcpu);if (r)return r;
arch/x86/kvm/mmu.h +2 -2
@@ -48,8 +48,8 @@#define PFERR_RSVD_MASK (1U << 3)#define PFERR_FETCH_MASK (1U << 4) -int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr, -u64 sptes[4 ]); +int handle_mmio_page_fault_common (struct kvm_vcpu *vcpu,u64 addr, +u64 sptes[4 ],int *nr_sptes,bool direct);int kvm_init_shadow_mmu (struct kvm_vcpu *vcpu,struct kvm_mmu *context);
arch/x86/kvm/paging_tmpl.h +31 -1
@@ -580,6 +580,10 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gva_t addr, u32 error_code,pgprintk ("%s: addr %lx err %x\n" , __func__, addr, error_code); + r =handle_mmio_page_fault (vcpu, addr, error_code,mmu_is_nested (vcpu)); +if (r) +return r; + r =mmu_topup_memory_caches (vcpu);if (r)return r;
@@ -779,6 +783,28 @@ static void FNAME(prefetch_page)(struct kvm_vcpu *vcpu, } } +static bool FNAME (sync_mmio_spte )(struct kvm_vcpu *vcpu, +struct kvm_mmu_page *sp,u64 *sptep, +pt_element_t gpte,int *nr_present) +{ +if (unlikely (is_mmio_spte (*sptep))) { +gfn_t gfn =gpte_to_gfn (gpte); +unsigned access = sp->role.access &FNAME (gpte_access)(vcpu, + gpte); + +if (gfn !=get_mmio_spte_gfn (*sptep)) { +__set_spte (sptep, shadow_trap_nonpresent_pte); +return true; + } + + (*nr_present)++; +mark_mmio_spte (sptep, gfn, access); +return true; + } + +return false; +} +/* * Using the cached information from sp->gfns is safe because: * - The spte has a reference to the struct page, so the pfn for a given gfn
@@ -814,7 +840,7 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)gpa_t pte_gpa;gfn_t gfn; -if (!is_shadow_present_pte (sp->spt[i])) +if (sp->spt[i] == shadow_trap_nonpresent_pte)continue ; pte_gpa = first_pte_gpa + i *sizeof (pt_element_t);
@@ -830,6 +856,10 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)continue ; } +if (FNAME (sync_mmio_spte)(vcpu, sp, &sp->spt[i], gpte, + &nr_present)) +continue ; +if (gfn != sp->gfns[i]) {drop_spte (vcpu->kvm, &sp->spt[i], shadow_trap_nonpresent_pte);
arch/x86/kvm/vmx.c +9 -3
@@ -4673,16 +4673,22 @@ static void ept_misconfig_inspect_spte(struct kvm_vcpu *vcpu, u64 spte,static int handle_ept_misconfig (struct kvm_vcpu *vcpu) {u64 sptes[4 ]; -int nr_sptes, i; +int nr_sptes, i, ret;gpa_t gpa; gpa =vmcs_read64 (GUEST_PHYSICAL_ADDRESS ); + ret =handle_mmio_page_fault_common (vcpu, gpa, sptes, &nr_sptes, true); +if (likely (ret ==1 )) +return x86_emulate_instruction (vcpu, gpa,0 ,NULL ,0 ) == +EMULATE_DONE ; +if (unlikely (!ret)) +return kvm_mmu_page_fault (vcpu, gpa,0 ,NULL ,0 ); + +/* It is the real ept misconfig */ printk (KERN_ERR "EPT: Misconfiguration.\n" );printk (KERN_ERR "EPT: GPA: 0x%llx\n" , gpa); - nr_sptes =kvm_mmu_walk_shadow_page_lockless (vcpu, gpa, sptes); -for (i =PT64_ROOT_LEVEL ; i >PT64_ROOT_LEVEL - nr_sptes; --i)ept_misconfig_inspect_spte (vcpu, sptes[i-1 ], i);
Signature · 1 line
1.7.4.4
PATCH 15/15 KVM: MMU: trace mmio page fault
In reply to Xiao GuangrongAdd tracepoints to trace mmio page fault
--- arch/x86/kvm/mmu.c | 4 +++ arch/x86/kvm/mmutrace.h | 48 ++++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 5 +++- include/trace/events/kvm.h | 24 ++++++++++++++++++++++ 4 files changed, 80 insertions(+), 1 deletions(-)
arch/x86/kvm/mmu.c +4 -0
@@ -207,6 +207,7 @@ static void mark_mmio_spte(u64 *sptep, u64 gfn, unsigned access) { access &=ACC_WRITE_MASK |ACC_USER_MASK ; +trace_mark_mmio_spte (sptep, gfn, access);__set_spte (sptep, shadow_mmio_mask | access | gfn <<PAGE_SHIFT ); }
@@ -1752,6 +1753,7 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm,free_mmu_pages_unlock_parts (invalid_list); sp = list_first_entry(invalid_list, struct kvm_mmu_page, link);list_del_init (invalid_list); +trace_kvm_mmu_delay_free_pages (sp);call_rcu (&sp->rcu, free_invalid_pages_rcu);return ; }
@@ -2765,6 +2767,8 @@ int handle_mmio_page_fault_common(struct kvm_vcpu *vcpu, u64 addr,if (direct) addr =0 ; + +trace_handle_mmio_page_fault (addr, gfn, access);vcpu_cache_mmio_info (vcpu, addr, gfn, access);return 1 ; }
arch/x86/kvm/mmutrace.h +48 -0
54 lines not on this page · raw message
arch/x86/kvm/x86.c +4 -1
@@ -3914,6 +3914,7 @@ static int vcpu_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva, vcpu->arch.access)) { *gpa = vcpu->arch.mmio_gfn <<PAGE_SHIFT | (gva & (PAGE_SIZE -1 )); +trace_vcpu_match_mmio (gva, *gpa, write, false);return 1 ; }
@@ -3929,8 +3930,10 @@ static int vcpu_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva,if ((*gpa &PAGE_MASK ) ==APIC_DEFAULT_PHYS_BASE )return 1 ; -if (vcpu_match_mmio_gpa (vcpu, *gpa)) +if (vcpu_match_mmio_gpa (vcpu, *gpa)) { +trace_vcpu_match_mmio (gva, *gpa, write, true);return 1 ; + }return 0 ; }
include/trace/events/kvm.h +24 -0
@@ -306,6 +306,30 @@ TRACE_EVENT(#endif +TRACE_EVENT ( + vcpu_match_mmio, + TP_PROTO(gva_t gva, gpa_t gpa, bool write, bool gpa_match), +TP_ARGS (gva, gpa, write, gpa_match), + +TP_STRUCT__entry ( + __field(gva_t, gva) + __field(gpa_t, gpa) + __field(bool, write) +__field (bool, gpa_match) + ), + +TP_fast_assign ( + __entry->gva = gva; + __entry->gpa = gpa; + __entry->write = write; + __entry->gpa_match = gpa_match + ), + +TP_printk ("gva %#lx gpa %#llx %s %s" , __entry->gva, __entry->gpa, + __entry->write ?"Write" :"Read" , + __entry->gpa_match ?"GPA" :"GVA" ) +); +#endif /* _TRACE_KVM_MAIN_H */ /* This part must be outside protection */
Signature · 1 line
1.7.4.4
Quoted 11 lines · On Tue, 07 Jun 2011 20:58:06 +0800 Xiao Guangrong…
The performance test result:
Netperf (
TCP_RR):
===========================
ept is enabled:Before After 1st 709.58 734.60 2nd 715.40 723.75 3rd 713.45 724.22
Quoted 7 lines
ept=0 bypass_guest_pf=0:
Before After 1st 706.10 709.63 2nd 709.38 715.80 3rd 695.90 710.70
In what condition, does TCP_RR perform so bad?
On 1Gbps network, directly connecting two Intel servers, I got 20 times better result before.
Even when I used a KVM guest as the netperf client, I got more than 10 times better result.
Could you tell me a bit more details of your test?
Quoted 16 lines
Kernbech (do not redirect output to
/dev/null)
==========================
ept is enabled:Before After 1st 2m34.749s 2m33.482s 2nd 2m34.651s 2m33.161s 3rd 2m34.543s 2m34.271sept=0 bypass_guest_pf=0:
Before After 1st 4m43.467s 4m41.873s 2nd 4m45.225s 4m41.668s 3rd 4m47.029s 4m40.128s
Re: [PATCH 05/15] KVM: MMU: optimize to handle dirty bit
In reply to Xiao GuangrongQuoted 8 lines · Xiao Guangrong
If dirty bit is not set, we can make the pte access read-only to avoid handing dirty bit everywhere
arch/x86/kvm/paging_tmpl.h +0
-0
arch/x86/kvm/paging_tmpl.h static unsigned FNAME(gpte_access)(struct kvm_vcpu *vcpu, pt_element_t gpte) · line 111
Quoted 6 lines
arch/x86/kvm/paging_tmpl.h +3
-0@@ -106,6 +106,9 @@ static unsigned FNAME(gpte_access)(struct kvm_vcpu *vcpu, pt_element_t gpte)unsigned access; access = (gpte & (PT_WRITABLE_MASK |PT_USER_MASK )) |ACC_EXEC_MASK ; +if (!is_dirty_gpte (gpte)) + access &= ~ACC_WRITE_MASK ; +
Sorry, it can break something: if the gpte is not on the last level and dirty bit is set later, below patch should fix it, i'll merge it into in the next version.
arch/x86/kvm/paging_tmpl.h +11 -9
@@ -101,12 +101,13 @@ static int FNAME(cmpxchg_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,return (ret != orig_pte); } -static unsigned FNAME (gpte_access )(struct kvm_vcpu *vcpu,pt_element_t gpte) +static unsigned FNAME (gpte_access )(struct kvm_vcpu *vcpu,pt_element_t gpte, +bool last) {unsigned access; access = (gpte & (PT_WRITABLE_MASK |PT_USER_MASK )) |ACC_EXEC_MASK ; -if (!is_dirty_gpte (gpte)) +if (last && !is_dirty_gpte (gpte)) access &= ~ACC_WRITE_MASK ;#if PTTYPE ==64
@@ -230,8 +231,6 @@ walk: pte |=PT_ACCESSED_MASK ; } - pte_access = pt_access &FNAME (gpte_access)(vcpu, pte); - walker->ptes[walker->level -1 ] = pte; if ((walker->level == PT_PAGE_TABLE_LEVEL) ||
@@ -266,7 +265,7 @@ walk:break ; } - pt_access = pte_access; + pt_access &=FNAME (gpte_access)(vcpu, pte, false); --walker->level; }
@@ -290,6 +289,7 @@ walk: walker->ptes[walker->level -1 ] = pte; } + pte_access = pt_access &FNAME (gpte_access)(vcpu, pte, true); walker->pt_access = pt_access; walker->pte_access = pte_access; pgprintk("%s: pte %llx pte_access %x pt_access %x\n",
@@ -369,7 +369,7 @@ static void FNAME(update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,return ;pgprintk ("%s: gpte %llx spte %p\n" , __func__, (u64 )gpte, spte); - pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte); + pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte, true); pfn =gfn_to_pfn_atomic (vcpu->kvm,gpte_to_gfn (gpte));if (mmu_invalid_pfn (pfn)) {kvm_release_pfn_clean (pfn);
@@ -444,7 +444,8 @@ static void FNAME(pte_prefetch)(struct kvm_vcpu *vcpu, struct guest_walker *gw,if (FNAME (prefetch_invalid_gpte)(vcpu, sp, spte, gpte))continue ; - pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte); + pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte, + true); gfn =gpte_to_gfn (gpte); pfn =pte_prefetch_gfn_to_pfn (vcpu, gfn, pte_access &ACC_WRITE_MASK );
@@ -790,7 +791,7 @@ static bool FNAME(sync_mmio_spte)(struct kvm_vcpu *vcpu,if (unlikely (is_mmio_spte (*sptep))) {gfn_t gfn =gpte_to_gfn (gpte);unsigned access = sp->role.access &FNAME (gpte_access)(vcpu, - gpte); + gpte, true);if (gfn !=get_mmio_spte_gfn (*sptep)) {__set_spte (sptep, shadow_trap_nonpresent_pte);
@@ -868,7 +869,8 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) } nr_present++; - pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte); + pte_access = sp->role.access &FNAME (gpte_access)(vcpu, gpte, + true); host_writable = sp->spt[i] &SPTE_HOST_WRITEABLE ; set_spte(vcpu, &sp->spt[i], pte_access, 0, 0,
Quoted 22 lines · Takuya Yoshikawa
Quoted 11 lines · On Tue, 07 Jun 2011 20:58:06 +0800 Xiao Guangrong…
The performance test result:
Netperf (
TCP_RR):
===========================
ept is enabled:Before After 1st 709.58 734.60 2nd 715.40 723.75 3rd 713.45 724.22Quoted 7 lines
ept=0 bypass_guest_pf=0:
Before After 1st 706.10 709.63 2nd 709.38 715.80 3rd 695.90 710.70
Quoted 8 lines
In what condition, does
TCP_RRperform so bad?On 1Gbps network, directly connecting two Intel servers, I got 20 times better result before.
Even when I used a KVM guest as the netperf client, I got more than 10 times better result.
Um, which case did you test? ept = 1 or ept=0 bypass_guest_pf=0 or both?
Quoted 2 lines
Could you tell me a bit more details of your test?
Sure, KVM guest is the client, and it uses e1000 NIC, and uses NAT network connect to the netperf server, the bandwidth of our network is 100M.
Quoted 32 lines · Xiao Guangrong
Quoted 22 lines · Takuya Yoshikawa
Quoted 11 lines · On Tue, 07 Jun 2011 20:58:06 +0800 Xiao Guangrong…
The performance test result:
Netperf (
TCP_RR):
===========================
ept is enabled:Before After 1st 709.58 734.60 2nd 715.40 723.75 3rd 713.45 724.22Quoted 7 lines
ept=0 bypass_guest_pf=0:
Before After 1st 706.10 709.63 2nd 709.38 715.80 3rd 695.90 710.70Quoted 8 lines
In what condition, does
TCP_RRperform so bad?On 1Gbps network, directly connecting two Intel servers, I got 20 times better result before.
Even when I used a KVM guest as the netperf client, I got more than 10 times better result.
Quoted 9 lines
Um, which case did you test? ept = 1 or ept=0 bypass_guest_pf=0 or both?
Quoted 2 lines
Could you tell me a bit more details of your test?
Sure, KVM guest is the client, and it uses e1000 NIC, and uses NAT network connect to the netperf server, the bandwidth of our network is 100M.
And this is my test script:
#!/bin/sh
echo 3 > /proc/sys/vm/drop_caches
./netperf -H $HOST_NAME -p $PORT -t TCP_RR -l 60
Quoted 29 lines · On Wed, 08 Jun 2011 11:32:12 +0800 Xiao Guangrong…
Quoted 28 lines · Xiao Guangrong
Quoted 27 lines · Takuya Yoshikawa
Quoted 11 lines · On Tue, 07 Jun 2011 20:58:06 +0800 Xiao Guangrong…
The performance test result:
Netperf (
TCP_RR):
===========================
ept is enabled:Before After 1st 709.58 734.60 2nd 715.40 723.75 3rd 713.45 724.22Quoted 7 lines
ept=0 bypass_guest_pf=0:
Before After 1st 706.10 709.63 2nd 709.38 715.80 3rd 695.90 710.70In what condition, does
TCP_RRperform so bad?On 1Gbps network, directly connecting two Intel servers, I got 20 times better result before.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Even when I used a KVM guest as the netperf client, I got more than 10 times better result.
Um, which case did you test? ept = 1 or ept=0 bypass_guest_pf=0 or both?
ept = 1 only.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Could you tell me a bit more details of your test?
Sure, KVM guest is the client, and it uses e1000 NIC, and uses NAT network connect to the netperf server, the bandwidth of our network is 100M.
I see the reason, thank you!
I used virtio-net and you used e1000.
You are using e1000 to see the MMIO performance change, right?
Takuya
Quoted 8 lines
And this is my test script:
#!/bin/sh
echo 3 >
/proc/sys/vm/drop_caches
./netperf -H $HOST_NAME -p $PORT -tTCP_RR-l 60
Quoted 10 lines · Takuya Yoshikawa
Quoted 4 lines
Quoted 4 lines
Sure, KVM guest is the client, and it uses e1000 NIC, and uses NAT network connect to the netperf server, the bandwidth of our network is 100M.
I see the reason, thank you!
I used virtio-net and you used e1000.
You are using e1000 to see the MMIO performance change, right?
Hi Takuya,
Please applied my fix path when you test it again, thanks! :-)
(http://www.spinics.net/lists/kvm/msg56017.html)
Just then, in order to affirm the performance result, i tested it again, and do not use our office network(since such many boxes in this network), just boot two guests, one runs netperf server, one runs netperf client, both use e1000 and NAT network.
I'll test the performance of virtio-net!
This is the result:
ept = 1:
============================
Before patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1182.27
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1185.84
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1181.58
16384 87380
After patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1205.65
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1216.06
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1215.70
16384 87380
ept = 0, bypass_guest_pf=0:
============================
Before patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1169.70
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1160.82
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1168.01
16384 87380
After patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1266.28
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1268.16
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 1267.18
16384 87380
To my surprise is: after patch, the performance of ept = 0, bypass_guest_pf=0 is better than the performance of ept = 1, maybe it is because MMIO is too much in network guests :-)
Quoted 10 lines · Takuya Yoshikawa
Quoted 4 lines
Quoted 4 lines
Sure, KVM guest is the client, and it uses e1000 NIC, and uses NAT network connect to the netperf server, the bandwidth of our network is 100M.
I see the reason, thank you!
I used virtio-net and you used e1000.
You are using e1000 to see the MMIO performance change, right?
Hi Takuya,
Now, i have done the performance test for virtio-net, the performance is improved very little, and it is not *regression* ;-)
The reason is, MMIO generated by virtio-net is very very little.
ept = 1:
============================
Before patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 972.21
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 971.01
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 974.44
16384 87380
After patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 973.45
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 973.63
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 976.25
16384 87380
ept = 0, bypass_guest_pf=0:
============================
Before patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 975.16
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 979.95
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 984.03
16384 87380
After patch:
--------------
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 974.30
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 976.33
16384 87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.122.247 (192.168.122.247) port 0 AF_INET Local /Remote
Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 60.00 981.45
16384 87380
Re: [PATCH 04/15] KVM: MMU: cache mmio info on page fault path
In reply to Xiao GuangrongQuoted 15 lines · Xiao Guangrong
If the page fault is caused by mmio, we can cache the mmio info, later, we do not need to walk guest page table and quickly know it is a mmio fault while we emulate the mmio instruction
Signed-off-by Xiao Guangrong <[email protected]>---
arch/x86/include/asm/kvm_host.h | 5 +++ arch/x86/kvm/mmu.c | 21 +++++---------- arch/x86/kvm/mmu.h | 23 +++++++++++++++++ arch/x86/kvm/paging_tmpl.h | 21 ++++++++++----- arch/x86/kvm/x86.c | 52 ++++++++++++++++++++++++++++++-------- arch/x86/kvm/x86.h | 36 +++++++++++++++++++++++++++ 6 files changed, 126 insertions(+), 32 deletions(-)
[...]
arch/x86/kvm/x86.c out: · line 3917
Quoted 12 lines
arch/x86/kvm/x86.c +12
-0+static int vcpu_gva_to_gpa (struct kvm_vcpu *vcpu,unsigned long gva, +gpa_t *gpa,struct x86_exception *exception, +bool write) +{ +u32 access = (kvm_x86_ops->get_cpl (vcpu) ==3 ) ?PFERR_USER_MASK :0 ; + +if (vcpu_match_mmio_gva (vcpu, gva) && +check_write_user_access (vcpu, write, access, + vcpu->arch.access)) { + *gpa = vcpu->arch.mmio_gfn <<PAGE_SHIFT | + (gva & (PAGE_SIZE -1 )); +return 1 ;
Hrm. Let me try to understand what you're doing.
Whenever a guest issues an MMIO, it triggers an #NPF or #PF and then we walk either the NPT or the guest PT to resolve the GPA to the fault and send off an MMIO.
Within that path, you remember the GVA->GPA mapping for the last MMIO request. If the next MMIO request is on the same GVA and kernel/user permissions still apply, you simply bypass the resolution. So far so good.
Now, what happens when the GVA is not identical to the GVA it was before? It's probably a purely theoretic case, but imagine the following:
1) guest issues MMIO on GVA 0x1000 (GPA 0x1000)
2) guest remaps page 0x1000 to GPA 0x2000
3) guest issues MMIO on GVA 0x1000
That would break with your current implementation, right? It sounds pretty theoretic, but imagine the following:
1) guest user space 1 maps MMIO region A to 0x1000
2) guest user space 2 maps MMIO region B to 0x1000
3) guest user space 1 issues MMIO on 0x1000
4) context switch; going to user space 2
5) user space 2 issues MMIO on 0x1000
That case could at least be identified by also comparing the guest's cr3 value during this hack. And considering things like UIO or microkernels, it's not too unlikely :).
Alex
Quoted 13 lines · On Wed, 08 Jun 2011 14:22:36 +0800 Xiao Guangrong…
Quoted 10 lines · Takuya Yoshikawa
Quoted 4 lines
Quoted 4 lines
Sure, KVM guest is the client, and it uses e1000 NIC, and uses NAT network connect to the netperf server, the bandwidth of our network is 100M.
I see the reason, thank you!
I used virtio-net and you used e1000.
You are using e1000 to see the MMIO performance change, right?
Quoted 7 lines
Hi Takuya,
Now, i have done the performance test for virtio-net, the performance is improved very little, and it is not *regression* ;-)
The reason is, MMIO generated by virtio-net is very very little.
Yes, so I thought you had chosen e1000 for this test :)
Thanks,
Takuya
Re: [PATCH 04/15] KVM: MMU: cache mmio info on page fault path
In reply to Alexander GrafQuoted 4 lines · Alexander Graf
Quoted 4 lines
arch/x86/kvm/x86.c +4
-0+static int vcpu_gva_to_gpa (struct kvm_vcpu *vcpu,unsigned long gva, +gpa_t *gpa,struct x86_exception *exception, +bool write) +{
Quoted 9 lines
arch/x86/kvm/x86.c out: · line 3917
Quoted 8 lines
arch/x86/kvm/x86.c +8
-0+u32 access = (kvm_x86_ops->get_cpl (vcpu) ==3 ) ?PFERR_USER_MASK :0 ; + +if (vcpu_match_mmio_gva (vcpu, gva) && +check_write_user_access (vcpu, write, access, + vcpu->arch.access)) { + *gpa = vcpu->arch.mmio_gfn <<PAGE_SHIFT | + (gva & (PAGE_SIZE -1 )); +return 1 ;
Hi Alexander,
Thanks for your review!
Quoted 5 lines
Hrm. Let me try to understand what you're doing.
Whenever a guest issues an MMIO, it triggers an #NPF or #PF and then we walk either the NPT or the guest PT to resolve the GPA to the fault and send off an MMIO.
Within that path, you remember theGVA->GPAmapping for the last MMIO request. If the next MMIO request is on the same GVA and kernel/user permissions still apply, you simply bypass the resolution. So far so good.
In this patch, we also introduced vcpu_clear_mmio_info() that clears mmio cache info on the vcpu, and it is called when guest flush tlb (reload CR3 or INVLPG).
Quoted 6 lines
Now, what happens when the GVA is not identical to the GVA it was before? It's probably a purely theoretic case, but imagine the following:
1) guest issues MMIO on GVA
0x1000(GPA0x1000)
2) guest remaps page0x1000to GPA0x2000
3) guest issues MMIO on GVA0x1000
If guest modify the page structure, base on x86 tlb rules, we should flush tlb to ensure the cpu use the new mapping.
When you remap GVA 0x1000 to 0x2000, you should flush tlb, then mmio cache info is cleared, so the later access is right.
Quoted 8 lines
That would break with your current implementation, right? It sounds pretty theoretic, but imagine the following:
1) guest user space 1 maps MMIO region A to
0x1000
2) guest user space 2 maps MMIO region B to0x1000
3) guest user space 1 issues MMIO on0x1000
4) context switch; going to user space 2
5) user space 2 issues MMIO on0x1000
Also, when context switched, CR3 is reloaded, mmio cache info can be cleared too. right? :-)
Re: [PATCH 04/15] KVM: MMU: cache mmio info on page fault path
Quoted 20 lines · Xiao Guangrong
Quoted 4 lines · Alexander Graf
Quoted 4 lines
arch/x86/kvm/x86.c +4
-0+static int vcpu_gva_to_gpa (struct kvm_vcpu *vcpu,unsigned long gva, +gpa_t *gpa,struct x86_exception *exception, +bool write) +{Quoted 9 lines
arch/x86/kvm/x86.c out: · line 3917
Quoted 8 lines
arch/x86/kvm/x86.c +8
-0+u32 access = (kvm_x86_ops->get_cpl (vcpu) ==3 ) ?PFERR_USER_MASK :0 ; + +if (vcpu_match_mmio_gva (vcpu, gva) && +check_write_user_access (vcpu, write, access, + vcpu->arch.access)) { + *gpa = vcpu->arch.mmio_gfn <<PAGE_SHIFT | + (gva & (PAGE_SIZE -1 )); +return 1 ;Hi Alexander,
Thanks for your review!
Quoted 8 lines
Quoted 5 lines
Hrm. Let me try to understand what you're doing.
Whenever a guest issues an MMIO, it triggers an #NPF or #PF and then we walk either the NPT or the guest PT to resolve the GPA to the fault and send off an MMIO.
Within that path, you remember theGVA->GPAmapping for the last MMIO request. If the next MMIO request is on the same GVA and kernel/user permissions still apply, you simply bypass the resolution. So far so good.In this patch, we also introduced
vcpu_clear_mmio_info()that clears mmio cache info on the vcpu, and it is called when guest flush tlb (reload CR3 or INVLPG).
Ah, that one solved the SPT case then of course.
Quoted 16 lines
Quoted 7 lines
Now, what happens when the GVA is not identical to the GVA it was before? It's probably a purely theoretic case, but imagine the following:
1) guest issues MMIO on GVA
0x1000(GPA0x1000)
2) guest remaps page0x1000to GPA0x2000
3) guest issues MMIO on GVA0x1000If guest modify the page structure, base on x86 tlb rules, we should flush tlb to ensure the cpu use the new mapping.
When you remap GVA
0x1000to0x2000, you should flush tlb, then mmio cache info is cleared, so the later access is right.Quoted 2 lines
That would break with your current implementation, right? It sounds pretty theoretic, but imagine the following:
Quoted 8 lines
Quoted 6 lines
1) guest user space 1 maps MMIO region A to
0x1000
2) guest user space 2 maps MMIO region B to0x1000
3) guest user space 1 issues MMIO on0x1000
4) context switch; going to user space 2
5) user space 2 issues MMIO on0x1000Also, when context switched, CR3 is reloaded, mmio cache info can be cleared too. right? :-)
Only when using SPT. In the NPT case, you will never see cr3 getting reloaded or INVLPG :).
Alex
Re: [PATCH 04/15] KVM: MMU: cache mmio info on page fault path
Quoted 5 lines · Alexander Graf
Quoted 2 lines
Also, when context switched, CR3 is reloaded, mmio cache info can be cleared too. right? :-)
Only when using SPT. In the NPT case, you will never see cr3 getting reloaded or INVLPG :).
In the NPT case, we only cache the GPA, GVA is not cached (vcpu.arch.mmio_gva is always 0) ;-)
Re: [PATCH 04/15] KVM: MMU: cache mmio info on page fault path
Quoted 9 lines · Xiao Guangrong
Quoted 5 lines · Alexander Graf
Quoted 2 lines
Also, when context switched, CR3 is reloaded, mmio cache info can be cleared too. right? :-)
Only when using SPT. In the NPT case, you will never see cr3 getting reloaded or INVLPG :).
In the NPT case, we only cache the GPA, GVA is not cached (vcpu.arch.mmio_gva is always 0) ;-)
Ah, very nice! :)
Alex
Re: [PATCH 03/15] KVM: x86: avoid unnecessarily guest page table walking
In reply to Xiao GuangrongQuoted 13 lines · Xiao Guangrong
We already get the guest physical address, so use it to read guest data directly to avoid walking guest page table again
Signed-off-by Xiao Guangrong<[email protected]>--- arch/x86/kvm/x86.c | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-)arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c static int emulator_read_emulated(struct x86_emulate_ctxt *ctxt, · line 3934
Quoted 7 lines
arch/x86/kvm/x86.c +1
-2@@ -3930,8 +3930,7 @@ static int emulator_read_emulated(struct x86_emulate_ctxt *ctxt,if ((gpa&PAGE_MASK ) ==APIC_DEFAULT_PHYS_BASE ) goto mmio; -if (kvm_read_guest_virt (ctxt, addr, val, bytes, exception) - ==X86EMUL_CONTINUE ) +if (!kvm_read_guest (vcpu->kvm, gpa, val, bytes))return X86EMUL_CONTINUE ;
This breaks is addr/bytes spans a page boundary.
(the current code is also broken, but only for mmio; the new code is broken for ram as well).
We need a gva_to_gpa() that returns a range of pages.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 09/15] KVM: MMU: split kvm_mmu_free_page
In reply to Xiao GuangrongQuoted 7 lines · Xiao Guangrong
Split
kvm_mmu_free_pagetokvm_mmu_free_lock_partsandkvm_mmu_free_unlock_partsOne is used to free the parts which is under mmu lock and the other is used to free the parts which can allow be freed out of mmu lock
It is used by later patch
Suggest kvm_mmu_isolate_page() and kvm_mmu_free_page(). Plus a comment explaining things, unless someone can come up with a better name.
Quoted 7 lines
arch/x86/kvm/mmu.c +1
-3-static void kvm_mmu_free_page (struct kvm_mmu_page *sp) +static void kvm_mmu_free_lock_parts (struct kvm_mmu_page *sp) {ASSERT (is_empty_shadow_page (sp->spt));hlist_del (&sp->hash_link); -list_del (&sp->link); -free_page ((unsigned long )sp->spt);
arch/x86/kvm/mmu.c static inline void kvm_mod_used_mmu_pages(struct kvm *kvm, int nr) · line 1055
Quoted 10 lines
arch/x86/kvm/mmu.c +6
-0if (!sp->role.direct)free_page ((unsigned long )sp->gfns); +} + +static void kvm_mmu_free_unlock_parts (struct kvm_mmu_page *sp) +{ +list_del (&sp->link); +free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); }
The list_del() must be run under a lock, no? it can access kvm->arch.active_mmu_pages.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 13/15] KVM: VMX: modify the default value of nontrap shadow pte
In reply to Xiao GuangrongQuoted 13 lines · Xiao Guangrong
Modify the default value to identify nontrap shadow pte and mmio shadow pte whill will be introduced in later patch
Signed-off-by Xiao Guangrong<[email protected]>--- arch/x86/kvm/vmx.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)arch/x86/kvm/vmx.c +0
-0
arch/x86/kvm/vmx.c static int __init vmx_init(void) · line 7114
Quoted 6 lines
arch/x86/kvm/vmx.c +1
-1@@ -7110,7 +7110,7 @@ static int __init vmx_init(void)kvm_disable_tdp ();if (bypass_guest_pf) -kvm_mmu_set_nonpresent_ptes (~0xffeull ,0ull ); +kvm_mmu_set_nonpresent_ptes (0xfull <<49 |1ull ,0ull );
This can break on newer processors (well, so can the original, but the new one will break earlier).
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 14/15] KVM: MMU: mmio page fault support
In reply to Xiao GuangrongQuoted 12 lines · Xiao Guangrong
The idea is from Avi:
| We could cache the result of a miss in an spte by using a reserved bit, and | checking the page fault error code (or seeing if we get an ept violation or | ept misconfiguration), so if we get repeated mmio on a page, we don't need to | search the slot list/tree. | (https://lkml.org/lkml/2011/2/22/221)When the page fault is caused by mmio, we cache the info in the shadow page table, and also set the reserved bits in the shadow page table, so if the mmio is caused again, we can quickly identify it and emulate it directly
Quoted 9 lines
Searching mmio gfn in memslots is heavy since we need to walk all memeslots, it can be reduced by this feature, and also avoid walking guest page table for soft mmu.
This feature can be disabled/enabled at the runtime, if
shadow_notrap_nonpresent_pteis enabled, the PFER.RSVD is always set, we need to walk shadow page table for all page fault, so disable this feature ifshadow_notrap_nonpresentis enabled.
Maybe it's time to kill off bypass_guest_pf=1. It's not as effective as it used to be, since unsync pages always use shadow_trap_nonpresent_pte, and since we convert between the two nonpresent_ptes during sync and unsync.
arch/x86/kvm/mmu.c module_param(dbg, bool, 0644); · line 95
Quoted 10 lines
arch/x86/kvm/mmu.c +2
-0@@ -91,6 +91,9 @@ module_param(dbg, bool, 0644);static int oos_shadow =1 ;module_param (oos_shadow, bool,0644 ); +static int __read_mostly mmio_pf =1 ; +module_param (mmio_pf, bool,0644 );
Why make it a module parameter?
arch/x86/kvm/mmu.c static u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ · line 211
Quoted 6 lines
arch/x86/kvm/mmu.c +6
-0+static void mark_mmio_spte (u64 *sptep,u64 gfn,unsigned access) +{ + access&=ACC_WRITE_MASK |ACC_USER_MASK ; + +__set_spte (sptep, shadow_mmio_mask | access | gfn<<PAGE_SHIFT ); +}
This can only work for shadow. Is it worth the complexity?
Also, shadow walking is not significantly faster than guest page table walking. And if we miss, we have to walk the guest page tables in any case.
arch/x86/kvm/mmu.c int kvm_mmu_walk_shadow_page_lockless(struct kvm_vcpu *vcpu, u64 addr, · line 2746
Quoted 11 lines
arch/x86/kvm/mmu.c +11
-0+ +static bool quickly_check_mmio_pf (struct kvm_vcpu *vcpu,u64 addr,bool direct) +{ +if (direct&&vcpu_match_mmio_gpa (vcpu, addr)) +return true; + +if (vcpu_match_mmio_gva (vcpu, addr)) +return true; + +return false; +}
There is also the case of nesting - it's not direct and it's not a gva.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Xiao Guangrong
The idea of this patchset is from Avi:
| We could cache the result of a miss in an spte by using a reserved bit, and | checking the page fault error code (or seeing if we get an ept violation or | ept misconfiguration), so if we get repeated mmio on a page, we don't need to | search the slot list/tree. | (https://lkml.org/lkml/2011/2/22/221)The aim of this patchset is to support fast mmio emulate, it reduce searching mmio gfn from memslots which is very expensive since we need to walk all slots for mmio gfn, and the other advantage is: we can reduce guest page table walking for soft mmu.
Lockless walk shadow page table is introduced in this patchset, it is the light way to check the page fault is the real mmio page fault or something is running out of our mind.
Quoted 6 lines
And, if
shadow_notrap_nonpresent_pteisenabled(bypass_guest_pf=1), mmio page fault and normal page fault is mixed(the reserved is set for all page fault), it has little regression, if the box can generate lots of mmio access, for example, the network server, it can disableshadow_notrap_nonpresent_pteand enable mmio pf, after all, we can enable/disable mmio pf at the runtime.
Okay, this is pretty complicated. And things are already complicated.
First, I think we should consider dropping bypass_guest_pf completely, just so we have less things to think about.
Second, I don't like two paths for accessing shadow page tables, it makes the code much larger. I'm also not sure RCU is enough protection - we can unlink a page in the middle of a hierarchy, and on i386 this causes an invalid pointer to appear when we fetch the two halves. But I guess, if the cpu can do it, so can we.
Maybe we can do something like
again:
fetch pointer to last level spte using RCU
if failed:
take lock
build spte hierarchy
drop lock
goto again
if sync:
if mmio:
do mmio
return
return
walk guest table
install spte
if mmio:
do mmio
(sync is always false for tdp)
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 10/15] KVM: MMU: lockless walking shadow page table
In reply to Xiao GuangrongQuoted 2 lines · Xiao Guangrong
Using rcu to protect shadow pages table to be freed, so we can safely walk it, it should run fast and is needed by mmio page fault
A couple of question below.
Thanx, Paul
Quoted 66 lines
Signed-off-by Xiao Guangrong <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++ arch/x86/kvm/mmu.c | 79 ++++++++++++++++++++++++++++++--------- arch/x86/kvm/mmu.h | 4 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 69 insertions(+), 20 deletions(-)arch/x86/include/asm/kvm_host.h +4
-0@@ -232,6 +232,8 @@ struct kvm_mmu_page {unsigned int unsync_children;unsigned long parent_ptes;/* Reverse mapping for parent_pte */ DECLARE_BITMAP (unsync_child_bitmap,512 ); + +struct rcu_head rcu; };struct kvm_pv_mmu_op_buffer {@@ -478,6 +480,8 @@ struct kvm_arch {u64 hv_guest_os_id;u64 hv_hypercall; +atomic_t reader_counter; +#ifdef CONFIG_KVM_MMU_AUDIT int audit_point;#endif arch/x86/kvm/mmu.c +24
-0@@ -1675,6 +1675,30 @@ static int kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,return ret; } +static void free_mmu_pages_unlock_parts (struct list_head *invalid_list) +{ +struct kvm_mmu_page *sp; + +list_for_each_entry (sp, invalid_list, link) +kvm_mmu_free_lock_parts (sp); +} + +static void free_invalid_pages_rcu (struct rcu_head *head) +{ +struct kvm_mmu_page *next, *sp; + + sp = container_of(head, struct kvm_mmu_page, rcu); +while (sp) { +if (!list_empty (&sp->link)) + next =list_first_entry (&sp->link, + struct kvm_mmu_page, link); +else + next =NULL ; +kvm_mmu_free_unlock_parts (sp); + sp = next; + } +} +static void kvm_mmu_commit_zap_page (struct kvm *kvm,struct list_head *invalid_list) {
arch/x86/kvm/mmu.c static void kvm_mmu_commit_zap_page(struct kvm *kvm, · line 1712
Quoted 4 lines
arch/x86/kvm/mmu.c +1
-0@@ -1685,6 +1709,14 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm,kvm_flush_remote_tlbs (kvm); +if (atomic_read (&kvm->arch.reader_counter)) {
This is the slowpath to be executed if there are currently readers in kvm->arch.reader_counter(), correct?
arch/x86/kvm/mmu.c static void kvm_mmu_commit_zap_page(struct kvm *kvm, · line 1718
Quoted 6 lines
arch/x86/kvm/mmu.c +6
-0+free_mmu_pages_unlock_parts (invalid_list); + sp = list_first_entry(invalid_list, struct kvm_mmu_page, link); +list_del_init (invalid_list); +call_rcu (&sp->rcu, free_invalid_pages_rcu); +return ; + }
OK, so it also looks like kvm->arch.reader_counter could transition from zero to non-zero at this point due to a concurrent call from a reader in the kvm_mmu_walk_shadow_page_lockless() function. Does the following code avoid messing up the reader? If so, why bother with the slowpath above?
Quoted 101 lines
arch/x86/kvm/mmu.c +30
-18+ do { sp = list_first_entry(invalid_list, struct kvm_mmu_page, link); WARN_ON(!sp->role.invalid || sp->root_count);@@ -2601,6 +2633,35 @@ static gpa_t nonpaging_gva_to_gpa_nested(struct kvm_vcpu *vcpu, gva_t vaddr,return vcpu->arch.nested_mmu.translate_gpa (vcpu, vaddr, access); } +int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr, +u64 sptes[4 ]) +{ +struct kvm_shadow_walk_iterator iterator; +int nr_sptes =0 ; + +rcu_read_lock (); + +atomic_inc (&vcpu->kvm->arch.reader_counter); +/* Increase the counter before walking shadow page table */ +smp_mb__after_atomic_inc (); + +for_each_shadow_entry (vcpu, addr, iterator) { + sptes[iterator.level-1 ] = *iterator.sptep; + nr_sptes++; +if (!is_shadow_present_pte (*iterator.sptep)) +break ; + } + +/* Decrease the counter after walking shadow page table finished */ +smp_mb__before_atomic_dec (); +atomic_dec (&vcpu->kvm->arch.reader_counter); + +rcu_read_unlock (); + +return nr_sptes; +} +EXPORT_SYMBOL_GPL (kvm_mmu_walk_shadow_page_lockless); +static int nonpaging_page_fault (struct kvm_vcpu *vcpu,gva_t gva,u32 error_code,bool prefault) {@@ -3684,24 +3745,6 @@ out:return r; } -int kvm_mmu_get_spte_hierarchy (struct kvm_vcpu *vcpu,u64 addr,u64 sptes[4 ]) -{ -struct kvm_shadow_walk_iterator iterator; -int nr_sptes =0 ; - -spin_lock (&vcpu->kvm->mmu_lock); -for_each_shadow_entry (vcpu, addr, iterator) { - sptes[iterator.level-1 ] = *iterator.sptep; - nr_sptes++; -if (!is_shadow_present_pte (*iterator.sptep)) -break ; - } -spin_unlock (&vcpu->kvm->mmu_lock); - -return nr_sptes; -} -EXPORT_SYMBOL_GPL (kvm_mmu_get_spte_hierarchy); -void kvm_mmu_destroy (struct kvm_vcpu *vcpu) {ASSERT (vcpu);arch/x86/kvm/mmu.h +3
-1@@ -48,7 +48,9 @@#define PFERR_RSVD_MASK (1U << 3)#define PFERR_FETCH_MASK (1U << 4) -int kvm_mmu_get_spte_hierarchy (struct kvm_vcpu *vcpu,u64 addr,u64 sptes[4 ]); +int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr, +u64 sptes[4 ]); +int kvm_init_shadow_mmu (struct kvm_vcpu *vcpu,struct kvm_mmu *context);static inline unsigned int kvm_mmu_available_pages (struct kvm *kvm)arch/x86/kvm/vmx.c +1
-1@@ -4681,7 +4681,7 @@ static int handle_ept_misconfig(struct kvm_vcpu *vcpu)printk (KERN_ERR "EPT: Misconfiguration.\n" );printk (KERN_ERR "EPT: GPA: 0x%llx\n" , gpa); - nr_sptes =kvm_mmu_get_spte_hierarchy (vcpu, gpa, sptes); + nr_sptes =kvm_mmu_walk_shadow_page_lockless (vcpu, gpa, sptes);for (i =PT64_ROOT_LEVEL ; i >PT64_ROOT_LEVEL - nr_sptes; --i)ept_misconfig_inspect_spte (vcpu, sptes[i-1 ], i);Signature · 7 lines
1.7.4.4 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 14/15] KVM: MMU: mmio page fault support
In reply to Avi KivityQuoted 3 lines · Avi Kivity
Maybe it's time to kill off bypass_guest_pf=1. It's not as effective as it used to be, since unsync pages always use
shadow_trap_nonpresent_pte, and since we convert between the twononpresent_ptesduring sync and unsync.
Reasonable!
Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/mmu.c +0
-0@@ -91,6 +91,9 @@ module_param(dbg, bool, 0644);static int oos_shadow =1 ;module_param (oos_shadow, bool,0644 );
Quoted 4 lines
arch/x86/kvm/mmu.c module_param(dbg, bool, 0644); · line 95
Quoted 2 lines
arch/x86/kvm/mmu.c +2
-0+static int __read_mostly mmio_pf =1 ; +module_param (mmio_pf, bool,0644 );Why make it a module parameter?
Will remove.
Quoted 10 lines
arch/x86/kvm/mmu.c static u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ · line 211
Quoted 7 lines
arch/x86/kvm/mmu.c +6
-0+static void mark_mmio_spte (u64 *sptep,u64 gfn,unsigned access) +{ + access&=ACC_WRITE_MASK |ACC_USER_MASK ; + +__set_spte (sptep, shadow_mmio_mask | access | gfn<<PAGE_SHIFT ); +}This can only work for shadow. Is it worth the complexity?
I think it is not bad, since it is really simple, and for tdp, we also need to set shadow_mmio_mask bits which causes misconfig/rsvd fault
Quoted 2 lines
Also, shadow walking is not significantly faster than guest page table walking. And if we miss, we have to walk the guest page tables in any case.
Um. i think walking guest page table is slower, it needs to walk memslots for many times and it triggers page fault if the host page is swapped.
And it is hardly missed, since for tdp, it infrequency zaps shadow pages, for soft mmu, the mmio spte is always unsync, and in guest, the mmio region is always mapped by kernel, so it is infrequency to be update and lazily flushed.
Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/mmu.c +5
-0+ +static bool quickly_check_mmio_pf (struct kvm_vcpu *vcpu,u64 addr,bool direct) +{ +if (direct&&vcpu_match_mmio_gpa (vcpu, addr)) +return true;
Quoted 9 lines
arch/x86/kvm/mmu.c int kvm_mmu_walk_shadow_page_lockless(struct kvm_vcpu *vcpu, u64 addr, · line 2746
Quoted 6 lines
arch/x86/kvm/mmu.c +6
-0+ +if (vcpu_match_mmio_gva (vcpu, addr)) +return true; + +return false; +}There is also the case of nesting - it's not direct and it's not a gva.
If it is direct, we only need to compare the pga, and direct=0, we only need to compare gva, i'll fix the code to make it clear.
Re: [PATCH 09/15] KVM: MMU: split kvm_mmu_free_page
In reply to Avi KivityQuoted 11 lines · Avi Kivity
Quoted 7 lines · Xiao Guangrong
Split
kvm_mmu_free_pagetokvm_mmu_free_lock_partsandkvm_mmu_free_unlock_partsOne is used to free the parts which is under mmu lock and the other is used to free the parts which can allow be freed out of mmu lock
It is used by later patch
Suggest
kvm_mmu_isolate_page()andkvm_mmu_free_page(). Plus a comment explaining things, unless someone can come up with a better name.
Good names, will fix.
Quoted 11 lines
Quoted 11 lines
arch/x86/kvm/mmu.c +3
-3-static void kvm_mmu_free_page (struct kvm_mmu_page *sp) +static void kvm_mmu_free_lock_parts (struct kvm_mmu_page *sp) {ASSERT (is_empty_shadow_page (sp->spt));hlist_del (&sp->hash_link); -list_del (&sp->link); -free_page ((unsigned long )sp->spt);if (!sp->role.direct)free_page ((unsigned long )sp->gfns); +} +
Quoted 9 lines
arch/x86/kvm/mmu.c static inline void kvm_mod_used_mmu_pages(struct kvm *kvm, int nr) · line 1055
Quoted 6 lines
arch/x86/kvm/mmu.c +4
-0+static void kvm_mmu_free_unlock_parts (struct kvm_mmu_page *sp) +{ +list_del (&sp->link); +free_page ((unsigned long )sp->spt);kmem_cache_free (mmu_page_header_cache, sp); }The
list_del()must be run under a lock, no? it can accesskvm->arch.active_mmu_pages.
In prepare path, we have moved the sp from active_mmu_pages to invlaid_list.
Re: [PATCH 03/15] KVM: x86: avoid unnecessarily guest page table walking
In reply to Avi KivityQuoted 6 lines · Avi Kivity
This breaks is addr/bytes spans a page boundary.
(the current code is also broken, but only for mmio; the new code is broken for ram as well).
We need a
gva_to_gpa()that returns a range of pages.
Thanks for you point it out, will fix it in the next version.
Quoted 2 lines · Avi Kivity
First, I think we should consider dropping
bypass_guest_pfcompletely, just so we have less things to think about.
I agree.
Quoted 1 line
I'm also not sure RCU is enough protection - we can unlink a page in the middle of a hierarchy,
I think it is ok, it just likes the page structure cache of real CPU, we can use the old mapping or new mapping here, if we missed, page fault path is called, it can fix the problem for us.
Quoted 2 lines
and on i386 this causes an invalid pointer to appear when we fetch the two halves. But I guess, if the cpu can do it, so can we.
Ah, maybe the cpu can not do it, we need a light way to get spte for i386 host...
Quoted 19 lines
Maybe we can do something like
again:
fetch pointer to last level spte using RCU
if failed:
take lock
build spte hierarchy
drop lock
goto again
if sync:
if mmio:
do mmio
return
return
walk guest table
install spte
if mmio:
do mmio
Quoted 2 lines
(sync is always false for tdp)
It seams it is more complex, the origin way is:
fetch last level spte
if failed or it is not a mmio spte:
call page fault
do mmio
and it has little heavy sine we need to walk guest page table, and build spte under mmu-lock.
Maybe i missed your meaning, could you please tell me the advantage? :-(
Re: [PATCH 10/15] KVM: MMU: lockless walking shadow page table
In reply to Paul E. McKenneyQuoted 5 lines · Paul E. McKenney
Quoted 2 lines · Xiao Guangrong
Using rcu to protect shadow pages table to be freed, so we can safely walk it, it should run fast and is needed by mmio page fault
A couple of question below.
Thanks for your review!
Quoted 5 lines
arch/x86/kvm/mmu.c static void kvm_mmu_commit_zap_page(struct kvm *kvm, · line 1712
Quoted 1 line
arch/x86/kvm/mmu.c +1
-0+if (atomic_read (&kvm->arch.reader_counter)) {This is the slowpath to be executed if there are currently readers in
kvm->arch.reader_counter(), correct?
Yes, we will free the pages in RCU context if it is in kvm->arch.reader_counter
Quoted 12 lines
arch/x86/kvm/mmu.c static void kvm_mmu_commit_zap_page(struct kvm *kvm, · line 1718
Quoted 6 lines
arch/x86/kvm/mmu.c +6
-0+free_mmu_pages_unlock_parts (invalid_list); + sp = list_first_entry(invalid_list, struct kvm_mmu_page, link); +list_del_init (invalid_list); +call_rcu (&sp->rcu, free_invalid_pages_rcu); +return ; + }OK, so it also looks like
kvm->arch.reader_countercould transition from zero to non-zero at this point due to a concurrent call from a reader in thekvm_mmu_walk_shadow_page_lockless()function. Does the following code avoid messing up the reader? If so, why bother with the slowpath above?
Actually, we have split the free operation to two steps, the first step is kvm_mmu_prepare_zap_page(), it isolates the page from shadow page table, so after call it, we can not get the page from the shadow page table, and the later steps is kvm_mmu_commit_zap_page(), it frees the page.
kvm_mmu_walk_shadow_page_lockless() get the page from shadow page table, so, even if kvm->arch.reader_counter transition from zero to non-zero in the fallowing code, we can sure the page is not used by kvm_mmu_walk_shadow_page_lockless(), so we can free the page directly.
Re: [PATCH 09/15] KVM: MMU: split kvm_mmu_free_page
In reply to Xiao GuangrongQuoted 11 lines · Xiao Guangrong
Quoted 9 lines
Quoted 6 lines
+static void
kvm_mmu_free_unlock_parts(struct kvm_mmu_page *sp)+{
+list_del(&sp->link);+ free_page((unsigned long)sp->spt); kmem_cache_free(mmu_page_header_cache, sp);}
The
list_del()must be run under a lock, no? it can accesskvm->arch.active_mmu_pages.In prepare path, we have moved the sp from
active_mmu_pagestoinvlaid_list.
It still needs to be accessed under a lock, no matter which list is used.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 14/15] KVM: MMU: mmio page fault support
In reply to Xiao GuangrongQuoted 5 lines · Xiao Guangrong
Quoted 2 lines
Also, shadow walking is not significantly faster than guest page table walking. And if we miss, we have to walk the guest page tables in any case.
Um. i think walking guest page table is slower, it needs to walk memslots for many times and it triggers page fault if the host page is swapped.
Well, if the page is swapped, we can't store anything in the spte.
And if we only store the mmio/ram condition in the spte (via the two types of page faults) we don't need to walk the spte. We know immediately if we need to search the slots or not.
Quoted 3 lines
And it is hardly missed, since for tdp, it infrequency zaps shadow pages, for soft mmu, the mmio spte is always unsync, and in guest, the mmio region is always mapped by kernel, so it is infrequency to be update and lazily flushed.
We still get frequent mmio misses.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
+
+static boolquickly_check_mmio_pf(struct kvm_vcpu *vcpu, u64 addr, bool direct)+{
+ if (direct&&vcpu_match_mmio_gpa(vcpu, addr))
+ return true;
+
+ if (vcpu_match_mmio_gva(vcpu, addr))
+ return true;
+
+ return false;
+}
Quoted 5 lines
Quoted 2 lines
There is also the case of nesting - it's not direct and it's not a gva.
If it is direct, we only need to compare the pga, and direct=0, we only need to compare gva, i'll fix the code to make it clear.
But for nested npt, we get the ngpa, not a gva.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Xiao Guangrong
Quoted 2 lines · Avi Kivity
First, I think we should consider dropping
bypass_guest_pfcompletely, just so we have less things to think about.I agree.
Great, please post a patch.
Quoted 10 lines
Quoted 1 line
I'm also not sure RCU is enough protection - we can unlink a page in the middle of a hierarchy,
I think it is ok, it just likes the page structure cache of real CPU, we can use the old mapping or new mapping here, if we missed, page fault path is called, it can fix the problem for us.
Quoted 2 lines
and on i386 this causes an invalid pointer to appear when we fetch the two halves. But I guess, if the cpu can do it, so can we.
Ah, maybe the cpu can not do it, we need a light way to get spte for i386 host...
Look at the comments in arch/x86/mm/gup.c - it does the same thing.
Quoted 19 lines
Quoted 19 lines
Maybe we can do something like
again:
fetch pointer to last level spte using RCU
if failed:
take lock
build spte hierarchy
drop lock
goto again
if sync:
if mmio:
do mmio
return
return
walk guest table
install spte
if mmio:
do mmio
Quoted 4 lines
Quoted 2 lines
(sync is always false for tdp)
It seams it is more complex,
It also doesn't work - we have to set up rmap under lock.
Quoted 9 lines
the origin way is:
fetch last level spte
if failed or it is not a mmio spte:
call page fault
do mmioand it has little heavy sine we need to walk guest page table, and build spte under mmu-lock.
For shadow, yes, this is a good optimization. But with nested paging it slow things down. We already have the gpa, so all we need to do is follow the mmio path. There's no need to walk the spte hierarchy.
Quoted 1 line
Maybe i missed your meaning, could you please tell me the advantage? :-(
I wanted to also service RAM faults without the lock, if the only thing missing was the spte (and the rest of the hierarchy was fine). But it can't be made to work without an overhaul of all of the locking.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 09/15] KVM: MMU: split kvm_mmu_free_page
In reply to Avi KivityQuoted 8 lines · Avi Kivity
Quoted 7 lines · Xiao Guangrong
Quoted 7 lines
Quoted 6 lines
+static void
kvm_mmu_free_unlock_parts(struct kvm_mmu_page *sp)+{
+list_del(&sp->link);+ free_page((unsigned long)sp->spt); kmem_cache_free(mmu_page_header_cache, sp);}
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
The
list_del()must be run under a lock, no? it can accesskvm->arch.active_mmu_pages.In prepare path, we have moved the sp from
active_mmu_pagestoinvlaid_list.It still needs to be accessed under a lock, no matter which list is used.
Actually, if we need to free page in RCU context, we unlink them from invalid_list firstly:
if (atomic_read(&kvm->arch.reader_counter)) {
......
list_del_init(invalid_list);
trace_kvm_mmu_delay_free_pages(sp);
call_rcu(&sp->rcu, free_invalid_pages_rcu);
return;
}Then, global list is not used anymore.
Re: [PATCH 14/15] KVM: MMU: mmio page fault support
In reply to Avi KivityQuoted 9 lines · Avi Kivity
Quoted 5 lines · Xiao Guangrong
Quoted 2 lines
Also, shadow walking is not significantly faster than guest page table walking. And if we miss, we have to walk the guest page tables in any case.
Um. i think walking guest page table is slower, it needs to walk memslots for many times and it triggers page fault if the host page is swapped.
Well, if the page is swapped, we can't store anything in the spte.
If we walk guest page table, we need to access guest page, and guest page can be swapped out anytime, but shadow page table is the kernel page, it is not swapped, that is why i think walking shadow page table is faster than guest page table.
Quoted 8 lines
And if we only store the mmio/ram condition in the spte (via the two types of page faults) we don't need to walk the spte. We know immediately if we need to search the slots or not.
Quoted 3 lines
And it is hardly missed, since for tdp, it infrequency zaps shadow pages, for soft mmu, the mmio spte is always unsync, and in guest, the mmio region is always mapped by kernel, so it is infrequency to be update and lazily flushed.
We still get frequent mmio misses.
I did the test, run three guests(4vcpu + 512M) on my box (4cores + 2G) and compile kernel in guests, for 1 hour, no mmio is missed(hard mmu and soft mmu), it means that usually we can catch almost all mmio by walking shadow page.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
+
+static boolquickly_check_mmio_pf(struct kvm_vcpu *vcpu, u64 addr, bool direct)+{
+ if (direct&&vcpu_match_mmio_gpa(vcpu, addr))
+ return true;
+
+ if (vcpu_match_mmio_gva(vcpu, addr))
+ return true;
+
+ return false;
+}
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
There is also the case of nesting - it's not direct and it's not a gva.
If it is direct, we only need to compare the pga, and direct=0, we only need to compare gva, i'll fix the code to make it clear.
But for nested npt, we get the ngpa, not a gva.
We treat nested npt as the 'direct' mmio:
r = handle_mmio_page_fault(vcpu, addr, error_code, mmu_is_nested(vcpu));
also do not cache gva for nested npt:
if (handle_abnormal_pfn(vcpu, mmu_is_nested(vcpu) ? 0 : addr,
walker.gfn, pfn, walker.pte_access, &r))
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Xiao Guangrong
Quoted 2 lines · Avi Kivity
First, I think we should consider dropping
bypass_guest_pfcompletely, just so we have less things to think about.I agree.
Great, please post a patch.
OK.
Quoted 4 lines
Quoted 1 line
Ah, maybe the cpu can not do it, we need a light way to get spte for i386 host...
Look at the comments in
arch/x86/mm/gup.c- it does the same thing.
Yeah, it is a good study case for me.
Quoted 7 lines
Quoted 7 lines
the origin way is:
fetch last level spte
if failed or it is not a mmio spte:
call page fault
do mmio
Quoted 5 lines
Quoted 2 lines
and it has little heavy sine we need to walk guest page table, and build spte under mmu-lock.
For shadow, yes, this is a good optimization. But with nested paging it slow things down. We already have the gpa, so all we need to do is follow the mmio path. There's no need to walk the spte hierarchy.
Yes, it is, i just want to detect BUG for KVM, it helps us to know if "ept misconfig" is the real MMIO or the BUG. I noticed some "ept misconfig" BUGs is reported before, so i think doing this is necessary, and i think it is not too bad, since walking spte hierarchy is lockless, it really fast.
Quoted 4 lines
Quoted 1 line
Maybe i missed your meaning, could you please tell me the advantage? :-(
I wanted to also service RAM faults without the lock, if the only thing missing was the spte (and the rest of the hierarchy was fine). But it can't be made to work without an overhaul of all of the locking.
Great, i have the same thought, anyway, it is a good start :-)
Quoted 11 lines · Xiao Guangrong
Quoted 6 lines
Quoted 3 lines
and it has little heavy sine we need to walk guest page table, and build spte under mmu-lock.
For shadow, yes, this is a good optimization. But with nested paging it slow things down. We already have the gpa, so all we need to do is follow the mmio path. There's no need to walk the spte hierarchy.
Yes, it is, i just want to detect BUG for KVM, it helps us to know if "ept misconfig" is the real MMIO or the BUG. I noticed some "ept misconfig" BUGs is reported before, so i think doing this is necessary, and i think it is not too bad, since walking spte hierarchy is lockless, it really fast.
Okay. We can later see if it show up on profiles.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 14/15] KVM: MMU: mmio page fault support
In reply to Xiao GuangrongQuoted 8 lines · Xiao Guangrong
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Xiao Guangrong
Quoted 2 lines
Also, shadow walking is not significantly faster than guest page table walking. And if we miss, we have to walk the guest page tables in any case.
Um. i think walking guest page table is slower, it needs to walk memslots for many times and it triggers page fault if the host page is swapped.
Quoted 6 lines
Quoted 2 lines
Well, if the page is swapped, we can't store anything in the spte.
If we walk guest page table, we need to access guest page, and guest page can be swapped out anytime, but shadow page table is the kernel page, it is not swapped, that is why i think walking shadow page table is faster than guest page table.
It's unlikely that the guest page table is swapped out since the hardware just walked it.
Quoted 6 lines
Quoted 6 lines
And if we only store the mmio/ram condition in the spte (via the two types of page faults) we don't need to walk the spte. We know immediately if we need to search the slots or not.
Quoted 3 lines
And it is hardly missed, since for tdp, it infrequency zaps shadow pages, for soft mmu, the mmio spte is always unsync, and in guest, the mmio region is always mapped by kernel, so it is infrequency to be update and lazily flushed.
Quoted 6 lines
Quoted 2 lines
We still get frequent mmio misses.
I did the test, run three guests(4vcpu + 512M) on my box (4cores + 2G) and compile kernel in guests, for 1 hour, no mmio is missed(hard mmu and soft mmu), it means that usually we can catch almost all mmio by walking shadow page.
Yes, but if you rely on EPT misconfig then you don't need that walk at all (conversely, if we do walk unconditionally, we can use EPT violations instead).
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
If it is direct, we only need to compare the pga, and direct=0, we only need to compare gva, i'll fix the code to make it clear.
But for nested npt, we get the ngpa, not a gva.
Quoted 6 lines
We treat nested npt as the 'direct' mmio:
r = handle_mmio_page_fault(vcpu, addr, error_code, mmu_is_nested(vcpu));also do not cache gva for nested npt:
if (handle_abnormal_pfn(vcpu,mmu_is_nested(vcpu)? 0 : addr,
walker.gfn, pfn, walker.pte_access,&r))
Okay.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 04/15] KVM: MMU: cache mmio info on page fault path
In reply to Xiao GuangrongQuoted 20 lines · Xiao Guangrong
If the page fault is caused by mmio, we can cache the mmio info, later, we do not need to walk guest page table and quickly know it is a mmio fault while we emulate the mmio instruction
Signed-off-by Xiao Guangrong <[email protected]>--- arch/x86/include/asm/kvm_host.h | 5 +++ arch/x86/kvm/mmu.c | 21 +++++---------- arch/x86/kvm/mmu.h | 23 +++++++++++++++++ arch/x86/kvm/paging_tmpl.h | 21 ++++++++++----- arch/x86/kvm/x86.c | 52 ++++++++++++++++++++++++++++++-------- arch/x86/kvm/x86.h | 36 +++++++++++++++++++++++++++ 6 files changed, 126 insertions(+), 32 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0@@ -414,6 +414,11 @@ struct kvm_vcpu_arch {u64 mcg_ctl;
arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 424
Quoted 10 lines
arch/x86/include/asm/kvm_host.h +5
-0u64 *mce_banks; +/* Cache MMIO info */ +u64 mmio_gva; +unsigned access; +gfn_t mmio_gfn; +/* used for guest single stepping over the given code position */ unsigned long singlestep_rip;
Why you're not implementing the original idea to cache the MMIO attribute of an address into the spte?
That solution is wider reaching than a one-entry cache, and was proposed to overcome large number of memslots.
If the access pattern switches between different addresses this one solution is doomed.
Re: [PATCH 04/15] KVM: MMU: cache mmio info on page fault path
Quoted 35 lines · Marcelo Tosatti
Quoted 20 lines · Xiao Guangrong
If the page fault is caused by mmio, we can cache the mmio info, later, we do not need to walk guest page table and quickly know it is a mmio fault while we emulate the mmio instruction
Signed-off-by Xiao Guangrong <[email protected]>--- arch/x86/include/asm/kvm_host.h | 5 +++ arch/x86/kvm/mmu.c | 21 +++++---------- arch/x86/kvm/mmu.h | 23 +++++++++++++++++ arch/x86/kvm/paging_tmpl.h | 21 ++++++++++----- arch/x86/kvm/x86.c | 52 ++++++++++++++++++++++++++++++-------- arch/x86/kvm/x86.h | 36 +++++++++++++++++++++++++++ 6 files changed, 126 insertions(+), 32 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0@@ -414,6 +414,11 @@ struct kvm_vcpu_arch {u64 mcg_ctl;arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 424
Quoted 10 lines
arch/x86/include/asm/kvm_host.h +5
-0u64 *mce_banks; +/* Cache MMIO info */ +u64 mmio_gva; +unsigned access; +gfn_t mmio_gfn; +/* used for guest single stepping over the given code position */ unsigned long singlestep_rip;Why you're not implementing the original idea to cache the MMIO attribute of an address into the spte?
Quoted 5 lines
That solution is wider reaching than a one-entry cache, and was proposed to overcome large number of memslots.
If the access pattern switches between different addresses this one solution is doomed.
Nevermind, its later in the series.
Re: [PATCH 02/15] KVM: MMU: do not update slot bitmap if spte is nonpresent
In reply to Xiao GuangrongQuoted 12 lines · Xiao Guangrong
Set slot bitmap only if the spte is present
Signed-off-by Xiao Guangrong <[email protected]>--- arch/x86/kvm/mmu.c | 15 +++++++-------- 1 files changed, 7 insertions(+), 8 deletions(-)arch/x86/kvm/mmu.c +0
-0
arch/x86/kvm/mmu.c static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) · line 748
Quoted 6 lines
arch/x86/kvm/mmu.c +0
-3@@ -743,9 +743,6 @@ static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)struct kvm_mmu_page *sp;unsigned long *rmapp; -if (!is_rmap_spte (*spte)) -return 0 ; -
Not sure if this is safe, what if the spte is set as nonpresent but rmap not removed?
BTW i don't see what patch 1 and this have to do with the goal of the series.
Re: [PATCH 10/15] KVM: MMU: lockless walking shadow page table
In reply to Xiao GuangrongQuoted 69 lines · Xiao Guangrong
Using rcu to protect shadow pages table to be freed, so we can safely walk it, it should run fast and is needed by mmio page fault
Signed-off-by Xiao Guangrong <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++ arch/x86/kvm/mmu.c | 79 ++++++++++++++++++++++++++++++--------- arch/x86/kvm/mmu.h | 4 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 69 insertions(+), 20 deletions(-)arch/x86/include/asm/kvm_host.h +4
-0@@ -232,6 +232,8 @@ struct kvm_mmu_page {unsigned int unsync_children;unsigned long parent_ptes;/* Reverse mapping for parent_pte */ DECLARE_BITMAP (unsync_child_bitmap,512 ); + +struct rcu_head rcu; };struct kvm_pv_mmu_op_buffer {@@ -478,6 +480,8 @@ struct kvm_arch {u64 hv_guest_os_id;u64 hv_hypercall; +atomic_t reader_counter; +#ifdef CONFIG_KVM_MMU_AUDIT int audit_point;#endif arch/x86/kvm/mmu.c +24
-0@@ -1675,6 +1675,30 @@ static int kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,return ret; } +static void free_mmu_pages_unlock_parts (struct list_head *invalid_list) +{ +struct kvm_mmu_page *sp; + +list_for_each_entry (sp, invalid_list, link) +kvm_mmu_free_lock_parts (sp); +} + +static void free_invalid_pages_rcu (struct rcu_head *head) +{ +struct kvm_mmu_page *next, *sp; + + sp = container_of(head, struct kvm_mmu_page, rcu); +while (sp) { +if (!list_empty (&sp->link)) + next =list_first_entry (&sp->link, + struct kvm_mmu_page, link); +else + next =NULL ; +kvm_mmu_free_unlock_parts (sp); + sp = next; + } +} +static void kvm_mmu_commit_zap_page (struct kvm *kvm,struct list_head *invalid_list) {
arch/x86/kvm/mmu.c static void kvm_mmu_commit_zap_page(struct kvm *kvm, · line 1718
Quoted 10 lines
arch/x86/kvm/mmu.c +7
-0@@ -1685,6 +1709,14 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm,kvm_flush_remote_tlbs (kvm); +if (atomic_read (&kvm->arch.reader_counter)) { +free_mmu_pages_unlock_parts (invalid_list); + sp = list_first_entry(invalid_list, struct kvm_mmu_page, link); +list_del_init (invalid_list); +call_rcu (&sp->rcu, free_invalid_pages_rcu); +return ; + }
This is probably wrong, the caller wants the page to be zapped by the time the function returns, not scheduled sometime in the future.
Quoted 16 lines
arch/x86/kvm/mmu.c +9
-0+ do { sp = list_first_entry(invalid_list, struct kvm_mmu_page, link); WARN_ON(!sp->role.invalid || sp->root_count);@@ -2601,6 +2633,35 @@ static gpa_t nonpaging_gva_to_gpa_nested(struct kvm_vcpu *vcpu, gva_t vaddr,return vcpu->arch.nested_mmu.translate_gpa (vcpu, vaddr, access); } +int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr, +u64 sptes[4 ]) +{ +struct kvm_shadow_walk_iterator iterator; +int nr_sptes =0 ; + +rcu_read_lock (); +
arch/x86/kvm/mmu.c static gpa_t nonpaging_gva_to_gpa_nested(struct kvm_vcpu *vcpu, gva_t vaddr, · line 2653
Quoted 10 lines
arch/x86/kvm/mmu.c +10
-0+atomic_inc (&vcpu->kvm->arch.reader_counter); +/* Increase the counter before walking shadow page table */ +smp_mb__after_atomic_inc (); + +for_each_shadow_entry (vcpu, addr, iterator) { + sptes[iterator.level-1 ] = *iterator.sptep; + nr_sptes++; +if (!is_shadow_present_pte (*iterator.sptep)) +break ; + }
Why is lockless access needed for the MMIO optimization? Note the spte contents are copied to the array here are used for debugging purposes only, their contents are potentially stale.
Re: [PATCH 02/15] KVM: MMU: do not update slot bitmap if spte is nonpresent
In reply to Marcelo TosattiQuoted 16 lines · Marcelo Tosatti
Quoted 15 lines · Xiao Guangrong
Set slot bitmap only if the spte is present
Signed-off-by Xiao Guangrong <[email protected]>--- arch/x86/kvm/mmu.c | 15 +++++++-------- 1 files changed, 7 insertions(+), 8 deletions(-)arch/x86/kvm/mmu.c +0
-0@@ -743,9 +743,6 @@ static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)struct kvm_mmu_page *sp;unsigned long *rmapp;
Quoted 6 lines
arch/x86/kvm/mmu.c static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) · line 748
Quoted 3 lines
arch/x86/kvm/mmu.c +0
-3-if (!is_rmap_spte (*spte)) -return 0 ; -Not sure if this is safe, what if the spte is set as nonpresent but rmap not removed?
It can not happen, since when we set the spte as nonpresent, we always use drop_spte to remove the rmap, we also do it in set_spte()
Quoted 5 lines
BTW i don't see what patch 1 and this have to do with the goal of the series.
There are the preparing work for mmio page fault:
- Patch 1 fix the bug in walking shadow page, so we can safely use it to lockless-ly walk shadow page
- Patch 2 can avoid add rmap for the mmio spte :-)
Re: [PATCH 10/15] KVM: MMU: lockless walking shadow page table
In reply to Marcelo TosattiQuoted 11 lines · Marcelo Tosatti
arch/x86/kvm/mmu.c static void kvm_mmu_commit_zap_page(struct kvm *kvm, · line 1718
Quoted 7 lines
arch/x86/kvm/mmu.c +7
-0+if (atomic_read (&kvm->arch.reader_counter)) { +free_mmu_pages_unlock_parts (invalid_list); + sp = list_first_entry(invalid_list, struct kvm_mmu_page, link); +list_del_init (invalid_list); +call_rcu (&sp->rcu, free_invalid_pages_rcu); +return ; + }This is probably wrong, the caller wants the page to be zapped by the time the function returns, not scheduled sometime in the future.
It can be freed soon and KVM does not reuse these pages anymore... it is not too bad, no?
Quoted 22 lines
Quoted 22 lines
arch/x86/kvm/mmu.c +15
-0+ do { sp = list_first_entry(invalid_list, struct kvm_mmu_page, link); WARN_ON(!sp->role.invalid || sp->root_count);@@ -2601,6 +2633,35 @@ static gpa_t nonpaging_gva_to_gpa_nested(struct kvm_vcpu *vcpu, gva_t vaddr,return vcpu->arch.nested_mmu.translate_gpa (vcpu, vaddr, access); } +int kvm_mmu_walk_shadow_page_lockless (struct kvm_vcpu *vcpu,u64 addr, +u64 sptes[4 ]) +{ +struct kvm_shadow_walk_iterator iterator; +int nr_sptes =0 ; + +rcu_read_lock (); + +atomic_inc (&vcpu->kvm->arch.reader_counter); +/* Increase the counter before walking shadow page table */ +smp_mb__after_atomic_inc (); + +for_each_shadow_entry (vcpu, addr, iterator) { + sptes[iterator.level-1 ] = *iterator.sptep;
Quoted 9 lines
arch/x86/kvm/mmu.c static gpa_t nonpaging_gva_to_gpa_nested(struct kvm_vcpu *vcpu, gva_t vaddr, · line 2653
Quoted 4 lines
arch/x86/kvm/mmu.c +4
-0+ nr_sptes++; +if (!is_shadow_present_pte (*iterator.sptep)) +break ; + }Why is lockless access needed for the MMIO optimization? Note the spte contents are copied to the array here are used for debugging purposes only, their contents are potentially stale.
Um, we can use it to check the mmio page fault if it is the real mmio access or the bug of KVM, i discussed it with Avi:
===============================================
Quoted 5 lines
Yes, it is, i just want to detect BUG for KVM, it helps us to know if "ept misconfig" is the real MMIO or the BUG. I noticed some "ept misconfig" BUGs is reported before, so i think doing this is necessary, and i think it is not too bad, since walking spte hierarchy is lockless, it really fast.
Okay. We can later see if it show up on profiles.
===============================================
And it is really fast, i will attach the 'perf result' when the v2 is posted.
Yes, their contents are potentially stale, we just use it to check mmio, after all, if we get the stale spte, we will call page fault path to fix it.