Nested Paging support for Nested SVM (aka NPT-Virtualization)
cover Nested Paging support for Nested SVM (aka NPT-Virtualization)
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)
Thanks,
Joerg
Diffstat:
arch/x86/include/asm/kvm_host.h | 21 ++++++ arch/x86/kvm/mmu.c | 152 ++++++++++++++++++++++++++++++--------- arch/x86/kvm/mmu.h | 2 + arch/x86/kvm/paging_tmpl.h | 81 ++++++++++++++++++--- arch/x86/kvm/svm.c | 126 +++++++++++++++++++++++++++----- arch/x86/kvm/vmx.c | 9 +++ arch/x86/kvm/x86.c | 19 +++++- include/linux/kvm.h | 1 + include/linux/kvm_host.h | 5 ++ 9 files changed, 354 insertions(+), 62 deletions(-)
Shortlog:
Joerg Roedel (18):
KVM: MMU: Check for root_level instead of long mode
KVM: MMU: Make tdp_enabled a mmu-context parameter
KVM: MMU: Make set_cr3 a function pointer in kvm_mmu
KVM: X86: Introduce a tdp_set_cr3 function
KVM: MMU: Introduce get_cr3 function pointer
KVM: MMU: Introduce inject_page_fault function pointer
KVM: SVM: Implement MMU helper functions for Nested Nested Paging
KVM: MMU: Change init_kvm_softmmu to take a context as parameter
KVM: MMU: Let is_rsvd_bits_set take mmu context instead of vcpu
KVM: MMU: Introduce generic walk_addr function
KVM: MMU: Add infrastructure for two-level page walker
KVM: MMU: Implement nested gva_to_gpa functions
KVM: MMU: Introduce Nested MMU context
KVM: SVM: Initialize Nested Nested MMU context on VMRUN
KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
KVM: X86: Add callback to let modules decide over some supported cpuid bits
KVM: SVM: Report Nested Paging support to userspace
KVM: X86: Add KVM_CAP_SVM_CPUID_FIXED
PATCH 1/18 KVM: MMU: Check for root_level instead of long mode
The walk_addr function checks for !is_long_mode in its 64 bit version. But what is meant here is a check for pae paging. Change the condition to really check for pae paging so that it also works with nested nested paging.
--- arch/x86/kvm/paging_tmpl.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/paging_tmpl.h +2 -2
@@ -128,7 +128,7 @@ walk: walker->level = vcpu->arch.mmu.root_level; pte = vcpu->arch.cr3;#if PTTYPE ==64 -if (!is_long_mode (vcpu)) { +if (vcpu->arch.mmu.root_level ==PT32E_ROOT_LEVEL ) { pte =kvm_pdptr_read (vcpu, (addr >>30 ) &3 );trace_kvm_mmu_paging_element (pte, walker->level);if (!is_present_gpte (pte))
@@ -194,7 +194,7 @@ walk: (PTTYPE == 64 || is_pse(vcpu))) || ((walker->level == PT_PDPE_LEVEL) && (pte & PT_PAGE_SIZE_MASK) && - is_long_mode(vcpu))) { + vcpu->arch.mmu.root_level == PT64_ROOT_LEVEL)) {int lvl = walker->level; walker->gfn =gpte_to_gfn_lvl (pte, lvl);
Signature · 1 line
1.7.0
PATCH 2/18 KVM: MMU: Make tdp_enabled a mmu-context parameter
In reply to Joerg RoedelThis patch changes the tdp_enabled flag from its global meaning to the mmu-context. This is necessary for Nested SVM with emulation of Nested Paging where we need an extra MMU context to shadow the Nested Nested Page Table.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 8 +++++--- 2 files changed, 6 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -254,6 +254,7 @@ struct kvm_mmu {int root_level;int shadow_root_level;union kvm_mmu_page_role base_role; +bool tdp_enabled;u64 *pae_root;u64 rsvd_bits_mask[2 ][4 ];
arch/x86/kvm/mmu.c +5 -3
@@ -1812,7 +1812,7 @@ static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep, spte |= shadow_user_mask;if (level >PT_PAGE_TABLE_LEVEL ) spte |=PT_PAGE_SIZE_MASK ; -if (tdp_enabled) +if (vcpu->arch.mmu.tdp_enabled) spte |= kvm_x86_ops->get_mt_mask (vcpu, gfn,kvm_is_mmio_pfn (pfn));
@@ -2077,7 +2077,7 @@ static int mmu_alloc_roots(struct kvm_vcpu *vcpu)hpa_t root = vcpu->arch.mmu.root_hpa;ASSERT (!VALID_PAGE (root)); -if (tdp_enabled) +if (vcpu->arch.mmu.tdp_enabled) direct =1 ;if (mmu_check_root (vcpu, root_gfn))return 1 ;
@@ -2090,7 +2090,7 @@ static int mmu_alloc_roots(struct kvm_vcpu *vcpu)return 0 ; } direct = !is_paging (vcpu); -if (tdp_enabled) +if (vcpu->arch.mmu.tdp_enabled) direct =1 ;for (i =0 ; i <4 ; ++i) {hpa_t root = vcpu->arch.mmu.pae_root[i];
@@ -2397,6 +2397,7 @@ static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) context->invlpg = nonpaging_invlpg; context->shadow_root_level = kvm_x86_ops->get_tdp_level (); context->root_hpa =INVALID_PAGE ; + vcpu->arch.mmu.tdp_enabled = true;if (!is_paging (vcpu)) { context->gva_to_gpa = nonpaging_gva_to_gpa;
@@ -2435,6 +2436,7 @@ static int init_kvm_softmmu(struct kvm_vcpu *vcpu) r =paging32_init_context (vcpu); vcpu->arch.mmu.base_role.glevels = vcpu->arch.mmu.root_level; + vcpu->arch.mmu.tdp_enabled = false;return r; }
Signature · 1 line
1.7.0
Re: [PATCH 02/18] KVM: MMU: Make tdp_enabled a mmu-context parameter
Quoted 11 lines · Joerg Roedel
This patch changes the
tdp_enabledflag from its global meaning to the mmu-context. This is necessary for Nested SVM with emulation of Nested Paging where we need an extra MMU context to shadow the Nested Nested Page Table.arch/x86/include/asm/kvm_host.h +0
-0
arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 259
Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -254,6 +254,7 @@ struct kvm_mmu {int root_level;int shadow_root_level;union kvm_mmu_page_role base_role; +bool tdp_enabled;
This needs a different name, since the old one is still around. Perhaps we could call it parent_mmu and make it a kvm_mmu pointer.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 02/18] KVM: MMU: Make tdp_enabled a mmu-context parameter
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Joerg Roedel
This patch changes the
tdp_enabledflag from its global meaning to the mmu-context. This is necessary for Nested SVM with emulation of Nested Paging where we need an extra MMU context to shadow the Nested Nested Page Table.
Quoted 13 lines
arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 258
Quoted 10 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -254,6 +254,7 @@ struct kvm_mmu {int root_level;int shadow_root_level;union kvm_mmu_page_role base_role; +bool tdp_enabled;This needs a different name, since the old one is still around. Perhaps we could call it
parent_mmuand make it akvm_mmupointer.
Hmm, how about renaming the global tdp_enabled variable to tdp_usable? The global variable indicates if tdp is _usable_ and we can _enable_ it for a mmu context.
Joerg
Re: [PATCH 02/18] KVM: MMU: Make tdp_enabled a mmu-context parameter
Quoted 24 lines · Joerg Roedel
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Joerg Roedel
This patch changes the
tdp_enabledflag from its global meaning to the mmu-context. This is necessary for Nested SVM with emulation of Nested Paging where we need an extra MMU context to shadow the Nested Nested Page Table.Quoted 14 lines
arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 259
Quoted 11 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -254,6 +254,7 @@ struct kvm_mmu {int root_level;int shadow_root_level;union kvm_mmu_page_role base_role; +bool tdp_enabled;This needs a different name, since the old one is still around. Perhaps we could call it
parent_mmuand make it akvm_mmupointer.
Quoted 4 lines
Hmm, how about renaming the global
tdp_enabledvariable totdp_usable? The global variable indicates if tdp is _usable_ and we can _enable_ it for a mmu context.
I think of the global flags as host tdp, and the mmu as guest tdp (but maybe this is wrong?). If that makes sense, the naming should reflect that.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 02/18] KVM: MMU: Make tdp_enabled a mmu-context parameter
Quoted 21 lines · Avi Kivity
Quoted 20 lines · Joerg Roedel
Quoted 19 lines · Avi Kivity
Quoted 11 lines · Joerg Roedel
This patch changes the
tdp_enabledflag from its global meaning to the mmu-context. This is necessary for Nested SVM with emulation of Nested Paging where we need an extra MMU context to shadow the Nested Nested Page Table.arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 258
Quoted 5 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -254,6 +254,7 @@ struct kvm_mmu {int root_level;int shadow_root_level;union kvm_mmu_page_role base_role; +bool tdp_enabled;This needs a different name, since the old one is still around. Perhaps we could call it
parent_mmuand make it akvm_mmupointer.
Quoted 7 lines
Quoted 3 lines
Hmm, how about renaming the global
tdp_enabledvariable totdp_usable? The global variable indicates if tdp is _usable_ and we can _enable_ it for a mmu context.I think of the global flags as host tdp, and the mmu as guest tdp (but maybe this is wrong?). If that makes sense, the naming should reflect that.
The basic flow of the mmu state with npt-npt is:
1. As long as the L1 is running the arch.mmu context is in tdp mode and builds a direct-mapped page table.
2. When vmrun is emulated and the nested vmcb enables nested paging, arch.mmu is switched to a shadow-mmu mode which now shadows the l1 nested page table.
So when the l2-guest runs with nested paging the arch.mmu.tdp_enabled variable on the host is false.
3. On a vmexit emulation the mmu is switched back to tdp handling state.
So the mmu.tdp_enabled parameter is about tdp being enabled for the mmu context (so mmu.tdp_enabled means that we build a l1-direct-mapped page table when true or shadow a l1-page-table when false). Thats why I think the 'tdp_enabled' name makes sense in the mmu-context. The global flag only shows if an mmu-context could be in tdp-state. So tdp_usable may be a good name for it.
Joerg
Re: [PATCH 02/18] KVM: MMU: Make tdp_enabled a mmu-context parameter
Quoted 50 lines · Joerg Roedel
Quoted 26 lines · Avi Kivity
Quoted 24 lines · Joerg Roedel
Quoted 22 lines · Avi Kivity
Quoted 11 lines · Joerg Roedel
This patch changes the
tdp_enabledflag from its global meaning to the mmu-context. This is necessary for Nested SVM with emulation of Nested Paging where we need an extra MMU context to shadow the Nested Nested Page Table.arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 259
Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -254,6 +254,7 @@ struct kvm_mmu {int root_level;int shadow_root_level;union kvm_mmu_page_role base_role; +bool tdp_enabled;This needs a different name, since the old one is still around. Perhaps we could call it
parent_mmuand make it akvm_mmupointer.Quoted 8 lines
Quoted 4 lines
Hmm, how about renaming the global
tdp_enabledvariable totdp_usable? The global variable indicates if tdp is _usable_ and we can _enable_ it for a mmu context.I think of the global flags as host tdp, and the mmu as guest tdp (but maybe this is wrong?). If that makes sense, the naming should reflect that.
The basic flow of the mmu state with npt-npt is:
1. As long as the L1 is running the arch.mmu context is in tdp mode and builds a direct-mapped page table.
2. When vmrun is emulated and the nested vmcb enables nested paging, arch.mmu is switched to a shadow-mmu mode which now shadows the l1 nested page table.
So when the l2-guest runs with nested paging the arch.mmu.tdp_enabled variable on the host is false.3. On a vmexit emulation the mmu is switched back to tdp handling state.
Quoted 8 lines
So the mmu.tdp_enabled parameter is about tdp being enabled for the mmu context (so mmu.tdp_enabled means that we build a l1-direct-mapped page table when true or shadow a l1-page-table when false). Thats why I think the '
tdp_enabled' name makes sense in the mmu-context. The global flag only shows if an mmu-context could be in tdp-state. Sotdp_usablemay be a good name for it.
tdp is still used in both cases, so that name is confusing. We could call it mmu.direct_map (and set it for real mode?) or mmu.virtual_map (with the opposite sense). Or something.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 02/18] KVM: MMU: Make tdp_enabled a mmu-context parameter
Quoted 4 lines · Avi Kivity
tdp is still used in both cases, so that name is confusing. We could call it mmu.direct_map (and set it for real mode?) or mmu.virtual_map (with the opposite sense). Or something.
I like the mmu.direct_map name. Its a good term too, I will change it in the patch.
Joerg
PATCH 3/18 KVM: MMU: Make set_cr3 a function pointer in kvm_mmu
In reply to Joerg RoedelThis is necessary to implement Nested Nested Paging. As a side effect this allows some cleanups in the SVM nested paging code.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 4 +++- 2 files changed, 4 insertions(+), 1 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -241,6 +241,7 @@ struct kvm_pio_request {*/ struct kvm_mmu {void (*new_cr3)(struct kvm_vcpu *vcpu); +void (*set_cr3)(struct kvm_vcpu *vcpu,unsigned long root);int (*page_fault)(struct kvm_vcpu *vcpu,gva_t gva,u32 err);void (*free)(struct kvm_vcpu *vcpu); gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t gva, u32 access,
arch/x86/kvm/mmu.c +3 -1
@@ -2398,6 +2398,7 @@ static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) context->shadow_root_level = kvm_x86_ops->get_tdp_level (); context->root_hpa =INVALID_PAGE ; vcpu->arch.mmu.tdp_enabled = true; + vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_cr3;if (!is_paging (vcpu)) { context->gva_to_gpa = nonpaging_gva_to_gpa;
@@ -2437,6 +2438,7 @@ static int init_kvm_softmmu(struct kvm_vcpu *vcpu) vcpu->arch.mmu.base_role.glevels = vcpu->arch.mmu.root_level; vcpu->arch.mmu.tdp_enabled = false; + vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_cr3;return r; }
@@ -2482,7 +2484,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu)if (r) goto out;/* set_cr3() should ensure TLB has been flushed */ - kvm_x86_ops->set_cr3 (vcpu, vcpu->arch.mmu.root_hpa); + vcpu->arch.mmu.set_cr3 (vcpu, vcpu->arch.mmu.root_hpa); out:return r; }
Signature · 1 line
1.7.0
PATCH 4/18 KVM: X86: Introduce a tdp_set_cr3 function
In reply to Joerg RoedelThis patch introduces a special set_tdp_cr3 function pointer in kvm_x86_ops which is only used for tpd enabled mmu contexts. This allows to remove some hacks from svm code.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 2 +- arch/x86/kvm/svm.c | 23 ++++++++++++++--------- arch/x86/kvm/vmx.c | 3 +++ 4 files changed, 19 insertions(+), 10 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -537,6 +537,7 @@ struct kvm_x86_ops {u64 (*get_mt_mask)(struct kvm_vcpu *vcpu,gfn_t gfn,bool is_mmio);int (*get_lpage_level)(void );bool (*rdtscp_supported)(void ); +void (*set_tdp_cr3)(struct kvm_vcpu *vcpu,unsigned long cr3);const struct trace_print_flags *exit_reasons_str; };
arch/x86/kvm/mmu.c +1 -1
@@ -2398,7 +2398,7 @@ static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) context->shadow_root_level = kvm_x86_ops->get_tdp_level (); context->root_hpa =INVALID_PAGE ; vcpu->arch.mmu.tdp_enabled = true; - vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_cr3; + vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_tdp_cr3;if (!is_paging (vcpu)) { context->gva_to_gpa = nonpaging_gva_to_gpa;
arch/x86/kvm/svm.c +14 -9
@@ -3044,9 +3044,6 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu) gs_selector =kvm_read_gs (); ldt_selector =kvm_read_ldt (); svm->vmcb->save.cr2 = vcpu->arch.cr2; -/* required for live migration with NPT */ -if (npt_enabled) - svm->vmcb->save.cr3 = vcpu->arch.cr3;clgi ();
@@ -3155,16 +3152,22 @@ static void svm_set_cr3(struct kvm_vcpu *vcpu, unsigned long root) {struct vcpu_svm *svm =to_svm (vcpu); -if (npt_enabled) { - svm->vmcb->control.nested_cr3 = root; -force_new_asid (vcpu); -return ; - } - svm->vmcb->save.cr3 = root;force_new_asid (vcpu); } +static void set_tdp_cr3 (struct kvm_vcpu *vcpu,unsigned long root) +{ +struct vcpu_svm *svm =to_svm (vcpu); + + svm->vmcb->control.nested_cr3 = root; + +/* Also sync guest cr3 here in case we live migrate */ + svm->vmcb->save.cr3 = vcpu->arch.cr3; + +force_new_asid (vcpu); +} +static int is_disabled (void ) {u64 vm_cr;
@@ -3361,6 +3364,8 @@ static struct kvm_x86_ops svm_x86_ops = { .cpuid_update = svm_cpuid_update, .rdtscp_supported = svm_rdtscp_supported, + + .set_tdp_cr3 = set_tdp_cr3, };static int __init svm_init (void )
arch/x86/kvm/vmx.c +3 -0
@@ -4212,11 +4212,14 @@ static struct kvm_x86_ops vmx_x86_ops = { .get_mt_mask = vmx_get_mt_mask, .exit_reasons_str = vmx_exit_reasons_str, + .get_lpage_level = vmx_get_lpage_level, .cpuid_update = vmx_cpuid_update, .rdtscp_supported = vmx_rdtscp_supported, + + .set_tdp_cr3 = vmx_set_cr3, };static int __init vmx_init (void )
Signature · 1 line
1.7.0
PATCH 5/18 KVM: MMU: Introduce get_cr3 function pointer
In reply to Joerg RoedelThis function pointer in the MMU context is required to implement Nested Nested Paging.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 9 ++++++++- arch/x86/kvm/paging_tmpl.h | 4 ++-- 3 files changed, 11 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -242,6 +242,7 @@ struct kvm_pio_request {struct kvm_mmu {void (*new_cr3)(struct kvm_vcpu *vcpu);void (*set_cr3)(struct kvm_vcpu *vcpu,unsigned long root); +unsigned long (*get_cr3)(struct kvm_vcpu *vcpu);int (*page_fault)(struct kvm_vcpu *vcpu,gva_t gva,u32 err);void (*free)(struct kvm_vcpu *vcpu); gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t gva, u32 access,
arch/x86/kvm/mmu.c +8 -1
@@ -2071,7 +2071,7 @@ static int mmu_alloc_roots(struct kvm_vcpu *vcpu)int direct =0 ;u64 pdptr; - root_gfn = vcpu->arch.cr3 >>PAGE_SHIFT ; + root_gfn = vcpu->arch.mmu.get_cr3 (vcpu) >>PAGE_SHIFT ;if (vcpu->arch.mmu.shadow_root_level ==PT64_ROOT_LEVEL ) {hpa_t root = vcpu->arch.mmu.root_hpa;
@@ -2254,6 +2254,11 @@ static void paging_new_cr3(struct kvm_vcpu *vcpu)mmu_free_roots (vcpu); } +static unsigned long get_cr3 (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.cr3; +} +static void inject_page_fault (struct kvm_vcpu *vcpu,u64 addr,u32 err_code)
@@ -2399,6 +2404,7 @@ static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) context->root_hpa =INVALID_PAGE ; vcpu->arch.mmu.tdp_enabled = true; vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_tdp_cr3; + vcpu->arch.mmu.get_cr3 = get_cr3;if (!is_paging (vcpu)) { context->gva_to_gpa = nonpaging_gva_to_gpa;
@@ -2439,6 +2445,7 @@ static int init_kvm_softmmu(struct kvm_vcpu *vcpu) vcpu->arch.mmu.base_role.glevels = vcpu->arch.mmu.root_level; vcpu->arch.mmu.tdp_enabled = false; vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_cr3; + vcpu->arch.mmu.get_cr3 = get_cr3;return r; }
arch/x86/kvm/paging_tmpl.h +2 -2
@@ -126,7 +126,7 @@ static int FNAME(walk_addr)(struct guest_walker *walker, fetch_fault); walk: walker->level = vcpu->arch.mmu.root_level; - pte = vcpu->arch.cr3; + pte = vcpu->arch.mmu.get_cr3 (vcpu);#if PTTYPE ==64 if (vcpu->arch.mmu.root_level ==PT32E_ROOT_LEVEL ) { pte =kvm_pdptr_read (vcpu, (addr >>30 ) &3 );
@@ -137,7 +137,7 @@ walk: }#endif ASSERT ((!is_long_mode (vcpu) &&is_pae (vcpu)) || - (vcpu->arch.cr3 &CR3_NONPAE_RESERVED_BITS ) ==0 ); + (vcpu->arch.mmu.get_cr3 (vcpu) &CR3_NONPAE_RESERVED_BITS ) ==0 ); pt_access =ACC_ALL ;
Signature · 1 line
1.7.0
PATCH 6/18 KVM: MMU: Introduce inject_page_fault function pointer
In reply to Joerg RoedelThis patch introduces an inject_page_fault function pointer into struct kvm_mmu which will be used to inject a page fault. This will be used later when Nested Nested Paging is implemented.
--- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu.c | 3 ++- 2 files changed, 5 insertions(+), 1 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -244,6 +244,9 @@ struct kvm_mmu {void (*set_cr3)(struct kvm_vcpu *vcpu,unsigned long root);unsigned long (*get_cr3)(struct kvm_vcpu *vcpu);int (*page_fault)(struct kvm_vcpu *vcpu,gva_t gva,u32 err); +void (*inject_page_fault)(struct kvm_vcpu *vcpu, +unsigned long addr, +u32 error_code);void (*free)(struct kvm_vcpu *vcpu);gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu,gva_t gva,u32 access,u32 *error);
arch/x86/kvm/mmu.c +2 -1
@@ -2263,7 +2263,7 @@ static void inject_page_fault(struct kvm_vcpu *vcpu, u64 addr, u32 err_code) { -kvm_inject_page_fault (vcpu ,addr ,err_code ); + vcpu->arch.mmu.inject_page_fault (vcpu, addr, err_code); }static void paging_free (struct kvm_vcpu *vcpu)
@@ -2446,6 +2446,7 @@ static int init_kvm_softmmu(struct kvm_vcpu *vcpu) vcpu->arch.mmu.tdp_enabled = false; vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_cr3; vcpu->arch.mmu.get_cr3 = get_cr3; + vcpu->arch.mmu.inject_page_fault = kvm_inject_page_fault;return r; }
Signature · 1 line
1.7.0
PATCH 7/18 KVM: SVM: Implement MMU helper functions for Nested Nested Paging
In reply to Joerg RoedelThis patch adds the helper functions which will be used in the mmu context for handling nested nested page faults.
--- arch/x86/kvm/svm.c | 33 +++++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
arch/x86/kvm/svm.c +33 -0
@@ -91,6 +91,9 @@ struct nested_state {u32 intercept_exceptions;u64 intercept; +/* Nested Paging related state */ +u64 nested_cr3; + };#define MSRPM_OFFSETS 16
@@ -1545,6 +1548,36 @@ static int vmmcall_interception(struct vcpu_svm *svm)return 1 ; } +static unsigned long nested_svm_get_tdp_cr3 (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); + +return svm->nested.nested_cr3; +} + +static void nested_svm_set_tdp_cr3 (struct kvm_vcpu *vcpu, +unsigned long root) +{ +struct vcpu_svm *svm =to_svm (vcpu); + + svm->vmcb->control.nested_cr3 = root; +force_new_asid (vcpu); +} + +static void nested_svm_inject_npf_exit (struct kvm_vcpu *vcpu, +unsigned long addr, +u32 error_code) +{ +struct vcpu_svm *svm =to_svm (vcpu); + + svm->vmcb->control.exit_code =SVM_EXIT_NPF ; + svm->vmcb->control.exit_code_hi =0 ; + svm->vmcb->control.exit_info_1 = error_code; + svm->vmcb->control.exit_info_2 = addr; + +nested_svm_vmexit (svm); +} +static int nested_svm_check_permissions (struct vcpu_svm *svm) { if (!(svm->vcpu.arch.efer & EFER_SVME)
Signature · 1 line
1.7.0
PATCH 8/18 KVM: MMU: Change init_kvm_softmmu to take a context as parameter
In reply to Joerg RoedelSome logic of this function is required to build the Nested Nested Paging context. So factor the required logic into a seperate function and export it.
Also make the whole init path suitable for more than one mmu context.
--- arch/x86/kvm/mmu.c | 60 ++++++++++++++++++++++++++++++--------------------- arch/x86/kvm/mmu.h | 1 + 2 files changed, 36 insertions(+), 25 deletions(-)
arch/x86/kvm/mmu.c +35 -25
@@ -2225,10 +2225,9 @@ static void nonpaging_free(struct kvm_vcpu *vcpu)mmu_free_roots (vcpu); } -static int nonpaging_init_context (struct kvm_vcpu *vcpu) +static int nonpaging_init_context (struct kvm_vcpu *vcpu, +struct kvm_mmu *context) { -struct kvm_mmu *context = &vcpu->arch.mmu; - context->new_cr3 = nonpaging_new_cr3; context->page_fault = nonpaging_page_fault; context->gva_to_gpa = nonpaging_gva_to_gpa;
@@ -2287,9 +2286,10 @@ static bool is_rsvd_bits_set(struct kvm_vcpu *vcpu, u64 gpte, int level)#include "paging_tmpl.h" #undef PTTYPE -static void reset_rsvds_bits_mask (struct kvm_vcpu *vcpu,int level) +static void reset_rsvds_bits_mask (struct kvm_vcpu *vcpu, +struct kvm_mmu *context, +int level) { -struct kvm_mmu *context = &vcpu->arch.mmu;int maxphyaddr =cpuid_maxphyaddr (vcpu);u64 exb_bit_rsvd =0 ;
@@ -2342,9 +2342,11 @@ static void reset_rsvds_bits_mask(struct kvm_vcpu *vcpu, int level) } } -static int paging64_init_context_common (struct kvm_vcpu *vcpu,int level) +static int paging64_init_context_common (struct kvm_vcpu *vcpu, +struct kvm_mmu *context, +int level) { -struct kvm_mmu *context = &vcpu->arch.mmu; +reset_rsvds_bits_mask (vcpu, context, level);ASSERT (is_pae (vcpu)); context->new_cr3 = paging_new_cr3;
@@ -2360,17 +2362,17 @@ static int paging64_init_context_common(struct kvm_vcpu *vcpu, int level)return 0 ; } -static int paging64_init_context (struct kvm_vcpu *vcpu) +static int paging64_init_context (struct kvm_vcpu *vcpu, +struct kvm_mmu *context) { -reset_rsvds_bits_mask (vcpu,PT64_ROOT_LEVEL ); -return paging64_init_context_common (vcpu,PT64_ROOT_LEVEL ); +return paging64_init_context_common (vcpu, context,PT64_ROOT_LEVEL ); } -static int paging32_init_context (struct kvm_vcpu *vcpu) +static int paging32_init_context (struct kvm_vcpu *vcpu, +struct kvm_mmu *context) { -struct kvm_mmu *context = &vcpu->arch.mmu; +reset_rsvds_bits_mask (vcpu, context,PT32_ROOT_LEVEL ); -reset_rsvds_bits_mask (vcpu,PT32_ROOT_LEVEL ); context->new_cr3 = paging_new_cr3; context->page_fault = paging32_page_fault; context->gva_to_gpa = paging32_gva_to_gpa;
@@ -2384,10 +2386,10 @@ static int paging32_init_context(struct kvm_vcpu *vcpu)return 0 ; } -static int paging32E_init_context (struct kvm_vcpu *vcpu) +static int paging32E_init_context (struct kvm_vcpu *vcpu, +struct kvm_mmu *context) { -reset_rsvds_bits_mask (vcpu,PT32E_ROOT_LEVEL ); -return paging64_init_context_common (vcpu,PT32E_ROOT_LEVEL ); +return paging64_init_context_common (vcpu, context,PT32E_ROOT_LEVEL ); }static int init_kvm_tdp_mmu (struct kvm_vcpu *vcpu)
@@ -2410,15 +2412,15 @@ static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) context->gva_to_gpa = nonpaging_gva_to_gpa; context->root_level =0 ; }else if (is_long_mode (vcpu)) { -reset_rsvds_bits_mask (vcpu,PT64_ROOT_LEVEL ); +reset_rsvds_bits_mask (vcpu, context,PT64_ROOT_LEVEL ); context->gva_to_gpa = paging64_gva_to_gpa; context->root_level =PT64_ROOT_LEVEL ; }else if (is_pae (vcpu)) { -reset_rsvds_bits_mask (vcpu,PT32E_ROOT_LEVEL ); +reset_rsvds_bits_mask (vcpu, context,PT32E_ROOT_LEVEL ); context->gva_to_gpa = paging64_gva_to_gpa; context->root_level =PT32E_ROOT_LEVEL ; }else { -reset_rsvds_bits_mask (vcpu,PT32_ROOT_LEVEL ); +reset_rsvds_bits_mask (vcpu, context,PT32_ROOT_LEVEL ); context->gva_to_gpa = paging32_gva_to_gpa; context->root_level =PT32_ROOT_LEVEL ; }
@@ -2426,24 +2428,32 @@ static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)return 0 ; } -static int init_kvm_softmmu (struct kvm_vcpu *vcpu) +int kvm_init_shadow_mmu (struct kvm_vcpu *vcpu,struct kvm_mmu *context) {int r; -ASSERT (vcpu);ASSERT (!VALID_PAGE (vcpu->arch.mmu.root_hpa));if (!is_paging (vcpu)) - r =nonpaging_init_context (vcpu); + r =nonpaging_init_context (vcpu, context);else if (is_long_mode (vcpu)) - r =paging64_init_context (vcpu); + r =paging64_init_context (vcpu, context);else if (is_pae (vcpu)) - r =paging32E_init_context (vcpu); + r =paging32E_init_context (vcpu, context);else - r =paging32_init_context (vcpu); + r =paging32_init_context (vcpu, context); vcpu->arch.mmu.base_role.glevels = vcpu->arch.mmu.root_level; vcpu->arch.mmu.tdp_enabled = false; + +return r; +} +EXPORT_SYMBOL_GPL (kvm_init_shadow_mmu); + +static int init_kvm_softmmu (struct kvm_vcpu *vcpu) +{ +int r =kvm_init_shadow_mmu (vcpu, &vcpu->arch.mmu); + vcpu->arch.mmu.set_cr3 = kvm_x86_ops->set_cr3; vcpu->arch.mmu.get_cr3 = get_cr3; vcpu->arch.mmu.inject_page_fault = kvm_inject_page_fault;
arch/x86/kvm/mmu.h +1 -0
@@ -49,6 +49,7 @@#define PFERR_FETCH_MASK (1U << 4)int kvm_mmu_get_spte_hierarchy (struct kvm_vcpu *vcpu,u64 addr,u64 sptes[4 ]); +int kvm_init_shadow_mmu (struct kvm_vcpu *vcpu,struct kvm_mmu *context);static inline void kvm_mmu_free_some_pages (struct kvm_vcpu *vcpu) {
Signature · 1 line
1.7.0
PATCH 9/18 KVM: MMU: Let is_rsvd_bits_set take mmu context instead of vcpu
In reply to Joerg RoedelThis patch changes is_rsvd_bits_set() function prototype to take only a kvm_mmu context instead of a full vcpu.
--- arch/x86/kvm/mmu.c | 4 ++-- arch/x86/kvm/paging_tmpl.h | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu.c +2 -2
@@ -2270,12 +2270,12 @@ static void paging_free(struct kvm_vcpu *vcpu)nonpaging_free (vcpu); } -static bool is_rsvd_bits_set (struct kvm_vcpu *vcpu,u64 gpte,int level) +static bool is_rsvd_bits_set (struct kvm_mmu *mmu,u64 gpte,int level) {int bit7; bit7 = (gpte >>7 ) &1 ; -return (gpte & vcpu->arch.mmu.rsvd_bits_mask[bit7][level-1 ]) !=0 ; +return (gpte & mmu->rsvd_bits_mask[bit7][level-1 ]) !=0 ; }#define PTTYPE 64
arch/x86/kvm/paging_tmpl.h +2 -1
@@ -158,7 +158,8 @@ walk:if (!is_present_gpte (pte)) goto not_present; - rsvd_fault =is_rsvd_bits_set (vcpu, pte, walker->level); + rsvd_fault =is_rsvd_bits_set (&vcpu->arch.mmu, pte, + walker->level);if (rsvd_fault) goto access_error;
Signature · 1 line
1.7.0
PATCH 10/18 KVM: MMU: Introduce generic walk_addr function
In reply to Joerg RoedelThis is the first patch in the series towards a generic walk_addr implementation which could walk two-dimensional page tables in the end. In this first step the walk_addr function is renamed into walk_addr_generic which takes an mmu context as an additional parameter.
--- arch/x86/kvm/paging_tmpl.h | 30 ++++++++++++++++++++---------- 1 files changed, 20 insertions(+), 10 deletions(-)
arch/x86/kvm/paging_tmpl.h +20 -10
@@ -112,9 +112,10 @@ static unsigned FNAME(gpte_access)(struct kvm_vcpu *vcpu, pt_element_t gpte)/* * Fetch a guest pte for a guest virtual address */ -static int FNAME (walk_addr )(struct guest_walker *walker, -struct kvm_vcpu *vcpu,gva_t addr, -int write_fault,int user_fault,int fetch_fault) +static int FNAME (walk_addr_generic )(struct guest_walker *walker, +struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +gva_t addr,int write_fault, +int user_fault,int fetch_fault) {pt_element_t pte;gfn_t table_gfn;
@@ -125,10 +126,12 @@ static int FNAME(walk_addr)(struct guest_walker *walker,trace_kvm_mmu_pagetable_walk (addr, write_fault, user_fault, fetch_fault); walk: - walker->level = vcpu->arch.mmu.root_level; - pte = vcpu->arch.mmu.get_cr3 (vcpu); + + walker->level = mmu->root_level; + pte = mmu->get_cr3 (vcpu); +#if PTTYPE ==64 -if (vcpu->arch.mmu.root_level ==PT32E_ROOT_LEVEL ) { +if (walker->level ==PT32E_ROOT_LEVEL ) { pte =kvm_pdptr_read (vcpu, (addr >>30 ) &3 );trace_kvm_mmu_paging_element (pte, walker->level);if (!is_present_gpte (pte))
@@ -137,7 +140,7 @@ walk: }#endif ASSERT ((!is_long_mode (vcpu) &&is_pae (vcpu)) || - (vcpu->arch.mmu.get_cr3 (vcpu) &CR3_NONPAE_RESERVED_BITS ) ==0 ); + (mmu->get_cr3 (vcpu) &CR3_NONPAE_RESERVED_BITS ) ==0 ); pt_access =ACC_ALL ;
@@ -158,8 +161,7 @@ walk:if (!is_present_gpte (pte)) goto not_present; - rsvd_fault =is_rsvd_bits_set (&vcpu->arch.mmu, pte, - walker->level); + rsvd_fault =is_rsvd_bits_set (mmu, pte, walker->level);if (rsvd_fault) goto access_error;
@@ -195,7 +197,7 @@ walk: (PTTYPE == 64 || is_pse(vcpu))) || ((walker->level == PT_PDPE_LEVEL) && (pte & PT_PAGE_SIZE_MASK) && - vcpu->arch.mmu.root_level == PT64_ROOT_LEVEL)) { + mmu->root_level == PT64_ROOT_LEVEL)) {int lvl = walker->level; walker->gfn =gpte_to_gfn_lvl (pte, lvl);
@@ -253,6 +255,14 @@ err:return 0 ; } +static int FNAME (walk_addr )(struct guest_walker *walker, +struct kvm_vcpu *vcpu,gva_t addr, +int write_fault,int user_fault,int fetch_fault) +{ +return FNAME (walk_addr_generic)(walker, vcpu, &vcpu->arch.mmu, addr, + write_fault, user_fault, fetch_fault); +} +static void FNAME (update_pte )(struct kvm_vcpu *vcpu,struct kvm_mmu_page *page,u64 *spte,const void *pte) {
Signature · 1 line
1.7.0
PATCH 11/18 KVM: MMU: Add infrastructure for two-level page walker
In reply to Joerg RoedelThis patch introduces a mmu-callback to translate gpa addresses in the walk_addr code. This is later used to translate l2_gpa addresses into l1_gpa addresses.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 7 +++++++ arch/x86/kvm/paging_tmpl.h | 19 +++++++++++++++++++ include/linux/kvm_host.h | 5 +++++ 4 files changed, 32 insertions(+), 0 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -250,6 +250,7 @@ struct kvm_mmu {void (*free)(struct kvm_vcpu *vcpu);gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu,gva_t gva,u32 access,u32 *error); +gpa_t (*translate_gpa)(struct kvm_vcpu *vcpu,gpa_t gpa,u32 *error);void (*prefetch_page)(struct kvm_vcpu *vcpu,struct kvm_mmu_page *page); int (*sync_page)(struct kvm_vcpu *vcpu,
arch/x86/kvm/mmu.c +7 -0
@@ -2149,6 +2149,11 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu)spin_unlock (&vcpu->kvm->mmu_lock); } +static gpa_t translate_gpa (struct kvm_vcpu *vcpu,gpa_t gpa,u32 *error) +{ +return gpa; +} +static gpa_t nonpaging_gva_to_gpa (struct kvm_vcpu *vcpu,gva_t vaddr,u32 access,u32 *error) {
@@ -2399,6 +2404,7 @@ static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) context->new_cr3 = nonpaging_new_cr3; context->page_fault = tdp_page_fault; context->free = nonpaging_free; + context->translate_gpa = translate_gpa; context->prefetch_page = nonpaging_prefetch_page; context->sync_page = nonpaging_sync_page; context->invlpg = nonpaging_invlpg;
@@ -2443,6 +2449,7 @@ int kvm_init_shadow_mmu(struct kvm_vcpu *vcpu, struct kvm_mmu *context)else r =paging32_init_context (vcpu, context); + vcpu->arch.mmu.translate_gpa = translate_gpa; vcpu->arch.mmu.base_role.glevels = vcpu->arch.mmu.root_level; vcpu->arch.mmu.tdp_enabled = false;
arch/x86/kvm/paging_tmpl.h +19 -0
@@ -122,6 +122,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,unsigned index, pt_access, pte_access;gpa_t pte_gpa;int rsvd_fault =0 ; +u32 error;trace_kvm_mmu_pagetable_walk (addr, write_fault, user_fault, fetch_fault);
@@ -150,6 +151,15 @@ walk: table_gfn =gpte_to_gfn (pte); pte_gpa =gfn_to_gpa (table_gfn); pte_gpa += index *sizeof (pt_element_t); + + pte_gpa = mmu->translate_gpa (vcpu, pte_gpa, &error); +if (pte_gpa ==UNMAPPED_GVA ) { + walker->error_code = error; +return 0 ; + } +/* pte_gpa might have changed - recalculate table_gfn */ + table_gfn =gpa_to_gfn (pte_gpa); + walker->table_gfn[walker->level -1 ] = table_gfn; walker->pte_gpa[walker->level -1 ] = pte_gpa;
@@ -209,6 +219,15 @@ walk: is_cpuid_PSE36()) walker->gfn +=pse36_gfn_delta (pte); +/* Do the final translation */ + pte_gpa =gfn_to_gpa (walker->gfn); + pte_gpa = mmu->translate_gpa (vcpu, pte_gpa, &error); +if (pte_gpa ==UNMAPPED_GVA ) { + walker->error_code = error; +return 0 ; + } + walker->gfn =gpa_to_gfn (pte_gpa); +break ; }
include/linux/kvm_host.h +5 -0
@@ -503,6 +503,11 @@ static inline gpa_t gfn_to_gpa(gfn_t gfn)return (gpa_t )gfn <<PAGE_SHIFT ; } +static inline gfn_t gpa_to_gfn (gpa_t gpa) +{ +return (gfn_t )gpa >>PAGE_SHIFT ; +} +static inline hpa_t pfn_to_hpa (pfn_t pfn) {return (hpa_t )pfn <<PAGE_SHIFT ;
Signature · 1 line
1.7.0
Re: [PATCH 11/18] KVM: MMU: Add infrastructure for two-level page walker
Quoted 18 lines · Joerg Roedel
This patch introduces a mmu-callback to translate gpa addresses in the
walk_addrcode. This is later used to translatel2_gpaaddresses intol1_gpaaddresses.Signed-off-by Joerg Roedel<[email protected]>--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 7 +++++++ arch/x86/kvm/paging_tmpl.h | 19 +++++++++++++++++++ include/linux/kvm_host.h | 5 +++++ 4 files changed, 32 insertions(+), 0 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0
arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 256
Quoted 7 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -250,6 +250,7 @@ struct kvm_mmu {void (*free)(struct kvm_vcpu *vcpu);gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu,gva_t gva,u32 access,u32 *error); +gpa_t (*translate_gpa)(struct kvm_vcpu *vcpu,gpa_t gpa,u32 *error);void (*prefetch_page)(struct kvm_vcpu *vcpu,struct kvm_mmu_page *page); int (*sync_page)(struct kvm_vcpu *vcpu,
I think placing this here means we will miss a few translations, namely when we do a physical access (say, reading PDPTEs or similar).
We need to do this on the level of kvm_read_guest() so we capture physical accesses:
kvm_read_guest_virt
-> walk_addr
-> kvm_read_guest_tdp
-> kvm_read_guest_virt
-> walk_addr
-> kvm_read_guest_tdp
-> kvm_read_guestOf course, not all accesses will use kvm_read_guest_tdp; for example kvmclock accesses should still go untranslated.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 11/18] KVM: MMU: Add infrastructure for two-level page walker
Quoted 35 lines · Avi Kivity
Quoted 17 lines · Joerg Roedel
This patch introduces a mmu-callback to translate gpa addresses in the
walk_addrcode. This is later used to translatel2_gpaaddresses intol1_gpaaddresses.Signed-off-by Joerg Roedel<[email protected]>--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 7 +++++++ arch/x86/kvm/paging_tmpl.h | 19 +++++++++++++++++++ include/linux/kvm_host.h | 5 +++++ 4 files changed, 32 insertions(+), 0 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 256
Quoted 7 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -250,6 +250,7 @@ struct kvm_mmu {void (*free)(struct kvm_vcpu *vcpu);gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu,gva_t gva,u32 access,u32 *error); +gpa_t (*translate_gpa)(struct kvm_vcpu *vcpu,gpa_t gpa,u32 *error);void (*prefetch_page)(struct kvm_vcpu *vcpu,struct kvm_mmu_page *page); int (*sync_page)(struct kvm_vcpu *vcpu,I think placing this here means we will miss a few translations, namely when we do a physical access (say, reading PDPTEs or similar).
We need to do this on the level of
kvm_read_guest()so we capture physical accesses:
kvm_read_guest_virt
->walk_addr
Quoted 8 lines
-> kvm_read_guest_tdp -> kvm_read_guest_virt -> walk_addr -> kvm_read_guest_tdp -> kvm_read_guestOf course, not all accesses will use
kvm_read_guest_tdp; for example kvmclock accesses should still go untranslated.
Ok, doing the translation in kvm_read_guest is certainly the more generic approach. I already fixed a bug related to loading l2 pdptr pointers. Doing the translation in kvm_read_guest makes the code a lot nicer.
Joerg
PATCH 12/18 KVM: MMU: Implement nested gva_to_gpa functions
In reply to Joerg RoedelThis patch adds the functions to do a nested l2_gva to l1_gpa page table walk.
--- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu.c | 8 ++++++++ arch/x86/kvm/paging_tmpl.h | 31 +++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 0 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -295,6 +295,9 @@ struct kvm_vcpu_arch {bool tpr_access_reporting;struct kvm_mmu mmu; + +struct kvm_mmu nested_mmu; +/* only needed in kvm_pv_mmu_op() path, but it's hot so * put it here to avoid allocation */ struct kvm_pv_mmu_op_buffer mmu_op_buffer;
arch/x86/kvm/mmu.c +8 -0
@@ -2162,6 +2162,14 @@ static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, gva_t vaddr,return vaddr; } +static gpa_t nonpaging_gva_to_gpa_nested (struct kvm_vcpu *vcpu,gva_t vaddr, +u32 access,u32 *error) +{ +if (error) + *error =0 ; +return vcpu->arch.nested_mmu.translate_gpa (vcpu, vaddr, error); +} +static int nonpaging_page_fault (struct kvm_vcpu *vcpu,gva_t gva,u32 error_code) {
arch/x86/kvm/paging_tmpl.h +31 -0
@@ -282,6 +282,16 @@ static int FNAME(walk_addr)(struct guest_walker *walker, write_fault, user_fault, fetch_fault); } +static int FNAME (walk_addr_nested )(struct guest_walker *walker, +struct kvm_vcpu *vcpu,gva_t addr, +int write_fault,int user_fault, +int fetch_fault) +{ +return FNAME (walk_addr_generic)(walker, vcpu, &vcpu->arch.nested_mmu, + addr, write_fault, user_fault, + fetch_fault); +} +static void FNAME (update_pte )(struct kvm_vcpu *vcpu,struct kvm_mmu_page *page,u64 *spte,const void *pte) {
@@ -541,6 +551,27 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t vaddr, u32 access,return gpa; } +static gpa_t FNAME (gva_to_gpa_nested )(struct kvm_vcpu *vcpu,gva_t vaddr, +u32 access,u32 *error) +{ +struct guest_walker walker; +gpa_t gpa =UNMAPPED_GVA ; +int r; + + r =FNAME (walk_addr_nested)(&walker, vcpu, vaddr, + !!(access &PFERR_WRITE_MASK ), + !!(access &PFERR_USER_MASK ), + !!(access &PFERR_FETCH_MASK )); + +if (r) { + gpa =gfn_to_gpa (walker.gfn); + gpa |= vaddr & ~PAGE_MASK ; + }else if (error) + *error = walker.error_code; + +return gpa; +} +static void FNAME (prefetch_page )(struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) {
Signature · 1 line
1.7.0
PATCH 13/18 KVM: MMU: Introduce Nested MMU context
In reply to Joerg RoedelThis patch introduces a second MMU context which will hold the paging information for the l2 guest.
--- arch/x86/include/asm/kvm_host.h | 8 ++++++ arch/x86/kvm/mmu.c | 48 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 55 insertions(+), 1 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -0
@@ -264,6 +264,13 @@ struct kvm_mmu {u64 *pae_root;u64 rsvd_bits_mask[2 ][4 ]; + +/* +* If true the mmu runs in two-level mode. +* vcpu->arch.nested_mmu needs to contain meaningful +* values then. +*/ +bool nested; };struct kvm_vcpu_arch {
@@ -296,6 +303,7 @@ struct kvm_vcpu_arch {struct kvm_mmu mmu; +/* This will hold the mmu context of the second level guest */ struct kvm_mmu nested_mmu;/* only needed in kvm_pv_mmu_op() path, but it's hot so
arch/x86/kvm/mmu.c +47 -1
64 lines not on this page · raw message
Signature · 1 line
1.7.0
PATCH 14/18 KVM: SVM: Initialize Nested Nested MMU context on VMRUN
In reply to Joerg RoedelThis patch adds code to initialize the Nested Nested Paging MMU context when the L1 guest executes a VMRUN instruction and has nested paging enabled in its VMCB.
--- arch/x86/kvm/mmu.c | 1 + arch/x86/kvm/svm.c | 56 ++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 47 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu.c +1 -0
@@ -2573,6 +2573,7 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu) {mmu_free_roots (vcpu); } +EXPORT_SYMBOL_GPL (kvm_mmu_unload); static void mmu_pte_write_zap_pte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
arch/x86/kvm/svm.c +46 -10
104 lines not on this page · raw message
Signature · 1 line
1.7.0
PATCH 15/18 KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
In reply to Joerg RoedelThis patch implements logic to make sure that either a page-fault/page-fault-vmexit or a nested-page-fault-vmexit is propagated back to the guest.
--- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/paging_tmpl.h | 2 ++ arch/x86/kvm/x86.c | 15 ++++++++++++++- 3 files changed, 17 insertions(+), 1 deletions(-)
arch/x86/kvm/mmu.h +1 -0
@@ -47,6 +47,7 @@#define PFERR_USER_MASK (1U << 2)#define PFERR_RSVD_MASK (1U << 3)#define PFERR_FETCH_MASK (1U << 4) +#define PFERR_NESTED_MASK (1U << 31)int kvm_mmu_get_spte_hierarchy (struct kvm_vcpu *vcpu,u64 addr,u64 sptes[4 ]);int kvm_init_shadow_mmu (struct kvm_vcpu *vcpu,struct kvm_mmu *context);
arch/x86/kvm/paging_tmpl.h +2 -0
@@ -154,6 +154,7 @@ walk: pte_gpa = mmu->translate_gpa (vcpu, pte_gpa, &error);if (pte_gpa ==UNMAPPED_GVA ) { + error |=PFERR_NESTED_MASK ; walker->error_code = error;return 0 ; }
@@ -223,6 +224,7 @@ walk: pte_gpa =gfn_to_gpa (walker->gfn); pte_gpa = mmu->translate_gpa (vcpu, pte_gpa, &error);if (pte_gpa ==UNMAPPED_GVA ) { + error |=PFERR_NESTED_MASK ; walker->error_code = error;return 0 ; }
arch/x86/kvm/x86.c +14 -1
@@ -314,6 +314,19 @@ void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long addr,kvm_queue_exception_e (vcpu,PF_VECTOR , error_code); } +void kvm_propagate_fault (struct kvm_vcpu *vcpu,unsigned long addr,u32 error_code) +{ +u32 nested, error; + + nested = error_code &PFERR_NESTED_MASK ; + error = error_code & ~PFERR_NESTED_MASK ; + +if (vcpu->arch.mmu.nested && !(error_code &&PFERR_NESTED_MASK )) + vcpu->arch.nested_mmu.inject_page_fault (vcpu, addr, error); +else + vcpu->arch.mmu.inject_page_fault (vcpu, addr, error); +} +void kvm_inject_nmi (struct kvm_vcpu *vcpu) { vcpu->arch.nmi_pending =1 ;
@@ -3546,7 +3559,7 @@ static int pio_copy_data(struct kvm_vcpu *vcpu) ret =kvm_read_guest_virt (q, p, bytes, vcpu, &error_code);if (ret ==X86EMUL_PROPAGATE_FAULT ) -kvm_inject_page_fault (vcpu, q, error_code); +kvm_propagate_fault (vcpu, q, error_code);return ret; }
Signature · 1 line
1.7.0
Re: [PATCH 15/18] KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
Quoted 6 lines · Joerg Roedel
arch/x86/kvm/x86.c +0
-0@@ -314,6 +314,19 @@ void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long addr,kvm_queue_exception_e (vcpu,PF_VECTOR , error_code)
arch/x86/kvm/x86.c void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long addr, · line 324
Quoted 10 lines
arch/x86/kvm/x86.c +8
-0} +void kvm_propagate_fault (struct kvm_vcpu *vcpu,unsigned long addr,u32 error_code) +{ +u32 nested, error; + + nested = error_code &PFERR_NESTED_MASK ; + error = error_code & ~PFERR_NESTED_MASK ; + +if (vcpu->arch.mmu.nested && !(error_code &&PFERR_NESTED_MASK ))
This looks incorrect, nested is unused.
At the very least it should be a binary & operation
if (vcpu->arch.mmu.nested && !(error_code & PFERR_NESTED_MASK))
which can be simplified to
if (vcpu->arch.mmu.nested && !nested)
but it seems wrong that the condition is that it is nested and not nested at the same time.
arch/x86/kvm/x86.c void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long addr, · line 332
Quoted 8 lines
arch/x86/kvm/x86.c +5
-0+ vcpu->arch.nested_mmu.inject_page_fault (vcpu, addr, error); + else + vcpu->arch.mmu.inject_page_fault(vcpu, addr, error); +} +void kvm_inject_nmi (struct kvm_vcpu *vcpu) { vcpu->arch.nmi_pending =1 ;
Daniel K.
Re: [PATCH 15/18] KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
Quoted 22 lines · Daniel K.
Quoted 6 lines · Joerg Roedel
arch/x86/kvm/x86.c +0
-0@@ -314,6 +314,19 @@ void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long addr,kvm_queue_exception_e (vcpu,PF_VECTOR , error_code)arch/x86/kvm/x86.c void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long addr, · line 324
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0} +void kvm_propagate_fault(struct kvm_vcpu *vcpu, unsigned long addr, u32 error_code) +{ + u32 nested, error; + + nested = error_code &PFERR_NESTED_MASK ; + error = error_code & ~PFERR_NESTED_MASK ; + +if (vcpu->arch.mmu.nested && !(error_code &&PFERR_NESTED_MASK ))This looks incorrect, nested is unused.
At the very least it should be a binary & operation
Quoted 8 lines
if (vcpu->arch.mmu.nested && !(error_code & PFERR_NESTED_MASK))which can be simplified to
if (vcpu->arch.mmu.nested && !nested)but it seems wrong that the condition is that it is nested and not nested at the same time.
Yes, this is already fixed in my local patch-stack. I found it during further testing (while fixing another bug). But thanks for your feedback :-)
Joerg
Re: [PATCH 15/18] KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
In reply to Joerg RoedelQuoted 16 lines · Joerg Roedel
This patch implements logic to make sure that either a page-fault/page-fault-vmexit or a nested-page-fault-vmexit is propagated back to the guest.
Signed-off-by Joerg Roedel<[email protected]>--- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/paging_tmpl.h | 2 ++ arch/x86/kvm/x86.c | 15 ++++++++++++++- 3 files changed, 17 insertions(+), 1 deletions(-)arch/x86/kvm/mmu.h +0
-0
arch/x86/kvm/mmu.h line 51
Quoted 5 lines
arch/x86/kvm/mmu.h +1
-0@@ -47,6 +47,7 @@#define PFERR_USER_MASK (1U<< 2)#define PFERR_RSVD_MASK (1U<< 3)#define PFERR_FETCH_MASK (1U<< 4) +#define PFERR_NESTED_MASK (1U<< 31)
Why is this needed? Queue an ordinary page fault page; the injection code should check the page fault intercept and #VMEXIT if needed.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Re: [PATCH 15/18] KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Joerg Roedel
This patch implements logic to make sure that either a page-fault/page-fault-vmexit or a nested-page-fault-vmexit is propagated back to the guest.
Signed-off-by Joerg Roedel<[email protected]>--- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/paging_tmpl.h | 2 ++ arch/x86/kvm/x86.c | 15 ++++++++++++++- 3 files changed, 17 insertions(+), 1 deletions(-)
Quoted 14 lines
arch/x86/kvm/mmu.h line 51
Quoted 10 lines
arch/x86/kvm/mmu.h +1
-0@@ -47,6 +47,7 @@#define PFERR_USER_MASK (1U<< 2)#define PFERR_RSVD_MASK (1U<< 3)#define PFERR_FETCH_MASK (1U<< 4) +#define PFERR_NESTED_MASK (1U<< 31)Why is this needed? Queue an ordinary page fault page; the injection code should check the page fault intercept and #VMEXIT if needed.
This is needed because we could have a nested page fault or an ordinary page fault which need to be propagated.
Joerg
Re: [PATCH 15/18] KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
Quoted 30 lines · Joerg Roedel
Quoted 13 lines · Avi Kivity
Quoted 11 lines · Joerg Roedel
This patch implements logic to make sure that either a page-fault/page-fault-vmexit or a nested-page-fault-vmexit is propagated back to the guest.
Signed-off-by Joerg Roedel<[email protected]>--- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/paging_tmpl.h | 2 ++ arch/x86/kvm/x86.c | 15 ++++++++++++++- 3 files changed, 17 insertions(+), 1 deletions(-)Quoted 15 lines
arch/x86/kvm/mmu.h line 52
Quoted 11 lines
arch/x86/kvm/mmu.h +1
-0@@ -47,6 +47,7 @@#define PFERR_USER_MASK (1U<< 2)#define PFERR_RSVD_MASK (1U<< 3)#define PFERR_FETCH_MASK (1U<< 4) +#define PFERR_NESTED_MASK (1U<< 31)Why is this needed? Queue an ordinary page fault page; the injection code should check the page fault intercept and #VMEXIT if needed.
Quoted 3 lines
This is needed because we could have a nested page fault or an ordinary page fault which need to be propagated.
Right.
Why is pio_copy_data() changed? One would think that it would be an all-or-nothing affair.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 15/18] KVM: MMU: Propagate the right fault back to the guest after gva_to_gpa
Quoted 31 lines · Avi Kivity
Quoted 30 lines · Joerg Roedel
Quoted 28 lines · Avi Kivity
Quoted 16 lines · Joerg Roedel
This patch implements logic to make sure that either a page-fault/page-fault-vmexit or a nested-page-fault-vmexit is propagated back to the guest.
Signed-off-by Joerg Roedel<[email protected]>--- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/paging_tmpl.h | 2 ++ arch/x86/kvm/x86.c | 15 ++++++++++++++- 3 files changed, 17 insertions(+), 1 deletions(-)arch/x86/kvm/mmu.h +0
-0arch/x86/kvm/mmu.h line 52
Quoted 6 lines
arch/x86/kvm/mmu.h +1
-0@@ -47,6 +47,7 @@#define PFERR_USER_MASK (1U<< 2)#define PFERR_RSVD_MASK (1U<< 3)#define PFERR_FETCH_MASK (1U<< 4) +#define PFERR_NESTED_MASK (1U<< 31)Why is this needed? Queue an ordinary page fault page; the injection code should check the page fault intercept and #VMEXIT if needed.
Quoted 8 lines
Quoted 3 lines
This is needed because we could have a nested page fault or an ordinary page fault which need to be propagated.
Right.
Why is
pio_copy_data()changed? One would think that it would be an all-or-nothing affair.
It was the only place I found where the PROPAGATE_FAULT value was checked and actually propagated.
Joerg
PATCH 16/18 KVM: X86: Add callback to let modules decide over some supported cpuid bits
In reply to Joerg RoedelThis patch adds the get_supported_cpuid callback to kvm_x86_ops. It will be used in do_cpuid_ent to delegate the decission about some supported cpuid bits to the architecture modules.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/svm.c | 6 ++++++ arch/x86/kvm/vmx.c | 6 ++++++ arch/x86/kvm/x86.c | 3 +++ 4 files changed, 17 insertions(+), 0 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -555,6 +555,8 @@ struct kvm_x86_ops {bool (*rdtscp_supported)(void );void (*set_tdp_cr3)(struct kvm_vcpu *vcpu,unsigned long cr3); +void (*set_supported_cpuid)(u32 func,struct kvm_cpuid_entry2 *entry); +const struct trace_print_flags *exit_reasons_str; };
arch/x86/kvm/svm.c +6 -0
@@ -3287,6 +3287,10 @@ static void svm_cpuid_update(struct kvm_vcpu *vcpu) { } +static void svm_set_supported_cpuid (u32 func,struct kvm_cpuid_entry2 *entry) +{ +} +static const struct trace_print_flags svm_exit_reasons_str[] = { {SVM_EXIT_READ_CR0 ,"read_cr0" }, {SVM_EXIT_READ_CR3 ,"read_cr3" },
@@ -3435,6 +3439,8 @@ static struct kvm_x86_ops svm_x86_ops = { .rdtscp_supported = svm_rdtscp_supported, .set_tdp_cr3 = set_tdp_cr3, + + .set_supported_cpuid = svm_set_supported_cpuid, };static int __init svm_init (void )
arch/x86/kvm/vmx.c +6 -0
@@ -4146,6 +4146,10 @@ static void vmx_cpuid_update(struct kvm_vcpu *vcpu) } } +static void vmx_set_supported_cpuid (u32 func,struct kvm_cpuid_entry2 *entry) +{ +} +static struct kvm_x86_ops vmx_x86_ops = { .cpu_has_kvm_support = cpu_has_kvm_support, .disabled_by_bios = vmx_disabled_by_bios,
@@ -4220,6 +4224,8 @@ static struct kvm_x86_ops vmx_x86_ops = { .rdtscp_supported = vmx_rdtscp_supported, .set_tdp_cr3 = vmx_set_cr3, + + .set_supported_cpuid = vmx_set_supported_cpuid, };static int __init vmx_init (void )
arch/x86/kvm/x86.c +3 -0
@@ -1921,6 +1921,9 @@ static void do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function, entry->ecx &= kvm_supported_word6_x86_features;break ; } + + kvm_x86_ops->set_supported_cpuid (function, entry); +put_cpu (); }
Signature · 1 line
1.7.0
PATCH 17/18 KVM: SVM: Report Nested Paging support to userspace
In reply to Joerg RoedelThis patch implements the reporting of the nested paging feature support to userspace.
--- arch/x86/kvm/svm.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
arch/x86/kvm/svm.c +10 -0
@@ -3289,6 +3289,16 @@ static void svm_cpuid_update(struct kvm_vcpu *vcpu)static void svm_set_supported_cpuid (u32 func,struct kvm_cpuid_entry2 *entry) { +switch (func) { +case 0x8000000A : +if (!npt_enabled) +break ; + +/* NPT feature is supported by Nested SVM */ + entry->edx =SVM_FEATURE_NPT ; + +break ; + } }static const struct trace_print_flags svm_exit_reasons_str[] = {
Signature · 1 line
1.7.0
Re: [PATCH 17/18] KVM: SVM: Report Nested Paging support to userspace
Quoted 13 lines · Joerg Roedel
This patch implements the reporting of the nested paging feature support to userspace.
Signed-off-by Joerg Roedel <[email protected]>---
arch/x86/kvm/svm.c| 10 ++++++++++1 files changed, 10 insertions(+), 0 deletions(-)arch/x86/kvm/svm.c +0
-0
arch/x86/kvm/svm.c static void svm_cpuid_update(struct kvm_vcpu *vcpu) · line 3295
Quoted 7 lines
arch/x86/kvm/svm.c +4
-0@@ -3289,6 +3289,16 @@ static void svm_cpuid_update(struct kvm_vcpu *vcpu)static void svm_set_supported_cpuid (u32 func,struct kvm_cpuid_entry2 *entry) { +switch (func) { +case 0x8000000A : +if (!npt_enabled) +break ;
if (!nested)
break;
Alex
Re: [PATCH 17/18] KVM: SVM: Report Nested Paging support to userspace
Quoted 17 lines · Alexander Graf
Quoted 14 lines · Joerg Roedel
This patch implements the reporting of the nested paging feature support to userspace.
Signed-off-by Joerg Roedel <[email protected]>---
arch/x86/kvm/svm.c| 10 ++++++++++1 files changed, 10 insertions(+), 0 deletions(-)arch/x86/kvm/svm.c +0
-0@@ -3289,6 +3289,16 @@ static void svm_cpuid_update(struct kvm_vcpu *vcpu)
Quoted 9 lines
arch/x86/kvm/svm.c static void svm_cpuid_update(struct kvm_vcpu *vcpu) · line 3295
Quoted 6 lines
arch/x86/kvm/svm.c +4
-0static void svm_set_supported_cpuid (u32 func,struct kvm_cpuid_entry2 *entry) { +switch (func) { +case 0x8000000A : +if (!npt_enabled) +break ;if (!nested)
break;
True, but shouldn't matter much because if nested is off the guest will not see the svm bit. It would only see that the processor could do nested paging if it had svm support ;-)
Joerg
PATCH 18/18 KVM: X86: Add KVM_CAP_SVM_CPUID_FIXED
In reply to Joerg RoedelThis capability shows userspace that is can trust the values of cpuid[0x8000000A] that it gets from the kernel. Old behavior was to just return the host cpuid values which is broken because all additional svm-features need support in the svm emulation code.
--- arch/x86/kvm/x86.c | 1 + include/linux/kvm.h | 1 + 2 files changed, 2 insertions(+), 0 deletions(-)
arch/x86/kvm/x86.c +1 -0
@@ -1557,6 +1557,7 @@ int kvm_dev_ioctl_check_extension(long ext)case KVM_CAP_PCI_SEGMENT :case KVM_CAP_DEBUGREGS :case KVM_CAP_X86_ROBUST_SINGLESTEP : +case KVM_CAP_SVM_CPUID_FIXED : r =1 ;break ;case KVM_CAP_COALESCED_MMIO :
include/linux/kvm.h +1 -0
@@ -507,6 +507,7 @@ struct kvm_ioeventfd {#define KVM_CAP_DEBUGREGS 50#endif #define KVM_CAP_X86_ROBUST_SINGLESTEP 51 +#define KVM_CAP_SVM_CPUID_FIXED 52#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
1.7.0
Re: [PATCH 18/18] KVM: X86: Add KVM_CAP_SVM_CPUID_FIXED
Quoted 7 lines · Joerg Roedel
This capability shows userspace that is can trust the values of cpuid[0x8000000A] that it gets from the kernel. Old behavior was to just return the host cpuid values which is broken because all additional svm-features need support in the svm emulation code.
A think we can simply fix the bug and push the fix to the various stable queues.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 18/18] KVM: X86: Add KVM_CAP_SVM_CPUID_FIXED
Quoted 10 lines · Avi Kivity
Quoted 6 lines · Joerg Roedel
This capability shows userspace that is can trust the values of cpuid[0x8000000A] that it gets from the kernel. Old behavior was to just return the host cpuid values which is broken because all additional svm-features need support in the svm emulation code.
A think we can simply fix the bug and push the fix to the various stable queues.
Ok, sounds good too. I have some more fixes queued up and send this one together with them.
Joerg
Quoted 8 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the
Wow!
Jan
Quoted 45 lines
situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)Thanks,
Joerg
Diffstat:
arch/x86/include/asm/kvm_host.h | 21 ++++++ arch/x86/kvm/mmu.c | 152 ++++++++++++++++++++++++++++++--------- arch/x86/kvm/mmu.h | 2 + arch/x86/kvm/paging_tmpl.h | 81 ++++++++++++++++++--- arch/x86/kvm/svm.c | 126 +++++++++++++++++++++++++++----- arch/x86/kvm/vmx.c | 9 +++ arch/x86/kvm/x86.c | 19 +++++- include/linux/kvm.h | 1 + include/linux/kvm_host.h | 5 ++ 9 files changed, 354 insertions(+), 62 deletions(-)Shortlog:
Joerg Roedel (18):
KVM: MMU: Check forroot_levelinstead of long mode
KVM: MMU: Maketdp_enableda mmu-context parameter
KVM: MMU: Makeset_cr3a function pointer inkvm_mmu
KVM: X86: Introduce atdp_set_cr3function
KVM: MMU: Introduceget_cr3function pointer
KVM: MMU: Introduceinject_page_faultfunction pointer
KVM: SVM: Implement MMU helper functions for Nested Nested Paging
KVM: MMU: Changeinit_kvm_softmmuto take a context as parameter
KVM: MMU: Letis_rsvd_bits_settake mmu context instead of vcpu
KVM: MMU: Introduce genericwalk_addrfunction
KVM: MMU: Add infrastructure for two-level page walker
KVM: MMU: Implement nestedgva_to_gpafunctions
KVM: MMU: Introduce Nested MMU context
KVM: SVM: Initialize Nested Nested MMU context on VMRUN
KVM: MMU: Propagate the right fault back to the guest aftergva_to_gpa
KVM: X86: Add callback to let modules decide over some supported cpuid bits
KVM: SVM: Report Nested Paging support to userspace
KVM: X86: AddKVM_CAP_SVM_CPUID_FIXED
Quoted 12 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)
Nice job! It's great to see you finally got around to it :-).
Have you tracked what slows down SMP l2 guests yet? So far I've been assuming that IPIs just completely kill the performance, but I guess it shouldn't be that bad, especially now where you have sped up the #VMEXIT path that much.
Alex
Quoted 16 lines · Alexander Graf
Quoted 12 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)
Quoted 6 lines
Nice job! It's great to see you finally got around to it :-).
Have you tracked what slows down SMP l2 guests yet? So far I've been assuming that IPIs just completely kill the performance, but I guess it shouldn't be that bad, especially now where you have sped up the #VMEXIT path that much.
I have not yet looked deeper into this issue. I also suspect lockholder preemption to be the cause for this. I did the test with a populated nested page table too and the slowdown is still there. But thats all guessing, I need to do some research for the exact reasons.
Joerg
Quoted 8 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor.
That's an awesome result.
Regards,
Anthony Liguori
Quoted 12 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)
Joerg,
What perf gain does this bring ? (i'm not aware of the current overhead).
Overall comments:
Can't you translate l2_gpa -> l1_gpa walking the current l1 nested pagetable, and pass that to the kvm tdp fault path (with the correct context setup)?
You probably need to include a flag in base_role to differentiate between l1 / l2 shadow tables (say if they use the same cr3 value).
Quoted 14 lines · Marcelo Tosatti
Quoted 12 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)
Quoted 4 lines
Joerg,
What perf gain does this bring ? (i'm not aware of the current overhead).
The benchmark was an allnoconfig kernel compile in tmpfs which took with the same guest image:
as l1-guest with npt:
2m23s
as l2-guest with l1(nested)-l2(shadow):
around 8-9 minutes
as l2-guest with l1(nested)-l2(shadow) without the recent msrpm optimization:
around 19 minutes
as l2-guest with l1(nested)-l2(nested) [this patchset]:
2m25s-2m30s
Quoted 5 lines
Overall comments:
Can't you translate
l2_gpa->l1_gpawalking the current l1 nested pagetable, and pass that to the kvm tdp fault path (with the correct context setup)?
If I understand your suggestion correctly, I think thats exactly whats done in the patches. Some words about the design:
For nested-nested we need to shadow the l1-nested-ptable on the host. This is done using the vcpu->arch.mmu context which holds the l1 paging modes while the l2 is running. On a npt-fault from the l2 we just instrument the shadow-ptable code. This is the common case. because it happens all the time while the l2 is running.
The other thing is that vcpu->arch.mmu.gva_to_gpa is expected to still work and translate virtual addresses of the l2 into physical addresses of the l1 (so it can be accessed with kvm functions).
To do this we need to be aware of the L2 paging mode. It is stored in vcpu->arch.nested_mmu context. This context is only used for gva_to_gpa translations. It is not used to build shadow page tables or anything else. Thats the reason only the parts necessary for gva_to_gpa translations of the nested_mmu context are initialized.
Since we can not use mmu.gva_to_gpa to translate only between l2_gpa and l1_gpa because this function is required to translate l2_gva to l1_gpa by other parts of kvm, the function which does this translation is moved to nested_mmu.gva_to_gpa. So basically the gva_to_gpa function pointers are swapped between mmu and nested_mmu.
The nested_mmu.gva_to_gpa function is used in translate_gpa_nested which is assigned to the newly introduced translate_gpa callback of nested_mmu context.
This callback is used in the walk_addr function to translate every l2_gpa address we read from cr3 or the guest ptes into l1_gpa to read the next step from the guest memory.
In the old unnested case the translate_gpa callback would point to a function which just returns the gpa it is passed to it unmodified. The walk_addr function is generalized and now there are basically two versions of it:
* walk_addr which translates using vcpu->arch.mmu context
* walk_addr_nested which translates using vcpu->arch.nested_mmu context
Thats pretty much how these patches work.
Quoted 2 lines
You probably need to include a flag in
base_roleto differentiate between l1 / l2 shadow tables (say if they use the same cr3 value).
Not sure if this is necessary. It may be necessary when large pages come into play. Otherwise the host npt pages are distinguished by the shadow npt pages by the direct-flag.
Joerg
Quoted 46 lines · Joerg Roedel
Quoted 14 lines · Marcelo Tosatti
Quoted 12 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)Quoted 4 lines
Joerg,
What perf gain does this bring ? (i'm not aware of the current overhead).
The benchmark was an allnoconfig kernel compile in tmpfs which took with the same guest image:
as l1-guest with npt:
2m23sas l2-guest with l1(nested)-l2(shadow):
around 8-9 minutes
as l2-guest with l1(nested)-l2(shadow) without the recent msrpm optimization:
around 19 minutes
as l2-guest with l1(nested)-l2(nested) [this patchset]:
2m25s-2m30sQuoted 5 lines
Overall comments:
Can't you translate
l2_gpa->l1_gpawalking the current l1 nested pagetable, and pass that to the kvm tdp fault path (with the correct context setup)?
Quoted 8 lines
If I understand your suggestion correctly, I think thats exactly whats done in the patches. Some words about the design:
For nested-nested we need to shadow the l1-nested-ptable on the host. This is done using the
vcpu->arch.mmucontext which holds the l1 paging modes while the l2 is running. On a npt-fault from the l2 we just instrument the shadow-ptable code. This is the common case. because it happens all the time while the l2 is running.
OK, makes sense now, I was missing the fact that the l1-nested-ptable needs to be shadowed and l1 translations to it must be write protected.
You should disable out of sync shadow so that l1 guest writes to l1-nested-ptables always trap. And in the trap case, you'd have to invalidate l2 shadow pagetable entries that used the (now obsolete) l1-nested-ptable entry. Does that happen automatically?
Quoted 44 lines
The other thing is that
vcpu->arch.mmu.gva_to_gpais expected to still work and translate virtual addresses of the l2 into physical addresses of the l1 (so it can be accessed with kvm functions).To do this we need to be aware of the L2 paging mode. It is stored in
vcpu->arch.nested_mmucontext. This context is only used forgva_to_gpatranslations. It is not used to build shadow page tables or anything else. Thats the reason only the parts necessary forgva_to_gpatranslations of thenested_mmucontext are initialized.Since we can not use mmu.gva_to_gpa to translate only between
l2_gpaandl1_gpabecause this function is required to translatel2_gvatol1_gpaby other parts of kvm, the function which does this translation is moved to nested_mmu.gva_to_gpa. So basically thegva_to_gpafunction pointers are swapped between mmu andnested_mmu.The nested_mmu.gva_to_gpa function is used in
translate_gpa_nestedwhich is assigned to the newly introducedtranslate_gpacallback ofnested_mmucontext.This callback is used in the
walk_addrfunction to translate everyl2_gpaaddress we read from cr3 or the guest ptes intol1_gpato read the next step from the guest memory.In the old unnested case the
translate_gpacallback would point to a function which just returns the gpa it is passed to it unmodified. Thewalk_addrfunction is generalized and now there are basically two versions of it:*
walk_addrwhich translates usingvcpu->arch.mmucontext
*walk_addr_nestedwhich translates usingvcpu->arch.nested_mmucontextThats pretty much how these patches work.
Quoted 2 lines
You probably need to include a flag in
base_roleto differentiate between l1 / l2 shadow tables (say if they use the same cr3 value).Not sure if this is necessary. It may be necessary when large pages come into play. Otherwise the host npt pages are distinguished by the shadow npt pages by the direct-flag.
Joerg
Quoted 8 lines · Marcelo Tosatti
Quoted 8 lines
Quoted 5 lines
Can't you translate
l2_gpa->l1_gpawalking the current l1 nested pagetable, and pass that to the kvm tdp fault path (with the correct context setup)?If I understand your suggestion correctly, I think thats exactly whats done in the patches. Some words about the design:
Quoted 9 lines
Quoted 6 lines
For nested-nested we need to shadow the l1-nested-ptable on the host. This is done using the
vcpu->arch.mmucontext which holds the l1 paging modes while the l2 is running. On a npt-fault from the l2 we just instrument the shadow-ptable code. This is the common case. because it happens all the time while the l2 is running.OK, makes sense now, I was missing the fact that the l1-nested-ptable needs to be shadowed and l1 translations to it must be write protected.
Shadow converts (gva -> gpa -> hpa) to (gva -> hpa) or (ngpa -> gpa -> hpa) to (ngpa -> hpa) equally well. In the second case npt still does (ngva -> ngpa).
Quoted 2 lines
You should disable out of sync shadow so that l1 guest writes to l1-nested-ptables always trap.
Why? The guest is under obligation to flush the tlb if it writes to a page table, and we will resync on that tlb flush.
Unsync makes just as much sense for nnpt. Think of khugepaged in the guest eating a page table and spitting out a PDE.
Quoted 4 lines
And in the trap case, you'd have to
invalidate l2 shadow pagetable entries that used the (now obsolete) l1-nested-ptable entry. Does that happen automatically?
What do you mean by 'l2 shadow ptable entries'? There are the guest's page tables (ordinary direct mapped, unless the guest's guest is also running an npt-enabled hypervisor), and the host page tables. When the guest writes to each page table, we invalidate the shadows.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 20 lines · Avi Kivity
Quoted 7 lines · Marcelo Tosatti
Quoted 7 lines
Quoted 4 lines
Can't you translate
l2_gpa->l1_gpawalking the current l1 nested pagetable, and pass that to the kvm tdp fault path (with the correct context setup)?If I understand your suggestion correctly, I think thats exactly whats done in the patches. Some words about the design:
Quoted 7 lines
Quoted 5 lines
For nested-nested we need to shadow the l1-nested-ptable on the host. This is done using the
vcpu->arch.mmucontext which holds the l1 paging modes while the l2 is running. On a npt-fault from the l2 we just instrument the shadow-ptable code. This is the common case. because it happens all the time while the l2 is running.OK, makes sense now, I was missing the fact that the l1-nested-ptable needs to be shadowed and l1 translations to it must be write protected.
Shadow converts (gva -> gpa -> hpa) to (gva -> hpa) or (ngpa -> gpa -> hpa) to (ngpa -> hpa) equally well. In the second case npt still does (ngva -> ngpa).
Quoted 5 lines
Quoted 2 lines
You should disable out of sync shadow so that l1 guest writes to l1-nested-ptables always trap.
Why? The guest is under obligation to flush the tlb if it writes to a page table, and we will resync on that tlb flush.
The guests hypervisor will not flush the tlb with invlpg for updates of its NPT pagetables. It'll create a new ASID, and KVM will not trap that.
Quoted 7 lines
Unsync makes just as much sense for nnpt. Think of khugepaged in the guest eating a page table and spitting out a PDE.
Quoted 3 lines
And in the trap case, you'd have to
invalidate l2 shadow pagetable entries that used the (now obsolete) l1-nested-ptable entry. Does that happen automatically?
Quoted 5 lines
What do you mean by 'l2 shadow ptable entries'? There are the guest's page tables (ordinary direct mapped, unless the guest's guest is also running an npt-enabled hypervisor), and the host page tables. When the guest writes to each page table, we invalidate the shadows.
With 'l2 shadow ptable entries' i mean the shadow pagetables that translate GPA-L2 -> HPA.
Quoted 11 lines · Marcelo Tosatti
Quoted 7 lines
Quoted 4 lines
You should disable out of sync shadow so that l1 guest writes to l1-nested-ptables always trap.
Why? The guest is under obligation to flush the tlb if it writes to a page table, and we will resync on that tlb flush.
The guests hypervisor will not flush the tlb with invlpg for updates of its NPT pagetables. It'll create a new ASID, and KVM will not trap that.
We'll get a kvm_set_cr3() on the next vmrun.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
And in the trap case, you'd have to
invalidate l2 shadow pagetable entries that used the (now obsolete) l1-nested-ptable entry. Does that happen automatically?
Quoted 9 lines
Quoted 6 lines
What do you mean by 'l2 shadow ptable entries'? There are the guest's page tables (ordinary direct mapped, unless the guest's guest is also running an npt-enabled hypervisor), and the host page tables. When the guest writes to each page table, we invalidate the shadows.
With 'l2 shadow ptable entries' i mean the shadow pagetables that translate GPA-L2 -> HPA.
kvm_mmu_pte_write() will invalidate those sptes and will also install new translations if possible.
Beautiful, isn't it?
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 7 lines · Joerg Roedel
Quoted 3 lines
You probably need to include a flag in
base_roleto differentiate between l1 / l2 shadow tables (say if they use the same cr3 value).Not sure if this is necessary. It may be necessary when large pages come into play. Otherwise the host npt pages are distinguished by the shadow npt pages by the direct-flag.
Hm, I think that direct maps for the same gfn can be shared.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 19 lines · Joerg Roedel
Hi,
here are the patches that implement nested paging support for nested svm. They are somewhat intrusive to the soft-mmu so I post them as RFC in the first round to get feedback about the general direction of the changes. Nevertheless I am proud to report that with these patches the famous kernel-compile benchmark runs only 4% slower in the l2 guest as in the l1 guest when l2 is single-processor. With SMP guests the situation is very different. The more vcpus the guest has the more is the performance drop from l1 to l2.
Anyway, this post is to get feedback about the overall concept of these patches. Please review and give feedback :-)Thanks,
Joerg
Diffstat:
Quoted 11 lines
arch/x86/include/asm/kvm_host.h | 21 ++++++ arch/x86/kvm/mmu.c | 152 ++++++++++++++++++++++++++++++--------- arch/x86/kvm/mmu.h | 2 + arch/x86/kvm/paging_tmpl.h | 81 ++++++++++++++++++--- arch/x86/kvm/svm.c | 126 +++++++++++++++++++++++++++----- arch/x86/kvm/vmx.c | 9 +++ arch/x86/kvm/x86.c | 19 +++++- include/linux/kvm.h | 1 + include/linux/kvm_host.h | 5 ++ 9 files changed, 354 insertions(+), 62 deletions(-)
Okay, this looks excellent overall, it's nice to see how well this fits with the existing mmu infrastructure (only ~300 lines added). The performance results are impressive.
Signature · 1 line
error compiling committee.c: too many arguments to function