KVM: nVMX: Add more statistics to KVM debugfs
cover KVM: nVMX: Add more statistics to KVM debugfs
Patch# 1: Renames 'nested_run' statistic and increments it from a more
fine-grained location.
Patch# 2: Adds a new statistic to show if a VCPU is running a nested guest. Patch# 3: Adds a new statistic to show number of VCPUs created in a given VM.
[PATCH 1/3] KVM: nVMX: Move 'nested_run' counter to [PATCH 2/3] KVM: nVMX: Add a new VCPU statistic to show if VCPU is [PATCH 3/3] KVM: x86: Add a new VM statistic to show number of VCPUs
arch/x86/include/asm/kvm_host.h | 4 +++- arch/x86/kvm/debugfs.c | 11 +++++++++++ arch/x86/kvm/kvm_cache_regs.h | 4 ++++ arch/x86/kvm/svm/nested.c | 2 -- arch/x86/kvm/vmx/nested.c | 2 -- arch/x86/kvm/x86.c | 4 +++- virt/kvm/kvm_main.c | 2 ++ 7 files changed, 23 insertions(+), 6 deletions(-)
Krish Sadhukhan (3):
KVM: nVMX: Move 'nested_run' counter to enter_guest_mode()
KVM: nVMX: Add a new VCPU statistic to show if VCPU is running nested guest
KVM: x86: Add a new VM statistic to show number of VCPUs created in a given VM
PATCH 1/3 KVM: nVMX: Move 'nested_run' counter to enter_guest_mode()
Move 'nested_run' counter to enter_guest_mode() because,
i) This counter is common to both Intel and AMD and can be incremented from a common place,
ii) guest mode is a more finer-grained state than the beginning of
nested_svm_vmrun() and nested_vmx_run().
Also, rename it to 'nested_runs'.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/kvm_cache_regs.h | 1 + arch/x86/kvm/svm/nested.c | 2 -- arch/x86/kvm/vmx/nested.c | 2 -- arch/x86/kvm/x86.c | 2 +- 5 files changed, 3 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1170,7 +1170,7 @@ struct kvm_vcpu_stat {u64 req_event;u64 halt_poll_success_ns;u64 halt_poll_fail_ns; -u64 nested_run; +u64 nested_runs;u64 directed_yield_attempted;u64 directed_yield_successful; };
arch/x86/kvm/kvm_cache_regs.h +1 -0
@@ -162,6 +162,7 @@ static inline u64 kvm_read_edx_eax(struct kvm_vcpu *vcpu)static inline void enter_guest_mode (struct kvm_vcpu *vcpu) { vcpu->arch.hflags |=HF_GUEST_MASK ; + ++vcpu->stat.nested_runs; }static inline void leave_guest_mode (struct kvm_vcpu *vcpu)
arch/x86/kvm/svm/nested.c +0 -2
@@ -596,8 +596,6 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)struct kvm_host_map map;u64 vmcb12_gpa; - ++vcpu->stat.nested_run; -if (is_smm (vcpu)) {kvm_queue_exception (vcpu,UD_VECTOR );return 1 ;
arch/x86/kvm/vmx/nested.c +0 -2
@@ -3454,8 +3454,6 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)u32 interrupt_shadow =vmx_get_interrupt_shadow (vcpu);enum nested_evmptrld_status evmptrld_status; - ++vcpu->stat.nested_run; -if (!nested_vmx_check_permission (vcpu))return 1 ;
arch/x86/kvm/x86.c +1 -1
@@ -243,7 +243,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("l1d_flush" , l1d_flush),VCPU_STAT ("halt_poll_success_ns" , halt_poll_success_ns),VCPU_STAT ("halt_poll_fail_ns" , halt_poll_fail_ns), -VCPU_STAT ("nested_run" , nested_run), +VCPU_STAT ("nested_runs" , nested_runs),VCPU_STAT ("directed_yield_attempted" , directed_yield_attempted),VCPU_STAT ("directed_yield_successful" , directed_yield_successful), VM_STAT("mmu_shadow_zapped", mmu_shadow_zapped),
Signature · 1 line
2.27.0
Sean Christopherson
maintainerRe: [PATCH 1/3] KVM: nVMX: Move 'nested_run' counter to enter_guest_mode()
Quoted 5 lines · Krish Sadhukhan
Move '
nested_run' counter toenter_guest_mode()because,
i) This counter is common to both Intel and AMD and can be incremented from a common place,
ii) guest mode is a more finer-grained state than the beginning of
nested_svm_vmrun()andnested_vmx_run().
Hooking enter_guest_mode() makes the name a misnomer since it will count cases such as setting nested state and resuming from SMI, neither of which is a nested run in the sense of L1 deliberately choosing to run L2.
And while bumping nested_run at the very beginning of VMLAUNCH/VMRESUME/VMRUN is arguably wrong in that it counts _attempts_ instead of successful VM-Enters, it's at least consistent. Moving this to enter_guest_mode() means it's arbitrarily counting VM-Enter that fails late, but not those that fail early.
If we really want it to mean "successful VM-Enter", then we should wait until after VM-Enter actual succeeds, and do it only for actual VM-Enter.
Re: [PATCH 1/3] KVM: nVMX: Move 'nested_run' counter to enter_guest_mode()
Quoted 7 lines · Sean Christopherson
Quoted 5 lines · Krish Sadhukhan
Move '
nested_run' counter toenter_guest_mode()because,
i) This counter is common to both Intel and AMD and can be incremented from a common place,
ii) guest mode is a more finer-grained state than the beginning of
nested_svm_vmrun()andnested_vmx_run().
Quoted 6 lines
Hooking
enter_guest_mode()makes the name a misnomer since it will count cases such as setting nested state and resuming from SMI, neither of which is a nested run in the sense of L1 deliberately choosing to run L2.And while bumping
nested_runat the very beginning of VMLAUNCH/VMRESUME/VMRUN is arguably wrong in that it counts _attempts_ instead of successful VM-Enters, it's
Yes, the original purpose was to track the attempt. That's why the counter is incremented at the beginning. It helps tell if there is any attempt to run L2 VM (and also if VM is actively running L2 VM by monitoring the counter).
This helps as sometimes VM owner does not realize software like jailhouse will involve the L2 VM.
Without the counter, we may need to temporarily use "/sys/kernel/debug/kvm/mmu_unsync" assuming shadow page table is not used by host if L2 VM is not involved.
Dongli Zhang
Quoted 6 lines
at least consistent. Moving this to
enter_guest_mode()means it's arbitrarily counting VM-Enter that fails late, but not those that fail early.If we really want it to mean "successful VM-Enter", then we should wait until after VM-Enter actual succeeds, and do it only for actual VM-Enter.
PATCH 2/3 KVM: nVMX: Add a new VCPU statistic to show if VCPU is running nested guest
In reply to Krish SadhukhanAdd the following per-VCPU statistic to KVM debugfs to show if a given VCPU is running a nested guest:
nested_guest_running
Also add this as a per-VM statistic to KVM debugfs to show the total number of VCPUs running a nested guest in a given VM.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/debugfs.c | 11 +++++++++++ arch/x86/kvm/kvm_cache_regs.h | 3 +++ arch/x86/kvm/x86.c | 1 + 4 files changed, 16 insertions(+)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1173,6 +1173,7 @@ struct kvm_vcpu_stat {u64 nested_runs;u64 directed_yield_attempted;u64 directed_yield_successful; +u64 nested_guest_running; };struct x86_instruction_info ;
arch/x86/kvm/debugfs.c +11 -0
@@ -17,6 +17,15 @@ static int vcpu_get_timer_advance_ns(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (vcpu_timer_advance_ns_fops, vcpu_get_timer_advance_ns,NULL ,"%llu\n" ); +static int vcpu_get_guest_mode (void *data,u64 *val) +{ +struct kvm_vcpu *vcpu = (struct kvm_vcpu *) data; + *val = vcpu->stat.nested_guest_running; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (vcpu_guest_mode_fops, vcpu_get_guest_mode,NULL ,"%lld\n" ); +static int vcpu_get_tsc_offset (void *data,u64 *val) {struct kvm_vcpu *vcpu = (struct kvm_vcpu *) data;
@@ -45,6 +54,8 @@ DEFINE_SIMPLE_ATTRIBUTE(vcpu_tsc_scaling_frac_fops, vcpu_get_tsc_scaling_frac_bivoid kvm_arch_create_vcpu_debugfs (struct kvm_vcpu *vcpu,struct dentry *debugfs_dentry) { +debugfs_create_file ("nested_guest_running" ,0444 , debugfs_dentry, vcpu, + &vcpu_guest_mode_fops);debugfs_create_file ("tsc-offset" ,0444 , debugfs_dentry, vcpu, &vcpu_tsc_offset_fops);
arch/x86/kvm/kvm_cache_regs.h +3 -0
@@ -163,6 +163,7 @@ static inline void enter_guest_mode(struct kvm_vcpu *vcpu) { vcpu->arch.hflags |=HF_GUEST_MASK ; ++vcpu->stat.nested_runs; + vcpu->stat.nested_guest_running =1 ; }static inline void leave_guest_mode (struct kvm_vcpu *vcpu)
@@ -173,6 +174,8 @@ static inline void leave_guest_mode(struct kvm_vcpu *vcpu) vcpu->arch.load_eoi_exitmap_pending = false;kvm_make_request (KVM_REQ_LOAD_EOI_EXITMAP , vcpu); } + + vcpu->stat.nested_guest_running =0 ; }static inline bool is_guest_mode (struct kvm_vcpu *vcpu)
arch/x86/kvm/x86.c +1 -0
@@ -246,6 +246,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("nested_runs" , nested_runs),VCPU_STAT ("directed_yield_attempted" , directed_yield_attempted),VCPU_STAT ("directed_yield_successful" , directed_yield_successful), +VCPU_STAT ("nested_guest_running" , nested_guest_running),VM_STAT ("mmu_shadow_zapped" , mmu_shadow_zapped),VM_STAT ("mmu_pte_write" , mmu_pte_write), VM_STAT("mmu_pde_zapped", mmu_pde_zapped),
Signature · 1 line
2.27.0
Re: [PATCH 2/3] KVM: nVMX: Add a new VCPU statistic to show if VCPU is running nested guest
Quoted 10 lines · Krish Sadhukhan
Add the following per-VCPU statistic to KVM debugfs to show if a given VCPU is running a nested guest:
nested_guest_runningAlso add this as a per-VM statistic to KVM debugfs to show the total number of VCPUs running a nested guest in a given VM.
Signed-off-by Krish Sadhukhan <[email protected]>
This is fine, but I don't really see its usefulness. OTOH, one statistic I would really like to see is how many vCPUs have *ever* run a nested guest.
Re: [PATCH 2/3] KVM: nVMX: Add a new VCPU statistic to show if VCPU is running nested guest
Quoted 5 lines · Jim Mattson
Quoted 3 lines · Krish Sadhukhan
Add the following per-VCPU statistic to KVM debugfs to show if a given VCPU is running a nested guest:
Quoted 7 lines
Quoted 6 lines
nested_guest_runningAlso add this as a per-VM statistic to KVM debugfs to show the total number of VCPUs running a nested guest in a given VM.
Signed-off-by Krish Sadhukhan <[email protected]>This is fine, but I don't really see its usefulness. OTOH, one
Two potential uses:
1. If Live Migration of L2 guests is broken/buggy, this can be used to determine a safer time to trigger Live Migration of L1 guests.
2. This can be used to create a time-graph of the load of L1 and L2 in a given VM as well across the host.
Quoted 2 lines
statistic I would really like to see is how many vCPUs have *ever* run a nested guest.
'nested_runs' statistic provides this data, though only till VCPUs are alive. We can convert 'nested_runs' to be persistent beyond VCPU destruction.
But I am curious about the usage you are thinking of with this.
Re: [PATCH 2/3] KVM: nVMX: Add a new VCPU statistic to show if VCPU is running nested guest
Quoted 1 line · Krish Sadhukhan
But I am curious about the usage you are thinking of with this.
I'm interested in tracking usage of nested virtualization.
Sean Christopherson
maintainerRe: [PATCH 2/3] KVM: nVMX: Add a new VCPU statistic to show if VCPU is running nested guest
In reply to Krish SadhukhanQuoted 12 lines · Krish Sadhukhan
Quoted 10 lines · Jim Mattson
Quoted 8 lines · Krish Sadhukhan
Add the following per-VCPU statistic to KVM debugfs to show if a given VCPU is running a nested guest:
nested_guest_runningAlso add this as a per-VM statistic to KVM debugfs to show the total number of VCPUs running a nested guest in a given VM.
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
Signed-off-by Krish Sadhukhan <[email protected]>This is fine, but I don't really see its usefulness. OTOH, one
Two potential uses:
1. If Live Migration of L2 guests is broken/buggy, this can be used to determine a safer time to trigger Live Migration of L1 guests.
This seems tenuous. The stats are inherently racy, so userspace would still need to check for "guest mode" after retrieving state. And wouldn't you want to wait until L1 turns VMX/SVM _off_? If migrating L2 is broken, simply waiting until L2 exits likely isn't going to help all that much.
Quoted 2 lines
2. This can be used to create a time-graph of the load of L1 and L2 in a given VM as well across the host.
Hrm, I like the idea of being able to observe how much time a vCPU is spending in L1 vs. L2, but cross-referencing guest time with "in L2" seems difficult and error prone. I wonder if we can do better, i.e. explicitly track L1 vs. L2+ usage. I think that would also grant Jim's wish of being able to more precisely track nested virtualization utilization.
Quoted 2 lines
Quoted 2 lines
statistic I would really like to see is how many vCPUs have *ever* run a nested guest.
PATCH 3/3 KVM: x86: Add a new VM statistic to show number of VCPUs created in a given VM
In reply to Krish Sadhukhan'struct kvm' already has a member for counting the number of VCPUs created for a given VM. Add this as a new VM statistic to KVM debugfs.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 1 + virt/kvm/kvm_main.c | 2 ++ 3 files changed, 4 insertions(+)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1138,6 +1138,7 @@ struct kvm_vm_stat {ulong lpages;ulong nx_lpage_splits;ulong max_mmu_page_hash_collisions; +u64 created_vcpus; };struct kvm_vcpu_stat {
arch/x86/kvm/x86.c +1 -0
@@ -258,6 +258,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { VM_STAT("largepages", lpages, .mode = 0444), VM_STAT("nx_largepages_splitted", nx_lpage_splits, .mode = 0444),VM_STAT ("max_mmu_page_hash_collisions" , max_mmu_page_hash_collisions), +VM_STAT ("created_vcpus" , created_vcpus), { NULL } };
virt/kvm/kvm_main.c +2 -0
@@ -3318,6 +3318,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id) } kvm->created_vcpus++; + kvm->stat.created_vcpus++;mutex_unlock (&kvm->lock); r =kvm_arch_vcpu_precreate (kvm, id);
@@ -3394,6 +3395,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id) vcpu_decrement:mutex_lock (&kvm->lock); kvm->created_vcpus--; + kvm->stat.created_vcpus--;mutex_unlock (&kvm->lock);return r; }
Signature · 1 line
2.27.0