Skip to contentSkip to conversation

Extend KVM trace_kvm_nested_vmrun() to support VMX

PATCH Applied Mingwei Zhang 2 participants 7 messages Aug 25, 2022 – Aug 30, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover Extend KVM trace_kvm_nested_vmrun() to support VMX

This patch set update trace_kvm_nested_vmrun() to support VMX. In addition, add the guest_pgd field to enrich the trace information.

v3 -> v4:
- Rebase to tip of kvm/queue.

v2 -> v3:
- Split the insertion of guest pgd field into a separate patch [seanjc].
- Update field names as suggested [seanjc].

v2:
- https://lore.kernel.org/lkml/[email protected]/T/

v1 link:
- https://lore.kernel.org/lkml/[email protected]/T/

David Matlack (1):
KVM: nVMX: Add tracepoint for nested vmenter

Mingwei Zhang (2):
KVM: x86: Update trace function for nested VM entry to support VMX
KVM: x86: Print guest pgd in kvm_nested_vmenter()

 arch/x86/kvm/svm/nested.c |  8 ++++++--
 arch/x86/kvm/trace.h      | 33 +++++++++++++++++++++++----------
 arch/x86/kvm/vmx/nested.c | 10 ++++++++++
 arch/x86/kvm/x86.c        |  2 +-
 4 files changed, 40 insertions(+), 13 deletions(-)

base-commit: 372d07084593dc7a399bf9bee815711b1fb1bcf2

Signature · 1 line
2.37.2.672.g94769d06f0-goog
#1 lore raw

PATCH v4 1/3 KVM: x86: Update trace function for nested VM entry to support VMX

Update trace function for nested VM entry to support VMX. Existing trace function only supports nested VMX and the information printed out is AMD specific.

So, update trace_kvm_nested_vmrun() to trace_kvm_nested_vmenter(), since 'vmenter' is generic. Add a new field 'isa' to recognize Intel and AMD; Update the output to print out VMX/SVM related naming respectively, eg., vmcb vs. vmcs; npt vs. ept.

Opportunistically update the call site of trace_kvm_nested_vmenter() to make one line per parameter.

---
 arch/x86/kvm/svm/nested.c |  6 ++++--
 arch/x86/kvm/trace.h      | 28 ++++++++++++++++++----------
 arch/x86/kvm/x86.c        |  2 +-
 3 files changed, 23 insertions(+), 13 deletions(-)
arch/x86/kvm/svm/nested.c +4 -2
index 76dcc8a3e849..835c508eed8e 100644
@@ -781,11 +781,13 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa,
 	struct vcpu_svm *svm = to_svm(vcpu);
 	int ret;
 
-	trace_kvm_nested_vmrun(svm->vmcb->save.rip, vmcb12_gpa,
+	trace_kvm_nested_vmenter(svm->vmcb->save.rip,
+			       vmcb12_gpa,
 			       vmcb12->save.rip,
 			       vmcb12->control.int_ctl,
 			       vmcb12->control.event_inj,
-			       vmcb12->control.nested_ctl);
+			       vmcb12->control.nested_ctl,
+			       KVM_ISA_SVM);
 
 	trace_kvm_nested_intercepts(vmcb12->control.intercepts[INTERCEPT_CR] & 0xffff,
 				    vmcb12->control.intercepts[INTERCEPT_CR] >> 16,
arch/x86/kvm/trace.h +18 -10
index 2120d7c060a9..e7f0da9474f0 100644
@@ -589,10 +589,11 @@ TRACE_EVENT(kvm_pv_eoi,
 /*
  * Tracepoint for nested VMRUN
  */
-TRACE_EVENT(kvm_nested_vmrun,
+TRACE_EVENT(kvm_nested_vmenter,
 	    TP_PROTO(__u64 rip, __u64 vmcb, __u64 nested_rip, __u32 int_ctl,
-		     __u32 event_inj, bool npt),
-	    TP_ARGS(rip, vmcb, nested_rip, int_ctl, event_inj, npt),
+		     __u32 event_inj, bool tdp_enabled, __u32 isa),
+	    TP_ARGS(rip, vmcb, nested_rip, int_ctl, event_inj, tdp_enabled,
+		    isa),
 
 	TP_STRUCT__entry(
 		__field(	__u64,		rip		)
@@ -600,7 +601,8 @@ TRACE_EVENT(kvm_nested_vmrun,
 		__field(	__u64,		nested_rip	)
 		__field(	__u32,		int_ctl		)
 		__field(	__u32,		event_inj	)
-		__field(	bool,		npt		)
+		__field(	bool,		tdp_enabled	)
+		__field(	__u32,		isa		)
 	),
 
 	TP_fast_assign(
@@ -609,14 +611,20 @@ TRACE_EVENT(kvm_nested_vmrun,
 		__entry->nested_rip	= nested_rip;
 		__entry->int_ctl	= int_ctl;
 		__entry->event_inj	= event_inj;
-		__entry->npt		= npt;
+		__entry->tdp_enabled	= tdp_enabled;
+		__entry->isa		= isa;
 	),
 
-	TP_printk("rip: 0x%016llx vmcb: 0x%016llx nrip: 0x%016llx int_ctl: 0x%08x "
-		  "event_inj: 0x%08x npt: %s",
-		__entry->rip, __entry->vmcb, __entry->nested_rip,
-		__entry->int_ctl, __entry->event_inj,
-		__entry->npt ? "on" : "off")
+	TP_printk("rip: 0x%016llx %s: 0x%016llx nested_rip: 0x%016llx "
+		  "int_ctl: 0x%08x event_inj: 0x%08x nested_%s: %s",
+		__entry->rip,
+		__entry->isa == KVM_ISA_VMX ? "vmcs" : "vmcb",
+		__entry->vmcb,
+		__entry->nested_rip,
+		__entry->int_ctl,
+		__entry->event_inj,
+		__entry->isa == KVM_ISA_VMX ? "ept" : "npt",
+		__entry->tdp_enabled ? "on" : "off")
 );
 
 TRACE_EVENT(kvm_nested_intercepts,
arch/x86/kvm/x86.c +1 -1
index d7374d768296..0c36528ab345 100644
@@ -13375,7 +13375,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_inj_virq);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_page_fault);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_msr);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_cr);
-EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmrun);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmenter);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmexit);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmexit_inject);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intr_vmexit);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
#2 lore raw

Re: [PATCH v4 1/3] KVM: x86: Update trace function for nested VM entry to support VMX

Quoted 26 lines · Mingwei Zhang

Update trace function for nested VM entry to support VMX. Existing trace function only supports nested VMX and the information printed out is AMD specific.

So, update trace_kvm_nested_vmrun() to trace_kvm_nested_vmenter(), since 'vmenter' is generic. Add a new field 'isa' to recognize Intel and AMD; Update the output to print out VMX/SVM related naming respectively, eg., vmcb vs. vmcs; npt vs. ept.

Opportunistically update the call site of trace_kvm_nested_vmenter() to make one line per parameter.

---
 arch/x86/kvm/svm/nested.c |  6 ++++--
 arch/x86/kvm/trace.h      | 28 ++++++++++++++++++----------
 arch/x86/kvm/x86.c        |  2 +-
 3 files changed, 23 insertions(+), 13 deletions(-)
arch/x86/kvm/svm/nested.c +0 -0
index 76dcc8a3e849..835c508eed8e 100644
@@ -781,11 +781,13 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa,
 	struct vcpu_svm *svm = to_svm(vcpu);
 	int ret;

arch/x86/kvm/svm/nested.c int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, · line 790

Quoted 10 lines
arch/x86/kvm/svm/nested.c +4 -2
 
-	trace_kvm_nested_vmrun(svm->vmcb->save.rip, vmcb12_gpa,
+	trace_kvm_nested_vmenter(svm->vmcb->save.rip,
+			       vmcb12_gpa,
 			       vmcb12->save.rip,
 			       vmcb12->control.int_ctl,
 			       vmcb12->control.event_inj,
-			       vmcb12->control.nested_ctl);
+			       vmcb12->control.nested_ctl,
+			       KVM_ISA_SVM);

Align indentation.

Quoted 9 lines
arch/x86/kvm/svm/nested.c +0 -0
 	trace_kvm_nested_intercepts(vmcb12->control.intercepts[INTERCEPT_CR] & 0xffff,
 				    vmcb12->control.intercepts[INTERCEPT_CR] >> 16,
arch/x86/kvm/trace.h +0 -0
index 2120d7c060a9..e7f0da9474f0 100644
@@ -589,10 +589,11 @@ TRACE_EVENT(kvm_pv_eoi,
 /*

arch/x86/kvm/trace.h TRACE_EVENT(kvm_pv_eoi, · line 596

Quoted 10 lines
arch/x86/kvm/trace.h +4 -3
  * Tracepoint for nested VMRUN
  */
-TRACE_EVENT(kvm_nested_vmrun,
+TRACE_EVENT(kvm_nested_vmenter,
 	    TP_PROTO(__u64 rip, __u64 vmcb, __u64 nested_rip, __u32 int_ctl,
-		     __u32 event_inj, bool npt),
-	    TP_ARGS(rip, vmcb, nested_rip, int_ctl, event_inj, npt),
+		     __u32 event_inj, bool tdp_enabled, __u32 isa),
+	    TP_ARGS(rip, vmcb, nested_rip, int_ctl, event_inj, tdp_enabled,
+		    isa),

Let this poke out, "isa" feels quite lonely here. It's a moot point when patch 3 comes along, mostly a "for future reference" thing.

Quoted 26 lines
arch/x86/kvm/trace.h +4 -7
 	TP_STRUCT__entry(
 		__field(	__u64,		rip		)
@@ -600,7 +601,8 @@ TRACE_EVENT(kvm_nested_vmrun,
 		__field(	__u64,		nested_rip	)
 		__field(	__u32,		int_ctl		)
 		__field(	__u32,		event_inj	)
-		__field(	bool,		npt		)
+		__field(	bool,		tdp_enabled	)
+		__field(	__u32,		isa		)
 	),
 
 	TP_fast_assign(
@@ -609,14 +611,20 @@ TRACE_EVENT(kvm_nested_vmrun,
 		__entry->nested_rip	= nested_rip;
 		__entry->int_ctl	= int_ctl;
 		__entry->event_inj	= event_inj;
-		__entry->npt		= npt;
+		__entry->tdp_enabled	= tdp_enabled;
+		__entry->isa		= isa;
 	),
 
-	TP_printk("rip: 0x%016llx vmcb: 0x%016llx nrip: 0x%016llx int_ctl: 0x%08x "
-		  "event_inj: 0x%08x npt: %s",
-		__entry->rip, __entry->vmcb, __entry->nested_rip,
-		__entry->int_ctl, __entry->event_inj,
-		__entry->npt ? "on" : "off")

arch/x86/kvm/trace.h TRACE_EVENT(kvm_nested_vmrun, · line 627

Quoted 10 lines
arch/x86/kvm/trace.h +10 -0
+	TP_printk("rip: 0x%016llx %s: 0x%016llx nested_rip: 0x%016llx "
+		  "int_ctl: 0x%08x event_inj: 0x%08x nested_%s: %s",
+		__entry->rip,
+		__entry->isa == KVM_ISA_VMX ? "vmcs" : "vmcb",
+		__entry->vmcb,
+		__entry->nested_rip,
+		__entry->int_ctl,
+		__entry->event_inj,
+		__entry->isa == KVM_ISA_VMX ? "ept" : "npt",
+		__entry->tdp_enabled ? "on" : "off")

Align indentation.

#3 lore raw

PATCH v4 2/3 KVM: nVMX: Add tracepoint for nested vmenter

In reply to Mingwei Zhang

From: David Matlack <[email protected]>

Call trace_kvm_nested_vmenter() during nested VMLAUNCH/VMRESUME to bring parity with nSVM's usage of the tracepoint during nested VMRUN.

Attempt to use analagous VMCS fields to the VMCB fields that are reported in the SVM case:

"int_ctl": 32-bit field of the VMCB that the CPU uses to deliver virtual interrupts. The analagous VMCS field is the 16-bit "guest interrupt status".

"event_inj": 32-bit field of VMCB that is used to inject events (exceptions and interrupts) into the guest. The analagous VMCS field is the "VM-entry interruption-information field".

"npt_enabled": 1 when the VCPU has enabled nested paging. The analagous VMCS field is the enable-EPT execution control.

"npt_addr": 64-bit field when the VCPU has enabled nested paging. The analagous VMCS field is the ept_pointer.

[move the code into the nested_vmx_enter_non_root_mode().]
---
 arch/x86/kvm/vmx/nested.c | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/x86/kvm/vmx/nested.c +8 -0
index ddd4367d4826..f72fe9452391 100644
@@ -3364,6 +3364,14 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 	};
 	u32 failed_index;
 
+	trace_kvm_nested_vmenter(kvm_rip_read(vcpu),
+				 vmx->nested.current_vmptr,
+				 vmcs12->guest_rip,
+				 vmcs12->guest_intr_status,
+				 vmcs12->vm_entry_intr_info_field,
+				 vmcs12->secondary_vm_exec_control & SECONDARY_EXEC_ENABLE_EPT,
+				 KVM_ISA_VMX);
+
 	kvm_service_local_tlb_flush_requests(vcpu);
 
 	evaluate_pending_interrupts = exec_controls_get(vmx) &
Signature · 1 line
2.37.2.672.g94769d06f0-goog
#4 lore raw

PATCH v4 3/3 KVM: x86: Print guest pgd in kvm_nested_vmenter()

In reply to Mingwei Zhang

Print guest pgd in kvm_nested_vmenter() to enrich the information for tracing. When tdp is enabled, print the value of tdp page table (EPT/NPT); when tdp is disabled, print the value of non-nested CR3.

---
 arch/x86/kvm/svm/nested.c |  2 ++
 arch/x86/kvm/trace.h      | 13 +++++++++----
 arch/x86/kvm/vmx/nested.c |  2 ++
 3 files changed, 13 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/nested.c +2 -0
index 835c508eed8e..05b7994244c5 100644
@@ -787,6 +787,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa,
 			       vmcb12->control.int_ctl,
 			       vmcb12->control.event_inj,
 			       vmcb12->control.nested_ctl,
+			       vmcb12->control.nested_cr3,
+			       vmcb12->save.cr3,
 			       KVM_ISA_SVM);
 
 	trace_kvm_nested_intercepts(vmcb12->control.intercepts[INTERCEPT_CR] & 0xffff,
arch/x86/kvm/trace.h +9 -4
index e7f0da9474f0..b2be0348bb14 100644
@@ -591,9 +591,10 @@ TRACE_EVENT(kvm_pv_eoi,
  */
 TRACE_EVENT(kvm_nested_vmenter,
 	    TP_PROTO(__u64 rip, __u64 vmcb, __u64 nested_rip, __u32 int_ctl,
-		     __u32 event_inj, bool tdp_enabled, __u32 isa),
+		     __u32 event_inj, bool tdp_enabled, __u64 guest_tdp,
+		     __u64 guest_cr3, __u32 isa),
 	    TP_ARGS(rip, vmcb, nested_rip, int_ctl, event_inj, tdp_enabled,
-		    isa),
+		    guest_tdp, guest_cr3, isa),
 
 	TP_STRUCT__entry(
 		__field(	__u64,		rip		)
@@ -602,6 +603,7 @@ TRACE_EVENT(kvm_nested_vmenter,
 		__field(	__u32,		int_ctl		)
 		__field(	__u32,		event_inj	)
 		__field(	bool,		tdp_enabled	)
+		__field(	__u64,		guest_pgd	)
 		__field(	__u32,		isa		)
 	),
 
@@ -612,11 +614,13 @@ TRACE_EVENT(kvm_nested_vmenter,
 		__entry->int_ctl	= int_ctl;
 		__entry->event_inj	= event_inj;
 		__entry->tdp_enabled	= tdp_enabled;
+		__entry->guest_pgd	= tdp_enabled ? guest_tdp : guest_cr3;
 		__entry->isa		= isa;
 	),
 
 	TP_printk("rip: 0x%016llx %s: 0x%016llx nested_rip: 0x%016llx "
-		  "int_ctl: 0x%08x event_inj: 0x%08x nested_%s: %s",
+		  "int_ctl: 0x%08x event_inj: 0x%08x nested_%s: %s, "
+		  "guest_pgd: 0x%016llx",
 		__entry->rip,
 		__entry->isa == KVM_ISA_VMX ? "vmcs" : "vmcb",
 		__entry->vmcb,
@@ -624,7 +628,8 @@ TRACE_EVENT(kvm_nested_vmenter,
 		__entry->int_ctl,
 		__entry->event_inj,
 		__entry->isa == KVM_ISA_VMX ? "ept" : "npt",
-		__entry->tdp_enabled ? "on" : "off")
+		__entry->tdp_enabled ? "on" : "off",
+		__entry->guest_pgd)
 );
 
 TRACE_EVENT(kvm_nested_intercepts,
arch/x86/kvm/vmx/nested.c +2 -0
index f72fe9452391..f963e5ce0a28 100644
@@ -3370,6 +3370,8 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 				 vmcs12->guest_intr_status,
 				 vmcs12->vm_entry_intr_info_field,
 				 vmcs12->secondary_vm_exec_control & SECONDARY_EXEC_ENABLE_EPT,
+				 vmcs12->ept_pointer,
+				 vmcs12->guest_cr3,
 				 KVM_ISA_VMX);
 
 	kvm_service_local_tlb_flush_requests(vcpu);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
#5 lore raw

Re: [PATCH v4 3/3] KVM: x86: Print guest pgd in kvm_nested_vmenter()

arch/x86/kvm/trace.h TRACE_EVENT(kvm_pv_eoi, · line 594

Quoted 11 lines · Mingwei Zhang

---

arch/x86/kvm/trace.h +1 -1
index e7f0da9474f0..b2be0348bb14 100644
@@ -591,9 +591,10 @@ TRACE_EVENT(kvm_pv_eoi,
  */
 TRACE_EVENT(kvm_nested_vmenter,
 	    TP_PROTO(__u64 rip, __u64 vmcb, __u64 nested_rip, __u32 int_ctl,
-		     __u32 event_inj, bool tdp_enabled, __u32 isa),
+		     __u32 event_inj, bool tdp_enabled, __u64 guest_tdp,

s/guest_tdp_pgd to differentiate it from "tdp_enabled"

arch/x86/kvm/trace.h TRACE_EVENT(kvm_nested_vmenter, · line 623

Quoted 4 lines
arch/x86/kvm/trace.h +2 -1
        TP_printk("rip: 0x%016llx %s: 0x%016llx nested_rip: 0x%016llx "
-                 "int_ctl: 0x%08x event_inj: 0x%08x nested_%s: %s",
+                 "int_ctl: 0x%08x event_inj: 0x%08x nested_%s: %s, "
+                 "guest_pgd: 0x%016llx",

It's a little gross, but this can spit out nested_eptp vs. nested_cr3 vs. guest_cr3.

arch/x86/kvm/trace.h TRACE_EVENT(kvm_nested_vmenter, · line 631

Quoted 10 lines
arch/x86/kvm/trace.h +1 -1
                __entry->rip,
                __entry->isa == KVM_ISA_VMX ? "vmcs" : "vmcb",
                __entry->vmcb,
@@ -624,7 +628,8 @@ TRACE_EVENT(kvm_nested_vmenter,
 		__entry->int_ctl,
 		__entry->event_inj,
 		__entry->isa == KVM_ISA_VMX ? "ept" : "npt",
-		__entry->tdp_enabled ? "on" : "off")
+		__entry->tdp_enabled ? "on" : "off",

To keep things aligned, and because "on nested_eptp" reads as a combined snippet

event_inj: 0x00000000 nested_ept: off guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept: off guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept: on nested_eptp: 0x0000000007ec501e
event_inj: 0x00000000 nested_ept: on nested_eptp: 0x0000000007ec501e

what about teaking the format so that the output looks like this?

event_inj: 0x00000000 nested_ept=n guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept=n guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept=y nested_eptp: 0x0000000007ec501e
event_inj: 0x00000000 nested_ept=y nested_eptp: 0x0000000007ec501e

Deviating from the ": " style bothers me, but I find this difficult to read. Again, letters delimited by whitespace get visually clumped together.

event_inj: 0x00000000 nested_ept: n guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept: n guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept: y nested_eptp: 0x0000000007ec501e
event_inj: 0x00000000 nested_ept: y nested_eptp: 0x0000000007ec501e

And this looks like a typo

event_inj: 0x00000000 nested_ept:n guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept:n guest_cr3: 0x0000000001007000
event_inj: 0x00000000 nested_ept:y nested_eptp: 0x0000000007ec501e
event_inj: 0x00000000 nested_ept:y nested_eptp: 0x0000000007ec501e

We could leave off the nested_{ept/npt} entirely and leave the differentation to the nested_eptp vs. nested_cr3 vs. guest_cr3, but I don't think that's worth shaving a few chars.

maintainer Applied
#6 lore raw
In reply to Mingwei Zhang
Quoted 2 lines · Mingwei Zhang

This patch set update trace_kvm_nested_vmrun() to support VMX. In addition, add the guest_pgd field to enrich the trace information.

...

Quoted 7 lines

David Matlack (1):
KVM: nVMX: Add tracepoint for nested vmenter

Mingwei Zhang (2):
KVM: x86: Update trace function for nested VM entry to support VMX
KVM: x86: Print guest pgd in kvm_nested_vmenter()

Quoted 5 lines
 arch/x86/kvm/svm/nested.c |  8 ++++++--
 arch/x86/kvm/trace.h      | 33 +++++++++++++++++++++++----------
 arch/x86/kvm/vmx/nested.c | 10 ++++++++++
 arch/x86/kvm/x86.c        |  2 +-
 4 files changed, 40 insertions(+), 13 deletions(-)

Some nits that I fixed up when applying (will respond to individual patches).

Pushed to branch for_paolo/6.1 at:

https://github.com/sean-jc/linux.git

Unless you hear otherwise, it will make its way the official KVM tree "soon".

Note, the commit IDs are not guaranteed to be stable.