introduce HVM tracing to unify SVM and VMX tracing
PATCH 1/1 introduce HVM tracing to unify SVM and VMX tracing
Hello,
this patch introduces HVM tracing: one tracing class for both, SVM and VMX.
It adds several new trace events. So we can differentiate between them in the xentrace formats file and format each event's data items appropriately. With this patch the xentrace_format output is much more informative.
The previous simple tracing in SVM and VMX is completely replaced. Unfortunately I could not yet test the VMX part beyond compiling. Any volunteers? Help is very much appreciated!
Some short notes just to mention: This patch does not break xenmon. I don't know any other users of xentrace. The manpage for xentrace is updated. The xentrace command line parameter -e now accepts "hvm".
Cheers
Thomas
# HG changeset patch
# User Thomas Friebel <[email protected]>
# Date 1172494345 -3600
# Node ID 58d3f1c6183bdc71180446b2ee9a8218a5996e55
# Parent 2d3ceb082114bdc77e53b5710a076a1827254710 [xentrace][HVM] introduce HVM tracing to unify SVM and VMX tracing
* new tracing class TRC_HVM replacing TRC_VMX
* 20 new trace events to differentiate the information carried by them ** added corresponding trace points in SVM and VMX code
** updated formats file: descriptive formatting of trace event data ** completely replacing previous simple tracing in VMX
* possibility to toggle single events on and off at compile time in include/asm-x86/hvm/trace.h
tools/xentrace/formats +22 -26
@@ -16,29 +16,25 @@ 0x0002f00c CPU%(cpu)d %(tsc)d (+%(relt 0x0002f00c CPU%(cpu)d %(tsc)d (+%(reltsc)8d) t_timer_fn 0x0002f00d CPU%(cpu)d %(tsc)d (+%(reltsc)8d) dom_timer_fn -0x00080001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMX_VMEXIT [ domid = 0x%(1)08x, eip = 0x%(2)08x, reason = 0x%(3)08x ] -0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMX_INTR [ domid = 0x%(1)08x, trap = 0x%(2)08x, va = 0x%(3)08x ] - -0x00081001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_0 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_0 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x - -0x00081002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_1 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_1 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x - -0x00081003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_2 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_2 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x - -0x00081004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_3 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_3 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x - -0x00081005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_4 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_4 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x - -0x00081006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_5 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_5 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x - -0x00081007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_6 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_6 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x - -0x00081008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT_7 0x%(1)08x 0x%(2)08x 0x%(3)08x -0x00082008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY_7 0x%(1)08x 0x%(2)08x 0x%(3)08x 0x%(4)08x 0x%(5)08x +0x00081001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMENTRY [ dom:vcpu = 0x%(1)08x ] +0x00081002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMEXIT [ dom:vcpu = 0x%(1)08x, rIP = 0x%(2)08x, exitcode = 0x%(3)08x ] +0x00082001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) PF_XEN [ dom:vcpu = 0x%(1)08x, virt = 0x%(2)08x, errorcode = 0x%(3)02x ] +0x00082002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) PF_INJECT [ dom:vcpu = 0x%(1)08x, virt = 0x%(2)08x, errorcode = 0x%(3)02x ] +0x00082003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) INJ_EXC [ dom:vcpu = 0x%(1)08x, vector = 0x%(2)02x, errorcode = 0x%(3)04x ] +0x00082004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) INJ_VIRQ [ dom:vcpu = 0x%(1)08x, vector = 0x%(2)02x, fake = %(3)d ] +0x00082005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) REINJ_VIRQ [ dom:vcpu = 0x%(1)08x, vector = 0x%(2)02x ] +0x00082006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) IO_READ [ dom:vcpu = 0x%(1)08x, port = 0x%(2)04x, size = %(3)d ] +0x00082007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) IO_WRITE [ dom:vcpu = 0x%(1)08x, port = 0x%(2)04x, size = %(3)d ] +0x00082008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) CR_READ [ dom:vcpu = 0x%(1)08x, CR# = %(2)d, value = 0x%(3)08x ] +0x00082009 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) CR_WRITE [ dom:vcpu = 0x%(1)08x, CR# = %(2)d, value = 0x%(3)08x ] +0x0008200A CPU%(cpu)d %(tsc)d (+%(reltsc)8d) DR_READ [ dom:vcpu = 0x%(1)08x ] +0x0008200B CPU%(cpu)d %(tsc)d (+%(reltsc)8d) DR_WRITE [ dom:vcpu = 0x%(1)08x ] +0x0008200C CPU%(cpu)d %(tsc)d (+%(reltsc)8d) MSR_READ [ dom:vcpu = 0x%(1)08x, MSR# = 0x%(2)08x, value = 0x%(3)08x ] +0x0008200D CPU%(cpu)d %(tsc)d (+%(reltsc)8d) MSR_WRITE [ dom:vcpu = 0x%(1)08x, MSR# = 0x%(2)08x, value = 0x%(3)08x ] +0x0008200E CPU%(cpu)d %(tsc)d (+%(reltsc)8d) CPUID [ dom:vcpu = 0x%(1)08x, func = 0x%(2)08x, eax:ebx = 0x%(3)016x, ecx:edx = 0x%(4)016x ] +0x0008200F CPU%(cpu)d %(tsc)d (+%(reltsc)8d) INTR [ dom:vcpu = 0x%(1)08x, vector = 0x%(2)02x ] +0x00082010 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) NMI [ dom:vcpu = 0x%(1)08x ] +0x00082011 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) SMI [ dom:vcpu = 0x%(1)08x ] +0x00082012 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) VMMCALL [ dom:vcpu = 0x%(1)08x, func = 0x%(2)08x ] +0x00082013 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) HLT [ dom:vcpu = 0x%(1)08x, intpending = %(2)d ] +0x00082014 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) INVLPG [ dom:vcpu = 0x%(1)08x, virt = 0x%(2)08x, invlpga = %(3)d, asid = 0x%(4)02x ]
tools/xentrace/xentrace.8 +35 -19
82 lines not on this page · raw message
tools/xentrace/xentrace.c +2 -2
@@ -350,8 +350,8 @@ int parse_evtmask(char *arg, struct argp setup->evt_mask |=TRC_SCHED ; }else if (strcmp (arg,"dom0op" ) ==0 ){ setup->evt_mask |=TRC_DOM0OP ; - }else if (strcmp (arg,"vmx" ) ==0 ){ - setup->evt_mask |=TRC_VMX ; + }else if (strcmp (arg,"hvm" ) ==0 ){ + setup->evt_mask |=TRC_HVM ; }else if (strcmp (arg,"all" ) ==0 ){ setup->evt_mask |=TRC_ALL ; }else {
xen/arch/x86/hvm/io.c +0 -1
@@ -414,7 +414,6 @@ static void hvm_pio_assist(struct cpu_usprintk ("Error: %s unknown port size\n" , __FUNCTION__);domain_crash_synchronous (); } -TRACE_VMEXIT (3 , regs->eax); } }
xen/arch/x86/hvm/svm/intr.c +6 -1
@@ -37,6 +37,7 @@#include <xen/kernel.h> #include <public/hvm/ioreq.h> #include <xen/domain_page.h> +#include <asm/hvm/trace.h> /* * Most of this code is copied from vmx_io.c and modified
@@ -108,6 +109,7 @@ asmlinkage void svm_intr_assist(void)if (irq_masked (vmcb->rflags) || vmcb->interrupt_shadow ) { vmcb->general1_intercepts |=GENERAL1_INTERCEPT_VINTR ; +HVMTRACE_2D (INJ_VIRQ , v,0x0 ,/*fake=*/ 1 );svm_inject_extint (v,0x0 );/* actual vector doesn't really matter */ return ; }
@@ -128,7 +130,10 @@ asmlinkage void svm_intr_assist(void)if ( re_injecting && (pt =is_pt_irq (v, intr_vector, intr_type)) ) ++pt->pending_intr_nr;/* let's inject this interrupt */ -TRACE_3D (TRC_VMX_INTR , v->domain->domain_id, intr_vector,0 ); +if (re_injecting) +HVMTRACE_1D (REINJ_VIRQ , v, intr_vector); +else +HVMTRACE_2D (INJ_VIRQ , v, intr_vector,/*fake=*/ 0 );svm_inject_extint (v, intr_vector);break ;case APIC_DM_SMI :
xen/arch/x86/hvm/svm/svm.c +55 -12
182 lines not on this page · raw message
xen/arch/x86/hvm/vmx/intr.c +2 -1
@@ -37,6 +37,7 @@#include <asm/hvm/vpic.h> #include <asm/hvm/vlapic.h> #include <public/hvm/ioreq.h> +#include <asm/hvm/trace.h> static inline void
@@ -162,8 +163,8 @@ asmlinkage void vmx_intr_assist(void)case APIC_DM_EXTINT :case APIC_DM_FIXED :case APIC_DM_LOWEST : +HVMTRACE_2D (INJ_VIRQ , v, highest_vector,/*fake=*/ 0 );vmx_inject_extint (v, highest_vector,VMX_DELIVER_NO_ERROR_CODE ); -TRACE_3D (TRC_VMX_INTR , v->domain->domain_id, highest_vector,0 );break ;case APIC_DM_SMI :
xen/arch/x86/hvm/vmx/vmx.c +30 -47
240 lines not on this page · raw message
xen/include/asm-x86/hvm/support.h +0 -3
@@ -116,9 +116,6 @@ extern unsigned int opt_hvm_debug_level; #define HVM_DBG_LOG(level, _f, _a...)#endif -#define TRACE_VMEXIT (index, value) \ - current->arch.hvm_vcpu.hvm_trace_values[index] = (value) -/* * Save/restore support */
xen/include/asm-x86/hvm/vcpu.h +0 -2
@@ -45,8 +45,6 @@ struct hvm_vcpu {/* Flags */ int flag_dr_dirty; -unsigned long hvm_trace_values[5 ]; -union {struct arch_vmx_struct vmx;struct arch_svm_struct svm;
xen/include/asm-x86/hvm/vmx/vmx.h +6 -0
@@ -25,6 +25,7 @@#include <asm/processor.h> #include <asm/hvm/vmx/vmcs.h> #include <asm/i387.h> +#include <asm/hvm/trace.h> extern void vmx_asm_vmexit_handler (struct cpu_user_regs );extern void vmx_asm_do_vmentry (void );
@@ -294,6 +295,11 @@ static inline void __vmx_inject_exceptio__vmwrite (VM_ENTRY_INSTRUCTION_LEN , ilen);__vmwrite (VM_ENTRY_INTR_INFO_FIELD , intr_fields); + +if (trap == TRAP_page_fault) +HVMTRACE_2D (PF_INJECT , v, v->arch.hvm_vmx.cpu_cr2, error_code); +else +HVMTRACE_2D (INJ_EXC , v, trap, error_code); } static inline void vmx_inject_hw_exception(
xen/include/public/trace.h +26 -9
@@ -31,17 +31,16 @@#define TRC_GEN 0x0001f000/* General trace */ #define TRC_SCHED 0x0002f000/* Xen Scheduler trace */ #define TRC_DOM0OP 0x0004f000/* Xen DOM0 operation trace */ -#define TRC_VMX 0x0008f000/* Xen VMX trace */ +#define TRC_HVM 0x0008f000/* Xen HVM trace */ #define TRC_MEM 0x0010f000/* Xen memory trace */ #define TRC_ALL 0xfffff000/* Trace subclasses */ #define TRC_SUBCLS_SHIFT 12 -/* trace subclasses for VMX */ -#define TRC_VMXEXIT 0x00081000/* VMX exit trace */ -#define TRC_VMXENTRY 0x00082000/* VMX exit trace */ -#define TRC_VMXINTR 0x00084000/* VMX interrupt trace */ +/* trace subclasses for SVM */ +#define TRC_HVM_ENTRYEXIT 0x00081000/* VMENTRY and #VMEXIT */ +#define TRC_HVM_HANDLER 0x00082000/* various HVM handlers */ /* Trace events per class */ #define TRC_LOST_RECORDS (TRC_GEN + 1)
@@ -67,10 +66,28 @@#define TRC_MEM_PAGE_GRANT_TRANSFER (TRC_MEM + 3)/* trace events per subclass */ -#define TRC_VMX_VMEXIT (TRC_VMXEXIT + 1) -#define TRC_VMX_VMENTRY (TRC_VMXENTRY + 1) -#define TRC_VMX_INTR (TRC_VMXINTR + 1) - +#define TRC_HVM_VMENTRY (TRC_HVM_ENTRYEXIT + 0x01) +#define TRC_HVM_VMEXIT (TRC_HVM_ENTRYEXIT + 0x02) +#define TRC_HVM_PF_XEN (TRC_HVM_HANDLER + 0x01) +#define TRC_HVM_PF_INJECT (TRC_HVM_HANDLER + 0x02) +#define TRC_HVM_INJ_EXC (TRC_HVM_HANDLER + 0x03) +#define TRC_HVM_INJ_VIRQ (TRC_HVM_HANDLER + 0x04) +#define TRC_HVM_REINJ_VIRQ (TRC_HVM_HANDLER + 0x05) +#define TRC_HVM_IO_READ (TRC_HVM_HANDLER + 0x06) +#define TRC_HVM_IO_WRITE (TRC_HVM_HANDLER + 0x07) +#define TRC_HVM_CR_READ (TRC_HVM_HANDLER + 0x08) +#define TRC_HVM_CR_WRITE (TRC_HVM_HANDLER + 0x09) +#define TRC_HVM_DR_READ (TRC_HVM_HANDLER + 0x0A) +#define TRC_HVM_DR_WRITE (TRC_HVM_HANDLER + 0x0B) +#define TRC_HVM_MSR_READ (TRC_HVM_HANDLER + 0x0C) +#define TRC_HVM_MSR_WRITE (TRC_HVM_HANDLER + 0x0D) +#define TRC_HVM_CPUID (TRC_HVM_HANDLER + 0x0E) +#define TRC_HVM_INTR (TRC_HVM_HANDLER + 0x0F) +#define TRC_HVM_NMI (TRC_HVM_HANDLER + 0x10) +#define TRC_HVM_SMI (TRC_HVM_HANDLER + 0x11) +#define TRC_HVM_VMMCALL (TRC_HVM_HANDLER + 0x12) +#define TRC_HVM_HLT (TRC_HVM_HANDLER + 0x13) +#define TRC_HVM_INVLPG (TRC_HVM_HANDLER + 0x14)/* This structure represents a single trace buffer record. */ struct t_rec {
xen/include/asm-x86/hvm/trace.h new file +44 -0
44 lines not on this page · raw message
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
+1 to the idea, seems like a good way to do things.
Premusably the resulting functionality is a superset of the previous tracing capabilities?
I have no objections to the Xentrace-related changes, all looks sensible and useful. Did spot anything in particular to complain about in the HVM code either, but I didn't look that hard at those parts.
Cheers,
Mark
Signature · 4 lines
Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel!
Quoted 2 lines · On Monday, 26.02.2007, 18:33 +0000 Mark Williamson wrote:
Premusably the resulting functionality is a superset of the previous tracing capabilities?
Yes this is what I intended. I hope I didn't miss a point.
Cheers,
Thomas
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
I'm glad someone from AMD is interested in this -- I've been using the VMX tracing extensively, and have long thought unifying them would be a good idea. Thanks for taking the initiative on this.
A couple of comments:
First, there's a lot of manual duplication here. This is just begging for the VMX and SVM code to get out of sync again. Would it be easier if we made a series of macros that each called; e.g.,
#define HVMTRACE_PF_XEN(v, _eq, _ec) \
HVMTRACE_2D(v, _eq, _ec)
Then, in the appropriate place in each file:
HVMTRACE_PF_XEN(v, exit_qualification, regs->error_code);
This way, if someone modifies the trace, they'll be prompted by the compiler to modify it in both places.
Second, about taking traces as you go along rather than storing up data to be traced at the next vmenter. I'm curious what your reasoning was behind this?
Your method has the advantage that the "info" trace itself has more data; rather than taking up 2 slots with vmentry and the "exit reason" (for VMX), you have more space available for other information.
There are some disadvantages though. First, it automatically increases the size of the trace by 50%. Many of my traces reach in the multi-gigabyte range, so this is a concern for me. We've discussed variable-size trace buffers on the list before, but I don't remember if anyone volunteered to actually do it.
One of the advantages of tracing it at the end is the ability to set little bits as you go. I have some private 'extensions' to the VMX tracing that gives a lot more information about, for example, the page fault path. One of the things that I do is have a set of flags that get set depending on different paths that get taken -- for example, was a page promoted? Did any accessed or dirty bits get set? And so on. With the current setup, I can just have the code flip the bit in hvm_trace_values[] directly. Under a setup like yours, I'd have to do that manually for each one.
Overall, I think the +50% trace size is a big minus for me.
I'll look over the actual changes to VMX a little more closely, and give it a test run tomorrow.
Peace,
-George
Quoted 28 lines · Thomas Friebel
Hello,
this patch introduces HVM tracing: one tracing class for both, SVM and VMX.
It adds several new trace events. So we can differentiate between them in the xentrace formats file and format each event's data items appropriately. With this patch the
xentrace_formatoutput is much more informative.The previous simple tracing in SVM and VMX is completely replaced. Unfortunately I could not yet test the VMX part beyond compiling. Any volunteers? Help is very much appreciated!
Some short notes just to mention: This patch does not break xenmon. I don't know any other users of xentrace. The manpage for xentrace is updated. The xentrace command line parameter -e now accepts "hvm".
Cheers
Thomas_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 4 lines
First, there's a lot of manual duplication here. This is just begging for the VMX and SVM code to get out of sync again. Would it be easier if we made a series of macros that each called; e.g.,
Quoted 8 lines
#define HVMTRACE_PF_XEN(v, _eq, _ec) \ HVMTRACE_2D(v, _eq, _ec)Then, in the appropriate place in each file:
HVMTRACE_PF_XEN(v, exit_qualification, regs->error_code);This way, if someone modifies the trace, they'll be prompted by the compiler to modify it in both places.
Yes, this way you can't easily add or remove parameters on one side (SVM/VMX) and ignore the other (VMX/SVM). But I think the improvement is not that big.
We could introduce one inline function per trace event. This way we'd additionally assure type correctness of the parameters.
Quoted 3 lines
Second, about taking traces as you go along rather than storing up data to be traced at the next vmenter. I'm curious what your reasoning was behind this?
The main reason was to be able to exploit the xentrace_format feature. For a nice formatting of the trace logs we need one event ID per format string. In the format file we dedicate one line per event ID and this way have readable and much easier understandable output.
This sums up to at least three trace entries per round trip instead of two:
* one entry on #VMEXIT just for the exit reason and the timestamp
* one or more entries giving information about the execution path
* one entry just before VMENTRY giving a final timestamp.
If we really want to optimize for trace file size we could remember the exit reason and timestamp on #VMEXIT and stick with saving the trace data until just before VMENTRY. Then all the data is packed into one trace entry with the specific event ID for formatting.
But by doing that there is no clean way of logging the execution path. So I decided to have the 3+ trace entries per roundtrip and to include a facility to toggle single events on and off. The easiest way was to introduce some #defines to do this at compile time.
Cheers,
Thomas