Implement support for external IPT monitoring
v5 cover Implement support for external IPT monitoring
Intel Processor Trace is an architectural extension available in modern Intel family CPUs. It allows recording the detailed trace of activity while the processor executes the code. One might use the recorded trace to reconstruct the code flow. It means, to find out the executed code paths, determine branches taken, and so forth.
The abovementioned feature is described in Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3C: System Programming Guide, Part 3, Chapter 36: "Intel Processor Trace."
This patch series implements an interface that Dom0 could use in order to enable IPT for particular vCPUs in DomU, allowing for external monitoring. Such a feature has numerous applications like malware monitoring, fuzzing, or performance testing.
Also thanks to Tamas K Lengyel for a few preliminary hints before first version of this patch was submitted to xen-devel.
Changed since v1:
* MSR_RTIT_CTL is managed using MSR load lists
* other PT-related MSRs are modified only when vCPU goes out of context
* trace buffer is now acquired as a resource
* added vmtrace_pt_size parameter in xl.cfg, the size of trace buffer must be specified in the moment of domain creation
* trace buffers are allocated on domain creation, destructed on domain destruction
* HVMOP_vmtrace_ipt_enable/disable is limited to enabling/disabling PT these calls don't manage buffer memory anymore
* lifted 32 MFN/GFN array limit when acquiring resources
* minor code style changes according to review
Changed since v2:
* trace buffer is now allocated on domain creation (in v2 it was allocated when hvm param was set)
* restored 32-item limit in mfn/gfn arrays in acquire_resource and instead implemented hypercall continuations
* code changes according to Jan's and Roger's review
Changed since v3:
* vmtrace HVMOPs are not implemented as DOMCTLs
* patches splitted up according to Andrew's comments
* code changes according to v3 review on the mailing list
Changed since v4:
* rebased to commit be63d9d4
* fixed dependencies between patches (earlier patches don't reference further patches)
* introduced preemption check in acquire_resource
* moved buffer allocation to common code
* splitted some patches according to code review
* minor fixes according to code review
This patch series is available on GitHub:
https://github.com/icedevml/xen/tree/ipt-patch-v5
Michal Leszczynski (11):
memory: batch processing in acquire_resource()
x86/vmx: add Intel PT MSR definitions
x86/vmx: add IPT cpu feature
common: add vmtrace_pt_size domain parameter
tools/libxl: add vmtrace_pt_size parameter
x86/hvm: processor trace interface in HVM
x86/vmx: implement IPT in VMX
x86/mm: add vmtrace_buf resource type
x86/domctl: add XEN_DOMCTL_vmtrace_op
tools/libxc: add xc_vmtrace_* functions
tools/proctrace: add proctrace tool
docs/man/xl.cfg.5.pod.in | 11 ++ tools/golang/xenlight/helpers.gen.go | 2 + tools/golang/xenlight/types.gen.go | 1 + tools/libxc/Makefile | 1 + tools/libxc/include/xenctrl.h | 39 +++++ tools/libxc/xc_vmtrace.c | 73 +++++++++ tools/libxl/libxl.h | 8 + tools/libxl/libxl_create.c | 1 + tools/libxl/libxl_types.idl | 2 + tools/proctrace/Makefile | 48 ++++++ tools/proctrace/proctrace.c | 163 ++++++++++++++++++++ tools/xl/xl_parse.c | 22 +++ xen/arch/x86/domain.c | 19 +++ xen/arch/x86/domctl.c | 48 ++++++ xen/arch/x86/hvm/vmx/vmcs.c | 15 +- xen/arch/x86/hvm/vmx/vmx.c | 109 +++++++++++++ xen/common/domain.c | 33 ++++ xen/common/memory.c | 77 ++++++++- xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/hvm/hvm.h | 20 +++ xen/include/asm-x86/hvm/vmx/vmcs.h | 4 + xen/include/asm-x86/hvm/vmx/vmx.h | 14 ++ xen/include/asm-x86/msr-index.h | 24 +++ xen/include/public/arch-x86/cpufeatureset.h | 1 + xen/include/public/domctl.h | 27 ++++ xen/include/public/memory.h | 1 + xen/include/xen/domain.h | 2 + xen/include/xen/sched.h | 8 + 28 files changed, 768 insertions(+), 6 deletions(-) create mode 100644 tools/libxc/xc_vmtrace.c create mode 100644 tools/proctrace/Makefile create mode 100644 tools/proctrace/proctrace.c
Signature · 1 line
2.17.1
PATCH v5 1/11 memory: batch processing in acquire_resource()
From: Michal Leszczynski <[email protected]>
Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.
Be aware that this modifies the behavior of acquire_resource call with frame_list=NULL. While previously it would return the size of internal array (32), with this patch it returns the maximal quantity of frames that could be requested at once, i.e. UINT_MAX >> MEMOP_EXTENT_SHIFT.
--- xen/common/memory.c | 49 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 5 deletions(-)
xen/common/memory.c +44 -5
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id, }static int acquire_resource ( -XEN_GUEST_HANDLE_PARAM (xen_mem_acquire_resource_t) arg) +XEN_GUEST_HANDLE_PARAM (xen_mem_acquire_resource_t) arg, +unsigned long *start_extent) {struct domain *d, *currd = current->domain;xen_mem_acquire_resource_t xmar; +uint32_t total_frames;/* * The mfn_list and gfn_list (below) arrays are ok on stack for the * moment since they are small, but if they need to grow in future
@@ -1069,7 +1071,7 @@ static int acquire_resource(if ( xmar.nr_frames )return -EINVAL ; - xmar.nr_frames =ARRAY_SIZE (mfn_list); + xmar.nr_frames =UINT_MAX >>MEMOP_EXTENT_SHIFT ;if (__copy_field_to_guest (arg, &xmar, nr_frames) )return -EFAULT ;
@@ -1077,8 +1079,28 @@ static int acquire_resource(return 0 ; } + total_frames = xmar.nr_frames; + +/* Is the size too large for us to encode a continuation? */ +if (unlikely (xmar.nr_frames > (UINT_MAX >>MEMOP_EXTENT_SHIFT )) ) +return -EINVAL ; + +if ( *start_extent ) + { +/* +* Check whether start_extent is in bounds, as this +* value if visible to the calling domain. +*/ +if ( *start_extent > xmar.nr_frames ) +return -EINVAL ; + + xmar.frame += *start_extent; + xmar.nr_frames -= *start_extent; +guest_handle_add_offset (xmar.frame_list, *start_extent); + } +if ( xmar.nr_frames >ARRAY_SIZE (mfn_list) ) -return -E2BIG ; + xmar.nr_frames =ARRAY_SIZE (mfn_list); rc =rcu_lock_remote_domain_by_id (xmar.domid, &d);if ( rc )
@@ -1135,6 +1157,14 @@ static int acquire_resource( } } +if ( !rc ) + { + *start_extent += xmar.nr_frames; + +if ( *start_extent != total_frames ) + rc = -ERESTART ; + } + out:rcu_unlock_domain (d);
@@ -1599,8 +1629,17 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)#endif case XENMEM_acquire_resource: - rc =acquire_resource ( -guest_handle_cast (arg, xen_mem_acquire_resource_t)); +do { + rc =acquire_resource ( +guest_handle_cast (arg, xen_mem_acquire_resource_t), + &start_extent); + +if (hypercall_preempt_check () ) +return hypercall_create_continuation ( + __HYPERVISOR_memory_op,"lh" , + op | (start_extent <<MEMOP_EXTENT_SHIFT ), arg); + }while ( rc == -ERESTART ); +break ;default :
Signature · 1 line
2.17.1
PATCH v5 2/11 x86/vmx: add Intel PT MSR definitions
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Define constants related to Intel Processor Trace features.
--- xen/include/asm-x86/msr-index.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
xen/include/asm-x86/msr-index.h +24 -0
@@ -72,7 +72,31 @@#define MSR_RTIT_OUTPUT_BASE 0x00000560#define MSR_RTIT_OUTPUT_MASK 0x00000561#define MSR_RTIT_CTL 0x00000570 +#define RTIT_CTL_TRACE_EN (_AC(1, ULL) << 0) +#define RTIT_CTL_CYC_EN (_AC(1, ULL) << 1) +#define RTIT_CTL_OS (_AC(1, ULL) << 2) +#define RTIT_CTL_USR (_AC(1, ULL) << 3) +#define RTIT_CTL_PWR_EVT_EN (_AC(1, ULL) << 4) +#define RTIT_CTL_FUP_ON_PTW (_AC(1, ULL) << 5) +#define RTIT_CTL_FABRIC_EN (_AC(1, ULL) << 6) +#define RTIT_CTL_CR3_FILTER (_AC(1, ULL) << 7) +#define RTIT_CTL_TOPA (_AC(1, ULL) << 8) +#define RTIT_CTL_MTC_EN (_AC(1, ULL) << 9) +#define RTIT_CTL_TSC_EN (_AC(1, ULL) << 10) +#define RTIT_CTL_DIS_RETC (_AC(1, ULL) << 11) +#define RTIT_CTL_PTW_EN (_AC(1, ULL) << 12) +#define RTIT_CTL_BRANCH_EN (_AC(1, ULL) << 13) +#define RTIT_CTL_MTC_FREQ (_AC(0xf, ULL) << 14) +#define RTIT_CTL_CYC_THRESH (_AC(0xf, ULL) << 19) +#define RTIT_CTL_PSB_FREQ (_AC(0xf, ULL) << 24) +#define RTIT_CTL_ADDR (n) (_AC(0xf, ULL) << (32 + 4 * (n)))#define MSR_RTIT_STATUS 0x00000571 +#define RTIT_STATUS_FILTER_EN (_AC(1, ULL) << 0) +#define RTIT_STATUS_CONTEXT_EN (_AC(1, ULL) << 1) +#define RTIT_STATUS_TRIGGER_EN (_AC(1, ULL) << 2) +#define RTIT_STATUS_ERROR (_AC(1, ULL) << 4) +#define RTIT_STATUS_STOPPED (_AC(1, ULL) << 5) +#define RTIT_STATUS_BYTECNT (_AC(0x1ffff, ULL) << 32)#define MSR_RTIT_CR3_MATCH 0x00000572#define MSR_RTIT_ADDR_A (n) (0x00000580 + (n) * 2)#define MSR_RTIT_ADDR_B (n) (0x00000581 + (n) * 2)
Signature · 1 line
2.17.1
PATCH v5 3/11 x86/vmx: add IPT cpu feature
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Check if Intel Processor Trace feature is supported by current processor. Define vmtrace_supported global variable.
--- xen/arch/x86/hvm/vmx/vmcs.c | 15 ++++++++++++++- xen/common/domain.c | 2 ++ xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/hvm/vmx/vmcs.h | 1 + xen/include/public/arch-x86/cpufeatureset.h | 1 + xen/include/xen/domain.h | 2 ++ 6 files changed, 21 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/vmx/vmcs.c +14 -1
@@ -291,6 +291,20 @@ static int vmx_init_vmcs_config(void) _vmx_cpu_based_exec_control &= ~(CPU_BASED_CR8_LOAD_EXITING |CPU_BASED_CR8_STORE_EXITING ); +rdmsrl (MSR_IA32_VMX_MISC , _vmx_misc_cap); + +/* Check whether IPT is supported in VMX operation. */ +if ( !smp_processor_id () ) + vmtrace_supported = cpu_has_ipt && + (_vmx_misc_cap &VMX_MISC_PROC_TRACE ); +else if ( vmtrace_supported && + !(_vmx_misc_cap &VMX_MISC_PROC_TRACE ) ) + { +printk ("VMX: IPT capabilities fatally differ between CPU%u and CPU0\n" , +smp_processor_id ()); +return -EINVAL ; + } +if ( _vmx_cpu_based_exec_control &CPU_BASED_ACTIVATE_SECONDARY_CONTROLS ) { min =0 ;
@@ -305,7 +319,6 @@ static int vmx_init_vmcs_config(void)SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |SECONDARY_EXEC_XSAVES | SECONDARY_EXEC_TSC_SCALING); -rdmsrl (MSR_IA32_VMX_MISC , _vmx_misc_cap);if ( _vmx_misc_cap &VMX_MISC_VMWRITE_ALL ) opt |=SECONDARY_EXEC_ENABLE_VMCS_SHADOWING ;if ( opt_vpid_enabled )
xen/common/domain.c +2 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;vcpu_info_t dummy_vcpu_info; +bool vmtrace_supported__read_mostly ; +static void __domain_finalise_shutdown (struct domain *d) {struct vcpu *v;
xen/include/asm-x86/cpufeature.h +1 -0
@@ -104,6 +104,7 @@#define cpu_has_clwb boot_cpu_has(X86_FEATURE_CLWB)#define cpu_has_avx512er boot_cpu_has(X86_FEATURE_AVX512ER)#define cpu_has_avx512cd boot_cpu_has(X86_FEATURE_AVX512CD) +#define cpu_has_ipt boot_cpu_has(X86_FEATURE_PROC_TRACE)#define cpu_has_sha boot_cpu_has(X86_FEATURE_SHA)#define cpu_has_avx512bw boot_cpu_has(X86_FEATURE_AVX512BW)#define cpu_has_avx512vl boot_cpu_has(X86_FEATURE_AVX512VL)
xen/include/asm-x86/hvm/vmx/vmcs.h +1 -0
@@ -283,6 +283,7 @@ extern u32 vmx_secondary_exec_control;#define VMX_VPID_INVVPID_SINGLE_CONTEXT_RETAINING_GLOBAL 0x80000000000ULLextern u64 vmx_ept_vpid_cap; +#define VMX_MISC_PROC_TRACE 0x00004000#define VMX_MISC_CR3_TARGET 0x01ff0000#define VMX_MISC_VMWRITE_ALL 0x20000000
xen/include/public/arch-x86/cpufeatureset.h +1 -0
@@ -217,6 +217,7 @@ XEN_CPUFEATURE(SMAP, 5*32+20) /*S Supervisor Mode Access Prevention */ XEN_CPUFEATURE(AVX512_IFMA, 5*32+21) /*A AVX-512 Integer Fused Multiply Add */ XEN_CPUFEATURE(CLFLUSHOPT, 5*32+23) /*A CLFLUSHOPT instruction */ XEN_CPUFEATURE(CLWB, 5*32+24) /*A CLWB instruction */ +XEN_CPUFEATURE(PROC_TRACE, 5*32+25) /* Processor Tracing feature */ XEN_CPUFEATURE(AVX512PF, 5*32+26) /*A AVX-512 Prefetch Instructions */ XEN_CPUFEATURE(AVX512ER, 5*32+27) /*A AVX-512 Exponent & Reciprocal Instrs */XEN_CPUFEATURE (AVX512CD ,5 *32 +28 )/*A AVX-512 Conflict Detection Instrs */
xen/include/xen/domain.h +2 -0
@@ -130,4 +130,6 @@ struct vnuma_info {void vnuma_destroy (struct vnuma_info *vnuma); +extern bool vmtrace_supported; +#endif /* __XEN_DOMAIN_H__ */
Signature · 1 line
2.17.1
PATCH v5 4/11 common: add vmtrace_pt_size domain parameter
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Add vmtrace_pt_size domain parameter in live domain and vmtrace_pt_order parameter in xen_domctl_createdomain.
--- xen/common/domain.c | 12 ++++++++++++ xen/include/public/domctl.h | 1 + xen/include/xen/sched.h | 4 ++++ 3 files changed, 17 insertions(+)
xen/common/domain.c +12 -0
@@ -338,6 +338,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( config->vmtrace_pt_order && !vmtrace_supported ) + { +dprintk (XENLOG_INFO ,"Processor tracing is not supported\n" ); +return -EINVAL ; + } +return arch_sanitise_domain_config (config); }
@@ -443,6 +449,12 @@ struct domain *domain_create(domid_t domid, d->nr_pirqs =min (d->nr_pirqs, nr_irqs);radix_tree_init (&d->pirq_tree); + +if ( config->vmtrace_pt_order ) + { +uint32_t shift_val = config->vmtrace_pt_order +PAGE_SHIFT ; + d->vmtrace_pt_size = (1ULL << shift_val); + } }if ( (err =arch_domain_create (d, config)) !=0 )
xen/include/public/domctl.h +1 -0
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {uint32_t max_evtchn_port;int32_t max_grant_frames;int32_t max_maptrack_frames; +uint8_t vmtrace_pt_order;struct xen_arch_domainconfig arch; };
xen/include/xen/sched.h +4 -0
@@ -457,6 +457,10 @@ struct domainunsigned pbuf_idx;spinlock_t pbuf_lock; +/* Used by vmtrace features */ +spinlock_t vmtrace_lock; +uint64_t vmtrace_pt_size; +/* OProfile support. */ struct xenoprof *xenoprof;
Signature · 1 line
2.17.1
Re: [PATCH v5 04/11] common: add vmtrace_pt_size domain parameter
----- 5 lip 2020 o 20:54, Michał Leszczyński [email protected] napisał(a):
Quoted 71 lines
From: Michal Leszczynski <[email protected]>
Add
vmtrace_pt_sizedomain parameter in live domain andvmtrace_pt_orderparameter inxen_domctl_createdomain.Signed-off-by Michal Leszczynski <[email protected]>---
xen/common/domain.c | 12 ++++++++++++ xen/include/public/domctl.h | 1 + xen/include/xen/sched.h | 4 ++++ 3 files changed, 17 insertions(+)xen/common/domain.c +12
-0@@ -338,6 +338,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( config->vmtrace_pt_order && !vmtrace_supported ) + { +dprintk (XENLOG_INFO ,"Processor tracing is not supported\n" ); +return -EINVAL ; + } +return arch_sanitise_domain_config (config); }@@ -443,6 +449,12 @@ struct domain *domain_create(domid_t domid, d->nr_pirqs =min (d->nr_pirqs, nr_irqs);radix_tree_init (&d->pirq_tree); + +if ( config->vmtrace_pt_order ) + { +uint32_t shift_val = config->vmtrace_pt_order +PAGE_SHIFT ; + d->vmtrace_pt_size = (1ULL << shift_val); + } }if ( (err =arch_domain_create (d, config)) !=0 )xen/include/public/domctl.h +1
-0@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {uint32_t max_evtchn_port;int32_t max_grant_frames;int32_t max_maptrack_frames; +uint8_t vmtrace_pt_order;struct xen_arch_domainconfig arch; };xen/include/xen/sched.h +4
-1@@ -457,6 +457,10 @@ struct domainunsigned pbuf_idx;spinlock_t pbuf_lock; +/* Used by vmtrace features */ +spinlock_t vmtrace_lock; +uint64_t vmtrace_pt_size; +/* OProfile support. */ struct xenoprof *xenoprof; --
Quoted 1 line
2.17.1
Just a note to myself: in v4 it was suggested by Jan that we should go with "number of kB" instead of "number of bytes" and the type could be uint32_t.
I will modify it in such way within the next version.
Best regards,
Michał Leszczyński
CERT Polska
Re: [PATCH v5 04/11] common: add vmtrace_pt_size domain parameter
In reply to Michał LeszczyńskiHi,
Quoted 17 lines · Michał Leszczyński
From: Michal Leszczynski <[email protected]>
Add
vmtrace_pt_sizedomain parameter in live domain andvmtrace_pt_orderparameter inxen_domctl_createdomain.Signed-off-by Michal Leszczynski <[email protected]>--- xen/common/domain.c | 12 ++++++++++++ xen/include/public/domctl.h | 1 + xen/include/xen/sched.h | 4 ++++ 3 files changed, 17 insertions(+)xen/common/domain.c +0
-0
xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 341
Quoted 4 lines
xen/common/domain.c +1
-0@@ -338,6 +338,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( config->vmtrace_pt_order && !vmtrace_supported )
Looking at the rest of the series, vmtrace will only be supported for x86 HVM guest. So don't you want to return -EINVAL for PV guests here?
This could be done in a new helper arch_vmtrace_supported() or possibly in the existing arch_sanitise_domain_config().
Cheers,
Signature · 1 line
Julien Grall
PATCH v5 5/11 tools/libxl: add vmtrace_pt_size parameter
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).
--- docs/man/xl.cfg.5.pod.in | 11 +++++++++++ tools/golang/xenlight/helpers.gen.go | 2 ++ tools/golang/xenlight/types.gen.go | 1 + tools/libxl/libxl.h | 8 ++++++++ tools/libxl/libxl_create.c | 1 + tools/libxl/libxl_types.idl | 2 ++ tools/xl/xl_parse.c | 22 ++++++++++++++++++++++ 7 files changed, 47 insertions(+)
docs/man/xl.cfg.5.pod.in +11 -0
@@ -278,6 +278,17 @@ memory=8096 will report significantly less memory available for use than a system with maxmem=8096 memory=8096 due to the memory overhead of having to track the unused pages. +=item B<processor_trace_buffer_size=BYTES> + +Specifies the size of processor trace buffer that would be allocated +for each vCPU belonging to this domain. Disabled (i.e. +B<processor_trace_buffer_size=0> by default. This must be set to +non-zero value in order to be able to use processor tracing features +with this domain. + +B<NOTE>: The size value must be between 4 kB and 4 GB and it must +be also a power of 2. + =back =head3 Guest Virtual NUMA Configuration
tools/golang/xenlight/helpers.gen.go +2 -0
@@ -1117,6 +1117,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)} x.ArchArm.GicVersion = GicVersion(xc.arch_arm.gic_version) x.ArchArm.Vuart = VuartType(xc.arch_arm.vuart) x.Altp2M = Altp2MMode(xc.altp2m) +x.VmtracePtOrder = int(xc.vmtrace_pt_order) return nil}
@@ -1592,6 +1593,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)} xc.arch_arm.gic_version = C.libxl_gic_version(x.ArchArm.GicVersion) xc.arch_arm.vuart = C.libxl_vuart_type(x.ArchArm.Vuart) xc.altp2m = C.libxl_altp2m_mode(x.Altp2M) +xc.vmtrace_pt_order = C.int(x.VmtracePtOrder) return nil }
tools/golang/xenlight/types.gen.go +1 -0
@@ -516,6 +516,7 @@ GicVersion GicVersion Vuart VuartType } Altp2M Altp2MMode +VmtracePtOrder int } type domainBuildInfoTypeUnion interface {
tools/libxl/libxl.h +8 -0
@@ -438,6 +438,14 @@*/ #define LIBXL_HAVE_CREATEINFO_PASSTHROUGH 1 +/* +* LIBXL_HAVE_VMTRACE_PT_ORDER indicates that +* libxl_domain_create_info has a vmtrace_pt_order parameter, which +* allows to enable pre-allocation of processor tracing buffers +* with the given order of size. +*/ +#define LIBXL_HAVE_VMTRACE_PT_ORDER 1 +/* * libxl ABI compatibility *
tools/libxl/libxl_create.c +1 -0
@@ -608,6 +608,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, .max_evtchn_port = b_info->event_channels, .max_grant_frames = b_info->max_grant_frames, .max_maptrack_frames = b_info->max_maptrack_frames, + .vmtrace_pt_order = b_info->vmtrace_pt_order, };if (info->type !=LIBXL_DOMAIN_TYPE_PV ) {
tools/libxl/libxl_types.idl +2 -0
@@ -645,6 +645,8 @@ libxl_domain_build_info = Struct("domain_build_info",[ # supported by x86 HVM and ARM support is planned. ("altp2m", libxl_altp2m_mode), + ("vmtrace_pt_order", integer), + ], dir=DIR_IN, copy_deprecated_fn="libxl__domain_build_info_copy_deprecated", )
tools/xl/xl_parse.c +22 -0
@@ -1861,6 +1861,28 @@ void parse_config_data(const char *config_source, } } +if (!xlu_cfg_get_long (config,"processor_trace_buffer_size" , &l,1 ) && l) { +int32_t shift =0 ; + +if (l & (l -1 )) + { +fprintf (stderr,"ERROR: processor_trace_buffer_size " +"- must be a power of 2\n" ); +exit (1 ); + } + +while (l >>=1 ) ++shift; + +if (shift <=XEN_PAGE_SHIFT ) + { +fprintf (stderr,"ERROR: processor_trace_buffer_size " +"- value is too small\n" ); +exit (1 ); + } + + b_info->vmtrace_pt_order = shift -XEN_PAGE_SHIFT ; + } +if (!xlu_cfg_get_list (config,"ioports" , &ioports, &num_ioports,0 )) { b_info->num_ioports = num_ioports; b_info->ioports =calloc (num_ioports,sizeof (*b_info->ioports));
Signature · 1 line
2.17.1
Re: [PATCH v5 05/11] tools/libxl: add vmtrace_pt_size parameter
----- 5 lip 2020 o 20:54, Michał Leszczyński [email protected] napisał(a):
Quoted 152 lines
From: Michal Leszczynski <[email protected]>
Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).
Signed-off-by Michal Leszczynski <[email protected]>---
docs/man/xl.cfg.5.pod.in | 11 +++++++++++ tools/golang/xenlight/helpers.gen.go | 2 ++ tools/golang/xenlight/types.gen.go | 1 + tools/libxl/libxl.h | 8 ++++++++ tools/libxl/libxl_create.c | 1 + tools/libxl/libxl_types.idl | 2 ++ tools/xl/xl_parse.c | 22 ++++++++++++++++++++++ 7 files changed, 47 insertions(+)docs/man/xl.cfg.5.pod.in +0
-0@@ -278,6 +278,17 @@ memory=8096 will report significantly less memory availablefor use
than a system with maxmem=8096 memory=8096 due to the memory overhead of having to track the unused pages.+=item B<processor_trace_buffer_size=BYTES>
docs/man/xl.cfg.5.pod.in +10
-0+ +Specifies the size of processor trace buffer that would be allocated +for each vCPU belonging to this domain. Disabled (i.e. +B<processor_trace_buffer_size=0> by default. This must be set to +non-zero value in order to be able to use processor tracing features +with this domain. + +B<NOTE>: The size value must be between 4 kB and 4 GB and it must +be also a power of 2. +=back
=head3 Guest Virtual NUMA Configuration
tools/golang/xenlight/helpers.gen.go +2
-0@@ -1117,6 +1117,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)} x.ArchArm.GicVersion = GicVersion(xc.arch_arm.gic_version) x.ArchArm.Vuart = VuartType(xc.arch_arm.vuart) x.Altp2M = Altp2MMode(xc.altp2m) +x.VmtracePtOrder = int(xc.vmtrace_pt_order) return nil}@@ -1592,6 +1593,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)} xc.arch_arm.gic_version = C.libxl_gic_version(x.ArchArm.GicVersion) xc.arch_arm.vuart = C.libxl_vuart_type(x.ArchArm.Vuart) xc.altp2m = C.libxl_altp2m_mode(x.Altp2M) +xc.vmtrace_pt_order = C.int(x.VmtracePtOrder) return nil }tools/golang/xenlight/types.gen.go +0
-0@@ -516,6 +516,7 @@ GicVersion GicVersionVuart VuartType
}
Altp2M Altp2MMode
+VmtracePtOrder int
}type domainBuildInfoTypeUnion interface {
tools/libxl/libxl.h +8
-0@@ -438,6 +438,14 @@*/ #define LIBXL_HAVE_CREATEINFO_PASSTHROUGH 1 +/* +* LIBXL_HAVE_VMTRACE_PT_ORDER indicates that +* libxl_domain_create_info has a vmtrace_pt_order parameter, which +* allows to enable pre-allocation of processor tracing buffers +* with the given order of size. +*/ +#define LIBXL_HAVE_VMTRACE_PT_ORDER 1 +/* * libxl ABI compatibility * tools/libxl/libxl_create.c +1
-0@@ -608,6 +608,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, .max_evtchn_port = b_info->event_channels, .max_grant_frames = b_info->max_grant_frames, .max_maptrack_frames = b_info->max_maptrack_frames, + .vmtrace_pt_order = b_info->vmtrace_pt_order, };if (info->type != LIBXL_DOMAIN_TYPE_PV) {tools/libxl/libxl_types.idl +2
-0@@ -645,6 +645,8 @@ libxl_domain_build_info = Struct("domain_build_info",[ # supported by x86 HVM and ARM support is planned. ("altp2m", libxl_altp2m_mode), + ("vmtrace_pt_order", integer), + ], dir=DIR_IN, copy_deprecated_fn="libxl__domain_build_info_copy_deprecated", )tools/xl/xl_parse.c +22
-1@@ -1861,6 +1861,28 @@ void parse_config_data(const char *config_source, } } +if (!xlu_cfg_get_long (config,"processor_trace_buffer_size" , &l,1 ) && l) { +int32_t shift =0 ; + +if (l & (l -1 )) + { +fprintf (stderr,"ERROR: processor_trace_buffer_size " +"- must be a power of 2\n" ); +exit (1 ); + } + +while (l >>=1 ) ++shift; + +if (shift <=XEN_PAGE_SHIFT ) + { +fprintf (stderr,"ERROR: processor_trace_buffer_size " +"- value is too small\n" ); +exit (1 ); + } + + b_info->vmtrace_pt_order = shift -XEN_PAGE_SHIFT ; + } +if (!xlu_cfg_get_list (config,"ioports" , &ioports, &num_ioports,0 )) { b_info->num_ioports = num_ioports; b_info->ioports =calloc (num_ioports,sizeof (*b_info->ioports)); --
Quoted 1 line
2.17.1
As there were many different ideas about how the naming scheme should be and what kinds of values should be passed where, I would like to discuss this particular topic. Right now we have it pretty confusing:
* user sets "processor_trace_buffer_size" option in xl.cfg
* domain creation hypercall uses "vmtrace_pt_order" (derived from above)
* hypervisor side stores "vmtrace_pt_size" (converted back to bytes)
Best regards,
Michał Leszczyński
CERT Polska
Re: [PATCH v5 05/11] tools/libxl: add vmtrace_pt_size parameter
Hi,
Quoted 27 lines · Michał Leszczyński
----- 5 lip 2020 o 20:54, Michał Leszczyński [email protected] napisał(a):
Quoted 25 lines
From: Michal Leszczynski <[email protected]>
Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).
Signed-off-by Michal Leszczynski <[email protected]>---
docs/man/xl.cfg.5.pod.in | 11 +++++++++++ tools/golang/xenlight/helpers.gen.go | 2 ++ tools/golang/xenlight/types.gen.go | 1 + tools/libxl/libxl.h | 8 ++++++++ tools/libxl/libxl_create.c | 1 + tools/libxl/libxl_types.idl | 2 ++ tools/xl/xl_parse.c | 22 ++++++++++++++++++++++ 7 files changed, 47 insertions(+)docs/man/xl.cfg.5.pod.in +0
-0@@ -278,6 +278,17 @@ memory=8096 will report significantly less memory availablefor use
than a system with maxmem=8096 memory=8096 due to the memory overhead of having to track the unused pages.
Quoted 10 lines
docs/man/xl.cfg.5.pod.in memory=8096 will report significantly less memory available for use · line 290
Quoted 10 lines
+=item B<processor_trace_buffer_size=BYTES>
docs/man/xl.cfg.5.pod.in +9
-0+ +Specifies the size of processor trace buffer that would be allocated +for each vCPU belonging to this domain. Disabled (i.e. +B<processor_trace_buffer_size=0> by default. This must be set to +non-zero value in order to be able to use processor tracing features +with this domain. + +B<NOTE>: The size value must be between 4 kB and 4 GB and it must +be also a power of 2.
This seems to suggest that 4 kB is allowed. But looking at the code below, you are forbidding the value.
[...]
Quoted 6 lines
As there were many different ideas about how the naming scheme should be and what kinds of values should be passed where, I would like to discuss this particular topic. Right now we have it pretty confusing:
* user sets "
processor_trace_buffer_size" option in xl.cfg
* domain creation hypercall uses "vmtrace_pt_order" (derived from above)
You don't only use the order in the hypercall but also the public interface of libxl.
Quoted 1 line
* hypervisor side stores "
vmtrace_pt_size" (converted back to bytes)
My preference would be to use the size everywhere, but if one still prefer to use the order in the hypercall then the libxl interface should use the size.
See my comment in v4 for the rationale.
Cheers,
Signature · 1 line
Julien Grall
PATCH v5 6/11 x86/hvm: processor trace interface in HVM
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Implement necessary changes in common code/HVM to support processor trace features. Define vmtrace_pt_* API and implement trace buffer allocation/deallocation in common code.
--- xen/arch/x86/domain.c | 19 +++++++++++++++++++ xen/common/domain.c | 19 +++++++++++++++++++ xen/include/asm-x86/hvm/hvm.h | 20 ++++++++++++++++++++ xen/include/xen/sched.h | 4 ++++ 4 files changed, 62 insertions(+)
xen/arch/x86/domain.c +19 -0
@@ -2199,6 +2199,25 @@ int domain_relinquish_resources(struct domain *d)altp2m_vcpu_disable_ve (v); } + for_each_vcpu ( d, v ) + { +unsigned int i; + +if ( !v->vmtrace.pt_buf ) +continue ; + +for ( i =0 ; i < (v->domain->vmtrace_pt_size >>PAGE_SHIFT ); i++ ) + { +struct page_info *pg =mfn_to_page ( +mfn_add (page_to_mfn (v->vmtrace.pt_buf), i)); +if ( (pg->count_info & PGC_count_mask) !=1 ) +return -EBUSY ; + } + +free_domheap_pages (v->vmtrace.pt_buf, +get_order_from_bytes (v->domain->vmtrace_pt_size)); + } +if (is_pv_domain (d) ) {for_each_vcpu ( d, v )
xen/common/domain.c +19 -0
@@ -137,6 +137,21 @@ static void vcpu_destroy(struct vcpu *v)free_vcpu_struct (v); } +static int vmtrace_alloc_buffers (struct vcpu *v) +{ +struct page_info *pg; +uint64_t size = v->domain->vmtrace_pt_size; + + pg =alloc_domheap_pages (v->domain,get_order_from_bytes (size), + MEMF_no_refcount); + +if ( !pg ) +return -ENOMEM ; + + v->vmtrace.pt_buf = pg; +return 0 ; +} +struct vcpu *vcpu_create (struct domain *d,unsigned int vcpu_id) {struct vcpu *v;
@@ -162,6 +177,9 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id) v->vcpu_id = vcpu_id; v->dirty_cpu =VCPU_CPU_CLEAN ; +if ( d->vmtrace_pt_size &&vmtrace_alloc_buffers (v) !=0 ) +return NULL ; +spin_lock_init (&v->virq_lock);tasklet_init (&v->continue_hypercall_tasklet,NULL ,NULL );
@@ -422,6 +440,7 @@ struct domain *domain_create(domid_t domid, d->shutdown_code =SHUTDOWN_CODE_INVALID ;spin_lock_init (&d->pbuf_lock); +spin_lock_init (&d->vmtrace_lock);rwlock_init (&d->vnuma_rwlock);
xen/include/asm-x86/hvm/hvm.h +20 -0
@@ -214,6 +214,10 @@ struct hvm_function_table {bool_t (*altp2m_vcpu_emulate_ve)(struct vcpu *v);int (*altp2m_vcpu_emulate_vmfunc)(const struct cpu_user_regs *regs); +/* vmtrace */ +int (*vmtrace_control_pt)(struct vcpu *v,bool enable); +int (*vmtrace_get_pt_offset)(struct vcpu *v,uint64_t *offset); +/* * Parameters and callbacks for hardware-assisted TSC scaling, * which are valid only when the hardware feature is available.
@@ -655,6 +659,22 @@ static inline bool altp2m_vcpu_emulate_ve(struct vcpu *v)return false; } +static inline int vmtrace_control_pt (struct vcpu *v,bool enable) +{ +if ( hvm_funcs.vmtrace_control_pt ) +return hvm_funcs.vmtrace_control_pt (v, enable); + +return -EOPNOTSUPP ; +} + +static inline int vmtrace_get_pt_offset (struct vcpu *v,uint64_t *offset) +{ +if ( hvm_funcs.vmtrace_get_pt_offset ) +return hvm_funcs.vmtrace_get_pt_offset (v, offset); + +return -EOPNOTSUPP ; +} +/* * This must be defined as a macro instead of an inline function, * because it uses 'struct vcpu' and 'struct domain' which have
xen/include/xen/sched.h +4 -0
@@ -253,6 +253,10 @@ struct vcpu/* vPCI per-vCPU area, used to store data for long running operations. */ struct vpci_vcpu vpci; +struct { +struct page_info *pt_buf; + } vmtrace; +struct arch_vcpu arch; };
Signature · 1 line
2.17.1
Re: [PATCH v5 06/11] x86/hvm: processor trace interface in HVM
----- 5 lip 2020 o 20:54, Michał Leszczyński [email protected] napisał(a):
Quoted 30 lines
From: Michal Leszczynski <[email protected]>
Implement necessary changes in common code/HVM to support processor trace features. Define vmtrace_pt_* API and implement trace buffer allocation/deallocation in common code.
Signed-off-by Michal Leszczynski <[email protected]>---
xen/arch/x86/domain.c | 19 +++++++++++++++++++ xen/common/domain.c | 19 +++++++++++++++++++ xen/include/asm-x86/hvm/hvm.h | 20 ++++++++++++++++++++ xen/include/xen/sched.h | 4 ++++ 4 files changed, 62 insertions(+)xen/arch/x86/domain.c +7
-0@@ -2199,6 +2199,25 @@ int domain_relinquish_resources(struct domain *d)altp2m_vcpu_disable_ve (v); } + for_each_vcpu ( d, v ) + { +unsigned int i; + +if ( !v->vmtrace.pt_buf ) +continue ; +
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2218
Quoted 10 lines
xen/arch/x86/domain.c +10
-0+for ( i =0 ; i < (v->domain->vmtrace_pt_size >>PAGE_SHIFT ); i++ ) + { +struct page_info *pg =mfn_to_page ( +mfn_add (page_to_mfn (v->vmtrace.pt_buf), i)); +if ( (pg->count_info & PGC_count_mask) !=1 ) +return -EBUSY ; + } + +free_domheap_pages (v->vmtrace.pt_buf, +get_order_from_bytes (v->domain->vmtrace_pt_size));
While this works, I don't feel that this is a good solution with this loop returning -EBUSY here. I would like to kindly ask for suggestions regarding this topic.
Best regards,
Michał Leszczyński
CERT Polska
Re: [PATCH v5 06/11] x86/hvm: processor trace interface in HVM
Quoted 21 lines · Michał Leszczyński
----- 5 lip 2020 o 20:54, Michał Leszczyński [email protected] napisał(a):
Quoted 20 lines
xen/arch/x86/domain.c +14
-0@@ -2199,6 +2199,25 @@ int domain_relinquish_resources(struct domain *d)altp2m_vcpu_disable_ve (v); } + for_each_vcpu ( d, v ) + { +unsigned int i; + +if ( !v->vmtrace.pt_buf ) +continue ; + +for ( i =0 ; i < (v->domain->vmtrace_pt_size >>PAGE_SHIFT ); i++ ) + { +struct page_info *pg =mfn_to_page ( +mfn_add (page_to_mfn (v->vmtrace.pt_buf), i)); +if ( (pg->count_info & PGC_count_mask) !=1 ) +return -EBUSY ; + }
Quoted 8 lines
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2218
Quoted 3 lines
xen/arch/x86/domain.c +3
-0+ +free_domheap_pages (v->vmtrace.pt_buf, +get_order_from_bytes (v->domain->vmtrace_pt_size));While this works, I don't feel that this is a good solution with this loop returning
-EBUSYhere. I would like to kindly ask for suggestions regarding this topic.
I'm sorry to ask, but with the previously give suggestions to mirror existing code, why do you still need to play with this function? You really shouldn't have a need to, just like e.g. the ioreq server page handling code didn't.
Jan
Re: [PATCH v5 06/11] x86/hvm: processor trace interface in HVM
----- 6 lip 2020 o 10:31, Jan Beulich [email protected] napisał(a):
Quoted 32 lines · Michał Leszczyński
Quoted 22 lines
----- 5 lip 2020 o 20:54, Michał Leszczyński [email protected] napisał(a):
Quoted 20 lines
xen/arch/x86/domain.c +14
-0@@ -2199,6 +2199,25 @@ int domain_relinquish_resources(struct domain *d)altp2m_vcpu_disable_ve (v); } + for_each_vcpu ( d, v ) + { +unsigned int i; + +if ( !v->vmtrace.pt_buf ) +continue ; + +for ( i =0 ; i < (v->domain->vmtrace_pt_size >>PAGE_SHIFT ); i++ ) + { +struct page_info *pg =mfn_to_page ( +mfn_add (page_to_mfn (v->vmtrace.pt_buf), i)); +if ( (pg->count_info & PGC_count_mask) !=1 ) +return -EBUSY ; + }Quoted 8 lines
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2218
Quoted 3 lines
xen/arch/x86/domain.c +3
-0+ +free_domheap_pages (v->vmtrace.pt_buf, +get_order_from_bytes (v->domain->vmtrace_pt_size));While this works, I don't feel that this is a good solution with this loop returning
-EBUSYhere. I would like to kindly ask for suggestions regarding this topic.
Quoted 6 lines
I'm sorry to ask, but with the previously give suggestions to mirror existing code, why do you still need to play with this function? You really shouldn't have a need to, just like e.g. the ioreq server page handling code didn't.
Jan
Ok, sorry. I think I've finally got it after latest Roger's suggestions :P
This will be fixed in the next version.
Best regards,
Michał Leszczyński
CERT Polska
Re: [PATCH v5 06/11] x86/hvm: processor trace interface in HVM
In reply to Michał LeszczyńskiQuoted 30 lines · Michał Leszczyński
From: Michal Leszczynski <[email protected]>
Implement necessary changes in common code/HVM to support processor trace features. Define vmtrace_pt_* API and implement trace buffer allocation/deallocation in common code.
Signed-off-by Michal Leszczynski <[email protected]>--- xen/arch/x86/domain.c | 19 +++++++++++++++++++ xen/common/domain.c | 19 +++++++++++++++++++ xen/include/asm-x86/hvm/hvm.h | 20 ++++++++++++++++++++ xen/include/xen/sched.h | 4 ++++ 4 files changed, 62 insertions(+)xen/arch/x86/domain.c +7
-0@@ -2199,6 +2199,25 @@ int domain_relinquish_resources(struct domain *d)altp2m_vcpu_disable_ve (v); } + for_each_vcpu ( d, v ) + { +unsigned int i; + +if ( !v->vmtrace.pt_buf ) +continue ; +
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2218
Quoted 10 lines
xen/arch/x86/domain.c +10
-0+for ( i =0 ; i < (v->domain->vmtrace_pt_size >>PAGE_SHIFT ); i++ ) + { +struct page_info *pg =mfn_to_page ( +mfn_add (page_to_mfn (v->vmtrace.pt_buf), i)); +if ( (pg->count_info & PGC_count_mask) !=1 ) +return -EBUSY ; + } + +free_domheap_pages (v->vmtrace.pt_buf, +get_order_from_bytes (v->domain->vmtrace_pt_size));
This is racy as a control domain could take a reference between the check and the freeing.
Quoted 17 lines
xen/arch/x86/domain.c +2
-0+ } +if (is_pv_domain (d) ) {for_each_vcpu ( d, v )xen/common/domain.c +4
-0@@ -137,6 +137,21 @@ static void vcpu_destroy(struct vcpu *v)free_vcpu_struct (v); } +static int vmtrace_alloc_buffers (struct vcpu *v) +{ +struct page_info *pg; +uint64_t size = v->domain->vmtrace_pt_size;
xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 153
Quoted 10 lines
xen/common/domain.c +10
-0+ + pg =alloc_domheap_pages (v->domain,get_order_from_bytes (size), + MEMF_no_refcount); + +if ( !pg ) +return -ENOMEM ; + + v->vmtrace.pt_buf = pg; +return 0 ; +}
I think we already agreed that you would use the same model as ioreq servers, where a reference is taken on allocation and then the pages are not explicitly freed on domain destruction and put_page_and_type is used. Is there some reason why that model doesn't work in this case?
If not, please see hvm_alloc_ioreq_mfn and hvm_free_ioreq_mfn.
Roger.
Re: [PATCH v5 06/11] x86/hvm: processor trace interface in HVM
----- 6 lip 2020 o 10:42, Roger Pau Monné [email protected] napisał(a):
Quoted 79 lines · Michał Leszczyński
Quoted 30 lines
From: Michal Leszczynski <[email protected]>
Implement necessary changes in common code/HVM to support processor trace features. Define vmtrace_pt_* API and implement trace buffer allocation/deallocation in common code.
Signed-off-by Michal Leszczynski <[email protected]>--- xen/arch/x86/domain.c | 19 +++++++++++++++++++ xen/common/domain.c | 19 +++++++++++++++++++ xen/include/asm-x86/hvm/hvm.h | 20 ++++++++++++++++++++ xen/include/xen/sched.h | 4 ++++ 4 files changed, 62 insertions(+)xen/arch/x86/domain.c +7
-0@@ -2199,6 +2199,25 @@ int domain_relinquish_resources(struct domain *d)altp2m_vcpu_disable_ve (v); } + for_each_vcpu ( d, v ) + { +unsigned int i; + +if ( !v->vmtrace.pt_buf ) +continue ; +xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2218
Quoted 10 lines
xen/arch/x86/domain.c +10
-0+for ( i =0 ; i < (v->domain->vmtrace_pt_size >>PAGE_SHIFT ); i++ ) + { +struct page_info *pg =mfn_to_page ( +mfn_add (page_to_mfn (v->vmtrace.pt_buf), i)); +if ( (pg->count_info & PGC_count_mask) !=1 ) +return -EBUSY ; + } + +free_domheap_pages (v->vmtrace.pt_buf, +get_order_from_bytes (v->domain->vmtrace_pt_size));This is racy as a control domain could take a reference between the check and the freeing.
Quoted 17 lines
xen/arch/x86/domain.c +2
-0+ } +if (is_pv_domain (d) ) {for_each_vcpu ( d, v )xen/common/domain.c +4
-0@@ -137,6 +137,21 @@ static void vcpu_destroy(struct vcpu *v)free_vcpu_struct (v); } +static int vmtrace_alloc_buffers (struct vcpu *v) +{ +struct page_info *pg; +uint64_t size = v->domain->vmtrace_pt_size;xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 153
Quoted 10 lines
xen/common/domain.c +10
-0+ + pg =alloc_domheap_pages (v->domain,get_order_from_bytes (size), + MEMF_no_refcount); + +if ( !pg ) +return -ENOMEM ; + + v->vmtrace.pt_buf = pg; +return 0 ; +}I think we already agreed that you would use the same model as ioreq servers, where a reference is taken on allocation and then the pages are not explicitly freed on domain destruction and
put_page_and_typeis used. Is there some reason why that model doesn't work in this case?
Quoted 3 lines
If not, please see
hvm_alloc_ioreq_mfnandhvm_free_ioreq_mfn.Roger.
Ok, I've got it, will do. Thanks for pointing out the examples.
One thing that is confusing to me is that I don't get what is the meaning of MEMF_no_refcount flag.
In the hvm_{alloc,free}_ioreq_mfn the memory is allocated explicitly but freed just by putting out the reference, so
I guess it's automatically detected that the refcount dropped to 0 and the page should be freed? If so, why the flag is named "no refcount"?
Best regards,
Michał Leszczyński
CERT Polska
Re: [PATCH v5 06/11] x86/hvm: processor trace interface in HVM
Quoted 83 lines · Michał Leszczyński
----- 6 lip 2020 o 10:42, Roger Pau Monné [email protected] napisał(a):
Quoted 81 lines · Michał Leszczyński
Quoted 30 lines
From: Michal Leszczynski <[email protected]>
Implement necessary changes in common code/HVM to support processor trace features. Define vmtrace_pt_* API and implement trace buffer allocation/deallocation in common code.
Signed-off-by Michal Leszczynski <[email protected]>--- xen/arch/x86/domain.c | 19 +++++++++++++++++++ xen/common/domain.c | 19 +++++++++++++++++++ xen/include/asm-x86/hvm/hvm.h | 20 ++++++++++++++++++++ xen/include/xen/sched.h | 4 ++++ 4 files changed, 62 insertions(+)xen/arch/x86/domain.c +7
-0@@ -2199,6 +2199,25 @@ int domain_relinquish_resources(struct domain *d)altp2m_vcpu_disable_ve (v); } + for_each_vcpu ( d, v ) + { +unsigned int i; + +if ( !v->vmtrace.pt_buf ) +continue ; +xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2218
Quoted 10 lines
xen/arch/x86/domain.c +10
-0+for ( i =0 ; i < (v->domain->vmtrace_pt_size >>PAGE_SHIFT ); i++ ) + { +struct page_info *pg =mfn_to_page ( +mfn_add (page_to_mfn (v->vmtrace.pt_buf), i)); +if ( (pg->count_info & PGC_count_mask) !=1 ) +return -EBUSY ; + } + +free_domheap_pages (v->vmtrace.pt_buf, +get_order_from_bytes (v->domain->vmtrace_pt_size));This is racy as a control domain could take a reference between the check and the freeing.
Quoted 17 lines
xen/arch/x86/domain.c +2
-0+ } +if (is_pv_domain (d) ) {for_each_vcpu ( d, v )xen/common/domain.c +4
-0@@ -137,6 +137,21 @@ static void vcpu_destroy(struct vcpu *v)free_vcpu_struct (v); } +static int vmtrace_alloc_buffers (struct vcpu *v) +{ +struct page_info *pg; +uint64_t size = v->domain->vmtrace_pt_size;xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 153
Quoted 10 lines
xen/common/domain.c +10
-0+ + pg =alloc_domheap_pages (v->domain,get_order_from_bytes (size), + MEMF_no_refcount); + +if ( !pg ) +return -ENOMEM ; + + v->vmtrace.pt_buf = pg; +return 0 ; +}I think we already agreed that you would use the same model as ioreq servers, where a reference is taken on allocation and then the pages are not explicitly freed on domain destruction and
put_page_and_typeis used. Is there some reason why that model doesn't work in this case?If not, please see
hvm_alloc_ioreq_mfnandhvm_free_ioreq_mfn.
Quoted 8 lines
Quoted 1 line
Roger.
Ok, I've got it, will do. Thanks for pointing out the examples.
One thing that is confusing to me is that I don't get what is the meaning of
MEMF_no_refcountflag.
That flag prevents the memory from being counted towards the amount of memory assigned to the domain. You want it that way so that trace buffers are not accounted as part of the memory assigned to the domain.
You then need to get a (extra) reference to the pages (there's always the 'allocated' reference AFAICT) so that when the last reference is dropped (either by the domain being destroyed or the memory being unmapped from the control domain) it will be freed.
Quoted 4 lines
In the hvm_{alloc,free}_ioreq_mfn the memory is allocated explicitly but freed just by putting out the reference, so I guess it's automatically detected that the refcount dropped to 0 and the page should be freed?
Yes, put_page_alloc_ref will remove the allocated flag and then when the last reference is dropped the page will be freed.
Quoted 1 line
If so, why the flag is named "no refcount"?
I'm not sure about the naming, but you can get references to pages allocated as MEMF_no_refcount, and change their types. They are however not accounted towards the memory usage of a domain.
Roger.
PATCH v5 7/11 x86/vmx: implement IPT in VMX
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Use Intel Processor Trace feature to provide vmtrace_pt_* interface for HVM/VMX.
--- xen/arch/x86/hvm/vmx/vmx.c | 109 +++++++++++++++++++++++++++++ xen/include/asm-x86/hvm/vmx/vmcs.h | 3 + xen/include/asm-x86/hvm/vmx/vmx.h | 14 ++++ 3 files changed, 126 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +109 -0
@@ -428,6 +428,56 @@ static void vmx_domain_relinquish_resources(struct domain *d)vmx_free_vlapic_mapping (d); } +static int vmx_init_pt (struct vcpu *v) +{ +int rc; +uint64_t size = v->domain->vmtrace_pt_size; + + v->arch.hvm.vmx.ipt_state = xzalloc(struct ipt_state); + +if ( !v->arch.hvm.vmx.ipt_state ) +return -ENOMEM ; + +if ( !v->vmtrace.pt_buf || !size ) +return -EINVAL ; + +/* +* We don't accept trace buffer size smaller than single page +* and the upper bound is defined as 4GB in the specification. +* The buffer size must be also a power of 2. +*/ +if ( size <PAGE_SIZE || size >GB (4 ) || (size & (size -1 )) ) +return -EINVAL ; + + v->arch.hvm.vmx.ipt_state->output_base = +page_to_maddr (v->vmtrace.pt_buf); + v->arch.hvm.vmx.ipt_state->output_mask.raw = size -1 ; + + rc =vmx_add_host_load_msr (v,MSR_RTIT_CTL ,0 ); + +if ( rc ) +return rc; + + rc =vmx_add_guest_msr (v,MSR_RTIT_CTL , +RTIT_CTL_TRACE_EN |RTIT_CTL_OS | +RTIT_CTL_USR |RTIT_CTL_BRANCH_EN ); + +if ( rc ) +return rc; + +return 0 ; +} + +static int vmx_destroy_pt (struct vcpu * v) +{ +if ( v->arch.hvm.vmx.ipt_state ) +xfree (v->arch.hvm.vmx.ipt_state); + + v->arch.hvm.vmx.ipt_state =NULL ; +return 0 ; +} + +static int vmx_vcpu_initialise (struct vcpu *v) {int rc;
@@ -471,6 +521,14 @@ static int vmx_vcpu_initialise(struct vcpu *v)vmx_install_vlapic_mapping (v); +if ( v->domain->vmtrace_pt_size ) + { + rc =vmx_init_pt (v); + +if ( rc ) +return rc; + } +return 0 ; }
@@ -483,6 +541,7 @@ static void vmx_vcpu_destroy(struct vcpu *v)* prior to vmx_domain_destroy so we need to disable PML for each vcpu * separately here. */ +vmx_destroy_pt (v);vmx_vcpu_disable_pml (v);vmx_destroy_vmcs (v);passive_domain_destroy (v);
@@ -513,6 +572,18 @@ static void vmx_save_guest_msrs(struct vcpu *v)* be updated at any time via SWAPGS, which we cannot trap. */ v->arch.hvm.vmx.shadow_gs =rdgsshadow (); + +if (unlikely (v->arch.hvm.vmx.ipt_state && + v->arch.hvm.vmx.ipt_state->active) ) + { +uint64_t rtit_ctl; +rdmsrl (MSR_RTIT_CTL , rtit_ctl); +BUG_ON (rtit_ctl &RTIT_CTL_TRACE_EN ); + +rdmsrl (MSR_RTIT_STATUS , v->arch.hvm.vmx.ipt_state->status); +rdmsrl (MSR_RTIT_OUTPUT_MASK , + v->arch.hvm.vmx.ipt_state->output_mask.raw); + } }static void vmx_restore_guest_msrs (struct vcpu *v)
@@ -524,6 +595,17 @@ static void vmx_restore_guest_msrs(struct vcpu *v)if ( cpu_has_msr_tsc_aux )wrmsr_tsc_aux (v->arch.msrs->tsc_aux); + +if (unlikely (v->arch.hvm.vmx.ipt_state && + v->arch.hvm.vmx.ipt_state->active) ) + { +wrmsrl (MSR_RTIT_OUTPUT_BASE , + v->arch.hvm.vmx.ipt_state->output_base); +wrmsrl (MSR_RTIT_OUTPUT_MASK , + v->arch.hvm.vmx.ipt_state->output_mask.raw); +wrmsrl (MSR_RTIT_STATUS , + v->arch.hvm.vmx.ipt_state->status); + } }void vmx_update_cpu_exec_control (struct vcpu *v)
@@ -2240,6 +2322,24 @@ static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info)return true; } +static int vmx_control_pt (struct vcpu *v,bool enable) +{ +if ( !v->arch.hvm.vmx.ipt_state ) +return -EINVAL ; + + v->arch.hvm.vmx.ipt_state->active = enable; +return 0 ; +} + +static int vmx_get_pt_offset (struct vcpu *v,uint64_t *offset) +{ +if ( !v->arch.hvm.vmx.ipt_state ) +return -EINVAL ; + + *offset = v->arch.hvm.vmx.ipt_state->output_mask.offset; +return 0 ; +} +static struct hvm_function_table __initdata vmx_function_table = { .name ="VMX" , .cpu_up_prepare = vmx_cpu_up_prepare,
@@ -2295,6 +2395,8 @@ static struct hvm_function_table __initdata vmx_function_table = { .altp2m_vcpu_update_vmfunc_ve = vmx_vcpu_update_vmfunc_ve, .altp2m_vcpu_emulate_ve = vmx_vcpu_emulate_ve, .altp2m_vcpu_emulate_vmfunc = vmx_vcpu_emulate_vmfunc, + .vmtrace_control_pt = vmx_control_pt, + .vmtrace_get_pt_offset = vmx_get_pt_offset, .tsc_scaling = { .max_ratio = VMX_TSC_MULTIPLIER_MAX, },
@@ -3674,6 +3776,13 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)hvm_invalidate_regs_fields (regs); +if (unlikely (v->arch.hvm.vmx.ipt_state && + v->arch.hvm.vmx.ipt_state->active) ) + { +rdmsrl (MSR_RTIT_OUTPUT_MASK , + v->arch.hvm.vmx.ipt_state->output_mask.raw); + } +if (paging_mode_hap (v->domain) ) {/*
xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0
@@ -186,6 +186,9 @@ struct vmx_vcpu {* pCPU and wakeup the related vCPU. */ struct pi_blocking_vcpu pi_blocking; + +/* State of processor trace feature */ +struct ipt_state *ipt_state; };int vmx_create_vmcs (struct vcpu *v);
xen/include/asm-x86/hvm/vmx/vmx.h +14 -0
@@ -689,4 +689,18 @@ typedef union ldt_or_tr_instr_info { }; } ldt_or_tr_instr_info_t; +/* Processor Trace state per vCPU */ +struct ipt_state { +bool active; +uint64_t status; +uint64_t output_base; +union { +uint64_t raw; +struct { +uint32_t size; +uint32_t offset; + }; + } output_mask; +}; +#endif /* __ASM_X86_HVM_VMX_VMX_H__ */
Signature · 1 line
2.17.1
PATCH v5 8/11 x86/mm: add vmtrace_buf resource type
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Allow to map processor trace buffer usingacquire_resource().
--- xen/common/memory.c | 28 ++++++++++++++++++++++++++++ xen/include/public/memory.h | 1 + 2 files changed, 29 insertions(+)
xen/common/memory.c +28 -0
@@ -1007,6 +1007,29 @@ static long xatp_permission_check(struct domain *d, unsigned int space)return xsm_add_to_physmap (XSM_TARGET , current->domain, d); } +static int acquire_vmtrace_buf (struct domain *d,unsigned int id, +unsigned long frame, +unsigned int nr_frames, +xen_pfn_t mfn_list[]) +{ +mfn_t mfn; +unsigned int i; +struct vcpu *v =domain_vcpu (d, id); + +if ( !v || !v->vmtrace.pt_buf ) +return -EINVAL ; + + mfn =page_to_mfn (v->vmtrace.pt_buf); + +if ( frame + nr_frames > (v->domain->vmtrace_pt_size >>PAGE_SHIFT ) ) +return -EINVAL ; + +for ( i =0 ; i < nr_frames; i++ ) + mfn_list[i] =mfn_x (mfn_add (mfn, frame + i)); + +return 0 ; +} + static int acquire_grant_table(struct domain *d, unsigned int id, unsigned long frame, unsigned int nr_frames,
@@ -1117,6 +1140,11 @@ static int acquire_resource( mfn_list);break ; +case XENMEM_resource_vmtrace_buf: + rc =acquire_vmtrace_buf (d, xmar.id, xmar.frame, xmar.nr_frames, + mfn_list); +break ; +default : rc =arch_acquire_resource (d, xmar.type, xmar.id, xmar.frame, xmar.nr_frames, mfn_list);
xen/include/public/memory.h +1 -0
@@ -625,6 +625,7 @@ struct xen_mem_acquire_resource {#define XENMEM_resource_ioreq_server 0#define XENMEM_resource_grant_table 1 +#define XENMEM_resource_vmtrace_buf 2/* * IN - a type-specific resource identifier, which must be zero
Signature · 1 line
2.17.1
Re: [PATCH v5 08/11] x86/mm: add vmtrace_buf resource type
Hi,
Quoted 16 lines · Michał Leszczyński
From: Michal Leszczynski <[email protected]>
Allow to map processor trace buffer using
acquire_resource().Signed-off-by Michal Leszczynski <[email protected]>--- xen/common/memory.c | 28 ++++++++++++++++++++++++++++ xen/include/public/memory.h | 1 + 2 files changed, 29 insertions(+)xen/common/memory.c +0
-0
xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1011
Quoted 5 lines
xen/common/memory.c +2
-0@@ -1007,6 +1007,29 @@ static long xatp_permission_check(struct domain *d, unsigned int space)return xsm_add_to_physmap (XSM_TARGET , current->domain, d); } +static int acquire_vmtrace_buf(struct domain *d, unsigned int id, + unsigned long frame,
Shouldn't this be uint64_t to avoid truncation?
xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1024
Quoted 13 lines
xen/common/memory.c +13
-0+ unsigned int nr_frames, + xen_pfn_t mfn_list[]) +{ +mfn_t mfn; +unsigned int i; +struct vcpu *v =domain_vcpu (d, id); + +if ( !v || !v->vmtrace.pt_buf ) +return -EINVAL ; + + mfn =page_to_mfn (v->vmtrace.pt_buf); + +if ( frame + nr_frames > (v->domain->vmtrace_pt_size >>PAGE_SHIFT ) )
frame + nr_frames could possibly overflow a 64-bit value and therefore still pass the check.
So I would suggest to use:
(frame > (v->domain_vm_ptrace_pt_size >> PAGE_SHIFT)) || (nr_frames > ((v->domain_vm_ptrace_pt_size >> PAGE_SHIFT) - frame))
Cheers,
Signature · 1 line
Julien Grall
PATCH v5 9/11 x86/domctl: add XEN_DOMCTL_vmtrace_op
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Implement domctl to manage the runtime state of
processor trace feature.
--- xen/arch/x86/domctl.c | 48 +++++++++++++++++++++++++++++++++++++ xen/include/public/domctl.h | 26 ++++++++++++++++++++ 2 files changed, 74 insertions(+)
xen/arch/x86/domctl.c +48 -0
@@ -322,6 +322,48 @@ void arch_get_domain_info(const struct domain *d, info->arch_config.emulation_flags = d->arch.emulation_flags; } +static int do_vmtrace_op (struct domain *d,struct xen_domctl_vmtrace_op *op, +XEN_GUEST_HANDLE_PARAM (xen_domctl_t) u_domctl) +{ +int rc; +struct vcpu *v; + +if ( !vmtrace_supported ) +return -EOPNOTSUPP ; + +if ( !is_hvm_domain (d) ) +return -EOPNOTSUPP ; + +if ( op->vcpu >= d->max_vcpus ) +return -EINVAL ; + + v =domain_vcpu (d, op->vcpu); + rc =0 ; + +switch ( op->cmd ) + { +case XEN_DOMCTL_vmtrace_pt_enable: +case XEN_DOMCTL_vmtrace_pt_disable: +vcpu_pause (v); +spin_lock (&d->vmtrace_lock); + + rc =vmtrace_control_pt (v, op->cmd == XEN_DOMCTL_vmtrace_pt_enable); + +spin_unlock (&d->vmtrace_lock); +vcpu_unpause (v); +break ; + +case XEN_DOMCTL_vmtrace_pt_get_offset: + rc =vmtrace_get_pt_offset (v, &op->offset); +break ; + +default : + rc = -EOPNOTSUPP ; + } + +return rc; +} +#define MAX_IOPORTS 0x10000 long arch_do_domctl(
@@ -337,6 +379,12 @@ long arch_do_domctl(switch ( domctl->cmd ) { +case XEN_DOMCTL_vmtrace_op: + ret =do_vmtrace_op (d, &domctl->u.vmtrace_op, u_domctl); +if ( !ret ) + copyback = true; +break ; +case XEN_DOMCTL_shadow_op: ret =paging_domctl (d, &domctl->u.shadow_op, u_domctl,0 );if ( ret == -ERESTART )
xen/include/public/domctl.h +26 -0
@@ -1136,6 +1136,28 @@ struct xen_domctl_vuart_op {*/ }; +/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */ +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +struct xen_domctl_vmtrace_op { +/* IN variable */ +uint32_t cmd; +/* Enable/disable external vmtrace for given domain */ +#define XEN_DOMCTL_vmtrace_pt_enable 1 +#define XEN_DOMCTL_vmtrace_pt_disable 2 +#define XEN_DOMCTL_vmtrace_pt_get_offset 3 +domid_t domain; +uint32_t vcpu; +uint64_aligned_t size; + +/* OUT variable */ +uint64_aligned_t offset; +}; +typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t ; +DEFINE_XEN_GUEST_HANDLE (xen_domctl_vmtrace_op_t); + +#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */ +struct xen_domctl {uint32_t cmd;#define XEN_DOMCTL_createdomain 1
@@ -1217,6 +1239,7 @@ struct xen_domctl {#define XEN_DOMCTL_vuart_op 81#define XEN_DOMCTL_get_cpu_policy 82#define XEN_DOMCTL_set_cpu_policy 83 +#define XEN_DOMCTL_vmtrace_op 84#define XEN_DOMCTL_gdbsx_guestmemio 1000#define XEN_DOMCTL_gdbsx_pausevcpu 1001#define XEN_DOMCTL_gdbsx_unpausevcpu 1002
@@ -1277,6 +1300,9 @@ struct xen_domctl {struct xen_domctl_monitor_op monitor_op;struct xen_domctl_psr_alloc psr_alloc;struct xen_domctl_vuart_op vuart_op; +#if defined(__XEN__) || defined(__XEN_TOOLS__) +struct xen_domctl_vmtrace_op vmtrace_op; +#endif uint8_t pad[128 ]; } u; };
Signature · 1 line
2.17.1
Re: [PATCH v5 09/11] x86/domctl: add XEN_DOMCTL_vmtrace_op
Hi Michal,
xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1149
Quoted 11 lines · Michał Leszczyński
xen/include/public/domctl.h +11
-0+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */ +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +struct xen_domctl_vmtrace_op { + /* IN variable */ + uint32_t cmd; +/* Enable/disable external vmtrace for given domain */ +#define XEN_DOMCTL_vmtrace_pt_enable 1 +#define XEN_DOMCTL_vmtrace_pt_disable 2 +#define XEN_DOMCTL_vmtrace_pt_get_offset 3 + domid_t domain;
AFAICT, there is a 16-bit implicit padding here and ...
Quoted 1 line
+
uint32_tvcpu;
... a 32-bit implicit padding here. I would suggest to make the two explicit.
We possibly want to check they are also always zero.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v5 09/11] x86/domctl: add XEN_DOMCTL_vmtrace_op
Quoted 16 lines · Julien Grall
xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1149
Quoted 11 lines · Michał Leszczyński
xen/include/public/domctl.h +11
-0+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */ +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +struct xen_domctl_vmtrace_op { + /* IN variable */ + uint32_t cmd; +/* Enable/disable external vmtrace for given domain */ +#define XEN_DOMCTL_vmtrace_pt_enable 1 +#define XEN_DOMCTL_vmtrace_pt_disable 2 +#define XEN_DOMCTL_vmtrace_pt_get_offset 3 + domid_t domain;AFAICT, there is a 16-bit implicit padding here and ...
Quoted 6 lines
Quoted 1 line
+
uint32_tvcpu;... a 32-bit implicit padding here. I would suggest to make the two explicit.
We possibly want to check they are also always zero.
Not just possibly imo - we should.
Jan
Re: [PATCH v5 09/11] x86/domctl: add XEN_DOMCTL_vmtrace_op
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Julien Grall
xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1149
Quoted 11 lines · Michał Leszczyński
xen/include/public/domctl.h +11
-0+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */ +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +struct xen_domctl_vmtrace_op { + /* IN variable */ + uint32_t cmd; +/* Enable/disable external vmtrace for given domain */ +#define XEN_DOMCTL_vmtrace_pt_enable 1 +#define XEN_DOMCTL_vmtrace_pt_disable 2 +#define XEN_DOMCTL_vmtrace_pt_get_offset 3 + domid_t domain;AFAICT, there is a 16-bit implicit padding here and ...
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
+
uint32_tvcpu;... a 32-bit implicit padding here. I would suggest to make the two explicit.
We possibly want to check they are also always zero.
Not just possibly imo - we should.
I wasn't sure given that DOMCTL is not a stable interface.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH v5 09/11] x86/domctl: add XEN_DOMCTL_vmtrace_op
Quoted 20 lines · Julien Grall
Quoted 19 lines · Jan Beulich
Quoted 18 lines · Julien Grall
xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1149
Quoted 11 lines · Michał Leszczyński
xen/include/public/domctl.h +11
-0+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */ +#if defined(__XEN__) || defined(__XEN_TOOLS__) + +struct xen_domctl_vmtrace_op { + /* IN variable */ + uint32_t cmd; +/* Enable/disable external vmtrace for given domain */ +#define XEN_DOMCTL_vmtrace_pt_enable 1 +#define XEN_DOMCTL_vmtrace_pt_disable 2 +#define XEN_DOMCTL_vmtrace_pt_get_offset 3 + domid_t domain;AFAICT, there is a 16-bit implicit padding here and ...
Quoted 1 line
+
uint32_tvcpu;
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
... a 32-bit implicit padding here. I would suggest to make the two explicit.
We possibly want to check they are also always zero.
Not just possibly imo - we should.
I wasn't sure given that DOMCTL is not a stable interface.
True; checking padding fields allows assigning meaning to them without bumping the domctl interface version.
Jan
PATCH v5 10/11 tools/libxc: add xc_vmtrace_* functions
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Add functions in libxc that use the new XEN_DOMCTL_vmtrace interface.
--- tools/libxc/Makefile | 1 + tools/libxc/include/xenctrl.h | 39 +++++++++++++++++++ tools/libxc/xc_vmtrace.c | 73 +++++++++++++++++++++++++++++++++++ 3 files changed, 113 insertions(+) create mode 100644 tools/libxc/xc_vmtrace.c
tools/libxc/Makefile +1 -0
@@ -27,6 +27,7 @@ CTRL_SRCS-y += xc_csched2.c CTRL_SRCS-y += xc_arinc653.c CTRL_SRCS-y += xc_rt.c CTRL_SRCS-y += xc_tbuf.c +CTRL_SRCS-y += xc_vmtrace.c CTRL_SRCS-y += xc_pm.c CTRL_SRCS-y += xc_cpu_hotplug.c CTRL_SRCS-y += xc_resume.c
tools/libxc/include/xenctrl.h +39 -0
@@ -1585,6 +1585,45 @@ int xc_tbuf_set_cpu_mask(xc_interface *xch, xc_cpumap_t mask);int xc_tbuf_set_evt_mask (xc_interface *xch,uint32_t mask); +/** +* Enable processor trace for given vCPU in given DomU. +* Allocate the trace ringbuffer with given size. +* +* @parm xch a handle to an open hypervisor interface +* @parm domid domain identifier +* @parm vcpu vcpu identifier +* @return 0 on success, -1 on failure +*/ +int xc_vmtrace_pt_enable (xc_interface *xch,uint32_t domid, +uint32_t vcpu); + +/** +* Disable processor trace for given vCPU in given DomU. +* Deallocate the trace ringbuffer. +* +* @parm xch a handle to an open hypervisor interface +* @parm domid domain identifier +* @parm vcpu vcpu identifier +* @return 0 on success, -1 on failure +*/ +int xc_vmtrace_pt_disable (xc_interface *xch,uint32_t domid, +uint32_t vcpu); + +/** +* Get current offset inside the trace ringbuffer. +* This allows to determine how much data was written into the buffer. +* Once buffer overflows, the offset will reset to 0 and the previous +* data will be overriden. +* +* @parm xch a handle to an open hypervisor interface +* @parm domid domain identifier +* @parm vcpu vcpu identifier +* @parm offset current offset inside trace buffer will be written there +* @return 0 on success, -1 on failure +*/ +int xc_vmtrace_pt_get_offset (xc_interface *xch,uint32_t domid, +uint32_t vcpu,uint64_t *offset); +int xc_domctl (xc_interface *xch,struct xen_domctl *domctl);int xc_sysctl (xc_interface *xch,struct xen_sysctl *sysctl);
tools/libxc/xc_vmtrace.c new file +73 -0
73 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v5 11/11 tools/proctrace: add proctrace tool
In reply to Michał LeszczyńskiFrom: Michal Leszczynski <[email protected]>
Add an demonstration tool that uses xc_vmtrace_* calls in order to manage external IPT monitoring for DomU.
--- tools/proctrace/Makefile | 48 +++++++++++ tools/proctrace/proctrace.c | 163 ++++++++++++++++++++++++++++++++++++ 2 files changed, 211 insertions(+) create mode 100644 tools/proctrace/Makefile create mode 100644 tools/proctrace/proctrace.c
tools/proctrace/Makefile new file +48 -0
@@ -0,0 +1,48 @@ +# Copyright (C) CERT Polska - NASK PIB +# Author: Michał Leszczyński <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; under version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +XEN_ROOT=$(CURDIR)/../.. +include $(XEN_ROOT)/tools/Rules.mk + +CFLAGS += -Werror +CFLAGS += $(CFLAGS_libxenevtchn) +CFLAGS += $(CFLAGS_libxenctrl) +LDLIBS += $(LDLIBS_libxenctrl) +LDLIBS += $(LDLIBS_libxenevtchn) +LDLIBS += $(LDLIBS_libxenforeignmemory) + +.PHONY: all +all: build + +.PHONY: build +build: proctrace + +.PHONY: install +install: build + $(INSTALL_DIR) $(DESTDIR)$(sbindir) + $(INSTALL_PROG) proctrace $(DESTDIR)$(sbindir)/proctrace + +.PHONY: uninstall +uninstall: + rm -f $(DESTDIR)$(sbindir)/proctrace + +.PHONY: clean +clean: + $(RM) -f $(DEPS_RM) + +.PHONY: distclean +distclean: clean + +iptlive: iptlive.o Makefile + $(CC) $(LDFLAGS) $< -o $@ $(LDLIBS) $(APPEND_LDFLAGS) + +-include $(DEPS_INCLUDE)
tools/proctrace/proctrace.c new file +163 -0
163 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v5 11/11] tools/proctrace: add proctrace tool
----- 5 lip 2020 o 20:55, Michał Leszczyński [email protected] napisał(a):
Quoted 21 lines
From: Michal Leszczynski <[email protected]>
Add an demonstration tool that uses xc_vmtrace_* calls in order to manage external IPT monitoring for DomU.
Signed-off-by Michal Leszczynski <[email protected]>---
tools/proctrace/Makefile| 48 +++++++++++tools/proctrace/proctrace.c| 163 ++++++++++++++++++++++++++++++++++++2 files changed, 211 insertions(+)create mode 100644
tools/proctrace/Makefile
create mode 100644tools/proctrace/proctrace.ctools/proctrace/proctrace.c new file +0
-0
tools/proctrace/proctrace.c line 34
Quoted 10 lines
tools/proctrace/proctrace.c new file +10
-0+#include <stdlib.h> +#include <stdio.h> +#include <sys/mman.h> +#include <signal.h> + +#include <xenctrl.h> +#include <xen/xen.h> +#include <xenforeignmemory.h> + +#define BUF_SIZE (16384 * XC_PAGE_SIZE)
I would like to discuss here, how we should retrieve the trace buffer size in runtime? Should there be a hypercall for it, or some extension to acquire_resource logic?
Best regards,
Michał Leszczyński
CERT Polska
Re: [PATCH v5 11/11] tools/proctrace: add proctrace tool
Quoted 17 lines · Michał Leszczyński
----- 5 lip 2020 o 20:55, Michał Leszczyński [email protected] napisał(a):
tools/proctrace/proctrace.c line 34
Quoted 12 lines
tools/proctrace/proctrace.c new file +10
-0+#include <stdlib.h> +#include <stdio.h> +#include <sys/mman.h> +#include <signal.h> + +#include <xenctrl.h> +#include <xen/xen.h> +#include <xenforeignmemory.h> + +#define BUF_SIZE (16384 * XC_PAGE_SIZE)I would like to discuss here, how we should retrieve the trace buffer size in runtime? Should there be a hypercall for it, or some extension to
acquire_resourcelogic?
Personally I'd prefer the latter, but the question is whether one can be made in a backwards compatible way.
Jan
Re: [PATCH v5 11/11] tools/proctrace: add proctrace tool
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Michał Leszczyński
----- 5 lip 2020 o 20:55, Michał Leszczyński [email protected] napisał(a):
tools/proctrace/proctrace.c line 34
Quoted 12 lines
tools/proctrace/proctrace.c new file +10
-0+#include <stdlib.h> +#include <stdio.h> +#include <sys/mman.h> +#include <signal.h> + +#include <xenctrl.h> +#include <xen/xen.h> +#include <xenforeignmemory.h> + +#define BUF_SIZE (16384 * XC_PAGE_SIZE)I would like to discuss here, how we should retrieve the trace buffer size in runtime? Should there be a hypercall for it, or some extension to
acquire_resourcelogic?
Quoted 2 lines
Personally I'd prefer the latter, but the question is whether one can be made in a backwards compatible way.
I already covered this in v4.
~Andrew
Re: [PATCH v5 11/11] tools/proctrace: add proctrace tool
----- 6 lip 2020 o 11:47, Andrew Cooper [email protected] napisał(a):
Quoted 19 lines · Jan Beulich
Quoted 18 lines · Michał Leszczyński
Quoted 17 lines
----- 5 lip 2020 o 20:55, Michał Leszczyński [email protected] napisał(a):
tools/proctrace/proctrace.c line 34
Quoted 12 lines
tools/proctrace/proctrace.c new file +10
-0+#include <stdlib.h> +#include <stdio.h> +#include <sys/mman.h> +#include <signal.h> + +#include <xenctrl.h> +#include <xen/xen.h> +#include <xenforeignmemory.h> + +#define BUF_SIZE (16384 * XC_PAGE_SIZE)I would like to discuss here, how we should retrieve the trace buffer size in runtime? Should there be a hypercall for it, or some extension to
acquire_resourcelogic?
Quoted 6 lines
Quoted 2 lines
Personally I'd prefer the latter, but the question is whether one can be made in a backwards compatible way.
I already covered this in v4.
~Andrew
Ok, sorry, I see:
Quoted 6 lines
The
guest_handle_is_null(xmar.frame_list)path
in Xen is supposed to report the size of the resource, not the size of Xen's local buffer, so userspace can ask "how large is this resource".I'll try and find some time to fix this and arrange for backports, but the current behaviour is nonsense, and problematic for new users.
So to make it clear: should I modify the acquire_resource logic in such way that NULL guest handle would report the actual size of the resource?
If I got it right, here:
https://lists.xen.org/archives/html/xen-devel/2020-07/msg00159.html
it was suggested that it should report the constant value of UINT_MAX >> MEMOP_EXTENT_SHIFT and as far as I understood, the expectation is that it would report how many frames might be requested at once, not what is the size of the resource we're asking for.
Best regards,
Michał Leszczyński
CERT Polska