Skip to contentSkip to conversation

Implement support for external IPT monitoring

PATCH In review Michał Leszczyński 4 participants 33 messages Jul 7, 2020 – Aug 27, 2020

lore
Share & tools
lore
#0 lore raw

v6 cover Implement support for external IPT monitoring

Intel Processor Trace is an architectural extension available in modern Intel family CPUs. It allows recording the detailed trace of activity while the processor executes the code. One might use the recorded trace to reconstruct the code flow. It means, to find out the executed code paths, determine branches taken, and so forth.

The abovementioned feature is described in Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3C: System Programming Guide, Part 3, Chapter 36: "Intel Processor Trace."

This patch series implements an interface that Dom0 could use in order to enable IPT for particular vCPUs in DomU, allowing for external monitoring. Such a feature has numerous applications like malware monitoring, fuzzing, or performance testing.

Also thanks to Tamas K Lengyel for a few preliminary hints before first version of this patch was submitted to xen-devel.

Changed since v1:
* MSR_RTIT_CTL is managed using MSR load lists
* other PT-related MSRs are modified only when vCPU goes out of context
* trace buffer is now acquired as a resource
* added vmtrace_pt_size parameter in xl.cfg, the size of trace buffer must be specified in the moment of domain creation
* trace buffers are allocated on domain creation, destructed on domain destruction
* HVMOP_vmtrace_ipt_enable/disable is limited to enabling/disabling PT these calls don't manage buffer memory anymore
* lifted 32 MFN/GFN array limit when acquiring resources
* minor code style changes according to review

Changed since v2:
* trace buffer is now allocated on domain creation (in v2 it was allocated when hvm param was set)
* restored 32-item limit in mfn/gfn arrays in acquire_resource and instead implemented hypercall continuations
* code changes according to Jan's and Roger's review

Changed since v3:
* vmtrace HVMOPs are not implemented as DOMCTLs
* patches splitted up according to Andrew's comments
* code changes according to v3 review on the mailing list

Changed since v4:
* rebased to commit be63d9d4
* fixed dependencies between patches (earlier patches don't reference further patches)
* introduced preemption check in acquire_resource
* moved buffer allocation to common code
* splitted some patches according to code review
* minor fixes according to code review

Changed since v5:
* trace buffer size is now dynamically determined by the proctrace tool
* trace buffer size variable is uniformly defined as uint32_t processor_trace_buf_kb in hypervisor, toolstack and ABI
* buffer pages are not freed explicitly but reference count is now used instead
* minor fixes according to code review

This patch series is available on GitHub:
https://github.com/icedevml/xen/tree/ipt-patch-v6

Michal Leszczynski (11):
memory: batch processing in acquire_resource()
x86/vmx: add Intel PT MSR definitions
x86/vmx: add IPT cpu feature
common: add vmtrace_pt_size domain parameter
tools/libxl: add vmtrace_pt_size parameter
x86/hvm: processor trace interface in HVM
x86/vmx: implement IPT in VMX
x86/mm: add vmtrace_buf resource type
x86/domctl: add XEN_DOMCTL_vmtrace_op
tools/libxc: add xc_vmtrace_* functions
tools/proctrace: add proctrace tool

 docs/man/xl.cfg.5.pod.in                    |  13 ++
 tools/golang/xenlight/helpers.gen.go        |   2 +
 tools/golang/xenlight/types.gen.go          |   1 +
 tools/libxc/Makefile                        |   1 +
 tools/libxc/include/xenctrl.h               |  40 +++++
 tools/libxc/xc_vmtrace.c                    |  87 ++++++++++
 tools/libxl/libxl.h                         |   8 +
 tools/libxl/libxl_create.c                  |   1 +
 tools/libxl/libxl_types.idl                 |   4 +
 tools/proctrace/Makefile                    |  45 +++++
 tools/proctrace/proctrace.c                 | 179 ++++++++++++++++++++
 tools/xl/xl_parse.c                         |  22 +++
 xen/arch/x86/domain.c                       |  27 +++
 xen/arch/x86/domctl.c                       |  50 ++++++
 xen/arch/x86/hvm/vmx/vmcs.c                 |  15 +-
 xen/arch/x86/hvm/vmx/vmx.c                  | 110 ++++++++++++
 xen/common/domain.c                         |  46 +++++
 xen/common/memory.c                         |  80 ++++++++-
 xen/include/asm-x86/cpufeature.h            |   1 +
 xen/include/asm-x86/hvm/hvm.h               |  20 +++
 xen/include/asm-x86/hvm/vmx/vmcs.h          |   4 +
 xen/include/asm-x86/hvm/vmx/vmx.h           |  14 ++
 xen/include/asm-x86/msr-index.h             |  24 +++
 xen/include/public/arch-x86/cpufeatureset.h |   1 +
 xen/include/public/domctl.h                 |  29 ++++
 xen/include/public/memory.h                 |   1 +
 xen/include/xen/domain.h                    |   2 +
 xen/include/xen/sched.h                     |   7 +
 28 files changed, 828 insertions(+), 6 deletions(-)
 create mode 100644 tools/libxc/xc_vmtrace.c
 create mode 100644 tools/proctrace/Makefile
 create mode 100644 tools/proctrace/proctrace.c
Signature · 1 line
2.17.1
#1 lore raw

PATCH v6 1/11 memory: batch processing in acquire_resource()

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

Be aware that this modifies the behavior of acquire_resource call with frame_list=NULL. While previously it would return the size of internal array (32), with this patch it returns the maximal quantity of frames that could be requested at once, i.e. UINT_MAX >> MEMOP_EXTENT_SHIFT.

---
 xen/common/memory.c | 49 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 44 insertions(+), 5 deletions(-)
xen/common/memory.c +44 -5
index 714077c1e5..eb42f883df 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
 }
 
 static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
 {
     struct domain *d, *currd = current->domain;
     xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
     /*
      * The mfn_list and gfn_list (below) arrays are ok on stack for the
      * moment since they are small, but if they need to grow in future
@@ -1069,7 +1071,7 @@ static int acquire_resource(
         if ( xmar.nr_frames )
             return -EINVAL;
 
-        xmar.nr_frames = ARRAY_SIZE(mfn_list);
+        xmar.nr_frames = UINT_MAX >> MEMOP_EXTENT_SHIFT;
 
         if ( __copy_field_to_guest(arg, &xmar, nr_frames) )
             return -EFAULT;
@@ -1077,8 +1079,28 @@ static int acquire_resource(
         return 0;
     }
 
+    total_frames = xmar.nr_frames;
+
+    /* Is the size too large for us to encode a continuation? */
+    if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )
+        return -EINVAL;
+
+    if ( *start_extent )
+    {
+        /*
+         * Check whether start_extent is in bounds, as this
+         * value if visible to the calling domain.
+         */
+        if ( *start_extent > xmar.nr_frames )
+            return -EINVAL;
+
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
     if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);
 
     rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
     if ( rc )
@@ -1135,6 +1157,14 @@ static int acquire_resource(
         }
     }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
  out:
     rcu_unlock_domain(d);
 
@@ -1599,8 +1629,17 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 #endif
 
     case XENMEM_acquire_resource:
-        rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+        do {
+            rc = acquire_resource(
+                guest_handle_cast(arg, xen_mem_acquire_resource_t),
+                &start_extent);
+
+            if ( hypercall_preempt_check() )
+                return hypercall_create_continuation(
+                    __HYPERVISOR_memory_op, "lh",
+                    op | (start_extent << MEMOP_EXTENT_SHIFT), arg);
+        } while ( rc == -ERESTART );
+
         break;
 
     default:
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v6 01/11] memory: batch processing in acquire_resource()

Quoted 5 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

Quoted 8 lines

Be aware that this modifies the behavior of acquire_resource call with frame_list=NULL. While previously it would return the size of internal array (32), with this patch it returns the maximal quantity of frames that could be requested at once, i.e. UINT_MAX >> MEMOP_EXTENT_SHIFT.

---

FWIW, I think I've also said on a previous version, I would prefer if the changelog between versions is added to each patch, having it on the cover letter is not very helpful as I usually care about specific changes made to each patch.

I've just got one comment that needs addressing below.

Quoted 76 lines
 xen/common/memory.c | 49 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 44 insertions(+), 5 deletions(-)
xen/common/memory.c +33 -3
index 714077c1e5..eb42f883df 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
 }
 
 static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
 {
     struct domain *d, *currd = current->domain;
     xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
     /*
      * The mfn_list and gfn_list (below) arrays are ok on stack for the
      * moment since they are small, but if they need to grow in future
@@ -1069,7 +1071,7 @@ static int acquire_resource(
         if ( xmar.nr_frames )
             return -EINVAL;
 
-        xmar.nr_frames = ARRAY_SIZE(mfn_list);
+        xmar.nr_frames = UINT_MAX >> MEMOP_EXTENT_SHIFT;
 
         if ( __copy_field_to_guest(arg, &xmar, nr_frames) )
             return -EFAULT;
@@ -1077,8 +1079,28 @@ static int acquire_resource(
         return 0;
     }
 
+    total_frames = xmar.nr_frames;
+
+    /* Is the size too large for us to encode a continuation? */
+    if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )
+        return -EINVAL;
+
+    if ( *start_extent )
+    {
+        /*
+         * Check whether start_extent is in bounds, as this
+         * value if visible to the calling domain.
+         */
+        if ( *start_extent > xmar.nr_frames )
+            return -EINVAL;
+
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
     if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);
 
     rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
     if ( rc )
@@ -1135,6 +1157,14 @@ static int acquire_resource(
         }
     }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
  out:
     rcu_unlock_domain(d);
 

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1635

Quoted 9 lines
xen/common/memory.c +4 -2
@@ -1599,8 +1629,17 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 #endif
 
     case XENMEM_acquire_resource:
-        rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+        do {
+            rc = acquire_resource(
+                guest_handle_cast(arg, xen_mem_acquire_resource_t),
+                &start_extent);

I think it would be interesting from a performance PoV to move the xmar copy_from_guest here, so that each call to acquire_resource in the loop doesn't need to perform a copy from guest. That's more relevant for translated callers, where a copy_from_guest involves a guest page table and a p2m walk.

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1637

Quoted 2 lines
xen/common/memory.c +2 -0
+
+            if ( hypercall_preempt_check() )

You are missing a rc == -ERESTART check here, you don't want to encode a continuation if rc is different than -ERESTART AFAICT.

Thanks, Roger.

#3 lore raw

Re: [PATCH v6 01/11] memory: batch processing in acquire_resource()

Quoted 4 lines · Roger Pau Monné
Quoted 3 lines · Michał Leszczyński
(quoted diff) +0 -0
@@ -1599,8 +1629,17 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 #endif
 
Quoted 13 lines

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1635

Quoted 7 lines
xen/common/memory.c +4 -2
     case XENMEM_acquire_resource:
-        rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+        do {
+            rc = acquire_resource(
+                guest_handle_cast(arg, xen_mem_acquire_resource_t),
+                &start_extent);

I think it would be interesting from a performance PoV to move the xmar copy_from_guest here, so that each call to acquire_resource in the loop doesn't need to perform a copy from guest. That's more relevant for translated callers, where a copy_from_guest involves a guest page table and a p2m walk.

This isn't just a nice-to-have for performance reasons, but a correctness/consistency thing: A rogue (or buggy) guest may alter the structure between two such reads. It _may_ be the case that we're dealing fine with this right now, but it would feel like a trap to fall into later on.

Quoted 5 lines

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1637

Quoted 2 lines
xen/common/memory.c +2 -0
+
+            if ( hypercall_preempt_check() )

You are missing a rc == -ERESTART check here, you don't want to encode a continuation if rc is different than -ERESTART AFAICT.

At which point the subsequent containing do/while() likely wants adjusting to, e.g. to "for( ; ; )".

Jan

#4 lore raw

Re: [PATCH v6 01/11] memory: batch processing in acquire_resource()

Quoted 19 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné
Quoted 3 lines · Michał Leszczyński
(quoted diff) +0 -0
@@ -1599,8 +1629,17 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 #endif
 
Quoted 13 lines

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1635

Quoted 7 lines
xen/common/memory.c +4 -2
     case XENMEM_acquire_resource:
-        rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+        do {
+            rc = acquire_resource(
+                guest_handle_cast(arg, xen_mem_acquire_resource_t),
+                &start_extent);

I think it would be interesting from a performance PoV to move the xmar copy_from_guest here, so that each call to acquire_resource in the loop doesn't need to perform a copy from guest. That's more relevant for translated callers, where a copy_from_guest involves a guest page table and a p2m walk.

Quoted 5 lines

This isn't just a nice-to-have for performance reasons, but a correctness/consistency thing: A rogue (or buggy) guest may alter the structure between two such reads. It _may_ be the case that we're dealing fine with this right now, but it would feel like a trap to fall into later on.

I *think* this is safe, given you copy from guest and perform the checks for each iteration. I agree the copy should be pulled out of the loop together with the checks. There's no point in performing it for every iteration.

Quoted 8 lines
Quoted 5 lines

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1637

Quoted 2 lines
xen/common/memory.c +2 -0
+
+            if ( hypercall_preempt_check() )

You are missing a rc == -ERESTART check here, you don't want to encode a continuation if rc is different than -ERESTART AFAICT.

At which point the subsequent containing do/while() likely wants adjusting to, e.g. to "for( ; ; )".

That's another option, but you would need to add an extra if ( rc != -ERESTART ) break; to the loop body if the while condition is removed.

Roger.

#5 lore raw

Re: [PATCH v6 01/11] memory: batch processing in acquire_resource()

In reply to Michał Leszczyński
Quoted 10 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

Be aware that this modifies the behavior of acquire_resource call with frame_list=NULL. While previously it would return the size of internal array (32), with this patch it returns the maximal quantity of frames that could be requested at once, i.e. UINT_MAX >> MEMOP_EXTENT_SHIFT.

This isn't really a behavioral change, and hence I'd prefer this to be re-worded: It was and is the upper bound on request sizes that gets reported here. It's just that this upper bound now changes.

xen/common/memory.c static int acquire_grant_table(struct domain *d, unsigned int id, · line 1054

Quoted 13 lines
xen/common/memory.c +3 -1
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
 }
 
 static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
 {
     struct domain *d, *currd = current->domain;
     xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;

Please don't use fixed width types when plain C types will do (unsigned int here).

Quoted 22 lines
xen/common/memory.c +10 -1
@@ -1069,7 +1071,7 @@ static int acquire_resource(
         if ( xmar.nr_frames )
             return -EINVAL;
 
-        xmar.nr_frames = ARRAY_SIZE(mfn_list);
+        xmar.nr_frames = UINT_MAX >> MEMOP_EXTENT_SHIFT;
 
         if ( __copy_field_to_guest(arg, &xmar, nr_frames) )
             return -EFAULT;
@@ -1077,8 +1079,28 @@ static int acquire_resource(
         return 0;
     }
 
+    total_frames = xmar.nr_frames;
+
+    /* Is the size too large for us to encode a continuation? */
+    if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )
+        return -EINVAL;
+
+    if ( *start_extent )
+    {
+        /*

xen/common/memory.c static int acquire_resource( · line 1100

Quoted 10 lines
xen/common/memory.c +10 -0
+         * Check whether start_extent is in bounds, as this
+         * value if visible to the calling domain.
+         */
+        if ( *start_extent > xmar.nr_frames )
+            return -EINVAL;
+
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }

May I ask that you drop the if() around this block of code?

Also, looking at this, I wonder whether it's a good idea to use the "start extent" model here anyway: You could as well update the struct (saying that it may be clobbered in the public header) and copy the whole thing back to the original guest struct. This would then remove the pretty arbitrary "UINT_MAX >> MEMOP_EXTENT_SHIFT" limit you currently need to enforce. The main question is whether we'd consider such an adjustment to an existing interface acceptable; there's an at least theoretical risk that it may break existing callers. Then again no existing caller can sensibly have specified a count above 32, and when the copying back would be limited to just the continuation case, no such caller would be affected in any way afaict.

Jan

#6 lore raw

Re: [PATCH v6 01/11] memory: batch processing in acquire_resource()

Hi,

Quoted 14 lines · Jan Beulich

May I ask that you drop the if() around this block of code?

Also, looking at this, I wonder whether it's a good idea to use the "start extent" model here anyway: You could as well update the struct (saying that it may be clobbered in the public header) and copy the whole thing back to the original guest struct. This would then remove the pretty arbitrary "UINT_MAX >> MEMOP_EXTENT_SHIFT" limit you currently need to enforce. The main question is whether we'd consider such an adjustment to an existing interface acceptable; there's an at least theoretical risk that it may break existing callers. Then again no existing caller can sensibly have specified a count above 32, and when the copying back would be limited to just the continuation case, no such caller would be affected in any way afaict.

I can see two risks with this approach:
1) There is no requirement for the 'arg' buffer to be in read-write
memory.
2) A guest may expect to re-use the value within the structure after
the hypercalls.

So I think the "start extent" model is better. This is already used in other memory sub-hypercalls, so the risk to break existing users is more limited.

Cheers,

Signature · 1 line
Julien Grall
#7 lore raw

PATCH v6 2/11 x86/vmx: add Intel PT MSR definitions

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Define constants related to Intel Processor Trace features.

---
 xen/include/asm-x86/msr-index.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
xen/include/asm-x86/msr-index.h +24 -0
index 0fe98af923..4fd54fb5c9 100644
@@ -72,7 +72,31 @@
 #define MSR_RTIT_OUTPUT_BASE                0x00000560
 #define MSR_RTIT_OUTPUT_MASK                0x00000561
 #define MSR_RTIT_CTL                        0x00000570
+#define  RTIT_CTL_TRACE_EN                  (_AC(1, ULL) <<  0)
+#define  RTIT_CTL_CYC_EN                    (_AC(1, ULL) <<  1)
+#define  RTIT_CTL_OS                        (_AC(1, ULL) <<  2)
+#define  RTIT_CTL_USR                       (_AC(1, ULL) <<  3)
+#define  RTIT_CTL_PWR_EVT_EN                (_AC(1, ULL) <<  4)
+#define  RTIT_CTL_FUP_ON_PTW                (_AC(1, ULL) <<  5)
+#define  RTIT_CTL_FABRIC_EN                 (_AC(1, ULL) <<  6)
+#define  RTIT_CTL_CR3_FILTER                (_AC(1, ULL) <<  7)
+#define  RTIT_CTL_TOPA                      (_AC(1, ULL) <<  8)
+#define  RTIT_CTL_MTC_EN                    (_AC(1, ULL) <<  9)
+#define  RTIT_CTL_TSC_EN                    (_AC(1, ULL) << 10)
+#define  RTIT_CTL_DIS_RETC                  (_AC(1, ULL) << 11)
+#define  RTIT_CTL_PTW_EN                    (_AC(1, ULL) << 12)
+#define  RTIT_CTL_BRANCH_EN                 (_AC(1, ULL) << 13)
+#define  RTIT_CTL_MTC_FREQ                  (_AC(0xf, ULL) << 14)
+#define  RTIT_CTL_CYC_THRESH                (_AC(0xf, ULL) << 19)
+#define  RTIT_CTL_PSB_FREQ                  (_AC(0xf, ULL) << 24)
+#define  RTIT_CTL_ADDR(n)                   (_AC(0xf, ULL) << (32 + 4 * (n)))
 #define MSR_RTIT_STATUS                     0x00000571
+#define  RTIT_STATUS_FILTER_EN              (_AC(1, ULL) <<  0)
+#define  RTIT_STATUS_CONTEXT_EN             (_AC(1, ULL) <<  1)
+#define  RTIT_STATUS_TRIGGER_EN             (_AC(1, ULL) <<  2)
+#define  RTIT_STATUS_ERROR                  (_AC(1, ULL) <<  4)
+#define  RTIT_STATUS_STOPPED                (_AC(1, ULL) <<  5)
+#define  RTIT_STATUS_BYTECNT                (_AC(0x1ffff, ULL) << 32)
 #define MSR_RTIT_CR3_MATCH                  0x00000572
 #define MSR_RTIT_ADDR_A(n)                 (0x00000580 + (n) * 2)
 #define MSR_RTIT_ADDR_B(n)                 (0x00000581 + (n) * 2)
Signature · 1 line
2.17.1
#8 lore raw

PATCH v6 3/11 x86/vmx: add IPT cpu feature

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Check if Intel Processor Trace feature is supported by current processor. Define vmtrace_supported global variable.

---
 xen/arch/x86/hvm/vmx/vmcs.c                 | 15 ++++++++++++++-
 xen/common/domain.c                         |  2 ++
 xen/include/asm-x86/cpufeature.h            |  1 +
 xen/include/asm-x86/hvm/vmx/vmcs.h          |  1 +
 xen/include/public/arch-x86/cpufeatureset.h |  1 +
 xen/include/xen/domain.h                    |  2 ++
 6 files changed, 21 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/vmx/vmcs.c +14 -1
index ca94c2bedc..3a53553f10 100644
@@ -291,6 +291,20 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    if ( !smp_processor_id() )
+        vmtrace_supported = cpu_has_ipt &&
+                            (_vmx_misc_cap & VMX_MISC_PROC_TRACE);
+    else if ( vmtrace_supported &&
+              !(_vmx_misc_cap & VMX_MISC_PROC_TRACE) )
+    {
+        printk("VMX: IPT capabilities fatally differ between CPU%u and CPU0\n",
+               smp_processor_id());
+        return -EINVAL;
+    }
+
     if ( _vmx_cpu_based_exec_control & CPU_BASED_ACTIVATE_SECONDARY_CONTROLS )
     {
         min = 0;
@@ -305,7 +319,6 @@ static int vmx_init_vmcs_config(void)
                SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                SECONDARY_EXEC_XSAVES |
                SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
         if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
             opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
         if ( opt_vpid_enabled )
xen/common/domain.c +2 -0
index 7cc9526139..a45cf023f7 100644
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
 
 vcpu_info_t dummy_vcpu_info;
 
+bool vmtrace_supported __read_mostly;
+
 static void __domain_finalise_shutdown(struct domain *d)
 {
     struct vcpu *v;
xen/include/asm-x86/cpufeature.h +1 -0
index f790d5c1f8..555f696a26 100644
@@ -104,6 +104,7 @@
 #define cpu_has_clwb            boot_cpu_has(X86_FEATURE_CLWB)
 #define cpu_has_avx512er        boot_cpu_has(X86_FEATURE_AVX512ER)
 #define cpu_has_avx512cd        boot_cpu_has(X86_FEATURE_AVX512CD)
+#define cpu_has_ipt             boot_cpu_has(X86_FEATURE_PROC_TRACE)
 #define cpu_has_sha             boot_cpu_has(X86_FEATURE_SHA)
 #define cpu_has_avx512bw        boot_cpu_has(X86_FEATURE_AVX512BW)
 #define cpu_has_avx512vl        boot_cpu_has(X86_FEATURE_AVX512VL)
xen/include/asm-x86/hvm/vmx/vmcs.h +1 -0
index 906810592f..6153ba6769 100644
@@ -283,6 +283,7 @@ extern u32 vmx_secondary_exec_control;
 #define VMX_VPID_INVVPID_SINGLE_CONTEXT_RETAINING_GLOBAL 0x80000000000ULL
 extern u64 vmx_ept_vpid_cap;
 
+#define VMX_MISC_PROC_TRACE                     0x00004000
 #define VMX_MISC_CR3_TARGET                     0x01ff0000
 #define VMX_MISC_VMWRITE_ALL                    0x20000000
 
xen/include/public/arch-x86/cpufeatureset.h +1 -0
index fe7492a225..2c91862f2d 100644
@@ -217,6 +217,7 @@ XEN_CPUFEATURE(SMAP,          5*32+20) /*S  Supervisor Mode Access Prevention */
 XEN_CPUFEATURE(AVX512_IFMA,   5*32+21) /*A  AVX-512 Integer Fused Multiply Add */
 XEN_CPUFEATURE(CLFLUSHOPT,    5*32+23) /*A  CLFLUSHOPT instruction */
 XEN_CPUFEATURE(CLWB,          5*32+24) /*A  CLWB instruction */
+XEN_CPUFEATURE(PROC_TRACE,    5*32+25) /*   Processor Tracing feature */
 XEN_CPUFEATURE(AVX512PF,      5*32+26) /*A  AVX-512 Prefetch Instructions */
 XEN_CPUFEATURE(AVX512ER,      5*32+27) /*A  AVX-512 Exponent & Reciprocal Instrs */
 XEN_CPUFEATURE(AVX512CD,      5*32+28) /*A  AVX-512 Conflict Detection Instrs */
xen/include/xen/domain.h +2 -0
index 7e51d361de..61ebc6c24d 100644
@@ -130,4 +130,6 @@ struct vnuma_info {
 
 void vnuma_destroy(struct vnuma_info *vnuma);
 
+extern bool vmtrace_supported;
+
 #endif /* __XEN_DOMAIN_H__ */
Signature · 1 line
2.17.1
Reviewed-by on v6 3/11
#9 lore raw

Re: [PATCH v6 03/11] x86/vmx: add IPT cpu feature

Quoted 4 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Check if Intel Processor Trace feature is supported by current processor. Define vmtrace_supported global variable.

IIRC there was some discussion in previous versions about whether vmtrace_supported should be globally exposed to all arches, since I see the symbol is defined in a common file I assume Arm maintainers agree to this approach, and hence it would be helpful to add a note to the commit message, ie:

"The vmtrace_supported global variable is defined in common code with the expectation that other arches also supporting processor tracing can make use of it."

Quoted 1 line

LGTM:

Quoted 14 lines
---
 xen/arch/x86/hvm/vmx/vmcs.c                 | 15 ++++++++++++++-
 xen/common/domain.c                         |  2 ++
 xen/include/asm-x86/cpufeature.h            |  1 +
 xen/include/asm-x86/hvm/vmx/vmcs.h          |  1 +
 xen/include/public/arch-x86/cpufeatureset.h |  1 +
 xen/include/xen/domain.h                    |  2 ++
 6 files changed, 21 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/vmx/vmcs.c +0 -0
index ca94c2bedc..3a53553f10 100644

xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(void) · line 299

Quoted 9 lines
xen/arch/x86/hvm/vmx/vmcs.c +6 -0
@@ -291,6 +291,20 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    if ( !smp_processor_id() )
+        vmtrace_supported = cpu_has_ipt &&
+                            (_vmx_misc_cap & VMX_MISC_PROC_TRACE);

Andrew also suggested to set vmtrace_supported to the value of cpu_has_ipt during CPU identification and then clear it here if VMX doesn't support IPT. Since this implementation won't add support for PV guests I'm not specially trilled and I think this approach is fine for the time being. If/When PV support is added we will have to re-arrange this a bit.

Thanks.

#10 lore raw

Re: [PATCH v6 03/11] x86/vmx: add IPT cpu feature

In reply to Michał Leszczyński

xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(void) · line 297

Quoted 10 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmcs.c +4 -0
@@ -291,6 +291,20 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    if ( !smp_processor_id() )

Despite us not supporting offlining and re-onlining of the BSP (i.e. CPU 0) yet, I'm trying hard to keep new code from making assumptions like this one. Please don't base this on CPU number, but some other property (system_state, if nothing else can be found).

Jan

#11 lore raw

PATCH v6 4/11 common: add vmtrace_pt_size domain parameter

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add vmtrace_pt_size domain parameter in live domain and vmtrace_pt_order parameter in xen_domctl_createdomain.

---
 xen/arch/x86/domain.c       | 6 ++++++
 xen/common/domain.c         | 9 +++++++++
 xen/include/public/domctl.h | 1 +
 xen/include/xen/sched.h     | 3 +++
 4 files changed, 19 insertions(+)
xen/arch/x86/domain.c +6 -0
index fee6c3931a..b75017b28b 100644
@@ -499,6 +499,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
          */
         config->flags |= XEN_DOMCTL_CDF_oos_off;
 
+    if ( !hvm && config->processor_trace_buf_kb )
+    {
+        dprintk(XENLOG_INFO, "Processor trace is not supported on non-HVM\n");
+        return -EINVAL;
+    }
+
     return 0;
 }
 
xen/common/domain.c +9 -0
index a45cf023f7..e6e8f88da1 100644
@@ -338,6 +338,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->processor_trace_buf_kb && !vmtrace_supported )
+    {
+        dprintk(XENLOG_INFO, "Processor tracing is not supported\n");
+        return -EINVAL;
+    }
+
     return arch_sanitise_domain_config(config);
 }
 
@@ -443,6 +449,9 @@ struct domain *domain_create(domid_t domid,
         d->nr_pirqs = min(d->nr_pirqs, nr_irqs);
 
         radix_tree_init(&d->pirq_tree);
+
+        if ( config->processor_trace_buf_kb )
+            d->processor_trace_buf_kb = config->processor_trace_buf_kb;
     }
 
     if ( (err = arch_domain_create(d, config)) != 0 )
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7681675a94 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint32_t processor_trace_buf_kb;
 
     struct xen_arch_domainconfig arch;
 };
xen/include/xen/sched.h +3 -0
index ac53519d7f..c046e59886 100644
@@ -457,6 +457,9 @@ struct domain
     unsigned    pbuf_idx;
     spinlock_t  pbuf_lock;
 
+    /* Used by vmtrace features */
+    uint32_t    processor_trace_buf_kb;
+
     /* OProfile support. */
     struct xenoprof *xenoprof;
 
Signature · 1 line
2.17.1
#12 lore raw

Re: [PATCH v6 04/11] common: add vmtrace_pt_size domain parameter

Quoted 74 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add vmtrace_pt_size domain parameter in live domain and vmtrace_pt_order parameter in xen_domctl_createdomain.

---
 xen/arch/x86/domain.c       | 6 ++++++
 xen/common/domain.c         | 9 +++++++++
 xen/include/public/domctl.h | 1 +
 xen/include/xen/sched.h     | 3 +++
 4 files changed, 19 insertions(+)
xen/arch/x86/domain.c +6 -0
index fee6c3931a..b75017b28b 100644
@@ -499,6 +499,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
          */
         config->flags |= XEN_DOMCTL_CDF_oos_off;
 
+    if ( !hvm && config->processor_trace_buf_kb )
+    {
+        dprintk(XENLOG_INFO, "Processor trace is not supported on non-HVM\n");
+        return -EINVAL;
+    }
+
     return 0;
 }
 
xen/common/domain.c +9 -0
index a45cf023f7..e6e8f88da1 100644
@@ -338,6 +338,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->processor_trace_buf_kb && !vmtrace_supported )
+    {
+        dprintk(XENLOG_INFO, "Processor tracing is not supported\n");
+        return -EINVAL;
+    }
+
     return arch_sanitise_domain_config(config);
 }
 
@@ -443,6 +449,9 @@ struct domain *domain_create(domid_t domid,
         d->nr_pirqs = min(d->nr_pirqs, nr_irqs);
 
         radix_tree_init(&d->pirq_tree);
+
+        if ( config->processor_trace_buf_kb )
+            d->processor_trace_buf_kb = config->processor_trace_buf_kb;
     }
 
     if ( (err = arch_domain_create(d, config)) != 0 )
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7681675a94 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint32_t processor_trace_buf_kb;
 
     struct xen_arch_domainconfig arch;
 };
xen/include/xen/sched.h +0 -0
index ac53519d7f..c046e59886 100644

xen/include/xen/sched.h struct domain · line 461

Quoted 5 lines
xen/include/xen/sched.h +2 -0
@@ -457,6 +457,9 @@ struct domain
     unsigned    pbuf_idx;
     spinlock_t  pbuf_lock;
 
+    /* Used by vmtrace features */
+    uint32_t    processor_trace_buf_kb;

Any reason for adding it here instead of doing it at the end of the struct? Also I think this should be unsigned int, there's no reason to use a specific width type here.

IMO it would be nice to have a Kconfig option for this, so that it can be build time disabled, and for example disabled by default on Arm (or on x86 if HVM is not built).

Most recently added features have followed this model for example Argo, where a Kconfig option is added in order to build time disable the added feature. Let me know if you need some tips about it, I'm happy to help in order to try to get this in Kconfig.

Thanks, Roger.

#13 lore raw

Re: [PATCH v6 04/11] common: add vmtrace_pt_size domain parameter

In reply to Michał Leszczyński

xen/common/domain.c struct domain *domain_create(domid_t domid, · line 454

Quoted 7 lines · Michał Leszczyński
xen/common/domain.c +3 -0
@@ -443,6 +449,9 @@ struct domain *domain_create(domid_t domid,
         d->nr_pirqs = min(d->nr_pirqs, nr_irqs);
 
         radix_tree_init(&d->pirq_tree);
+
+        if ( config->processor_trace_buf_kb )
+            d->processor_trace_buf_kb = config->processor_trace_buf_kb;

There's no real need for the if, is there?

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 7 lines
xen/include/public/domctl.h +1 -0
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint32_t processor_trace_buf_kb;

Adding a new field to an existing interface struct requires bumping of the interface version, unless that has already happened during a release cycle.

Here I agree with the choice of uint32_t, but ...

xen/include/xen/sched.h struct domain · line 461

Quoted 8 lines
xen/include/xen/sched.h +2 -0
@@ -457,6 +457,9 @@ struct domain
     unsigned    pbuf_idx;
     spinlock_t  pbuf_lock;
 
+    /* Used by vmtrace features */
+    uint32_t    processor_trace_buf_kb;

... why a fixed size type here? unsigned int will do fine, won't it?

Jan

#14 lore raw

PATCH v6 5/11 tools/libxl: add vmtrace_pt_size parameter

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).

---
 docs/man/xl.cfg.5.pod.in             | 13 +++++++++++++
 tools/golang/xenlight/helpers.gen.go |  2 ++
 tools/golang/xenlight/types.gen.go   |  1 +
 tools/libxl/libxl.h                  |  8 ++++++++
 tools/libxl/libxl_create.c           |  1 +
 tools/libxl/libxl_types.idl          |  4 ++++
 tools/xl/xl_parse.c                  | 22 ++++++++++++++++++++++
 7 files changed, 51 insertions(+)
docs/man/xl.cfg.5.pod.in +13 -0
index 0532739c1f..ddef9b6014 100644
@@ -683,6 +683,19 @@ If this option is not specified then it will default to B<false>.
 
 =back
 
+=item B<processor_trace_buf_kb=KBYTES>
+
+Specifies the size of processor trace buffer that would be allocated
+for each vCPU belonging to this domain. Disabled (i.e.
+B<processor_trace_buf_kb=0> by default. This must be set to
+non-zero value in order to be able to use processor tracing features
+with this domain.
+
+B<NOTE>: In order to use Intel Processor Trace feature, this value
+must be between 8 kB and 4 GB and it must be a power of 2.
+
+=back
+
 =head2 Devices
 
 The following options define the paravirtual, emulated and physical
tools/golang/xenlight/helpers.gen.go +2 -0
index 152c7e8e6b..3ce6f2374b 100644
@@ -1117,6 +1117,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)}
 x.ArchArm.GicVersion = GicVersion(xc.arch_arm.gic_version)
 x.ArchArm.Vuart = VuartType(xc.arch_arm.vuart)
 x.Altp2M = Altp2MMode(xc.altp2m)
+x.ProcessorTraceBufKb = int(xc.processor_trace_buf_kb)
 
  return nil}
 
@@ -1592,6 +1593,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)}
 xc.arch_arm.gic_version = C.libxl_gic_version(x.ArchArm.GicVersion)
 xc.arch_arm.vuart = C.libxl_vuart_type(x.ArchArm.Vuart)
 xc.altp2m = C.libxl_altp2m_mode(x.Altp2M)
+xc.processor_trace_buf_kb = C.int(x.ProcessorTraceBufKb)
 
  return nil
  }
tools/golang/xenlight/types.gen.go +1 -0
index 663c1e86b4..f4bc16c0fd 100644
@@ -516,6 +516,7 @@ GicVersion GicVersion
 Vuart VuartType
 }
 Altp2M Altp2MMode
+ProcessorTraceBufKb int
 }
 
 type domainBuildInfoTypeUnion interface {
tools/libxl/libxl.h +8 -0
index 1cd6c38e83..fbf222967a 100644
@@ -438,6 +438,14 @@
  */
 #define LIBXL_HAVE_CREATEINFO_PASSTHROUGH 1
 
+/*
+ * LIBXL_HAVE_PROCESSOR_TRACE_BUF_KB indicates that
+ * libxl_domain_create_info has a processor_trace_buf_kb parameter, which
+ * allows to enable pre-allocation of processor tracing buffers of given
+ * size.
+ */
+#define LIBXL_HAVE_PROCESSOR_TRACE_BUF_KB 1
+
 /*
  * libxl ABI compatibility
  *
tools/libxl/libxl_create.c +1 -0
index 2814818e34..4d6318124a 100644
@@ -608,6 +608,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
             .max_evtchn_port = b_info->event_channels,
             .max_grant_frames = b_info->max_grant_frames,
             .max_maptrack_frames = b_info->max_maptrack_frames,
+            .processor_trace_buf_kb = b_info->processor_trace_buf_kb,
         };
 
         if (info->type != LIBXL_DOMAIN_TYPE_PV) {
tools/libxl/libxl_types.idl +4 -0
index 9d3f05f399..748fde65ab 100644
@@ -645,6 +645,10 @@ libxl_domain_build_info = Struct("domain_build_info",[
     # supported by x86 HVM and ARM support is planned.
     ("altp2m", libxl_altp2m_mode),
 
+    # Size of preallocated processor trace buffers (in KBYTES).
+    # Use zero value to disable this feature.
+    ("processor_trace_buf_kb", integer),
+
     ], dir=DIR_IN,
        copy_deprecated_fn="libxl__domain_build_info_copy_deprecated",
 )
tools/xl/xl_parse.c +22 -0
index 61b4ef7b7e..87e373b413 100644
@@ -1861,6 +1861,28 @@ void parse_config_data(const char *config_source,
         }
     }
 
+    if (!xlu_cfg_get_long(config, "processor_trace_buf_kb", &l, 1) && l) {
+        if (l & (l - 1)) {
+            fprintf(stderr, "ERROR: processor_trace_buf_kb"
+                            " - must be a power of 2\n");
+            exit(1);
+        }
+
+        if (l < 8) {
+            fprintf(stderr, "ERROR: processor_trace_buf_kb"
+                            " - value is too small\n");
+            exit(1);
+        }
+
+        if (l > 1024*1024*4) {
+            fprintf(stderr, "ERROR: processor_trace_buf_kb"
+                            " - value is too large\n");
+            exit(1);
+        }
+
+        b_info->processor_trace_buf_kb = l;
+    }
+
     if (!xlu_cfg_get_list(config, "ioports", &ioports, &num_ioports, 0)) {
         b_info->num_ioports = num_ioports;
         b_info->ioports = calloc(num_ioports, sizeof(*b_info->ioports));
Signature · 1 line
2.17.1
#15 lore raw

Re: [PATCH v6 05/11] tools/libxl: add vmtrace_pt_size parameter

Quoted 24 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).

---
 docs/man/xl.cfg.5.pod.in             | 13 +++++++++++++
 tools/golang/xenlight/helpers.gen.go |  2 ++
 tools/golang/xenlight/types.gen.go   |  1 +
 tools/libxl/libxl.h                  |  8 ++++++++
 tools/libxl/libxl_create.c           |  1 +
 tools/libxl/libxl_types.idl          |  4 ++++
 tools/xl/xl_parse.c                  | 22 ++++++++++++++++++++++
 7 files changed, 51 insertions(+)
docs/man/xl.cfg.5.pod.in +0 -0
index 0532739c1f..ddef9b6014 100644
@@ -683,6 +683,19 @@ If this option is not specified then it will default to B<false>.
 
 =back
 

docs/man/xl.cfg.5.pod.in If this option is not specified then it will default to B<false>. · line 695

Quoted 10 lines
docs/man/xl.cfg.5.pod.in +10 -0
+=item B<processor_trace_buf_kb=KBYTES>
+
+Specifies the size of processor trace buffer that would be allocated
+for each vCPU belonging to this domain. Disabled (i.e.
+B<processor_trace_buf_kb=0> by default. This must be set to
+non-zero value in order to be able to use processor tracing features
+with this domain.
+
+B<NOTE>: In order to use Intel Processor Trace feature, this value
+must be between 8 kB and 4 GB and it must be a power of 2.

I think the minimum that we could support is 4KB? (ie: one page?). Not that it matters much, as I don't think anyone would use such small buffers.

tools/libxl/libxl_types.idl libxl_domain_build_info = Struct("domain_build_info",[ · line 650

Quoted 11 lines
tools/libxl/libxl_types.idl +3 -0
index 9d3f05f399..748fde65ab 100644
@@ -645,6 +645,10 @@ libxl_domain_build_info = Struct("domain_build_info",[
     # supported by x86 HVM and ARM support is planned.
     ("altp2m", libxl_altp2m_mode),
 
+    # Size of preallocated processor trace buffers (in KBYTES).
+    # Use zero value to disable this feature.
+    ("processor_trace_buf_kb", integer),

MemKB should be used here instead of integer.

Quoted 19 lines
tools/libxl/libxl_types.idl +1 -0
+
     ], dir=DIR_IN,
        copy_deprecated_fn="libxl__domain_build_info_copy_deprecated",
 )
tools/xl/xl_parse.c +7 -0
index 61b4ef7b7e..87e373b413 100644
@@ -1861,6 +1861,28 @@ void parse_config_data(const char *config_source,
         }
     }
 
+    if (!xlu_cfg_get_long(config, "processor_trace_buf_kb", &l, 1) && l) {
+        if (l & (l - 1)) {
+            fprintf(stderr, "ERROR: processor_trace_buf_kb"
+                            " - must be a power of 2\n");
+            exit(1);
+        }
+

tools/xl/xl_parse.c void parse_config_data(const char *config_source, · line 1880

Quoted 10 lines
tools/xl/xl_parse.c +10 -0
+        if (l < 8) {
+            fprintf(stderr, "ERROR: processor_trace_buf_kb"
+                            " - value is too small\n");
+            exit(1);
+        }
+
+        if (l > 1024*1024*4) {
+            fprintf(stderr, "ERROR: processor_trace_buf_kb"
+                            " - value is too large\n");
+            exit(1);

Those checks shouldn't be here, this is libxl common code, and those limitations are specific to the Intel implementation. Those should be inside of the hypervisor IMO, or if we really want to have them in libxl for some reason they should be moved to libxl_x86.c

Thanks.

#16 lore raw

PATCH v6 6/11 x86/hvm: processor trace interface in HVM

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Implement necessary changes in common code/HVM to support processor trace features. Define vmtrace_pt_* API and implement trace buffer allocation/deallocation in common code.

---
 xen/arch/x86/domain.c         | 21 +++++++++++++++++++++
 xen/common/domain.c           | 35 +++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/hvm.h | 20 ++++++++++++++++++++
 xen/include/xen/sched.h       |  4 ++++
 4 files changed, 80 insertions(+)
xen/arch/x86/domain.c +21 -0
index b75017b28b..8ce2ab6b8f 100644
@@ -2205,6 +2205,27 @@ int domain_relinquish_resources(struct domain *d)
                 altp2m_vcpu_disable_ve(v);
         }
 
+        for_each_vcpu ( d, v )
+        {
+            unsigned int i;
+            uint64_t nr_pages = v->domain->processor_trace_buf_kb * KB(1);
+            nr_pages >>= PAGE_SHIFT;
+
+            if ( !v->vmtrace.pt_buf )
+                continue;
+
+            for ( i = 0; i < nr_pages; i++ )
+            {
+                struct page_info *pg = mfn_to_page(
+                    mfn_add(page_to_mfn(v->vmtrace.pt_buf), i));
+
+                put_page_alloc_ref(pg);
+                put_page_and_type(pg);
+            }
+
+            v->vmtrace.pt_buf = NULL;
+        }
+
         if ( is_pv_domain(d) )
         {
             for_each_vcpu ( d, v )
xen/common/domain.c +35 -0
index e6e8f88da1..193099a2ab 100644
@@ -137,6 +137,38 @@ static void vcpu_destroy(struct vcpu *v)
     free_vcpu_struct(v);
 }
 
+static int vmtrace_alloc_buffers(struct vcpu *v)
+{
+    unsigned int i;
+    struct page_info *pg;
+    uint64_t size = v->domain->processor_trace_buf_kb * KB(1);
+
+    pg = alloc_domheap_pages(v->domain, get_order_from_bytes(size),
+                             MEMF_no_refcount);
+
+    if ( !pg )
+        return -ENOMEM;
+
+    for ( i = 0; i < (size >> PAGE_SHIFT); i++ )
+    {
+        struct page_info *pg_iter = mfn_to_page(
+            mfn_add(page_to_mfn(pg), i));
+
+        if ( !get_page_and_type(pg_iter, v->domain, PGT_writable_page) )
+        {
+            /*
+             * The domain can't possibly know about this page yet, so failure
+             * here is a clear indication of something fishy going on.
+             */
+            domain_crash(v->domain);
+            return -ENODATA;
+        }
+    }
+
+    v->vmtrace.pt_buf = pg;
+    return 0;
+}
+
 struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
 {
     struct vcpu *v;
@@ -162,6 +194,9 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
     v->vcpu_id = vcpu_id;
     v->dirty_cpu = VCPU_CPU_CLEAN;
 
+    if ( d->processor_trace_buf_kb && vmtrace_alloc_buffers(v) != 0 )
+        return NULL;
+
     spin_lock_init(&v->virq_lock);
 
     tasklet_init(&v->continue_hypercall_tasklet, NULL, NULL);
xen/include/asm-x86/hvm/hvm.h +20 -0
index 1eb377dd82..476a216205 100644
@@ -214,6 +214,10 @@ struct hvm_function_table {
     bool_t (*altp2m_vcpu_emulate_ve)(struct vcpu *v);
     int (*altp2m_vcpu_emulate_vmfunc)(const struct cpu_user_regs *regs);
 
+    /* vmtrace */
+    int (*vmtrace_control_pt)(struct vcpu *v, bool enable);
+    int (*vmtrace_get_pt_offset)(struct vcpu *v, uint64_t *offset, uint64_t *size);
+
     /*
      * Parameters and callbacks for hardware-assisted TSC scaling,
      * which are valid only when the hardware feature is available.
@@ -655,6 +659,22 @@ static inline bool altp2m_vcpu_emulate_ve(struct vcpu *v)
     return false;
 }
 
+static inline int vmtrace_control_pt(struct vcpu *v, bool enable)
+{
+    if ( hvm_funcs.vmtrace_control_pt )
+        return hvm_funcs.vmtrace_control_pt(v, enable);
+
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_get_pt_offset(struct vcpu *v, uint64_t *offset, uint64_t *size)
+{
+    if ( hvm_funcs.vmtrace_get_pt_offset )
+        return hvm_funcs.vmtrace_get_pt_offset(v, offset, size);
+
+    return -EOPNOTSUPP;
+}
+
 /*
  * This must be defined as a macro instead of an inline function,
  * because it uses 'struct vcpu' and 'struct domain' which have
xen/include/xen/sched.h +4 -0
index c046e59886..b6f39233aa 100644
@@ -253,6 +253,10 @@ struct vcpu
     /* vPCI per-vCPU area, used to store data for long running operations. */
     struct vpci_vcpu vpci;
 
+    struct {
+        struct page_info *pt_buf;
+    } vmtrace;
+
     struct arch_vcpu arch;
 };
 
Signature · 1 line
2.17.1
#17 lore raw

Re: [PATCH v6 06/11] x86/hvm: processor trace interface in HVM

Quoted 20 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Implement necessary changes in common code/HVM to support processor trace features. Define vmtrace_pt_* API and implement trace buffer allocation/deallocation in common code.

---
 xen/arch/x86/domain.c         | 21 +++++++++++++++++++++
 xen/common/domain.c           | 35 +++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/hvm.h | 20 ++++++++++++++++++++
 xen/include/xen/sched.h       |  4 ++++
 4 files changed, 80 insertions(+)
xen/arch/x86/domain.c +0 -0
index b75017b28b..8ce2ab6b8f 100644

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2212

Quoted 8 lines
xen/arch/x86/domain.c +5 -0
@@ -2205,6 +2205,27 @@ int domain_relinquish_resources(struct domain *d)
                 altp2m_vcpu_disable_ve(v);
         }
 
+        for_each_vcpu ( d, v )
+        {
+            unsigned int i;
+            uint64_t nr_pages = v->domain->processor_trace_buf_kb * KB(1);
+            nr_pages >>= PAGE_SHIFT;

It would be easier as:

unsigned int nr_pages = d->processor_trace_buf_kb / KB(4);

Or maybe:

unsigned int nr_pages = d->processor_trace_buf_kb >> (PAGE_SHIFT - 10);

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2220

Quoted 8 lines
xen/arch/x86/domain.c +8 -0
+
+            if ( !v->vmtrace.pt_buf )
+                continue;
+
+            for ( i = 0; i < nr_pages; i++ )
+            {
+                struct page_info *pg = mfn_to_page(
+                    mfn_add(page_to_mfn(v->vmtrace.pt_buf), i));

You can just do:

struct page_info *pg = v->vmtrace.pt_buf[i];

Quoted 16 lines
xen/arch/x86/domain.c +8 -0
+
+                put_page_alloc_ref(pg);
+                put_page_and_type(pg);
+            }
+
+            v->vmtrace.pt_buf = NULL;
+        }
+
         if ( is_pv_domain(d) )
         {
             for_each_vcpu ( d, v )
xen/common/domain.c +0 -0
index e6e8f88da1..193099a2ab 100644

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 144

Quoted 8 lines
xen/common/domain.c +5 -0
@@ -137,6 +137,38 @@ static void vcpu_destroy(struct vcpu *v)
     free_vcpu_struct(v);
 }
 
+static int vmtrace_alloc_buffers(struct vcpu *v)
+{
+    unsigned int i;
+    struct page_info *pg;
+    uint64_t size = v->domain->processor_trace_buf_kb * KB(1);

Same here, you could just use a number of pages directly and turn size into 'unsigned int nr_pages', and then use get_order_from_pages below.

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 148

Quoted 4 lines
xen/common/domain.c +4 -0
+
+    pg = alloc_domheap_pages(v->domain, get_order_from_bytes(size),
+                             MEMF_no_refcount);
+

Extra newline.

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 155

Quoted 7 lines
xen/common/domain.c +7 -0
+    if ( !pg )
+        return -ENOMEM;
+
+    for ( i = 0; i < (size >> PAGE_SHIFT); i++ )
+    {
+        struct page_info *pg_iter = mfn_to_page(
+            mfn_add(page_to_mfn(pg), i));

Same as above here, just use pg[i],

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 164

Quoted 9 lines
xen/common/domain.c +9 -0
+
+        if ( !get_page_and_type(pg_iter, v->domain, PGT_writable_page) )
+        {
+            /*
+             * The domain can't possibly know about this page yet, so failure
+             * here is a clear indication of something fishy going on.
+             */
+            domain_crash(v->domain);
+            return -ENODATA;

ENODATA is IMO a weird return code, ENOMEM would likely be better.

What about the pg array of pages, don't you need to free it somehow? (and likely drop the references to pages before pg[i] on the array)

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 166

Quoted 2 lines
xen/common/domain.c +2 -0
+        }
+    }

Also you seem to assume that size is a power of 2, but I think that's only guaranteed by the current Intel implementation, and hence other implementations could have a more lax requirement (or even Intel when using TOPA).

So you need to free the remaining pages if (1 << get_order_from_pages(nr_pages)) != nr_pages.

Quoted 9 lines
xen/common/domain.c +5 -0
+
+    v->vmtrace.pt_buf = pg;
+    return 0;
+}
+
 struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
 {
     struct vcpu *v;

xen/common/domain.c struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id) · line 198

Quoted 5 lines
xen/common/domain.c +2 -0
@@ -162,6 +194,9 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
     v->vcpu_id = vcpu_id;
     v->dirty_cpu = VCPU_CPU_CLEAN;
 
+    if ( d->processor_trace_buf_kb && vmtrace_alloc_buffers(v) != 0 )
+        return NULL;

Don't you need to do some cleanup here in case of failure? AFAICT this seems to leak the allocated v at least.

Quoted 27 lines
xen/common/domain.c +0 -0
     spin_lock_init(&v->virq_lock);
 
     tasklet_init(&v->continue_hypercall_tasklet, NULL, NULL);
xen/include/asm-x86/hvm/hvm.h +9 -0
index 1eb377dd82..476a216205 100644
@@ -214,6 +214,10 @@ struct hvm_function_table {
     bool_t (*altp2m_vcpu_emulate_ve)(struct vcpu *v);
     int (*altp2m_vcpu_emulate_vmfunc)(const struct cpu_user_regs *regs);
 
+    /* vmtrace */
+    int (*vmtrace_control_pt)(struct vcpu *v, bool enable);
+    int (*vmtrace_get_pt_offset)(struct vcpu *v, uint64_t *offset, uint64_t *size);
+
     /*
      * Parameters and callbacks for hardware-assisted TSC scaling,
      * which are valid only when the hardware feature is available.
@@ -655,6 +659,22 @@ static inline bool altp2m_vcpu_emulate_ve(struct vcpu *v)
     return false;
 }
 
+static inline int vmtrace_control_pt(struct vcpu *v, bool enable)
+{
+    if ( hvm_funcs.vmtrace_control_pt )
+        return hvm_funcs.vmtrace_control_pt(v, enable);
+

xen/include/asm-x86/hvm/hvm.h static inline bool altp2m_vcpu_emulate_ve(struct vcpu *v) · line 676

Quoted 10 lines
xen/include/asm-x86/hvm/hvm.h +10 -0
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_get_pt_offset(struct vcpu *v, uint64_t *offset, uint64_t *size)
+{
+    if ( hvm_funcs.vmtrace_get_pt_offset )
+        return hvm_funcs.vmtrace_get_pt_offset(v, offset, size);
+
+    return -EOPNOTSUPP;
+}

I think this API would be better placed together with the VMX implementation of those functions, introducing it in this patch is not required since there are no callers?

Thanks.

#18 lore raw

Re: [PATCH v6 06/11] x86/hvm: processor trace interface in HVM

In reply to Michał Leszczyński
Quoted 16 lines · Michał Leszczyński
xen/arch/x86/domain.c +10 -0
@@ -2205,6 +2205,27 @@ int domain_relinquish_resources(struct domain *d)
                 altp2m_vcpu_disable_ve(v);
         }
 
+        for_each_vcpu ( d, v )
+        {
+            unsigned int i;
+            uint64_t nr_pages = v->domain->processor_trace_buf_kb * KB(1);
+            nr_pages >>= PAGE_SHIFT;
+
+            if ( !v->vmtrace.pt_buf )
+                continue;
+
+            for ( i = 0; i < nr_pages; i++ )

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2227

Quoted 10 lines
xen/arch/x86/domain.c +10 -0
+            {
+                struct page_info *pg = mfn_to_page(
+                    mfn_add(page_to_mfn(v->vmtrace.pt_buf), i));
+
+                put_page_alloc_ref(pg);
+                put_page_and_type(pg);
+            }
+
+            v->vmtrace.pt_buf = NULL;
+        }

This needs to allow for preemption. Also this isn't x86-specific, so should be implemented in common code (just like allocation is).

Jan

#19 lore raw

PATCH v6 7/11 x86/vmx: implement IPT in VMX

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Use Intel Processor Trace feature to provide vmtrace_pt_* interface for HVM/VMX.

---
 xen/arch/x86/hvm/vmx/vmx.c         | 110 +++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/vmx/vmcs.h |   3 +
 xen/include/asm-x86/hvm/vmx/vmx.h  |  14 ++++
 3 files changed, 127 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +110 -0
index cc6d4ece22..63a5a76e16 100644
@@ -428,6 +428,56 @@ static void vmx_domain_relinquish_resources(struct domain *d)
     vmx_free_vlapic_mapping(d);
 }
 
+static int vmx_init_pt(struct vcpu *v)
+{
+    int rc;
+    uint64_t size = v->domain->processor_trace_buf_kb * KB(1);
+
+    if ( !v->vmtrace.pt_buf || !size )
+        return -EINVAL;
+
+    /*
+     * We don't accept trace buffer size smaller than single page
+     * and the upper bound is defined as 4GB in the specification.
+     * The buffer size must be also a power of 2.
+     */
+    if ( size < PAGE_SIZE || size > GB(4) || (size & (size - 1)) )
+        return -EINVAL;
+
+    v->arch.hvm.vmx.ipt_state = xzalloc(struct ipt_state);
+
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -ENOMEM;
+
+    v->arch.hvm.vmx.ipt_state->output_base =
+        page_to_maddr(v->vmtrace.pt_buf);
+    v->arch.hvm.vmx.ipt_state->output_mask.raw = size - 1;
+
+    rc = vmx_add_host_load_msr(v, MSR_RTIT_CTL, 0);
+
+    if ( rc )
+        return rc;
+
+    rc = vmx_add_guest_msr(v, MSR_RTIT_CTL,
+                              RTIT_CTL_TRACE_EN | RTIT_CTL_OS |
+                              RTIT_CTL_USR | RTIT_CTL_BRANCH_EN);
+
+    if ( rc )
+        return rc;
+
+    return 0;
+}
+
+static int vmx_destroy_pt(struct vcpu* v)
+{
+    if ( v->arch.hvm.vmx.ipt_state )
+        xfree(v->arch.hvm.vmx.ipt_state);
+
+    v->arch.hvm.vmx.ipt_state = NULL;
+    return 0;
+}
+
+
 static int vmx_vcpu_initialise(struct vcpu *v)
 {
     int rc;
@@ -471,6 +521,14 @@ static int vmx_vcpu_initialise(struct vcpu *v)
 
     vmx_install_vlapic_mapping(v);
 
+    if ( v->domain->processor_trace_buf_kb )
+    {
+        rc = vmx_init_pt(v);
+
+        if ( rc )
+            return rc;
+    }
+
     return 0;
 }
 
@@ -483,6 +541,7 @@ static void vmx_vcpu_destroy(struct vcpu *v)
      * prior to vmx_domain_destroy so we need to disable PML for each vcpu
      * separately here.
      */
+    vmx_destroy_pt(v);
     vmx_vcpu_disable_pml(v);
     vmx_destroy_vmcs(v);
     passive_domain_destroy(v);
@@ -513,6 +572,18 @@ static void vmx_save_guest_msrs(struct vcpu *v)
      * be updated at any time via SWAPGS, which we cannot trap.
      */
     v->arch.hvm.vmx.shadow_gs = rdgsshadow();
+
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )
+    {
+        uint64_t rtit_ctl;
+        rdmsrl(MSR_RTIT_CTL, rtit_ctl);
+        BUG_ON(rtit_ctl & RTIT_CTL_TRACE_EN);
+
+        rdmsrl(MSR_RTIT_STATUS, v->arch.hvm.vmx.ipt_state->status);
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.ipt_state->output_mask.raw);
+    }
 }
 
 static void vmx_restore_guest_msrs(struct vcpu *v)
@@ -524,6 +595,17 @@ static void vmx_restore_guest_msrs(struct vcpu *v)
 
     if ( cpu_has_msr_tsc_aux )
         wrmsr_tsc_aux(v->arch.msrs->tsc_aux);
+
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )
+    {
+        wrmsrl(MSR_RTIT_OUTPUT_BASE,
+               v->arch.hvm.vmx.ipt_state->output_base);
+        wrmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.ipt_state->output_mask.raw);
+        wrmsrl(MSR_RTIT_STATUS,
+               v->arch.hvm.vmx.ipt_state->status);
+    }
 }
 
 void vmx_update_cpu_exec_control(struct vcpu *v)
@@ -2240,6 +2322,25 @@ static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info)
     return true;
 }
 
+static int vmx_control_pt(struct vcpu *v, bool enable)
+{
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -EINVAL;
+
+    v->arch.hvm.vmx.ipt_state->active = enable;
+    return 0;
+}
+
+static int vmx_get_pt_offset(struct vcpu *v, uint64_t *offset, uint64_t *size)
+{
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -EINVAL;
+
+    *offset = v->arch.hvm.vmx.ipt_state->output_mask.offset;
+    *size = v->arch.hvm.vmx.ipt_state->output_mask.size + 1;
+    return 0;
+}
+
 static struct hvm_function_table __initdata vmx_function_table = {
     .name                 = "VMX",
     .cpu_up_prepare       = vmx_cpu_up_prepare,
@@ -2295,6 +2396,8 @@ static struct hvm_function_table __initdata vmx_function_table = {
     .altp2m_vcpu_update_vmfunc_ve = vmx_vcpu_update_vmfunc_ve,
     .altp2m_vcpu_emulate_ve = vmx_vcpu_emulate_ve,
     .altp2m_vcpu_emulate_vmfunc = vmx_vcpu_emulate_vmfunc,
+    .vmtrace_control_pt = vmx_control_pt,
+    .vmtrace_get_pt_offset = vmx_get_pt_offset,
     .tsc_scaling = {
         .max_ratio = VMX_TSC_MULTIPLIER_MAX,
     },
@@ -3674,6 +3777,13 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
 
     hvm_invalidate_regs_fields(regs);
 
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )
+    {
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.ipt_state->output_mask.raw);
+    }
+
     if ( paging_mode_hap(v->domain) )
     {
         /*
xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0
index 6153ba6769..65971fa6ad 100644
@@ -186,6 +186,9 @@ struct vmx_vcpu {
      * pCPU and wakeup the related vCPU.
      */
     struct pi_blocking_vcpu pi_blocking;
+
+    /* State of processor trace feature */
+    struct ipt_state      *ipt_state;
 };
 
 int vmx_create_vmcs(struct vcpu *v);
xen/include/asm-x86/hvm/vmx/vmx.h +14 -0
index 111ccd7e61..8d7c67e43d 100644
@@ -689,4 +689,18 @@ typedef union ldt_or_tr_instr_info {
     };
 } ldt_or_tr_instr_info_t;
 
+/* Processor Trace state per vCPU */
+struct ipt_state {
+    bool active;
+    uint64_t status;
+    uint64_t output_base;
+    union {
+        uint64_t raw;
+        struct {
+            uint32_t size;
+            uint32_t offset;
+        };
+    } output_mask;
+};
+
 #endif /* __ASM_X86_HVM_VMX_VMX_H__ */
Signature · 1 line
2.17.1
#20 lore raw

Re: [PATCH v6 07/11] x86/vmx: implement IPT in VMX

Quoted 17 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Use Intel Processor Trace feature to provide vmtrace_pt_* interface for HVM/VMX.

---
 xen/arch/x86/hvm/vmx/vmx.c         | 110 +++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/vmx/vmcs.h |   3 +
 xen/include/asm-x86/hvm/vmx/vmx.h  |  14 ++++
 3 files changed, 127 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index cc6d4ece22..63a5a76e16 100644

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 434

Quoted 7 lines
xen/arch/x86/hvm/vmx/vmx.c +4 -0
@@ -428,6 +428,56 @@ static void vmx_domain_relinquish_resources(struct domain *d)
     vmx_free_vlapic_mapping(d);
 }
 
+static int vmx_init_pt(struct vcpu *v)
+{
+    int rc;
+    uint64_t size = v->domain->processor_trace_buf_kb * KB(1);

As I commented in other patches, I don't think there's a need to have the size in bytes, and hence you could just convert to number of pages?

You might have to check that the value is rounded to a page boundary.

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 445

Quoted 11 lines
xen/arch/x86/hvm/vmx/vmx.c +11 -0
+
+    if ( !v->vmtrace.pt_buf || !size )
+        return -EINVAL;
+
+    /*
+     * We don't accept trace buffer size smaller than single page
+     * and the upper bound is defined as 4GB in the specification.
+     * The buffer size must be also a power of 2.
+     */
+    if ( size < PAGE_SIZE || size > GB(4) || (size & (size - 1)) )
+        return -EINVAL;

IMO there should be a hook to sanitize the buffer size before you go and allocate it. It makes no sense to allocate a buffer to come here and realize it's not suitable.

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 448

Quoted 3 lines
xen/arch/x86/hvm/vmx/vmx.c +3 -0
+
+    v->arch.hvm.vmx.ipt_state = xzalloc(struct ipt_state);
+

Extra newline.

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 453

Quoted 5 lines
xen/arch/x86/hvm/vmx/vmx.c +5 -0
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -ENOMEM;
+
+    v->arch.hvm.vmx.ipt_state->output_base =
+        page_to_maddr(v->vmtrace.pt_buf);

The above fits on a single line now. You could also avoid having an output_base field and just do the conversion in vmx_restore_guest_msrs, I'm not sure there's much value in having this cached here.

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 466

Quoted 13 lines
xen/arch/x86/hvm/vmx/vmx.c +13 -0
+    v->arch.hvm.vmx.ipt_state->output_mask.raw = size - 1;
+
+    rc = vmx_add_host_load_msr(v, MSR_RTIT_CTL, 0);
+
+    if ( rc )
+        return rc;
+
+    rc = vmx_add_guest_msr(v, MSR_RTIT_CTL,
+                              RTIT_CTL_TRACE_EN | RTIT_CTL_OS |
+                              RTIT_CTL_USR | RTIT_CTL_BRANCH_EN);
+
+    if ( rc )
+        return rc;

We don't usually leave an empty line between setting and testing rc.

Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c +4 -0
+
+    return 0;
+}
+

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 480

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +10 -0
+static int vmx_destroy_pt(struct vcpu* v)
+{
+    if ( v->arch.hvm.vmx.ipt_state )
+        xfree(v->arch.hvm.vmx.ipt_state);
+
+    v->arch.hvm.vmx.ipt_state = NULL;
+    return 0;
+}
+
+

Double newline, just one newline please between functions.

xen/arch/x86/hvm/vmx/vmx.c static int vmx_vcpu_initialise(struct vcpu *v) · line 524

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c +1 -0
 static int vmx_vcpu_initialise(struct vcpu *v)
 {
     int rc;
@@ -471,6 +521,14 @@ static int vmx_vcpu_initialise(struct vcpu *v)
 
     vmx_install_vlapic_mapping(v);
 
+    if ( v->domain->processor_trace_buf_kb )

Can you move this check inside of vmx_init_pt, so that here you just do:

return vmx_init_pt(v);

Quoted 19 lines
xen/arch/x86/hvm/vmx/vmx.c +8 -0
+    {
+        rc = vmx_init_pt(v);
+
+        if ( rc )
+            return rc;
+    }
+
     return 0;
 }
 
@@ -483,6 +541,7 @@ static void vmx_vcpu_destroy(struct vcpu *v)
      * prior to vmx_domain_destroy so we need to disable PML for each vcpu
      * separately here.
      */
+    vmx_destroy_pt(v);
     vmx_vcpu_disable_pml(v);
     vmx_destroy_vmcs(v);
     passive_domain_destroy(v);

xen/arch/x86/hvm/vmx/vmx.c static void vmx_save_guest_msrs(struct vcpu *v) · line 579

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmx.c +5 -0
@@ -513,6 +572,18 @@ static void vmx_save_guest_msrs(struct vcpu *v)
      * be updated at any time via SWAPGS, which we cannot trap.
      */
     v->arch.hvm.vmx.shadow_gs = rdgsshadow();
+
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )
+    {
+        uint64_t rtit_ctl;

Missing newline.

Quoted 29 lines
xen/arch/x86/hvm/vmx/vmx.c +18 -0
+        rdmsrl(MSR_RTIT_CTL, rtit_ctl);
+        BUG_ON(rtit_ctl & RTIT_CTL_TRACE_EN);
+
+        rdmsrl(MSR_RTIT_STATUS, v->arch.hvm.vmx.ipt_state->status);
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.ipt_state->output_mask.raw);
+    }
 }
 
 static void vmx_restore_guest_msrs(struct vcpu *v)
@@ -524,6 +595,17 @@ static void vmx_restore_guest_msrs(struct vcpu *v)
 
     if ( cpu_has_msr_tsc_aux )
         wrmsr_tsc_aux(v->arch.msrs->tsc_aux);
+
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )
+    {
+        wrmsrl(MSR_RTIT_OUTPUT_BASE,
+               v->arch.hvm.vmx.ipt_state->output_base);
+        wrmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.ipt_state->output_mask.raw);
+        wrmsrl(MSR_RTIT_STATUS,
+               v->arch.hvm.vmx.ipt_state->status);
+    }
 }
 
 void vmx_update_cpu_exec_control(struct vcpu *v)

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2330

Quoted 9 lines
xen/arch/x86/hvm/vmx/vmx.c +6 -0
@@ -2240,6 +2322,25 @@ static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info)
     return true;
 }
 
+static int vmx_control_pt(struct vcpu *v, bool enable)
+{
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -EINVAL;
+
+    v->arch.hvm.vmx.ipt_state->active = enable;

I think you should assert that the vCPU is paused? As doing this on a non-paused vCPU is not going to work reliably?

Quoted 26 lines
xen/arch/x86/hvm/vmx/vmx.c +15 -0
+    return 0;
+}
+
+static int vmx_get_pt_offset(struct vcpu *v, uint64_t *offset, uint64_t *size)
+{
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -EINVAL;
+
+    *offset = v->arch.hvm.vmx.ipt_state->output_mask.offset;
+    *size = v->arch.hvm.vmx.ipt_state->output_mask.size + 1;
+    return 0;
+}
+
 static struct hvm_function_table __initdata vmx_function_table = {
     .name                 = "VMX",
     .cpu_up_prepare       = vmx_cpu_up_prepare,
@@ -2295,6 +2396,8 @@ static struct hvm_function_table __initdata vmx_function_table = {
     .altp2m_vcpu_update_vmfunc_ve = vmx_vcpu_update_vmfunc_ve,
     .altp2m_vcpu_emulate_ve = vmx_vcpu_emulate_ve,
     .altp2m_vcpu_emulate_vmfunc = vmx_vcpu_emulate_vmfunc,
+    .vmtrace_control_pt = vmx_control_pt,
+    .vmtrace_get_pt_offset = vmx_get_pt_offset,
     .tsc_scaling = {
         .max_ratio = VMX_TSC_MULTIPLIER_MAX,
     },

xen/arch/x86/hvm/vmx/vmx.c void vmx_vmexit_handler(struct cpu_user_regs *regs) · line 3785

Quoted 9 lines
xen/arch/x86/hvm/vmx/vmx.c +6 -0
@@ -3674,6 +3777,13 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
 
     hvm_invalidate_regs_fields(regs);
 
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )
+    {
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.ipt_state->output_mask.raw);
+    }

Unneeded braces.

Thanks.

#21 lore raw

Re: [PATCH v6 07/11] x86/vmx: implement IPT in VMX

In reply to Michał Leszczyński
Quoted 29 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmx.c +23 -0
@@ -428,6 +428,56 @@ static void vmx_domain_relinquish_resources(struct domain *d)
     vmx_free_vlapic_mapping(d);
 }
 
+static int vmx_init_pt(struct vcpu *v)
+{
+    int rc;
+    uint64_t size = v->domain->processor_trace_buf_kb * KB(1);
+
+    if ( !v->vmtrace.pt_buf || !size )
+        return -EINVAL;
+
+    /*
+     * We don't accept trace buffer size smaller than single page
+     * and the upper bound is defined as 4GB in the specification.
+     * The buffer size must be also a power of 2.
+     */
+    if ( size < PAGE_SIZE || size > GB(4) || (size & (size - 1)) )
+        return -EINVAL;
+
+    v->arch.hvm.vmx.ipt_state = xzalloc(struct ipt_state);
+
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -ENOMEM;
+
+    v->arch.hvm.vmx.ipt_state->output_base =
+        page_to_maddr(v->vmtrace.pt_buf);

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 463

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +10 -0
+    v->arch.hvm.vmx.ipt_state->output_mask.raw = size - 1;
+
+    rc = vmx_add_host_load_msr(v, MSR_RTIT_CTL, 0);
+
+    if ( rc )
+        return rc;
+
+    rc = vmx_add_guest_msr(v, MSR_RTIT_CTL,
+                              RTIT_CTL_TRACE_EN | RTIT_CTL_OS |
+                              RTIT_CTL_USR | RTIT_CTL_BRANCH_EN);

Indentation is off by three here, and ...

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 469

Quoted 5 lines
xen/arch/x86/hvm/vmx/vmx.c +5 -0
+    if ( rc )
+        return rc;
+
+    return 0;
+}

... this whole thing would be shorter (and hence easier to read) as

return vmx_add_guest_msr(v, MSR_RTIT_CTL,
RTIT_CTL_TRACE_EN | RTIT_CTL_OS | RTIT_CTL_USR | RTIT_CTL_BRANCH_EN);

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 474

Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c +4 -0
+static int vmx_destroy_pt(struct vcpu* v)
+{
+    if ( v->arch.hvm.vmx.ipt_state )
+        xfree(v->arch.hvm.vmx.ipt_state);

No need for the if().

xen/arch/x86/hvm/vmx/vmx.c static void vmx_domain_relinquish_resources(struct domain *d) · line 476

Quoted 1 line
xen/arch/x86/hvm/vmx/vmx.c +1 -0
+    v->arch.hvm.vmx.ipt_state = NULL;

And everything together is actually simply
"XFREE(v->arch.hvm.vmx.ipt_state);".

Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c +0 -0
@@ -471,6 +521,14 @@ static int vmx_vcpu_initialise(struct vcpu *v)
 
     vmx_install_vlapic_mapping(v);
 

xen/arch/x86/hvm/vmx/vmx.c static int vmx_vcpu_initialise(struct vcpu *v) · line 533

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +8 -0
+    if ( v->domain->processor_trace_buf_kb )
+    {
+        rc = vmx_init_pt(v);
+
+        if ( rc )
+            return rc;
+    }
+
     return 0;
 }

Is there no cleanup you need to do in case of failure? The caller will invoke vmx_vcpu_destroy() only for failures subsequent to one coming from here.

xen/arch/x86/hvm/vmx/vmx.c static void vmx_save_guest_msrs(struct vcpu *v) · line 577

Quoted 7 lines
xen/arch/x86/hvm/vmx/vmx.c +3 -0
@@ -513,6 +572,18 @@ static void vmx_save_guest_msrs(struct vcpu *v)
      * be updated at any time via SWAPGS, which we cannot trap.
      */
     v->arch.hvm.vmx.shadow_gs = rdgsshadow();
+
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )

likely() / unlikely(), for being efficient, don't want && or || in their expressions. Please limit to just the left side of &&.

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2330

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +6 -0
@@ -2240,6 +2322,25 @@ static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info)
     return true;
 }
 
+static int vmx_control_pt(struct vcpu *v, bool enable)
+{
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -EINVAL;
+
+    v->arch.hvm.vmx.ipt_state->active = enable;

Peeking ahead into patch 9, the vCPU is paused at this point. Please add a respective ASSERT(), if only for documentation purposes.

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2340

Quoted 7 lines
xen/arch/x86/hvm/vmx/vmx.c +7 -0
+static int vmx_get_pt_offset(struct vcpu *v, uint64_t *offset, uint64_t *size)
+{
+    if ( !v->arch.hvm.vmx.ipt_state )
+        return -EINVAL;
+
+    *offset = v->arch.hvm.vmx.ipt_state->output_mask.offset;
+    *size = v->arch.hvm.vmx.ipt_state->output_mask.size + 1;

Either the function parameter or the struct field is misnamed, or else there shouldn't be an addition of 1 here.

xen/arch/x86/hvm/vmx/vmx.c static struct hvm_function_table __initdata vmx_function_table = { · line 2400

Quoted 6 lines
xen/arch/x86/hvm/vmx/vmx.c +2 -0
@@ -2295,6 +2396,8 @@ static struct hvm_function_table __initdata vmx_function_table = {
     .altp2m_vcpu_update_vmfunc_ve = vmx_vcpu_update_vmfunc_ve,
     .altp2m_vcpu_emulate_ve = vmx_vcpu_emulate_ve,
     .altp2m_vcpu_emulate_vmfunc = vmx_vcpu_emulate_vmfunc,
+    .vmtrace_control_pt = vmx_control_pt,
+    .vmtrace_get_pt_offset = vmx_get_pt_offset,

Better install these hooks only if the underlying feature is available (like we do for several other hooks)?

xen/arch/x86/hvm/vmx/vmx.c void vmx_vmexit_handler(struct cpu_user_regs *regs) · line 3784

Quoted 9 lines
xen/arch/x86/hvm/vmx/vmx.c +5 -0
@@ -3674,6 +3777,13 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
 
     hvm_invalidate_regs_fields(regs);
 
+    if ( unlikely(v->arch.hvm.vmx.ipt_state &&
+                  v->arch.hvm.vmx.ipt_state->active) )
+    {
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.ipt_state->output_mask.raw);

Don't you also need to latch RTIT_STATUS?

xen/include/asm-x86/hvm/vmx/vmx.h typedef union ldt_or_tr_instr_info { · line 696

Quoted 11 lines
xen/include/asm-x86/hvm/vmx/vmx.h +5 -0
@@ -689,4 +689,18 @@ typedef union ldt_or_tr_instr_info {
     };
 } ldt_or_tr_instr_info_t;
 
+/* Processor Trace state per vCPU */
+struct ipt_state {
+    bool active;
+    uint64_t status;
+    uint64_t output_base;

maddr_t according to its use.

Jan

#22 lore raw

PATCH v6 8/11 x86/mm: add vmtrace_buf resource type

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to map processor trace buffer using
acquire_resource().

---
 xen/common/memory.c         | 31 +++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  1 +
 2 files changed, 32 insertions(+)
xen/common/memory.c +31 -0
index eb42f883df..c0a22eb60f 100644
@@ -1007,6 +1007,32 @@ static long xatp_permission_check(struct domain *d, unsigned int space)
     return xsm_add_to_physmap(XSM_TARGET, current->domain, d);
 }
 
+static int acquire_vmtrace_buf(struct domain *d, unsigned int id,
+                               uint64_t frame,
+                               uint64_t nr_frames,
+                               xen_pfn_t mfn_list[])
+{
+    mfn_t mfn;
+    unsigned int i;
+    uint64_t size;
+    struct vcpu *v = domain_vcpu(d, id);
+
+    if ( !v || !v->vmtrace.pt_buf )
+        return -EINVAL;
+
+    mfn = page_to_mfn(v->vmtrace.pt_buf);
+    size = v->domain->processor_trace_buf_kb * KB(1);
+
+    if ( (frame > (size >> PAGE_SHIFT)) ||
+         (nr_frames > ((size >> PAGE_SHIFT) - frame)) )
+        return -EINVAL;
+
+    for ( i = 0; i < nr_frames; i++ )
+        mfn_list[i] = mfn_x(mfn_add(mfn, frame + i));
+
+    return 0;
+}
+
 static int acquire_grant_table(struct domain *d, unsigned int id,
                                unsigned long frame,
                                unsigned int nr_frames,
@@ -1117,6 +1143,11 @@ static int acquire_resource(
                                  mfn_list);
         break;
 
+    case XENMEM_resource_vmtrace_buf:
+        rc = acquire_vmtrace_buf(d, xmar.id, xmar.frame, xmar.nr_frames,
+                                 mfn_list);
+        break;
+
     default:
         rc = arch_acquire_resource(d, xmar.type, xmar.id, xmar.frame,
                                    xmar.nr_frames, mfn_list);
xen/include/public/memory.h +1 -0
index 21057ed78e..f4c905a10e 100644
@@ -625,6 +625,7 @@ struct xen_mem_acquire_resource {
 
 #define XENMEM_resource_ioreq_server 0
 #define XENMEM_resource_grant_table 1
+#define XENMEM_resource_vmtrace_buf 2
 
     /*
      * IN - a type-specific resource identifier, which must be zero
Signature · 1 line
2.17.1
#23 lore raw

Re: [PATCH v6 08/11] x86/mm: add vmtrace_buf resource type

Quoted 16 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to map processor trace buffer using
acquire_resource().

---
 xen/common/memory.c         | 31 +++++++++++++++++++++++++++++++
 xen/include/public/memory.h |  1 +
 2 files changed, 32 insertions(+)
xen/common/memory.c +0 -0
index eb42f883df..c0a22eb60f 100644

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1012

Quoted 6 lines
xen/common/memory.c +3 -0
@@ -1007,6 +1007,32 @@ static long xatp_permission_check(struct domain *d, unsigned int space)
     return xsm_add_to_physmap(XSM_TARGET, current->domain, d);
 }
 
+static int acquire_vmtrace_buf(struct domain *d, unsigned int id,
+                               uint64_t frame,
+                               uint64_t nr_frames,

frame and nr_frames should be unsigned int, as I think they can't overflow an unsigned integer?

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1024

Quoted 12 lines
xen/common/memory.c +12 -0
+                               xen_pfn_t mfn_list[])
+{
+    mfn_t mfn;
+    unsigned int i;
+    uint64_t size;
+    struct vcpu *v = domain_vcpu(d, id);
+
+    if ( !v || !v->vmtrace.pt_buf )
+        return -EINVAL;
+
+    mfn = page_to_mfn(v->vmtrace.pt_buf);
+    size = v->domain->processor_trace_buf_kb * KB(1);

I think this should be the number of pages, and then you want to directly access d, there's no need to do v->domain->...

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1027

Quoted 3 lines
xen/common/memory.c +3 -0
+
+    if ( (frame > (size >> PAGE_SHIFT)) ||
+         (nr_frames > ((size >> PAGE_SHIFT) - frame)) )

Isn't this off by one (should use >=)? If size is 8 pages, you can get pages [0,7], but requesting page 8 would be out of the range?

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1031

Quoted 4 lines
xen/common/memory.c +4 -0
+        return -EINVAL;
+
+    for ( i = 0; i < nr_frames; i++ )
+        mfn_list[i] = mfn_x(mfn_add(mfn, frame + i));

You could drop the mfn variable and just do:

mfn_list[i] = mfn_x(page_to_mfn(v->vmtrace.pt_buf[frame + i]));

Thanks.

#24 lore raw

PATCH v6 9/11 x86/domctl: add XEN_DOMCTL_vmtrace_op

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Implement domctl to manage the runtime state of
processor trace feature.

---
 xen/arch/x86/domctl.c       | 50 +++++++++++++++++++++++++++++++++++++
 xen/include/public/domctl.h | 28 +++++++++++++++++++++
 2 files changed, 78 insertions(+)
xen/arch/x86/domctl.c +50 -0
index 6f2c69788d..6132499db4 100644
@@ -322,6 +322,50 @@ void arch_get_domain_info(const struct domain *d,
     info->arch_config.emulation_flags = d->arch.emulation_flags;
 }
 
+static int do_vmtrace_op(struct domain *d, struct xen_domctl_vmtrace_op *op,
+                         XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
+{
+    int rc;
+    struct vcpu *v;
+
+    if ( op->pad1 || op->pad2 )
+        return -EINVAL;
+
+    if ( !vmtrace_supported )
+        return -EOPNOTSUPP;
+
+    if ( !is_hvm_domain(d) )
+        return -EOPNOTSUPP;
+
+    if ( op->vcpu >= d->max_vcpus )
+        return -EINVAL;
+
+    v = domain_vcpu(d, op->vcpu);
+    rc = 0;
+
+    switch ( op->cmd )
+    {
+    case XEN_DOMCTL_vmtrace_pt_enable:
+    case XEN_DOMCTL_vmtrace_pt_disable:
+        vcpu_pause(v);
+        rc = vmtrace_control_pt(v, op->cmd == XEN_DOMCTL_vmtrace_pt_enable);
+        vcpu_unpause(v);
+        break;
+
+    case XEN_DOMCTL_vmtrace_pt_get_offset:
+        rc = vmtrace_get_pt_offset(v, &op->offset, &op->size);
+
+        if ( !rc && d->is_dying )
+            rc = ENODATA;
+        break;
+
+    default:
+        rc = -EOPNOTSUPP;
+    }
+
+    return rc;
+}
+
 #define MAX_IOPORTS 0x10000
 
 long arch_do_domctl(
@@ -337,6 +381,12 @@ long arch_do_domctl(
     switch ( domctl->cmd )
     {
 
+    case XEN_DOMCTL_vmtrace_op:
+        ret = do_vmtrace_op(d, &domctl->u.vmtrace_op, u_domctl);
+        if ( !ret )
+            copyback = true;
+        break;
+
     case XEN_DOMCTL_shadow_op:
         ret = paging_domctl(d, &domctl->u.shadow_op, u_domctl, 0);
         if ( ret == -ERESTART )
xen/include/public/domctl.h +28 -0
index 7681675a94..73c7ccbd16 100644
@@ -1136,6 +1136,30 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+struct xen_domctl_vmtrace_op {
+    /* IN variable */
+    uint32_t cmd;
+/* Enable/disable external vmtrace for given domain */
+#define XEN_DOMCTL_vmtrace_pt_enable      1
+#define XEN_DOMCTL_vmtrace_pt_disable     2
+#define XEN_DOMCTL_vmtrace_pt_get_offset  3
+    domid_t domain;
+    uint16_t pad1;
+    uint32_t vcpu;
+    uint16_t pad2;
+
+    /* OUT variable */
+    uint64_aligned_t size;
+    uint64_aligned_t offset;
+};
+typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
+
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 struct xen_domctl {
     uint32_t cmd;
 #define XEN_DOMCTL_createdomain                   1
@@ -1217,6 +1241,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_vuart_op                      81
 #define XEN_DOMCTL_get_cpu_policy                82
 #define XEN_DOMCTL_set_cpu_policy                83
+#define XEN_DOMCTL_vmtrace_op                    84
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002
@@ -1277,6 +1302,9 @@ struct xen_domctl {
         struct xen_domctl_monitor_op        monitor_op;
         struct xen_domctl_psr_alloc         psr_alloc;
         struct xen_domctl_vuart_op          vuart_op;
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+        struct xen_domctl_vmtrace_op        vmtrace_op;
+#endif
         uint8_t                             pad[128];
     } u;
 };
Signature · 1 line
2.17.1
#25 lore raw

Re: [PATCH v6 09/11] x86/domctl: add XEN_DOMCTL_vmtrace_op

Quoted 23 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Implement domctl to manage the runtime state of
processor trace feature.

---
 xen/arch/x86/domctl.c       | 50 +++++++++++++++++++++++++++++++++++++
 xen/include/public/domctl.h | 28 +++++++++++++++++++++
 2 files changed, 78 insertions(+)
xen/arch/x86/domctl.c +4 -0
index 6f2c69788d..6132499db4 100644
@@ -322,6 +322,50 @@ void arch_get_domain_info(const struct domain *d,
     info->arch_config.emulation_flags = d->arch.emulation_flags;
 }
 
+static int do_vmtrace_op(struct domain *d, struct xen_domctl_vmtrace_op *op,
+                         XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
+{
+    int rc;

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 338

Quoted 10 lines
xen/arch/x86/domctl.c +10 -0
+    struct vcpu *v;
+
+    if ( op->pad1 || op->pad2 )
+        return -EINVAL;
+
+    if ( !vmtrace_supported )
+        return -EOPNOTSUPP;
+
+    if ( !is_hvm_domain(d) )
+        return -EOPNOTSUPP;

You can join both if into a single one if they both return -EOPNOTSUPP.

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 341

Quoted 3 lines
xen/arch/x86/domctl.c +3 -0
+
+    if ( op->vcpu >= d->max_vcpus )
+        return -EINVAL;

You can remove this check and just use the return value of domain_vcpu instead, if it's NULL the vCPU ID is wrong.

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 344

Quoted 3 lines
xen/arch/x86/domctl.c +3 -0
+
+    v = domain_vcpu(d, op->vcpu);
+    rc = 0;

No need to init rc to 0, as it would be unconditionally initialized in the switch below due to the default case.

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 356

Quoted 12 lines
xen/arch/x86/domctl.c +12 -0
+
+    switch ( op->cmd )
+    {
+    case XEN_DOMCTL_vmtrace_pt_enable:
+    case XEN_DOMCTL_vmtrace_pt_disable:
+        vcpu_pause(v);
+        rc = vmtrace_control_pt(v, op->cmd == XEN_DOMCTL_vmtrace_pt_enable);
+        vcpu_unpause(v);
+        break;
+
+    case XEN_DOMCTL_vmtrace_pt_get_offset:
+        rc = vmtrace_get_pt_offset(v, &op->offset, &op->size);

In order to get consistent values here the vCPU should be paused, or else you just get stale values from the last vmexit?

Maybe that's fine for your use case?

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 359

Quoted 3 lines
xen/arch/x86/domctl.c +3 -0
+
+        if ( !rc && d->is_dying )
+            rc = ENODATA;

This check here seems weird, if this is really required shouldn't it be done before attempting to fetch the data?

Quoted 13 lines
xen/arch/x86/domctl.c +9 -0
+        break;
+
+    default:
+        rc = -EOPNOTSUPP;
+    }
+
+    return rc;
+}
+
 #define MAX_IOPORTS 0x10000
 
 long arch_do_domctl(

xen/arch/x86/domctl.c long arch_do_domctl( · line 388

Quoted 8 lines
xen/arch/x86/domctl.c +5 -0
@@ -337,6 +381,12 @@ long arch_do_domctl(
     switch ( domctl->cmd )
     {
 
+    case XEN_DOMCTL_vmtrace_op:
+        ret = do_vmtrace_op(d, &domctl->u.vmtrace_op, u_domctl);
+        if ( !ret )
+            copyback = true;
+        break;

Nit: new additions would usually got at the end of the switch.

Quoted 16 lines
xen/arch/x86/domctl.c +1 -0
+
     case XEN_DOMCTL_shadow_op:
         ret = paging_domctl(d, &domctl->u.shadow_op, u_domctl, 0);
         if ( ret == -ERESTART )
xen/include/public/domctl.h +4 -0
index 7681675a94..73c7ccbd16 100644
@@ -1136,6 +1136,30 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+struct xen_domctl_vmtrace_op {

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1152

Quoted 10 lines
xen/include/public/domctl.h +10 -0
+    /* IN variable */
+    uint32_t cmd;
+/* Enable/disable external vmtrace for given domain */
+#define XEN_DOMCTL_vmtrace_pt_enable      1
+#define XEN_DOMCTL_vmtrace_pt_disable     2
+#define XEN_DOMCTL_vmtrace_pt_get_offset  3
+    domid_t domain;
+    uint16_t pad1;
+    uint32_t vcpu;
+    uint16_t pad2;

pad2 should be a uint32_t? Or else there's a padding field added by the compiler? (maybe I'm missing something, I haven't checked with pahole).

Quoted 22 lines
xen/include/public/domctl.h +11 -0
+
+    /* OUT variable */
+    uint64_aligned_t size;
+    uint64_aligned_t offset;
+};
+typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
+
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 struct xen_domctl {
     uint32_t cmd;
 #define XEN_DOMCTL_createdomain                   1
@@ -1217,6 +1241,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_vuart_op                      81
 #define XEN_DOMCTL_get_cpu_policy                82
 #define XEN_DOMCTL_set_cpu_policy                83
+#define XEN_DOMCTL_vmtrace_op                    84
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002

xen/include/public/domctl.h struct xen_domctl { · line 1307

Quoted 6 lines
xen/include/public/domctl.h +3 -0
@@ -1277,6 +1302,9 @@ struct xen_domctl {
         struct xen_domctl_monitor_op        monitor_op;
         struct xen_domctl_psr_alloc         psr_alloc;
         struct xen_domctl_vuart_op          vuart_op;
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+        struct xen_domctl_vmtrace_op        vmtrace_op;
+#endif

No need for the preprocessor conditionals here, the whole domctl.h is only to be used by the tools or Xen, and is already properly guarded (see the #error at the top of the file).

Thanks.

#26 lore raw

Re: [PATCH v6 09/11] x86/domctl: add XEN_DOMCTL_vmtrace_op

In reply to Michał Leszczyński
Quoted 19 lines · Michał Leszczyński
xen/include/public/domctl.h +13 -0
@@ -1136,6 +1136,30 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+struct xen_domctl_vmtrace_op {
+    /* IN variable */
+    uint32_t cmd;
+/* Enable/disable external vmtrace for given domain */
+#define XEN_DOMCTL_vmtrace_pt_enable      1
+#define XEN_DOMCTL_vmtrace_pt_disable     2
+#define XEN_DOMCTL_vmtrace_pt_get_offset  3
+    domid_t domain;
+    uint16_t pad1;
+    uint32_t vcpu;

xen/include/public/domctl.h struct xen_domctl_vuart_op { · line 1161

Quoted 10 lines
xen/include/public/domctl.h +10 -0
+    uint16_t pad2;
+
+    /* OUT variable */
+    uint64_aligned_t size;
+    uint64_aligned_t offset;
+};
+typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
+
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */

Here and ...

xen/include/public/domctl.h struct xen_domctl { · line 1307

Quoted 7 lines
xen/include/public/domctl.h +3 -0
@@ -1277,6 +1302,9 @@ struct xen_domctl {
         struct xen_domctl_monitor_op        monitor_op;
         struct xen_domctl_psr_alloc         psr_alloc;
         struct xen_domctl_vuart_op          vuart_op;
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+        struct xen_domctl_vmtrace_op        vmtrace_op;
+#endif

... here I'm struggling with the #ifdef-s - see the very top of the file.

Jan

#27 lore raw

PATCH v6 10/11 tools/libxc: add xc_vmtrace_* functions

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add functions in libxc that use the new XEN_DOMCTL_vmtrace interface.

---
 tools/libxc/Makefile          |  1 +
 tools/libxc/include/xenctrl.h | 40 ++++++++++++++++
 tools/libxc/xc_vmtrace.c      | 87 +++++++++++++++++++++++++++++++++++
 3 files changed, 128 insertions(+)
 create mode 100644 tools/libxc/xc_vmtrace.c
tools/libxc/Makefile +1 -0
index fae5969a73..605e44501d 100644
@@ -27,6 +27,7 @@ CTRL_SRCS-y       += xc_csched2.c
 CTRL_SRCS-y       += xc_arinc653.c
 CTRL_SRCS-y       += xc_rt.c
 CTRL_SRCS-y       += xc_tbuf.c
+CTRL_SRCS-y       += xc_vmtrace.c
 CTRL_SRCS-y       += xc_pm.c
 CTRL_SRCS-y       += xc_cpu_hotplug.c
 CTRL_SRCS-y       += xc_resume.c
tools/libxc/include/xenctrl.h +40 -0
index 4c89b7294c..491b2c3236 100644
@@ -1585,6 +1585,46 @@ int xc_tbuf_set_cpu_mask(xc_interface *xch, xc_cpumap_t mask);
 
 int xc_tbuf_set_evt_mask(xc_interface *xch, uint32_t mask);
 
+/**
+ * Enable processor trace for given vCPU in given DomU.
+ * Allocate the trace ringbuffer with given size.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_enable(xc_interface *xch, uint32_t domid,
+                         uint32_t vcpu);
+
+/**
+ * Disable processor trace for given vCPU in given DomU.
+ * Deallocate the trace ringbuffer.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_disable(xc_interface *xch, uint32_t domid,
+                          uint32_t vcpu);
+
+/**
+ * Get current offset inside the trace ringbuffer.
+ * This allows to determine how much data was written into the buffer.
+ * Once buffer overflows, the offset will reset to 0 and the previous
+ * data will be overriden.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @parm offset current offset inside trace buffer will be written there
+ * @parm size the total size of the trace buffer (in bytes)
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_get_offset(xc_interface *xch, uint32_t domid,
+                             uint32_t vcpu, uint64_t *offset, uint64_t *size);
+
 int xc_domctl(xc_interface *xch, struct xen_domctl *domctl);
 int xc_sysctl(xc_interface *xch, struct xen_sysctl *sysctl);
 
tools/libxc/xc_vmtrace.c new file +87 -0

87 lines not on this page · raw message

Signature · 1 line
2.17.1
#28 lore raw

Re: [PATCH v6 10/11] tools/libxc: add xc_vmtrace_* functions

Quoted 51 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add functions in libxc that use the new XEN_DOMCTL_vmtrace interface.

---
 tools/libxc/Makefile          |  1 +
 tools/libxc/include/xenctrl.h | 40 ++++++++++++++++
 tools/libxc/xc_vmtrace.c      | 87 +++++++++++++++++++++++++++++++++++
 3 files changed, 128 insertions(+)
 create mode 100644 tools/libxc/xc_vmtrace.c
tools/libxc/Makefile +1 -0
index fae5969a73..605e44501d 100644
@@ -27,6 +27,7 @@ CTRL_SRCS-y       += xc_csched2.c
 CTRL_SRCS-y       += xc_arinc653.c
 CTRL_SRCS-y       += xc_rt.c
 CTRL_SRCS-y       += xc_tbuf.c
+CTRL_SRCS-y       += xc_vmtrace.c
 CTRL_SRCS-y       += xc_pm.c
 CTRL_SRCS-y       += xc_cpu_hotplug.c
 CTRL_SRCS-y       += xc_resume.c
tools/libxc/include/xenctrl.h +19 -0
index 4c89b7294c..491b2c3236 100644
@@ -1585,6 +1585,46 @@ int xc_tbuf_set_cpu_mask(xc_interface *xch, xc_cpumap_t mask);
 
 int xc_tbuf_set_evt_mask(xc_interface *xch, uint32_t mask);
 
+/**
+ * Enable processor trace for given vCPU in given DomU.
+ * Allocate the trace ringbuffer with given size.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_enable(xc_interface *xch, uint32_t domid,
+                         uint32_t vcpu);
+
+/**
+ * Disable processor trace for given vCPU in given DomU.
+ * Deallocate the trace ringbuffer.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier

tools/libxc/include/xenctrl.h int xc_tbuf_set_cpu_mask(xc_interface *xch, xc_cpumap_t mask); · line 1616

Quoted 10 lines
tools/libxc/include/xenctrl.h +10 -0
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_disable(xc_interface *xch, uint32_t domid,
+                          uint32_t vcpu);
+
+/**
+ * Get current offset inside the trace ringbuffer.
+ * This allows to determine how much data was written into the buffer.
+ * Once buffer overflows, the offset will reset to 0 and the previous
+ * data will be overriden.

^ overridden.

Quoted 41 lines
tools/libxc/include/xenctrl.h +11 -0
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @parm offset current offset inside trace buffer will be written there
+ * @parm size the total size of the trace buffer (in bytes)
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_get_offset(xc_interface *xch, uint32_t domid,
+                             uint32_t vcpu, uint64_t *offset, uint64_t *size);
+
 int xc_domctl(xc_interface *xch, struct xen_domctl *domctl);
 int xc_sysctl(xc_interface *xch, struct xen_sysctl *sysctl);
 
tools/libxc/xc_vmtrace.c new file +21 -0
new file mode 100644
index 0000000000..ee034da8d3
@@ -0,0 +1,87 @@
+/******************************************************************************
+ * xc_vmtrace.c
+ *
+ * API for manipulating hardware tracing features
+ *
+ * Copyright (c) 2020, Michal Leszczynski
+ *
+ * Copyright 2020 CERT Polska. All rights reserved.
+ * Use is subject to license terms.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation;
+ * version 2.1 of the License.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public

tools/libxc/xc_vmtrace.c line 31

Quoted 10 lines
tools/libxc/xc_vmtrace.c new file +10 -0
+ * License along with this library; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "xc_private.h"
+#include <xen/trace.h>
+
+int xc_vmtrace_pt_enable(
+        xc_interface *xch, uint32_t domid, uint32_t vcpu)
+{
+    DECLARE_DOMCTL;

You could do:

DECLARE_DOMCTL = {
.cmd = XEN_DOMCTL_vmtrace_op,
.domain = domid,
...
};

And avoid setting the fields below, the same applies to the rest of the functions. Note that when doing this there's no need to set the padding fields, as they will be zeroed by the initialization.

tools/libxc/xc_vmtrace.c line 42

Quoted 11 lines
tools/libxc/xc_vmtrace.c new file +11 -0
+    int rc;
+
+    domctl.cmd = XEN_DOMCTL_vmtrace_op;
+    domctl.domain = domid;
+    domctl.u.vmtrace_op.cmd = XEN_DOMCTL_vmtrace_pt_enable;
+    domctl.u.vmtrace_op.vcpu = vcpu;
+    domctl.u.vmtrace_op.pad1 = 0;
+    domctl.u.vmtrace_op.pad2 = 0;
+
+    rc = do_domctl(xch, &domctl);
+    return rc;

Just do 'return do_domctl(xch, &domctl);', and you can drop the rc variable (here and in xc_vmtrace_pt_disable).

Quoted 10 lines
tools/libxc/xc_vmtrace.c new file +10 -0
+}
+
+int xc_vmtrace_pt_get_offset(
+        xc_interface *xch, uint32_t domid, uint32_t vcpu,
+        uint64_t *offset, uint64_t *size)
+{
+    DECLARE_DOMCTL;
+    int rc;
+
+    domctl.cmd = XEN_DOMCTL_vmtrace_op;

tools/libxc/xc_vmtrace.c line 62

Quoted 10 lines
tools/libxc/xc_vmtrace.c new file +10 -0
+    domctl.domain = domid;
+    domctl.u.vmtrace_op.cmd = XEN_DOMCTL_vmtrace_pt_get_offset;
+    domctl.u.vmtrace_op.vcpu = vcpu;
+    domctl.u.vmtrace_op.pad1 = 0;
+    domctl.u.vmtrace_op.pad2 = 0;
+
+    rc = do_domctl(xch, &domctl);
+    if ( !rc )
+    {
+        if (offset)

^ missing spaces.

Thanks.

#29 lore raw

PATCH v6 11/11 tools/proctrace: add proctrace tool

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add an demonstration tool that uses xc_vmtrace_* calls in order to manage external IPT monitoring for DomU.

---
 tools/proctrace/Makefile    |  45 +++++++++
 tools/proctrace/proctrace.c | 179 ++++++++++++++++++++++++++++++++++++
 2 files changed, 224 insertions(+)
 create mode 100644 tools/proctrace/Makefile
 create mode 100644 tools/proctrace/proctrace.c
tools/proctrace/Makefile new file +45 -0
new file mode 100644
index 0000000000..9c135229b9
@@ -0,0 +1,45 @@
+# Copyright (C) CERT Polska - NASK PIB
+# Author: Michał Leszczyński <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; under version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+XEN_ROOT=$(CURDIR)/../..
+include $(XEN_ROOT)/tools/Rules.mk
+
+CFLAGS  += -Werror
+CFLAGS  += $(CFLAGS_libxenevtchn)
+CFLAGS  += $(CFLAGS_libxenctrl)
+LDLIBS  += $(LDLIBS_libxenctrl)
+LDLIBS  += $(LDLIBS_libxenevtchn)
+LDLIBS  += $(LDLIBS_libxenforeignmemory)
+
+.PHONY: all
+all: build
+
+.PHONY: build
+build: proctrace
+
+.PHONY: install
+install: build
+	$(INSTALL_DIR) $(DESTDIR)$(sbindir)
+	$(INSTALL_PROG) proctrace $(DESTDIR)$(sbindir)/proctrace
+
+.PHONY: uninstall
+uninstall:
+	rm -f $(DESTDIR)$(sbindir)/proctrace
+
+.PHONY: clean
+clean:
+	$(RM) -f proctrace $(DEPS_RM)
+
+.PHONY: distclean
+distclean: clean
+
+-include $(DEPS_INCLUDE)
tools/proctrace/proctrace.c new file +179 -0

179 lines not on this page · raw message

Signature · 1 line
2.17.1
#30 lore raw

Re: [PATCH v6 11/11] tools/proctrace: add proctrace tool

Quoted 3 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add an demonstration tool that uses xc_vmtrace_* calls in order

      ^ a
Quoted 93 lines

to manage external IPT monitoring for DomU.

---
 tools/proctrace/Makefile    |  45 +++++++++
 tools/proctrace/proctrace.c | 179 ++++++++++++++++++++++++++++++++++++
 2 files changed, 224 insertions(+)
 create mode 100644 tools/proctrace/Makefile
 create mode 100644 tools/proctrace/proctrace.c
tools/proctrace/Makefile new file +45 -0
new file mode 100644
index 0000000000..9c135229b9
@@ -0,0 +1,45 @@
+# Copyright (C) CERT Polska - NASK PIB
+# Author: Michał Leszczyński <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; under version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+XEN_ROOT=$(CURDIR)/../..
+include $(XEN_ROOT)/tools/Rules.mk
+
+CFLAGS  += -Werror
+CFLAGS  += $(CFLAGS_libxenevtchn)
+CFLAGS  += $(CFLAGS_libxenctrl)
+LDLIBS  += $(LDLIBS_libxenctrl)
+LDLIBS  += $(LDLIBS_libxenevtchn)
+LDLIBS  += $(LDLIBS_libxenforeignmemory)
+
+.PHONY: all
+all: build
+
+.PHONY: build
+build: proctrace
+
+.PHONY: install
+install: build
+	$(INSTALL_DIR) $(DESTDIR)$(sbindir)
+	$(INSTALL_PROG) proctrace $(DESTDIR)$(sbindir)/proctrace
+
+.PHONY: uninstall
+uninstall:
+	rm -f $(DESTDIR)$(sbindir)/proctrace
+
+.PHONY: clean
+clean:
+	$(RM) -f proctrace $(DEPS_RM)
+
+.PHONY: distclean
+distclean: clean
+
+-include $(DEPS_INCLUDE)
tools/proctrace/proctrace.c new file +26 -0
new file mode 100644
index 0000000000..3c1ccccee8
@@ -0,0 +1,179 @@
+/******************************************************************************
+ * tools/proctrace.c
+ *
+ * Demonstrative tool for collecting Intel Processor Trace data from Xen.
+ *  Could be used to externally monitor a given vCPU in given DomU.
+ *
+ * Copyright (C) 2020 by CERT Polska - NASK PIB
+ *
+ * Authors: Michał Leszczyński, [email protected]
+ * Date:    June, 2020
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; under version 2 of the License.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <stdlib.h>
+#include <stdio.h>

tools/proctrace/proctrace.c line 36

Quoted 10 lines
tools/proctrace/proctrace.c new file +10 -0
+#include <sys/mman.h>
+#include <signal.h>
+#include <errno.h>
+
+#include <xenctrl.h>
+#include <xen/xen.h>
+#include <xenforeignmemory.h>
+
+volatile int interrupted = 0;
+volatile int domain_down = 0;

No need for the initialization, globals are already initialized to 0.

Quoted 27 lines
tools/proctrace/proctrace.c new file +27 -0
+void term_handler(int signum) {
+    interrupted = 1;
+}
+
+int main(int argc, char* argv[]) {
+    xc_interface *xc;
+    uint32_t domid;
+    uint32_t vcpu_id;
+    uint64_t size;
+
+    int rc = -1;
+    uint8_t *buf = NULL;
+    uint64_t last_offset = 0;
+
+    xenforeignmemory_handle *fmem;
+    xenforeignmemory_resource_handle *fres;
+
+    if (signal(SIGINT, term_handler) == SIG_ERR)
+    {
+        fprintf(stderr, "Failed to register signal handler\n");
+        return 1;
+    }
+
+    if (argc != 3) {
+        fprintf(stderr, "Usage: %s <domid> <vcpu_id>\n", argv[0]);
+        fprintf(stderr, "It's recommended to redirect this"
+                        "program's output to file\n");

tools/proctrace/proctrace.c line 74

Quoted 10 lines
tools/proctrace/proctrace.c new file +10 -0
+        fprintf(stderr, "or to pipe it's output to xxd or other program.\n");
+        return 1;
+    }
+
+    domid = atoi(argv[1]);
+    vcpu_id = atoi(argv[2]);
+
+    xc = xc_interface_open(0, 0, 0);
+
+    fmem = xenforeignmemory_open(0, 0);

I think you also need to test that fmem is set? (like you do for xc).

Quoted 31 lines
tools/proctrace/proctrace.c new file +31 -0
+
+    if (!xc) {
+        fprintf(stderr, "Failed to open xc interface\n");
+        return 1;
+    }
+
+    rc = xc_vmtrace_pt_enable(xc, domid, vcpu_id);
+
+    if (rc) {
+        fprintf(stderr, "Failed to call xc_vmtrace_pt_enable\n");
+        return 1;
+    }
+    
+    rc = xc_vmtrace_pt_get_offset(xc, domid, vcpu_id, NULL, &size);
+
+    if (rc) {
+        fprintf(stderr, "Failed to get trace buffer size\n");
+        return 1;
+    }
+
+    fres = xenforeignmemory_map_resource(
+        fmem, domid, XENMEM_resource_vmtrace_buf,
+        /* vcpu: */ vcpu_id,
+        /* frame: */ 0,
+        /* num_frames: */ size >> XC_PAGE_SHIFT,
+        (void **)&buf,
+        PROT_READ, 0);
+
+    if (!buf) {
+        fprintf(stderr, "Failed to map trace buffer\n");
+        return 1;

tools/proctrace/proctrace.c line 115

Quoted 10 lines
tools/proctrace/proctrace.c new file +10 -0
+    }
+
+    while (!interrupted) {
+        uint64_t offset;
+        rc = xc_vmtrace_pt_get_offset(xc, domid, vcpu_id, &offset, NULL);
+
+        if (rc == ENODATA) {
+            interrupted = 1;
+            domain_down = 1;
+	} else if (rc) {

Hard tab.

tools/proctrace/proctrace.c line 116

Quoted 1 line
tools/proctrace/proctrace.c new file +1 -0
+            fprintf(stderr, "Failed to call xc_vmtrace_pt_get_offset\n");

Should you try to disable vmtrace here before exiting?

tools/proctrace/proctrace.c line 126

Quoted 10 lines
tools/proctrace/proctrace.c new file +10 -0
+            return 1;
+        }
+
+        if (offset > last_offset)
+        {
+            fwrite(buf + last_offset, offset - last_offset, 1, stdout);
+        }
+        else if (offset < last_offset)
+        {
+            // buffer wrapped

I know this is a test utility, but I would prefer if you could use the C comment style /* */.

Quoted 17 lines
tools/proctrace/proctrace.c new file +17 -0
+            fwrite(buf + last_offset, size - last_offset, 1, stdout);
+            fwrite(buf, offset, 1, stdout);
+        }
+
+        last_offset = offset;
+        usleep(1000 * 100);
+    }
+
+    rc = xenforeignmemory_unmap_resource(fmem, fres);
+
+    if (rc) {
+        fprintf(stderr, "Failed to unmap resource\n");
+        return 1;
+    }
+
+    rc = xenforeignmemory_close(fmem);
+

tools/proctrace/proctrace.c line 153

Quoted 10 lines
tools/proctrace/proctrace.c new file +10 -0
+    if (rc) {
+        fprintf(stderr, "Failed to close fmem\n");
+        return 1;
+    }
+
+    /*
+     * Don't try to disable PT if the domain is already dying.
+     */
+    if (!domain_down) {
+        rc = xc_vmtrace_pt_disable(xc, domid, vcpu_id);

I'm not sure you can assume a domain is dying just because xc_vmtrace_pt_get_offset has returned ENODATA. Is there any harm in unconditionally attempting to disable vmtrace?

Thanks.

#31 lore raw
In reply to Michał Leszczyński

----- 7 lip 2020 o 21:39, Michał Leszczyński [email protected] napisał(a):

Quoted 107 lines

Intel Processor Trace is an architectural extension available in modern Intel family CPUs. It allows recording the detailed trace of activity while the processor executes the code. One might use the recorded trace to reconstruct the code flow. It means, to find out the executed code paths, determine branches taken, and so forth.

The abovementioned feature is described in Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3C: System Programming Guide, Part 3, Chapter 36: "Intel Processor Trace."

This patch series implements an interface that Dom0 could use in order to enable IPT for particular vCPUs in DomU, allowing for external monitoring. Such a feature has numerous applications like malware monitoring, fuzzing, or performance testing.

Also thanks to Tamas K Lengyel for a few preliminary hints before first version of this patch was submitted to xen-devel.

Changed since v1:
* MSR_RTIT_CTL is managed using MSR load lists
* other PT-related MSRs are modified only when vCPU goes out of context
* trace buffer is now acquired as a resource
* added vmtrace_pt_size parameter in xl.cfg, the size of trace buffer must be specified in the moment of domain creation
* trace buffers are allocated on domain creation, destructed on domain destruction
* HVMOP_vmtrace_ipt_enable/disable is limited to enabling/disabling PT these calls don't manage buffer memory anymore
* lifted 32 MFN/GFN array limit when acquiring resources
* minor code style changes according to review

Changed since v2:
* trace buffer is now allocated on domain creation (in v2 it was allocated when hvm param was set)
* restored 32-item limit in mfn/gfn arrays in acquire_resource and instead implemented hypercall continuations
* code changes according to Jan's and Roger's review

Changed since v3:
* vmtrace HVMOPs are not implemented as DOMCTLs
* patches splitted up according to Andrew's comments
* code changes according to v3 review on the mailing list

Changed since v4:
* rebased to commit be63d9d4
* fixed dependencies between patches (earlier patches don't reference further patches)
* introduced preemption check in acquire_resource
* moved buffer allocation to common code
* splitted some patches according to code review
* minor fixes according to code review

Changed since v5:
* trace buffer size is now dynamically determined by the proctrace tool
* trace buffer size variable is uniformly defined as uint32_t processor_trace_buf_kb in hypervisor, toolstack and ABI
* buffer pages are not freed explicitly but reference count is now used instead
* minor fixes according to code review

This patch series is available on GitHub:
https://github.com/icedevml/xen/tree/ipt-patch-v6

Michal Leszczynski (11):
memory: batch processing in acquire_resource()
x86/vmx: add Intel PT MSR definitions
x86/vmx: add IPT cpu feature
common: add vmtrace_pt_size domain parameter
tools/libxl: add vmtrace_pt_size parameter
x86/hvm: processor trace interface in HVM
x86/vmx: implement IPT in VMX
x86/mm: add vmtrace_buf resource type
x86/domctl: add XEN_DOMCTL_vmtrace_op
tools/libxc: add xc_vmtrace_* functions
tools/proctrace: add proctrace tool

docs/man/xl.cfg.5.pod.in                    |  13 ++
tools/golang/xenlight/helpers.gen.go        |   2 +
tools/golang/xenlight/types.gen.go          |   1 +
tools/libxc/Makefile                        |   1 +
tools/libxc/include/xenctrl.h               |  40 +++++
tools/libxc/xc_vmtrace.c                    |  87 ++++++++++
tools/libxl/libxl.h                         |   8 +
tools/libxl/libxl_create.c                  |   1 +
tools/libxl/libxl_types.idl                 |   4 +
tools/proctrace/Makefile                    |  45 +++++
tools/proctrace/proctrace.c                 | 179 ++++++++++++++++++++
tools/xl/xl_parse.c                         |  22 +++
xen/arch/x86/domain.c                       |  27 +++
xen/arch/x86/domctl.c                       |  50 ++++++
xen/arch/x86/hvm/vmx/vmcs.c                 |  15 +-
xen/arch/x86/hvm/vmx/vmx.c                  | 110 ++++++++++++
xen/common/domain.c                         |  46 +++++
xen/common/memory.c                         |  80 ++++++++-
xen/include/asm-x86/cpufeature.h            |   1 +
xen/include/asm-x86/hvm/hvm.h               |  20 +++
xen/include/asm-x86/hvm/vmx/vmcs.h          |   4 +
xen/include/asm-x86/hvm/vmx/vmx.h           |  14 ++
xen/include/asm-x86/msr-index.h             |  24 +++
xen/include/public/arch-x86/cpufeatureset.h |   1 +
xen/include/public/domctl.h                 |  29 ++++
xen/include/public/memory.h                 |   1 +
xen/include/xen/domain.h                    |   2 +
xen/include/xen/sched.h                     |   7 +
28 files changed, 828 insertions(+), 6 deletions(-)
Quoted 6 lines

create mode 100644 tools/libxc/xc_vmtrace.c
create mode 100644 tools/proctrace/Makefile
create mode 100644 tools/proctrace/proctrace.c

--
2.17.1

Kind reminder about this new patch version for external IPT monitoring.

Best regards,
Michał Leszczyński
CERT Polska

#32 lore raw
Quoted 1 line · Michał Leszczyński

Kind reminder about this new patch version for external IPT monitoring.

It's on my queue, but with XenSummit I haven't been able to take a look, will try to do between today and tomorrow.

Roger.