Skip to contentSkip to conversation

Implement support for external IPT monitoring

PATCH Superseded Michał Leszczyński 10 participants 75 messages Jun 30, 2020 – Jul 21, 2020

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v4 cover Implement support for external IPT monitoring

From: Michal Leszczynski <[email protected]>

Intel Processor Trace is an architectural extension available in modern Intel family CPUs. It allows recording the detailed trace of activity while the processor executes the code. One might use the recorded trace to reconstruct the code flow. It means, to find out the executed code paths, determine branches taken, and so forth.

The abovementioned feature is described in Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3C: System Programming Guide, Part 3, Chapter 36: "Intel Processor Trace."

This patch series implements an interface that Dom0 could use in order to enable IPT for particular vCPUs in DomU, allowing for external monitoring. Such a feature has numerous applications like malware monitoring, fuzzing, or performance testing.

Also thanks to Tamas K Lengyel for a few preliminary hints before first version of this patch was submitted to xen-devel.

Changed since v1:
* MSR_RTIT_CTL is managed using MSR load lists
* other PT-related MSRs are modified only when vCPU goes out of context
* trace buffer is now acquired as a resource
* added vmtrace_pt_size parameter in xl.cfg, the size of trace buffer must be specified in the moment of domain creation
* trace buffers are allocated on domain creation, destructed on domain destruction
* HVMOP_vmtrace_ipt_enable/disable is limited to enabling/disabling PT these calls don't manage buffer memory anymore
* lifted 32 MFN/GFN array limit when acquiring resources
* minor code style changes according to review

Changed since v2:
* trace buffer is now allocated on domain creation (in v2 it was allocated when hvm param was set)
* restored 32-item limit in mfn/gfn arrays in acquire_resource and instead implemented hypercall continuations
* code changes according to Jan's and Roger's review

Changed since v3:
* vmtrace HVMOPs are not implemented as DOMCTLs
* patches splitted up according to Andrew's comments
* code changes according to v3 review on the mailing list

Michal Leszczynski (10):
x86/vmx: add Intel PT MSR definitions
x86/vmx: add IPT cpu feature
tools/libxl: add vmtrace_pt_size parameter
x86/vmx: implement processor tracing for VMX
common/domain: allocate vmtrace_pt_buffer
memory: batch processing in acquire_resource()
x86/mm: add vmtrace_buf resource type
x86/domctl: add XEN_DOMCTL_vmtrace_op
tools/libxc: add xc_vmtrace_* functions
tools/proctrace: add proctrace tool

 docs/man/xl.cfg.5.pod.in                    |  10 +
 tools/golang/xenlight/helpers.gen.go        |   2 +
 tools/golang/xenlight/types.gen.go          |   1 +
 tools/libxc/Makefile                        |   1 +
 tools/libxc/include/xenctrl.h               |  39 +++
 tools/libxc/xc_vmtrace.c                    |  73 +++++
 tools/libxl/libxl.h                         |   8 +
 tools/libxl/libxl_create.c                  |   1 +
 tools/libxl/libxl_types.idl                 |   2 +
 tools/proctrace/COPYING                     | 339 ++++++++++++++++++++
 tools/proctrace/Makefile                    |  48 +++
 tools/proctrace/proctrace.c                 | 163 ++++++++++
 tools/xl/xl_parse.c                         |  20 ++
 xen/arch/x86/domain.c                       |  11 +
 xen/arch/x86/domctl.c                       |  48 +++
 xen/arch/x86/hvm/vmx/vmcs.c                 |   7 +-
 xen/arch/x86/hvm/vmx/vmx.c                  |  89 +++++
 xen/arch/x86/mm.c                           |  25 ++
 xen/common/domain.c                         |  46 +++
 xen/common/memory.c                         |  32 +-
 xen/include/asm-x86/cpufeature.h            |   1 +
 xen/include/asm-x86/domain.h                |   4 +
 xen/include/asm-x86/hvm/hvm.h               |  38 +++
 xen/include/asm-x86/hvm/vmx/vmcs.h          |   4 +
 xen/include/asm-x86/hvm/vmx/vmx.h           |  14 +
 xen/include/asm-x86/msr-index.h             |  37 +++
 xen/include/public/arch-x86/cpufeatureset.h |   1 +
 xen/include/public/domctl.h                 |  27 ++
 xen/include/public/memory.h                 |   1 +
 xen/include/xen/domain.h                    |   2 +
 xen/include/xen/sched.h                     |   4 +
 31 files changed, 1094 insertions(+), 4 deletions(-)
 create mode 100644 tools/libxc/xc_vmtrace.c
 create mode 100644 tools/proctrace/COPYING
 create mode 100644 tools/proctrace/Makefile
 create mode 100644 tools/proctrace/proctrace.c
Signature · 1 line
2.20.1
#1 lore raw

PATCH v4 1/10 x86/vmx: add Intel PT MSR definitions

From: Michal Leszczynski <[email protected]>

Define constants related to Intel Processor Trace features.

---
 xen/include/asm-x86/msr-index.h | 37 +++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
xen/include/asm-x86/msr-index.h +37 -0
index b328a47ed8..0203029be9 100644
@@ -69,6 +69,43 @@
 #define MSR_MCU_OPT_CTRL                    0x00000123
 #define  MCU_OPT_CTRL_RNGDS_MITG_DIS        (_AC(1, ULL) <<  0)
 
+/* Intel PT MSRs */
+#define MSR_RTIT_OUTPUT_BASE                0x00000560
+
+#define MSR_RTIT_OUTPUT_MASK                0x00000561
+
+#define MSR_RTIT_CTL                        0x00000570
+#define  RTIT_CTL_TRACEEN                    (_AC(1, ULL) <<  0)
+#define  RTIT_CTL_CYCEN                      (_AC(1, ULL) <<  1)
+#define  RTIT_CTL_OS                         (_AC(1, ULL) <<  2)
+#define  RTIT_CTL_USR                        (_AC(1, ULL) <<  3)
+#define  RTIT_CTL_PWR_EVT_EN                 (_AC(1, ULL) <<  4)
+#define  RTIT_CTL_FUP_ON_PTW                 (_AC(1, ULL) <<  5)
+#define  RTIT_CTL_FABRIC_EN                  (_AC(1, ULL) <<  6)
+#define  RTIT_CTL_CR3_FILTER                 (_AC(1, ULL) <<  7)
+#define  RTIT_CTL_TOPA                       (_AC(1, ULL) <<  8)
+#define  RTIT_CTL_MTC_EN                     (_AC(1, ULL) <<  9)
+#define  RTIT_CTL_TSC_EN                     (_AC(1, ULL) <<  10)
+#define  RTIT_CTL_DIS_RETC                   (_AC(1, ULL) <<  11)
+#define  RTIT_CTL_PTW_EN                     (_AC(1, ULL) <<  12)
+#define  RTIT_CTL_BRANCH_EN                  (_AC(1, ULL) <<  13)
+#define  RTIT_CTL_MTC_FREQ                   (_AC(0x0F, ULL) <<  14)
+#define  RTIT_CTL_CYC_THRESH                 (_AC(0x0F, ULL) <<  19)
+#define  RTIT_CTL_PSB_FREQ                   (_AC(0x0F, ULL) <<  24)
+#define  RTIT_CTL_ADDR(n)                    (_AC(0x0F, ULL) <<  (32 + (4 * (n))))
+
+#define MSR_RTIT_STATUS                     0x00000571
+#define  RTIT_STATUS_FILTER_EN               (_AC(1, ULL) <<  0)
+#define  RTIT_STATUS_CONTEXT_EN              (_AC(1, ULL) <<  1)
+#define  RTIT_STATUS_TRIGGER_EN              (_AC(1, ULL) <<  2)
+#define  RTIT_STATUS_ERROR                   (_AC(1, ULL) <<  4)
+#define  RTIT_STATUS_STOPPED                 (_AC(1, ULL) <<  5)
+#define  RTIT_STATUS_BYTECNT                 (_AC(0x1FFFF, ULL) <<  32)
+
+#define MSR_RTIT_CR3_MATCH                  0x00000572
+#define MSR_RTIT_ADDR_A(n)                  (0x00000580 + (n) * 2)
+#define MSR_RTIT_ADDR_B(n)                  (0x00000581 + (n) * 2)
+
 #define MSR_U_CET                           0x000006a0
 #define MSR_S_CET                           0x000006a2
 #define  CET_SHSTK_EN                       (_AC(1, ULL) <<  0)
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v4 01/10] x86/vmx: add Intel PT MSR definitions

Quoted 5 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Define constants related to Intel Processor Trace features.

This needs re-basing onto current staging, now that Andrew's patch to add the MSR numbers has gone in. Apart from this a couple of cosmetic requests:

xen/include/asm-x86/msr-index.h line 78

Quoted 13 lines
xen/include/asm-x86/msr-index.h +7 -0
@@ -69,6 +69,43 @@
 #define MSR_MCU_OPT_CTRL                    0x00000123
 #define  MCU_OPT_CTRL_RNGDS_MITG_DIS        (_AC(1, ULL) <<  0)
 
+/* Intel PT MSRs */
+#define MSR_RTIT_OUTPUT_BASE                0x00000560
+
+#define MSR_RTIT_OUTPUT_MASK                0x00000561
+
+#define MSR_RTIT_CTL                        0x00000570
+#define  RTIT_CTL_TRACEEN                    (_AC(1, ULL) <<  0)

The right side is indented one space too many - see the similar #define in context above.

xen/include/asm-x86/msr-index.h line 88

Quoted 10 lines
xen/include/asm-x86/msr-index.h +10 -0
+#define  RTIT_CTL_CYCEN                      (_AC(1, ULL) <<  1)
+#define  RTIT_CTL_OS                         (_AC(1, ULL) <<  2)
+#define  RTIT_CTL_USR                        (_AC(1, ULL) <<  3)
+#define  RTIT_CTL_PWR_EVT_EN                 (_AC(1, ULL) <<  4)
+#define  RTIT_CTL_FUP_ON_PTW                 (_AC(1, ULL) <<  5)
+#define  RTIT_CTL_FABRIC_EN                  (_AC(1, ULL) <<  6)
+#define  RTIT_CTL_CR3_FILTER                 (_AC(1, ULL) <<  7)
+#define  RTIT_CTL_TOPA                       (_AC(1, ULL) <<  8)
+#define  RTIT_CTL_MTC_EN                     (_AC(1, ULL) <<  9)
+#define  RTIT_CTL_TSC_EN                     (_AC(1, ULL) <<  10)

The double blanks on the earlier lines exist such that here you can reduce to a single one. You'll also find examples of this further up in the file.

xen/include/asm-x86/msr-index.h line 92

Quoted 4 lines
xen/include/asm-x86/msr-index.h +4 -0
+#define  RTIT_CTL_DIS_RETC                   (_AC(1, ULL) <<  11)
+#define  RTIT_CTL_PTW_EN                     (_AC(1, ULL) <<  12)
+#define  RTIT_CTL_BRANCH_EN                  (_AC(1, ULL) <<  13)
+#define  RTIT_CTL_MTC_FREQ                   (_AC(0x0F, ULL) <<  14)

0xf please (i.e. lower case and no random number of leading zeros).

xen/include/asm-x86/msr-index.h line 95

Quoted 3 lines
xen/include/asm-x86/msr-index.h +3 -0
+#define  RTIT_CTL_CYC_THRESH                 (_AC(0x0F, ULL) <<  19)
+#define  RTIT_CTL_PSB_FREQ                   (_AC(0x0F, ULL) <<  24)
+#define  RTIT_CTL_ADDR(n)                    (_AC(0x0F, ULL) <<  (32 + (4 * (n))))

Strictly speaking we don't need the parentheses around the operands of binary * here - in mathematics precedence between + and * is well defined. (We do parenthesize certain other expressions, when the precedence may not be as well known.)

Thanks, Jan

#3 lore raw

Re: [PATCH v4 01/10] x86/vmx: add Intel PT MSR definitions

In reply to Michał Leszczyński
Quoted 6 lines · Michał Leszczyński
xen/include/asm-x86/msr-index.h +0 -0
index b328a47ed8..0203029be9 100644
@@ -69,6 +69,43 @@
 #define MSR_MCU_OPT_CTRL                    0x00000123

xen/include/asm-x86/msr-index.h line 79

Quoted 10 lines
xen/include/asm-x86/msr-index.h +8 -0
 #define  MCU_OPT_CTRL_RNGDS_MITG_DIS        (_AC(1, ULL) <<  0)
 
+/* Intel PT MSRs */
+#define MSR_RTIT_OUTPUT_BASE                0x00000560
+
+#define MSR_RTIT_OUTPUT_MASK                0x00000561
+
+#define MSR_RTIT_CTL                        0x00000570
+#define  RTIT_CTL_TRACEEN                    (_AC(1, ULL) <<  0)
+#define  RTIT_CTL_CYCEN                      (_AC(1, ULL) <<  1)

In addition to what Jan has said, please can we be consistent with an underscore (or not) before EN.  Preferably with, so these would become TRACE_EN and CYC_EN.

That said, there are a lot of bit definitions which aren't used at all. IMO, it would be better to introduce defines when you use them.

Thanks,

~Andrew

#4 lore raw

Re: [PATCH v4 01/10] x86/vmx: add Intel PT MSR definitions

Quoted 23 lines · Andrew Cooper
Quoted 6 lines · Michał Leszczyński
xen/include/asm-x86/msr-index.h +0 -0
index b328a47ed8..0203029be9 100644
@@ -69,6 +69,43 @@
 #define MSR_MCU_OPT_CTRL                    0x00000123

xen/include/asm-x86/msr-index.h line 79

Quoted 10 lines
xen/include/asm-x86/msr-index.h +8 -0
 #define  MCU_OPT_CTRL_RNGDS_MITG_DIS        (_AC(1, ULL) <<  0)
 
+/* Intel PT MSRs */
+#define MSR_RTIT_OUTPUT_BASE                0x00000560
+
+#define MSR_RTIT_OUTPUT_MASK                0x00000561
+
+#define MSR_RTIT_CTL                        0x00000570
+#define  RTIT_CTL_TRACEEN                    (_AC(1, ULL) <<  0)
+#define  RTIT_CTL_CYCEN                      (_AC(1, ULL) <<  1)

In addition to what Jan has said, please can we be consistent with an underscore (or not) before EN. Preferably with, so these would become TRACE_EN and CYC_EN.

Quoted 2 lines

That said, there are a lot of bit definitions which aren't used at all. IMO, it would be better to introduce defines when you use them.

In the past I found it very valuable when this type of plumbing was already present in Xen instead of me having to go into the SDM to digg out the magic numbers. So while some of the bits might not be used right now I also don't see any downside in having them, just in case.

Tamas

#5 lore raw

Re: [PATCH v4 01/10] x86/vmx: add Intel PT MSR definitions

----- 30 cze 2020 o 20:03, Tamas K Lengyel [email protected] napisał(a):

Quoted 28 lines
Quoted 23 lines · Andrew Cooper
Quoted 6 lines · Michał Leszczyński
xen/include/asm-x86/msr-index.h +0 -0
index b328a47ed8..0203029be9 100644
@@ -69,6 +69,43 @@
 #define MSR_MCU_OPT_CTRL                    0x00000123

xen/include/asm-x86/msr-index.h line 79

Quoted 10 lines
xen/include/asm-x86/msr-index.h +8 -0
 #define  MCU_OPT_CTRL_RNGDS_MITG_DIS        (_AC(1, ULL) <<  0)
 
+/* Intel PT MSRs */
+#define MSR_RTIT_OUTPUT_BASE                0x00000560
+
+#define MSR_RTIT_OUTPUT_MASK                0x00000561
+
+#define MSR_RTIT_CTL                        0x00000570
+#define  RTIT_CTL_TRACEEN                    (_AC(1, ULL) <<  0)
+#define  RTIT_CTL_CYCEN                      (_AC(1, ULL) <<  1)

In addition to what Jan has said, please can we be consistent with an underscore (or not) before EN. Preferably with, so these would become TRACE_EN and CYC_EN.

Quoted 2 lines

That said, there are a lot of bit definitions which aren't used at all. IMO, it would be better to introduce defines when you use them.

Quoted 6 lines

In the past I found it very valuable when this type of plumbing was already present in Xen instead of me having to go into the SDM to digg out the magic numbers. So while some of the bits might not be used right now I also don't see any downside in having them, just in case.

Tamas

+1 for keeping the unused #defines, this is a helpful piece of knowledge which speeds up further patch development. It doesn't affect the compilation nor runtime time and it doesn't occupy too much space in the code so I would opt for keep it.

I will rebase this series onto latest master within patch v5. The remaining patches in this series are not affected and still could be reviewed, so I will wait a few days before posting the new version.

Best regards,
Michał Leszczyński
CERT Polska

Acked-by on v4 1/10
#6 lore raw

Re: [PATCH v4 01/10] x86/vmx: add Intel PT MSR definitions

In reply to Michał Leszczyński
Quoted 5 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Define constants related to Intel Processor Trace features.

I wanted to have a play with the series, and have ended up having to do the rebase anyway.

As we're in code freeze for 4.14, I've started x86-next in its usual location
(https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen.git;a=shortlog;h=refs/heads/x86-next)
and will commit this (and any other accumulated patches) once 4.15 opens.

~Andrew

#7 lore raw

PATCH v4 2/10 x86/vmx: add IPT cpu feature

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Check if Intel Processor Trace feature is supported by current processor. Define vmtrace_supported global variable.

---
 xen/arch/x86/hvm/vmx/vmcs.c                 | 7 ++++++-
 xen/common/domain.c                         | 2 ++
 xen/include/asm-x86/cpufeature.h            | 1 +
 xen/include/asm-x86/hvm/vmx/vmcs.h          | 1 +
 xen/include/public/arch-x86/cpufeatureset.h | 1 +
 xen/include/xen/domain.h                    | 2 ++
 6 files changed, 13 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/vmx/vmcs.c +6 -1
index ca94c2bedc..b73d824357 100644
@@ -291,6 +291,12 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    vmtrace_supported = cpu_has_ipt &&
+                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED);
+
     if ( _vmx_cpu_based_exec_control & CPU_BASED_ACTIVATE_SECONDARY_CONTROLS )
     {
         min = 0;
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)
                SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                SECONDARY_EXEC_XSAVES |
                SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
         if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
             opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
         if ( opt_vpid_enabled )
xen/common/domain.c +2 -0
index 7cc9526139..0a33e0dfd6 100644
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
 
 vcpu_info_t dummy_vcpu_info;
 
+bool_t vmtrace_supported;
+
 static void __domain_finalise_shutdown(struct domain *d)
 {
     struct vcpu *v;
xen/include/asm-x86/cpufeature.h +1 -0
index f790d5c1f8..8d7955dd87 100644
@@ -104,6 +104,7 @@
 #define cpu_has_clwb            boot_cpu_has(X86_FEATURE_CLWB)
 #define cpu_has_avx512er        boot_cpu_has(X86_FEATURE_AVX512ER)
 #define cpu_has_avx512cd        boot_cpu_has(X86_FEATURE_AVX512CD)
+#define cpu_has_ipt             boot_cpu_has(X86_FEATURE_IPT)
 #define cpu_has_sha             boot_cpu_has(X86_FEATURE_SHA)
 #define cpu_has_avx512bw        boot_cpu_has(X86_FEATURE_AVX512BW)
 #define cpu_has_avx512vl        boot_cpu_has(X86_FEATURE_AVX512VL)
xen/include/asm-x86/hvm/vmx/vmcs.h +1 -0
index 906810592f..0e9a0b8de6 100644
@@ -283,6 +283,7 @@ extern u32 vmx_secondary_exec_control;
 #define VMX_VPID_INVVPID_SINGLE_CONTEXT_RETAINING_GLOBAL 0x80000000000ULL
 extern u64 vmx_ept_vpid_cap;
 
+#define VMX_MISC_PT_SUPPORTED                   0x00004000
 #define VMX_MISC_CR3_TARGET                     0x01ff0000
 #define VMX_MISC_VMWRITE_ALL                    0x20000000
 
xen/include/public/arch-x86/cpufeatureset.h +1 -0
index 5ca35d9d97..0d3f15f628 100644
@@ -217,6 +217,7 @@ XEN_CPUFEATURE(SMAP,          5*32+20) /*S  Supervisor Mode Access Prevention */
 XEN_CPUFEATURE(AVX512_IFMA,   5*32+21) /*A  AVX-512 Integer Fused Multiply Add */
 XEN_CPUFEATURE(CLFLUSHOPT,    5*32+23) /*A  CLFLUSHOPT instruction */
 XEN_CPUFEATURE(CLWB,          5*32+24) /*A  CLWB instruction */
+XEN_CPUFEATURE(IPT,           5*32+25) /*   Intel Processor Trace */
 XEN_CPUFEATURE(AVX512PF,      5*32+26) /*A  AVX-512 Prefetch Instructions */
 XEN_CPUFEATURE(AVX512ER,      5*32+27) /*A  AVX-512 Exponent & Reciprocal Instrs */
 XEN_CPUFEATURE(AVX512CD,      5*32+28) /*A  AVX-512 Conflict Detection Instrs */
xen/include/xen/domain.h +2 -0
index 7e51d361de..6c786a56c2 100644
@@ -130,4 +130,6 @@ struct vnuma_info {
 
 void vnuma_destroy(struct vnuma_info *vnuma);
 
+extern bool_t vmtrace_supported;
+
 #endif /* __XEN_DOMAIN_H__ */
Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 20 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Check if Intel Processor Trace feature is supported by current processor. Define vmtrace_supported global variable.

---
 xen/arch/x86/hvm/vmx/vmcs.c                 | 7 ++++++-
 xen/common/domain.c                         | 2 ++
 xen/include/asm-x86/cpufeature.h            | 1 +
 xen/include/asm-x86/hvm/vmx/vmcs.h          | 1 +
 xen/include/public/arch-x86/cpufeatureset.h | 1 +
 xen/include/xen/domain.h                    | 2 ++
 6 files changed, 13 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/vmx/vmcs.c +0 -0
index ca94c2bedc..b73d824357 100644

xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(void) · line 298

Quoted 8 lines
xen/arch/x86/hvm/vmx/vmcs.c +5 -0
@@ -291,6 +291,12 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    vmtrace_supported = cpu_has_ipt &&
+                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED);

This function gets called for every CPU that's brought up, so you need to set it on the BSP, and then check that the APs also support the feature or else fail to bring them up AFAICT. If not you could end up with a non working system.

I agree it's very unlikely to boot on a system with such differences between CPUs, but better be safe than sorry.

Quoted 17 lines
xen/arch/x86/hvm/vmx/vmcs.c +1 -1
+
     if ( _vmx_cpu_based_exec_control & CPU_BASED_ACTIVATE_SECONDARY_CONTROLS )
     {
         min = 0;
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)
                SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                SECONDARY_EXEC_XSAVES |
                SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
         if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
             opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
         if ( opt_vpid_enabled )
xen/common/domain.c +0 -0
index 7cc9526139..0a33e0dfd6 100644

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 85

Quoted 4 lines
xen/common/domain.c +1 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
 
 vcpu_info_t dummy_vcpu_info;
 
+bool_t vmtrace_supported;

Plain bool, and I think it wants to be __read_mostly.

I'm also unsure whether this is the best place to put such variable, since there are no users introduced on this patch it's hard to tell.

Thanks, Roger.

#9 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

In reply to Michał Leszczyński
Quoted 13 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmcs.c +0 -1
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)
                 SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                 SECONDARY_EXEC_XSAVES |
                 SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
          if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
              opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
          if ( opt_vpid_enabled )
xen/common/domain.c +0 -0
index 7cc9526139..0a33e0dfd6 100644

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 85

Quoted 4 lines
xen/common/domain.c +1 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
 
  vcpu_info_t dummy_vcpu_info;
 
+bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented in common code?

Cheers,

Signature · 1 line
Julien Grall
#10 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 11 lines · Julien Grall
Quoted 10 lines · Michał Leszczyński
(quoted diff) +0 -0
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)

                 SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS | SECONDARY_EXEC_XSAVES | SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap); if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL ) opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING; if ( opt_vpid_enabled )

xen/common/domain.c +0 -0
index 7cc9526139..0a33e0dfd6 100644
Quoted 8 lines

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 83

Quoted 5 lines
xen/common/domain.c +0 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
     vcpu_info_t dummy_vcpu_info;
   +bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented in common code?

There were some questions directed specifically at the ARM maintainers about CoreSight, which have gone unanswered.

~Andrew

#11 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 21 lines · Andrew Cooper
Quoted 11 lines · Julien Grall
Quoted 10 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmcs.c +0 -1
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)
                  SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                  SECONDARY_EXEC_XSAVES |
                  SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
           if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
               opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
           if ( opt_vpid_enabled )
xen/common/domain.c +0 -0
index 7cc9526139..0a33e0dfd6 100644
Quoted 8 lines

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 83

Quoted 5 lines
xen/common/domain.c +0 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
     vcpu_info_t dummy_vcpu_info;
   +bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented in common code?

Quoted 2 lines

There were some questions directed specifically at the ARM maintainers about CoreSight, which have gone unanswered.

I can only find one question related to the size. Is there any other?

I don't know how the interface will look like given that AFAICT the buffer may be embedded in the HW. We would need to investigate how to differentiate between two domUs in this case without impacting the performance in the common code.

So I think it is a little premature to implement this in common code and always compiled in for Arm. It would be best if this stay in x86 code.

Cheers,

Signature · 1 line
Julien Grall
#12 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 24 lines · Julien Grall
Quoted 21 lines · Andrew Cooper
Quoted 19 lines · Julien Grall
Quoted 13 lines · Michał Leszczyński
(quoted diff) +0 -0
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)

                  SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS | SECONDARY_EXEC_XSAVES | SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap); if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL ) opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING; if ( opt_vpid_enabled )

xen/common/domain.c +0 -0
index 7cc9526139..0a33e0dfd6 100644

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 83

Quoted 2 lines
xen/common/domain.c +0 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
      vcpu_info_t dummy_vcpu_info;
    +bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented in common code?

Quoted 6 lines
Quoted 2 lines

There were some questions directed specifically at the ARM maintainers about CoreSight, which have gone unanswered.

I can only find one question related to the size. Is there any other?

I don't know how the interface will look like given that AFAICT the buffer may be embedded in the HW. We would need to investigate how to differentiate between two domUs in this case without impacting the performance in the common code.

s/in the common code/during the context switch/

Quoted 4 lines

So I think it is a little premature to implement this in common code and always compiled in for Arm. It would be best if this stay in x86 code.

Cheers,

Signature · 1 line
Julien Grall
#13 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 30 lines · Julien Grall
Quoted 27 lines · Julien Grall
Quoted 23 lines · Andrew Cooper
Quoted 11 lines · Julien Grall
Quoted 10 lines · Michał Leszczyński
(quoted diff) +0 -0
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)

                  SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS | SECONDARY_EXEC_XSAVES | SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap); if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL ) opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING; if ( opt_vpid_enabled )

xen/common/domain.c +0 -0
index 7cc9526139..0a33e0dfd6 100644
Quoted 8 lines

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 83

Quoted 5 lines
xen/common/domain.c +0 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
      vcpu_info_t dummy_vcpu_info;
    +bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented in common code?

There were some questions directed specifically at the ARM maintainers about CoreSight, which have gone unanswered.

Quoted 12 lines
Quoted 6 lines

I can only find one question related to the size. Is there any other?

I don't know how the interface will look like given that AFAICT the buffer may be embedded in the HW. We would need to investigate how to differentiate between two domUs in this case without impacting the performance in the common code.

s/in the common code/during the context switch/

Quoted 3 lines

So I think it is a little premature to implement this in common code and always compiled in for Arm. It would be best if this stay in x86 code.

I've just checked with a colleague.  CoreSight can dump to a memory buffer - there's even a decode library for the packet stream
https://github.com/Linaro/OpenCSD, although ultimately it is platform specific as to whether the feature is supported.

Furthermore, the choice isn't "x86 vs ARM", now that RISCv support is on-list, and Power9 is floating on the horizon.

For the sake of what is literally just one byte in common code, I stand my original suggestion of this being a common interface.  It is not something which should be x86 specific.

~Andrew

#14 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Hi,

Quoted 49 lines · Andrew Cooper
Quoted 30 lines · Julien Grall
Quoted 27 lines · Julien Grall
Quoted 23 lines · Andrew Cooper
Quoted 11 lines · Julien Grall
Quoted 10 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmcs.c +0 -1
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)
                   SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                   SECONDARY_EXEC_XSAVES |
                   SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
            if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
                opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
            if ( opt_vpid_enabled )
xen/common/domain.c +0 -0
index 7cc9526139..0a33e0dfd6 100644
Quoted 8 lines

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 83

Quoted 5 lines
xen/common/domain.c +0 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
      vcpu_info_t dummy_vcpu_info;
    +bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented in common code?

There were some questions directed specifically at the ARM maintainers about CoreSight, which have gone unanswered.

Quoted 12 lines
Quoted 6 lines

I can only find one question related to the size. Is there any other?

I don't know how the interface will look like given that AFAICT the buffer may be embedded in the HW. We would need to investigate how to differentiate between two domUs in this case without impacting the performance in the common code.

s/in the common code/during the context switch/

Quoted 3 lines

So I think it is a little premature to implement this in common code and always compiled in for Arm. It would be best if this stay in x86 code.

I've just checked with a colleague.  CoreSight can dump to a memory buffer - there's even a decode library for the packet stream
https://github.com/Linaro/OpenCSD, although ultimately it is platform specific as to whether the feature is supported.

Quoted 6 lines

Furthermore, the choice isn't "x86 vs ARM", now that RISCv support is on-list, and Power9 is floating on the horizon.

For the sake of what is literally just one byte in common code, I stand my original suggestion of this being a common interface.  It is not something which should be x86 specific.

This argument can also be used against putting in common code. What I am the most concern of is we are trying to guess how the interface will look like for another architecture. Your suggested interface may work, but this also may end up to be a complete mess.

So I think we want to wait for a new architecture to use vmtrace before moving to common code. This is not going to be a massive effort to move that bit in common if needed.

Cheers,

Signature · 1 line
Julien Grall
#15 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 61 lines · Julien Grall

Hi,

Quoted 51 lines · Andrew Cooper
Quoted 32 lines · Julien Grall
Quoted 29 lines · Julien Grall
Quoted 25 lines · Andrew Cooper
Quoted 12 lines · Julien Grall
Quoted 11 lines · Michał Leszczyński
(quoted diff) +0 -0
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)

                   SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS | SECONDARY_EXEC_XSAVES | SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap); if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL ) opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING; if ( opt_vpid_enabled )

xen/common/domain.c +0 -1
index 7cc9526139..0a33e0dfd6 100644
--- a/xen/common/domain.c
Quoted 8 lines

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 83

Quoted 4 lines

+++ b/xen/common/domain.c

xen/common/domain.c +0 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
       vcpu_info_t dummy_vcpu_info;
     +bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented
in common code?

There were some questions directed specifically at the ARM maintainers
about CoreSight, which have gone unanswered.

Quoted 12 lines
Quoted 6 lines

I can only find one question related to the size. Is there any other?

I don't know how the interface will look like given that AFAICT the buffer may be embedded in the HW. We would need to investigate how to differentiate between two domUs in this case without impacting the performance in the common code.

s/in the common code/during the context switch/

Quoted 3 lines

So I think it is a little premature to implement this in common code and always compiled in for Arm. It would be best if this stay in x86 code.

I've just checked with a colleague.  CoreSight can dump to a memory buffer - there's even a decode library for the packet stream
https://github.com/Linaro/OpenCSD, although ultimately it is platform specific as to whether the feature is supported.

Quoted 6 lines

Furthermore, the choice isn't "x86 vs ARM", now that RISCv support is on-list, and Power9 is floating on the horizon.

For the sake of what is literally just one byte in common code, I stand my original suggestion of this being a common interface.  It is not something which should be x86 specific.

Quoted 8 lines

This argument can also be used against putting in common code. What I am the most concern of is we are trying to guess how the interface will look like for another architecture. Your suggested interface may work, but this also may end up to be a complete mess.

So I think we want to wait for a new architecture to use vmtrace before moving to common code. This is not going to be a massive effort to move that bit in common if needed.

I suggest you read the series.

The only thing in common code is the bit of the interface saying "I'd like buffers this big please".

~Andrew

#16 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 67 lines · Andrew Cooper
Quoted 66 lines · Julien Grall

Hi,

Quoted 51 lines · Andrew Cooper
Quoted 32 lines · Julien Grall
Quoted 29 lines · Julien Grall
Quoted 25 lines · Andrew Cooper
Quoted 12 lines · Julien Grall
Quoted 11 lines · Michał Leszczyński
(quoted diff) +0 -1
@@ -305,7 +311,6 @@ static int vmx_init_vmcs_config(void)
                    SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                    SECONDARY_EXEC_XSAVES |
                    SECONDARY_EXEC_TSC_SCALING);
-        rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
             if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
                 opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
             if ( opt_vpid_enabled )
xen/common/domain.c +0 -1
index 7cc9526139..0a33e0dfd6 100644
--- a/xen/common/domain.c
Quoted 8 lines

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 83

Quoted 4 lines

+++ b/xen/common/domain.c

xen/common/domain.c +0 -0
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
       vcpu_info_t dummy_vcpu_info;
     +bool_t vmtrace_supported;

All the code looks x86 specific. So may I ask why this was implemented
in common code?

There were some questions directed specifically at the ARM maintainers
about CoreSight, which have gone unanswered.

Quoted 12 lines
Quoted 6 lines

I can only find one question related to the size. Is there any other?

I don't know how the interface will look like given that AFAICT the buffer may be embedded in the HW. We would need to investigate how to differentiate between two domUs in this case without impacting the performance in the common code.

s/in the common code/during the context switch/

Quoted 3 lines

So I think it is a little premature to implement this in common code and always compiled in for Arm. It would be best if this stay in x86 code.

I've just checked with a colleague.  CoreSight can dump to a memory buffer - there's even a decode library for the packet stream
https://github.com/Linaro/OpenCSD, although ultimately it is platform specific as to whether the feature is supported.

Quoted 6 lines

Furthermore, the choice isn't "x86 vs ARM", now that RISCv support is on-list, and Power9 is floating on the horizon.

For the sake of what is literally just one byte in common code, I stand my original suggestion of this being a common interface.  It is not something which should be x86 specific.

This argument can also be used against putting in common code. What I am the most concern of is we are trying to guess how the interface will look like for another architecture. Your suggested interface may work, but this also may end up to be a complete mess.

Quoted 5 lines
Quoted 3 lines

So I think we want to wait for a new architecture to use vmtrace before moving to common code. This is not going to be a massive effort to move that bit in common if needed.

I suggest you read the series.

Already went through the series and ...

Quoted 3 lines

The only thing in common code is the bit of the interface saying "I'd like buffers this big please".

... I stand by my point. There is no need to have this code in common code until someone else need it. This code can be easily implemented in arch_domain_create().

Cheers,

Signature · 1 line
Julien Grall
#17 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 21 lines · Julien Grall
Quoted 11 lines · Andrew Cooper
Quoted 10 lines · Julien Grall
Quoted 3 lines · Andrew Cooper

For the sake of what is literally just one byte in common code, I stand my original suggestion of this being a common interface.  It is not something which should be x86 specific.

This argument can also be used against putting in common code. What I am the most concern of is we are trying to guess how the interface will look like for another architecture. Your suggested interface may work, but this also may end up to be a complete mess.

Quoted 5 lines
Quoted 3 lines

So I think we want to wait for a new architecture to use vmtrace before moving to common code. This is not going to be a massive effort to move that bit in common if needed.

I suggest you read the series.

Already went through the series and ...

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

The only thing in common code is the bit of the interface saying "I'd like buffers this big please".

... I stand by my point. There is no need to have this code in common code until someone else need it. This code can be easily implemented in arch_domain_create().

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values. For example, unless there is proof that for all architectures of interest currently or in the not too distant future an order value is fine (as opposed to a size one), then an order field would be fine to live in common code imo. Otherwise it would need to be a size one, with per-arch enforcement of further imposed restrictions (like needing to be a power of 2).

Jan

#18 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Hi Jan,

Quoted 25 lines · Jan Beulich
Quoted 24 lines · Julien Grall
Quoted 11 lines · Andrew Cooper
Quoted 10 lines · Julien Grall
Quoted 3 lines · Andrew Cooper

For the sake of what is literally just one byte in common code, I stand my original suggestion of this being a common interface.  It is not something which should be x86 specific.

This argument can also be used against putting in common code. What I am the most concern of is we are trying to guess how the interface will look like for another architecture. Your suggested interface may work, but this also may end up to be a complete mess.

Quoted 5 lines
Quoted 3 lines

So I think we want to wait for a new architecture to use vmtrace before moving to common code. This is not going to be a massive effort to move that bit in common if needed.

I suggest you read the series.

Already went through the series and ...

Quoted 3 lines

The only thing in common code is the bit of the interface saying "I'd like buffers this big please".

Quoted 7 lines
Quoted 3 lines

... I stand by my point. There is no need to have this code in common code until someone else need it. This code can be easily implemented in arch_domain_create().

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

Cheers,

Signature · 1 line
Julien Grall
#19 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Quoted 8 lines

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

I don't think I agree here - I see nothing wrong with exposing of the memory being arch specific, when allocation is generic. This is no different from, in just x86, allocation logic being common to PV and HVM, but exposing being different for both.

Jan

#20 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 10 lines · Jan Beulich
Quoted 9 lines · Julien Grall
Quoted 3 lines · Jan Beulich

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

Quoted 8 lines
Quoted 3 lines

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

I don't think I agree here - I see nothing wrong with exposing of the memory being arch specific, when allocation is generic. This is no different from, in just x86, allocation logic being common to PV and HVM, but exposing being different for both.

Are you suggesting that the way it would be exposed may be different for other architecture?

If so, this is one more reason to not impose a way for allocating the buffer in the common code until another arch add support for vmtrace.

Cheers,

Signature · 1 line
Julien Grall
#21 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 17 lines · Julien Grall
Quoted 14 lines · Jan Beulich
Quoted 12 lines · Julien Grall
Quoted 3 lines · Jan Beulich

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

Quoted 7 lines
Quoted 4 lines

I don't think I agree here - I see nothing wrong with exposing of the memory being arch specific, when allocation is generic. This is no different from, in just x86, allocation logic being common to PV and HVM, but exposing being different for both.

Are you suggesting that the way it would be exposed may be different for other architecture?

Why not? To take a possibly extreme example - consider an arch where (for bare metal) the buffer is specified to appear at a fixed range of addresses. This would then want to be this way in the virtualized case as well. There'd be no point in using any common logic mapping the buffer at a guest requested address. Instead it would simply appear at the arch mandated one, without the guest needing to take any action.

Quoted 2 lines

If so, this is one more reason to not impose a way for allocating the buffer in the common code until another arch add support for vmtrace.

I'm still not seeing why allocation and exposure need to be done at the same place.

Jan

#22 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Hi,

Quoted 23 lines · Jan Beulich
Quoted 22 lines · Julien Grall
Quoted 18 lines · Jan Beulich
Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Quoted 8 lines

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

I don't think I agree here - I see nothing wrong with exposing of the memory being arch specific, when allocation is generic. This is no different from, in just x86, allocation logic being common to PV and HVM, but exposing being different for both.

Quoted 6 lines
Quoted 2 lines

Are you suggesting that the way it would be exposed may be different for other architecture?

Why not? To take a possibly extreme example - consider an arch where (for bare metal) the buffer is specified to appear at a fixed range of addresses.

I am probably missing something here... The current goal is the buffer will be mapped in the dom0. Most likely the way to map it will be using the acquire hypercall (unless you invent a brand new one...).

For a guest, you could possibly reserve a fixed range and then map it when creating the vCPU in Xen. But then, you will likely want a fixed size... So why would you bother to ask the user to define the size?

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

Quoted 6 lines
Quoted 3 lines

If so, this is one more reason to not impose a way for allocating the buffer in the common code until another arch add support for vmtrace.

I'm still not seeing why allocation and exposure need to be done at the same place.

If I were going to add support for CoreSight on Arm, then the acquire hypercall is likely going to be the way to go for mapping the resource in the tools. At which point this will need to be common.

I am still not entirely happy to define the interface yet, but this would still be better than trying to make the allocation in common code and the leaving the mapping aside. After all, this is a "little bit" more code.

Cheers,

Signature · 1 line
Julien Grall
#23 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 33 lines · Julien Grall

Hi,

Quoted 23 lines · Jan Beulich
Quoted 22 lines · Julien Grall
Quoted 18 lines · Jan Beulich
Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Quoted 8 lines

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

I don't think I agree here - I see nothing wrong with exposing of the memory being arch specific, when allocation is generic. This is no different from, in just x86, allocation logic being common to PV and HVM, but exposing being different for both.

Quoted 6 lines
Quoted 2 lines

Are you suggesting that the way it would be exposed may be different for other architecture?

Why not? To take a possibly extreme example - consider an arch where (for bare metal) the buffer is specified to appear at a fixed range of addresses.

Quoted 7 lines

I am probably missing something here... The current goal is the buffer will be mapped in the dom0. Most likely the way to map it will be using the acquire hypercall (unless you invent a brand new one...).

For a guest, you could possibly reserve a fixed range and then map it when creating the vCPU in Xen. But then, you will likely want a fixed size... So why would you bother to ask the user to define the size?

Because there may be the option to only populate part of the fixed range?

Quoted 3 lines

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

There may not be any mapping to do in such a contrived, fixed-range environment. This scenario was specifically to demonstrate that the way the mapping gets done may be arch-specific (here: a no-op) despite the allocation not being so.

Jan

#24 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Hi,

Quoted 38 lines · Jan Beulich
Quoted 37 lines · Julien Grall

Hi,

Quoted 23 lines · Jan Beulich
Quoted 22 lines · Julien Grall
Quoted 18 lines · Jan Beulich
Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Quoted 8 lines

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

I don't think I agree here - I see nothing wrong with exposing of the memory being arch specific, when allocation is generic. This is no different from, in just x86, allocation logic being common to PV and HVM, but exposing being different for both.

Quoted 6 lines
Quoted 2 lines

Are you suggesting that the way it would be exposed may be different for other architecture?

Why not? To take a possibly extreme example - consider an arch where (for bare metal) the buffer is specified to appear at a fixed range of addresses.

I am probably missing something here... The current goal is the buffer will be mapped in the dom0. Most likely the way to map it will be using the acquire hypercall (unless you invent a brand new one...).

Quoted 6 lines
Quoted 3 lines

For a guest, you could possibly reserve a fixed range and then map it when creating the vCPU in Xen. But then, you will likely want a fixed size... So why would you bother to ask the user to define the size?

Because there may be the option to only populate part of the fixed range?

It was yet another extreme case ;).

Quoted 9 lines
Quoted 4 lines

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

There may not be any mapping to do in such a contrived, fixed-range environment. This scenario was specifically to demonstrate that the way the mapping gets done may be arch-specific (here: a no-op) despite the allocation not being so.

You are arguing on extreme cases which I don't think is really helpful here. Yes if you want to map at a fixed address in a guest you may not need the acquire hypercall. But in most of the other cases (see has for the tools) you will need it.

So what's the problem with requesting to have the acquire hypercall implemented in common code?

Cheers,

Signature · 1 line
Julien Grall
#25 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 41 lines · Julien Grall

Hi,

Quoted 38 lines · Jan Beulich
Quoted 37 lines · Julien Grall

Hi,

Quoted 23 lines · Jan Beulich
Quoted 22 lines · Julien Grall
Quoted 18 lines · Jan Beulich
Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich

I'm with Andrew here, fwiw, as long as the little bit of code that is actually put in common/ or include/xen/ doesn't imply arbitrary restrictions on acceptable values.

Quoted 8 lines

Well yes the code is simple. However, the code as it is wouldn't be usuable on other architecture without additional work (aside arch specific code). For instance, there is no way to map the buffer outside of Xen as it is all x86 specific.

If you want the allocation to be in the common code, then the infrastructure to map/unmap the buffer should also be in common code. Otherwise, there is no point to allocate it in common.

I don't think I agree here - I see nothing wrong with exposing of the memory being arch specific, when allocation is generic. This is no different from, in just x86, allocation logic being common to PV and HVM, but exposing being different for both.

Quoted 6 lines
Quoted 2 lines

Are you suggesting that the way it would be exposed may be different for other architecture?

Why not? To take a possibly extreme example - consider an arch where (for bare metal) the buffer is specified to appear at a fixed range of addresses.

I am probably missing something here... The current goal is the buffer will be mapped in the dom0. Most likely the way to map it will be using the acquire hypercall (unless you invent a brand new one...).

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

For a guest, you could possibly reserve a fixed range and then map it when creating the vCPU in Xen. But then, you will likely want a fixed size... So why would you bother to ask the user to define the size?

Because there may be the option to only populate part of the fixed range?

It was yet another extreme case ;).

Yes, sure - just to demonstrate my point.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

There may not be any mapping to do in such a contrived, fixed-range environment. This scenario was specifically to demonstrate that the way the mapping gets done may be arch-specific (here: a no-op) despite the allocation not being so.

Quoted 7 lines

You are arguing on extreme cases which I don't think is really helpful here. Yes if you want to map at a fixed address in a guest you may not need the acquire hypercall. But in most of the other cases (see has for the tools) you will need it.

So what's the problem with requesting to have the acquire hypercall implemented in common code?

Didn't we start out by you asking that there be as little common code as possible for the time being? I have no issue with putting the acquire implementation there ...

Jan

#26 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 21 lines · Jan Beulich
Quoted 20 lines · Julien Grall
Quoted 6 lines · Jan Beulich
Quoted 5 lines · Julien Grall
Quoted 4 lines · Jan Beulich
Quoted 3 lines · Julien Grall
Quoted 2 lines · Jan Beulich
Quoted 1 line · Julien Grall

On 02/07/2020 09:29, Jan Beulich wrote:

Quoted 8 lines
Quoted 3 lines

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

There may not be any mapping to do in such a contrived, fixed-range environment. This scenario was specifically to demonstrate that the way the mapping gets done may be arch-specific (here: a no-op) despite the allocation not being so.

You are arguing on extreme cases which I don't think is really helpful here. Yes if you want to map at a fixed address in a guest you may not need the acquire hypercall. But in most of the other cases (see has for the tools) you will need it.

Quoted 5 lines
Quoted 2 lines

So what's the problem with requesting to have the acquire hypercall implemented in common code?

Didn't we start out by you asking that there be as little common code as possible for the time being?

Well as I said I am not in favor of having the allocation in common code, but if you want to keep it then you also want to implement map/unmap in the common code ([1], [2]).

Quoted 2 lines

I have no issue with putting the
acquire implementation there ...

This was definitely not clear given how you argued with extreme cases...

Cheers,

[1] <[email protected]>
[2] <[email protected]>

Signature · 1 line
Julien Grall
#27 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

----- 2 lip 2020 o 16:31, Julien Grall [email protected] napisał(a):

Quoted 36 lines · Jan Beulich
Quoted 21 lines · Julien Grall
Quoted 20 lines · Jan Beulich
Quoted 6 lines · Julien Grall
Quoted 5 lines · Jan Beulich
Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Quoted 1 line · Jan Beulich
Quoted 8 lines
Quoted 3 lines

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

There may not be any mapping to do in such a contrived, fixed-range environment. This scenario was specifically to demonstrate that the way the mapping gets done may be arch-specific (here: a no-op) despite the allocation not being so.

You are arguing on extreme cases which I don't think is really helpful here. Yes if you want to map at a fixed address in a guest you may not need the acquire hypercall. But in most of the other cases (see has for the tools) you will need it.

Quoted 5 lines
Quoted 2 lines

So what's the problem with requesting to have the acquire hypercall implemented in common code?

Didn't we start out by you asking that there be as little common code as possible for the time being?

Well as I said I am not in favor of having the allocation in common code, but if you want to keep it then you also want to implement map/unmap in the common code ([1], [2]).

Quoted 2 lines

I have no issue with putting the
acquire implementation there ...

This was definitely not clear given how you argued with extreme cases...

Quoted 7 lines

Cheers,

[1] <[email protected]>
[2] <[email protected]>

--
Julien Grall

Guys,

could you express your final decision on this topic?

While I understand the discussion and the arguments you've raised, I would like to know what particular elements should be moved where.

So are we going abstract way, or non-abstract-x86 only way?

Best regards,
Michał Leszczyński
CERT Polska

#28 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Hi,

Quoted 43 lines · Michał Leszczyński

----- 2 lip 2020 o 16:31, Julien Grall [email protected] napisał(a):

Quoted 41 lines · Jan Beulich
Quoted 21 lines · Julien Grall
Quoted 20 lines · Jan Beulich
Quoted 6 lines · Julien Grall
Quoted 5 lines · Jan Beulich
Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Quoted 1 line · Jan Beulich
Quoted 8 lines
Quoted 3 lines

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

There may not be any mapping to do in such a contrived, fixed-range environment. This scenario was specifically to demonstrate that the way the mapping gets done may be arch-specific (here: a no-op) despite the allocation not being so.

You are arguing on extreme cases which I don't think is really helpful here. Yes if you want to map at a fixed address in a guest you may not need the acquire hypercall. But in most of the other cases (see has for the tools) you will need it.

Quoted 5 lines
Quoted 2 lines

So what's the problem with requesting to have the acquire hypercall implemented in common code?

Didn't we start out by you asking that there be as little common code as possible for the time being?

Well as I said I am not in favor of having the allocation in common code, but if you want to keep it then you also want to implement map/unmap in the common code ([1], [2]).

Quoted 2 lines

I have no issue with putting the
acquire implementation there ...

This was definitely not clear given how you argued with extreme cases...

Cheers,

[1] <[email protected]>
[2] <[email protected]>

Quoted 7 lines
Quoted 2 lines

--
Julien Grall

Guys,

could you express your final decision on this topic?

Can you move the acquire implementation from x86 to common code?

Cheers,

Signature · 1 line
Julien Grall
#29 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

----- 3 lip 2020 o 9:58, Julien Grall [email protected] napisał(a):

Quoted 52 lines

Hi,

Quoted 49 lines · Michał Leszczyński

----- 2 lip 2020 o 16:31, Julien Grall [email protected] napisał(a):

Quoted 36 lines · Jan Beulich
Quoted 21 lines · Julien Grall
Quoted 20 lines · Jan Beulich
Quoted 6 lines · Julien Grall
Quoted 5 lines · Jan Beulich
Quoted 4 lines · Julien Grall
Quoted 3 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Quoted 1 line · Jan Beulich
Quoted 8 lines
Quoted 3 lines

Another way to do it, would be the toolstack to do the mapping. At which point, you still need an hypercall to do the mapping (probably the hypercall acquire).

There may not be any mapping to do in such a contrived, fixed-range environment. This scenario was specifically to demonstrate that the way the mapping gets done may be arch-specific (here: a no-op) despite the allocation not being so.

You are arguing on extreme cases which I don't think is really helpful here. Yes if you want to map at a fixed address in a guest you may not need the acquire hypercall. But in most of the other cases (see has for the tools) you will need it.

Quoted 5 lines
Quoted 2 lines

So what's the problem with requesting to have the acquire hypercall implemented in common code?

Didn't we start out by you asking that there be as little common code as possible for the time being?

Well as I said I am not in favor of having the allocation in common code, but if you want to keep it then you also want to implement map/unmap in the common code ([1], [2]).

Quoted 2 lines

I have no issue with putting the
acquire implementation there ...

This was definitely not clear given how you argued with extreme cases...

Quoted 7 lines

Cheers,

[1] <[email protected]>
[2] <[email protected]>

--
Julien Grall

Guys,

Quoted 8 lines
Quoted 1 line

could you express your final decision on this topic?

Can you move the acquire implementation from x86 to common code?

Cheers,

--
Julien Grall

Ok, sure. This will be done within the patch v5.

Best regards,
Michał Leszczyński
CERT Polska

#30 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

In reply to Michał Leszczyński

xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(void) · line 298

Quoted 13 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmcs.c +5 -0
index ca94c2bedc..b73d824357 100644
@@ -291,6 +291,12 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    vmtrace_supported = cpu_has_ipt &&
+                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED);

There is a subtle corner case here.  vmx_init_vmcs_config() is called on all CPUs, and is supposed to level things down safely if we find any asymmetry.

If instead you go with something like this:

xen/arch/x86/hvm/vmx/vmcs.c +0 -0
index b73d824357..6960109183 100644
@@ -294,8 +294,8 @@ static int vmx_init_vmcs_config(void)

     rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);

     /* Check whether IPT is supported in VMX operation. */ -    vmtrace_supported = cpu_has_ipt &&
-                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED); +    if ( !(_vmx_misc_cap & VMX_MISC_PT_SUPPORTED) )
+        vmtrace_supported = false;

     if ( _vmx_cpu_based_exec_control & CPU_BASED_ACTIVATE_SECONDARY_CONTROLS )
     {

xen/arch/x86/setup.c +0 -0
index c9b6af826d..9d7822e006 100644
@@ -1092,6 +1092,9 @@ void __init noreturn __start_xen(unsigned long mbi_p)

 #endif
     }

+    /* Set a default for VMTrace before HVM setup occurs. */ +    vmtrace_supported = cpu_has_ipt;
+
     /* Sanitise the raw E820 map to produce a final clean version. */ max_page = raw_max_page = init_e820(memmap_type, &e820_raw);

Then you'll also get a vmtrace_supported=true which works correctly in the Broadwell and no-VT-x case as well.

xen/common/domain.c struct vcpu *idle_vcpu[NR_CPUS] __read_mostly; · line 85

Quoted 9 lines
xen/common/domain.c +1 -0
index 7cc9526139..0a33e0dfd6 100644
@@ -82,6 +82,8 @@ struct vcpu *idle_vcpu[NR_CPUS] __read_mostly;
 
 vcpu_info_t dummy_vcpu_info;
 
+bool_t vmtrace_supported;

bool please.  We're in the process of converting over to C99 bools, and objection was taken to a tree-wide cleanup.

Quoted 21 lines
xen/common/domain.c +1 -0
+
 static void __domain_finalise_shutdown(struct domain *d)
 {
     struct vcpu *v;
xen/include/asm-x86/cpufeature.h +1 -0
index f790d5c1f8..8d7955dd87 100644
@@ -104,6 +104,7 @@
 #define cpu_has_clwb            boot_cpu_has(X86_FEATURE_CLWB)
 #define cpu_has_avx512er        boot_cpu_has(X86_FEATURE_AVX512ER)
 #define cpu_has_avx512cd        boot_cpu_has(X86_FEATURE_AVX512CD)
+#define cpu_has_ipt             boot_cpu_has(X86_FEATURE_IPT)
 #define cpu_has_sha             boot_cpu_has(X86_FEATURE_SHA)
 #define cpu_has_avx512bw        boot_cpu_has(X86_FEATURE_AVX512BW)
 #define cpu_has_avx512vl        boot_cpu_has(X86_FEATURE_AVX512VL)
xen/include/asm-x86/hvm/vmx/vmcs.h +0 -0
index 906810592f..0e9a0b8de6 100644

xen/include/asm-x86/hvm/vmx/vmcs.h extern u32 vmx_secondary_exec_control; · line 286

Quoted 4 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +1 -0
@@ -283,6 +283,7 @@ extern u32 vmx_secondary_exec_control;
 #define VMX_VPID_INVVPID_SINGLE_CONTEXT_RETAINING_GLOBAL 0x80000000000ULL
 extern u64 vmx_ept_vpid_cap;
 
+#define VMX_MISC_PT_SUPPORTED                   0x00004000

VMX_MISC_PROC_TRACE, and ...

xen/include/public/arch-x86/cpufeatureset.h XEN_CPUFEATURE(SMAP, 5*32+20) /*S Supervisor Mode Access Prevention */ · line 220

Quoted 12 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +0 -0
 #define VMX_MISC_CR3_TARGET                     0x01ff0000
 #define VMX_MISC_VMWRITE_ALL                    0x20000000
 
xen/include/public/arch-x86/cpufeatureset.h +1 -0
index 5ca35d9d97..0d3f15f628 100644
@@ -217,6 +217,7 @@ XEN_CPUFEATURE(SMAP,          5*32+20) /*S  Supervisor Mode Access Prevention */
 XEN_CPUFEATURE(AVX512_IFMA,   5*32+21) /*A  AVX-512 Integer Fused Multiply Add */
 XEN_CPUFEATURE(CLFLUSHOPT,    5*32+23) /*A  CLFLUSHOPT instruction */
 XEN_CPUFEATURE(CLWB,          5*32+24) /*A  CLWB instruction */
+XEN_CPUFEATURE(IPT,           5*32+25) /*   Intel Processor Trace */

.. any chance we can spell this out as PROC_TRACE?  The "Intel" part won't be true if any of the other vendors choose to implement this interface to the spec.

Otherwise, LGTM.

~Andrew

#31 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 27 lines · Andrew Cooper

xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(void) · line 298

Quoted 13 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmcs.c +5 -0
index ca94c2bedc..b73d824357 100644
@@ -291,6 +291,12 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    vmtrace_supported = cpu_has_ipt &&
+                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED);

There is a subtle corner case here.  vmx_init_vmcs_config() is called on all CPUs, and is supposed to level things down safely if we find any asymmetry.

If instead you go with something like this:

xen/arch/x86/hvm/vmx/vmcs.c +0 -0
index b73d824357..6960109183 100644
@@ -294,8 +294,8 @@ static int vmx_init_vmcs_config(void)
      rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
Quoted 6 lines


     /* Check whether IPT is supported in VMX operation. */ -    vmtrace_supported = cpu_has_ipt &&
-                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED); +    if ( !(_vmx_misc_cap & VMX_MISC_PT_SUPPORTED) )
+        vmtrace_supported = false;

This is also used during hotplug, so I'm not sure it's safe to turn vmtrace_supported off during runtime, where VMs might be already using it. IMO it would be easier to just set it on the BSP, and then refuse to bring up any AP that doesn't have the feature. TBH I don't think we are likely to find any system with such configuration, but seems more robust than changing vmtrace_supported at runtime.

Roger.

#32 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

Quoted 31 lines · Roger Pau Monné
Quoted 30 lines · Andrew Cooper

xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(void) · line 298

Quoted 13 lines · Michał Leszczyński
xen/arch/x86/hvm/vmx/vmcs.c +5 -0
index ca94c2bedc..b73d824357 100644
@@ -291,6 +291,12 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    vmtrace_supported = cpu_has_ipt &&
+                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED);

There is a subtle corner case here.  vmx_init_vmcs_config() is called on all CPUs, and is supposed to level things down safely if we find any asymmetry.

If instead you go with something like this:

xen/arch/x86/hvm/vmx/vmcs.c +0 -0
index b73d824357..6960109183 100644
@@ -294,8 +294,8 @@ static int vmx_init_vmcs_config(void)
      rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);


     /* Check whether IPT is supported in VMX operation. */ -    vmtrace_supported = cpu_has_ipt &&

Quoted 8 lines
Quoted 3 lines

-                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED); +    if ( !(_vmx_misc_cap & VMX_MISC_PT_SUPPORTED) )
+        vmtrace_supported = false;

This is also used during hotplug, so I'm not sure it's safe to turn vmtrace_supported off during runtime, where VMs might be already using it. IMO it would be easier to just set it on the BSP, and then refuse to bring up any AP that doesn't have the feature.

+1

IOW I also don't think that "vmx_init_vmcs_config() ... is supposed to level things down safely". Instead I think the expectation is for CPU onlining to fail if a CPU lacks features compared to the BSP. As can be implied from what Roger says, doing like what you suggest may be fine during boot, but past that only at times where we know there's no user of a certain feature, and where discarding the feature flag won't lead to other inconsistencies (which may very well mean "never").

Jan

#33 lore raw

Re: [PATCH v4 02/10] x86/vmx: add IPT cpu feature

----- 2 lip 2020 o 10:34, Jan Beulich [email protected] napisał(a):

Quoted 51 lines · Roger Pau Monné
Quoted 31 lines · Andrew Cooper
Quoted 30 lines · Michał Leszczyński

xen/arch/x86/hvm/vmx/vmcs.c static int vmx_init_vmcs_config(void) · line 298

Quoted 13 lines
xen/arch/x86/hvm/vmx/vmcs.c +5 -0
index ca94c2bedc..b73d824357 100644
@@ -291,6 +291,12 @@ static int vmx_init_vmcs_config(void)
         _vmx_cpu_based_exec_control &=
             ~(CPU_BASED_CR8_LOAD_EXITING | CPU_BASED_CR8_STORE_EXITING);
 
+    rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
+
+    /* Check whether IPT is supported in VMX operation. */
+    vmtrace_supported = cpu_has_ipt &&
+                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED);

There is a subtle corner case here.  vmx_init_vmcs_config() is called on all CPUs, and is supposed to level things down safely if we find any asymmetry.

If instead you go with something like this:

xen/arch/x86/hvm/vmx/vmcs.c +0 -0
index b73d824357..6960109183 100644
@@ -294,8 +294,8 @@ static int vmx_init_vmcs_config(void)
      rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);


     /* Check whether IPT is supported in VMX operation. */ -    vmtrace_supported = cpu_has_ipt &&

Quoted 8 lines
Quoted 3 lines

-                        (_vmx_misc_cap & VMX_MISC_PT_SUPPORTED); +    if ( !(_vmx_misc_cap & VMX_MISC_PT_SUPPORTED) )
+        vmtrace_supported = false;

This is also used during hotplug, so I'm not sure it's safe to turn vmtrace_supported off during runtime, where VMs might be already using it. IMO it would be easier to just set it on the BSP, and then refuse to bring up any AP that doesn't have the feature.

+1

IOW I also don't think that "vmx_init_vmcs_config() ... is supposed to level things down safely". Instead I think the expectation is for CPU onlining to fail if a CPU lacks features compared to the BSP. As can be implied from what Roger says, doing like what you suggest may be fine during boot, but past that only at times where we know there's no user of a certain feature, and where discarding the feature flag won't lead to other inconsistencies (which may very well mean "never").

Quoted 1 line

Jan

Ok, I will modify it in a way Roger suggested for the previous patch version. CPU onlining will fail if there is an inconsistency.

Best regards,
Michał Leszczyński
CERT Polska

#34 lore raw

PATCH v4 3/10 tools/libxl: add vmtrace_pt_size parameter

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).

---
 docs/man/xl.cfg.5.pod.in             | 10 ++++++++++
 tools/golang/xenlight/helpers.gen.go |  2 ++
 tools/golang/xenlight/types.gen.go   |  1 +
 tools/libxl/libxl.h                  |  8 ++++++++
 tools/libxl/libxl_create.c           |  1 +
 tools/libxl/libxl_types.idl          |  2 ++
 tools/xl/xl_parse.c                  | 20 ++++++++++++++++++++
 xen/common/domain.c                  | 12 ++++++++++++
 xen/include/public/domctl.h          |  1 +
 9 files changed, 57 insertions(+)
docs/man/xl.cfg.5.pod.in +10 -0
index 0532739c1f..78f434b722 100644
@@ -278,6 +278,16 @@ memory=8096 will report significantly less memory available for use
 than a system with maxmem=8096 memory=8096 due to the memory overhead
 of having to track the unused pages.
 
+=item B<vmtrace_pt_size=BYTES>
+
+Specifies the size of processor trace buffer that would be allocated
+for each vCPU belonging to this domain. Disabled (i.e. B<vmtrace_pt_size=0>
+by default. This must be set to non-zero value in order to be able to
+use processor tracing features with this domain.
+
+B<NOTE>: The size value must be between 4 kB and 4 GB and it must
+be also a power of 2.
+
 =back
 
 =head3 Guest Virtual NUMA Configuration
tools/golang/xenlight/helpers.gen.go +2 -0
index 935d3bc50a..ecace9634e 100644
@@ -1117,6 +1117,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)}
 x.ArchArm.GicVersion = GicVersion(xc.arch_arm.gic_version)
 x.ArchArm.Vuart = VuartType(xc.arch_arm.vuart)
 x.Altp2M = Altp2MMode(xc.altp2m)
+x.VmtracePtOrder = int(xc.vmtrace_pt_order)
 
  return nil}
 
@@ -1592,6 +1593,7 @@ return fmt.Errorf("invalid union key '%v'", x.Type)}
 xc.arch_arm.gic_version = C.libxl_gic_version(x.ArchArm.GicVersion)
 xc.arch_arm.vuart = C.libxl_vuart_type(x.ArchArm.Vuart)
 xc.altp2m = C.libxl_altp2m_mode(x.Altp2M)
+xc.vmtrace_pt_order = C.int(x.VmtracePtOrder)
 
  return nil
  }
tools/golang/xenlight/types.gen.go +1 -0
index 663c1e86b4..f9b07ac862 100644
@@ -516,6 +516,7 @@ GicVersion GicVersion
 Vuart VuartType
 }
 Altp2M Altp2MMode
+VmtracePtOrder int
 }
 
 type domainBuildInfoTypeUnion interface {
tools/libxl/libxl.h +8 -0
index 71709dc585..891e8e28d6 100644
@@ -438,6 +438,14 @@
  */
 #define LIBXL_HAVE_CREATEINFO_PASSTHROUGH 1
 
+/*
+ * LIBXL_HAVE_VMTRACE_PT_ORDER indicates that
+ * libxl_domain_create_info has a vmtrace_pt_order parameter, which
+ * allows to enable pre-allocation of processor tracing buffers
+ * with the given order of size.
+ */
+#define LIBXL_HAVE_VMTRACE_PT_ORDER 1
+
 /*
  * libxl ABI compatibility
  *
tools/libxl/libxl_create.c +1 -0
index 75862dc6ed..651d1f4c0f 100644
@@ -608,6 +608,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
             .max_evtchn_port = b_info->event_channels,
             .max_grant_frames = b_info->max_grant_frames,
             .max_maptrack_frames = b_info->max_maptrack_frames,
+            .vmtrace_pt_order = b_info->vmtrace_pt_order,
         };
 
         if (info->type != LIBXL_DOMAIN_TYPE_PV) {
tools/libxl/libxl_types.idl +2 -0
index 9d3f05f399..1c5dd43e4d 100644
@@ -645,6 +645,8 @@ libxl_domain_build_info = Struct("domain_build_info",[
     # supported by x86 HVM and ARM support is planned.
     ("altp2m", libxl_altp2m_mode),
 
+    ("vmtrace_pt_order", integer),
+
     ], dir=DIR_IN,
        copy_deprecated_fn="libxl__domain_build_info_copy_deprecated",
 )
tools/xl/xl_parse.c +20 -0
index 61b4ef7b7e..4eba224590 100644
@@ -1861,6 +1861,26 @@ void parse_config_data(const char *config_source,
         }
     }
 
+    if (!xlu_cfg_get_long(config, "vmtrace_pt_size", &l, 1) && l) {
+        int32_t shift = 0;
+
+        if (l & (l - 1))
+        {
+            fprintf(stderr, "ERROR: pt buffer size must be a power of 2\n");
+            exit(1);
+        }
+
+        while (l >>= 1) ++shift;
+
+        if (shift <= XEN_PAGE_SHIFT)
+        {
+            fprintf(stderr, "ERROR: too small pt buffer\n");
+            exit(1);
+        }
+
+        b_info->vmtrace_pt_order = shift - XEN_PAGE_SHIFT;
+    }
+
     if (!xlu_cfg_get_list(config, "ioports", &ioports, &num_ioports, 0)) {
         b_info->num_ioports = num_ioports;
         b_info->ioports = calloc(num_ioports, sizeof(*b_info->ioports));
xen/common/domain.c +12 -0
index 0a33e0dfd6..27dcfbac8c 100644
@@ -338,6 +338,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->vmtrace_pt_order && !vmtrace_supported )
+    {
+        dprintk(XENLOG_INFO, "Processor tracing is not supported\n");
+        return -EINVAL;
+    }
+
     return arch_sanitise_domain_config(config);
 }
 
@@ -443,6 +449,12 @@ struct domain *domain_create(domid_t domid,
         d->nr_pirqs = min(d->nr_pirqs, nr_irqs);
 
         radix_tree_init(&d->pirq_tree);
+
+        if ( config->vmtrace_pt_order )
+        {
+            uint32_t shift_val = config->vmtrace_pt_order + PAGE_SHIFT;
+            d->vmtrace_pt_size = (1ULL << shift_val);
+        }
     }
 
     if ( (err = arch_domain_create(d, config)) != 0 )
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;
 
     struct xen_arch_domainconfig arch;
 };
Signature · 1 line
2.20.1
#35 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 24 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).

---
 docs/man/xl.cfg.5.pod.in             | 10 ++++++++++
 tools/golang/xenlight/helpers.gen.go |  2 ++
 tools/golang/xenlight/types.gen.go   |  1 +
 tools/libxl/libxl.h                  |  8 ++++++++
 tools/libxl/libxl_create.c           |  1 +
 tools/libxl/libxl_types.idl          |  2 ++
 tools/xl/xl_parse.c                  | 20 ++++++++++++++++++++
 xen/common/domain.c                  | 12 ++++++++++++
 xen/include/public/domctl.h          |  1 +
 9 files changed, 57 insertions(+)
docs/man/xl.cfg.5.pod.in +0 -0
index 0532739c1f..78f434b722 100644
@@ -278,6 +278,16 @@ memory=8096 will report significantly less memory available for use
 than a system with maxmem=8096 memory=8096 due to the memory overhead

docs/man/xl.cfg.5.pod.in memory=8096 will report significantly less memory available for use · line 288

Quoted 10 lines
docs/man/xl.cfg.5.pod.in +8 -0
 of having to track the unused pages.
 
+=item B<vmtrace_pt_size=BYTES>
+
+Specifies the size of processor trace buffer that would be allocated
+for each vCPU belonging to this domain. Disabled (i.e. B<vmtrace_pt_size=0>
+by default. This must be set to non-zero value in order to be able to
+use processor tracing features with this domain.
+
+B<NOTE>: The size value must be between 4 kB and 4 GB and it must

I think the minimum value is 8kB, since 4kB would be order 0, which is used to signal that the feature is disabled?

tools/xl/xl_parse.c void parse_config_data(const char *config_source, · line 1865

Quoted 10 lines
tools/xl/xl_parse.c +2 -0
index 61b4ef7b7e..4eba224590 100644
@@ -1861,6 +1861,26 @@ void parse_config_data(const char *config_source,
         }
     }
 
+    if (!xlu_cfg_get_long(config, "vmtrace_pt_size", &l, 1) && l) {
+        int32_t shift = 0;

unsigned int? I don't think there's a reason for this to be a fixed width signed integer.

Quoted 39 lines
tools/xl/xl_parse.c +18 -0
+
+        if (l & (l - 1))
+        {
+            fprintf(stderr, "ERROR: pt buffer size must be a power of 2\n");
+            exit(1);
+        }
+
+        while (l >>= 1) ++shift;
+
+        if (shift <= XEN_PAGE_SHIFT)
+        {
+            fprintf(stderr, "ERROR: too small pt buffer\n");
+            exit(1);
+        }
+
+        b_info->vmtrace_pt_order = shift - XEN_PAGE_SHIFT;
+    }
+
     if (!xlu_cfg_get_list(config, "ioports", &ioports, &num_ioports, 0)) {
         b_info->num_ioports = num_ioports;
         b_info->ioports = calloc(num_ioports, sizeof(*b_info->ioports));
xen/common/domain.c +6 -0
index 0a33e0dfd6..27dcfbac8c 100644
@@ -338,6 +338,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( config->vmtrace_pt_order && !vmtrace_supported )
+    {
+        dprintk(XENLOG_INFO, "Processor tracing is not supported\n");
+        return -EINVAL;
+    }
+
     return arch_sanitise_domain_config(config);
 }
 

xen/common/domain.c struct domain *domain_create(domid_t domid, · line 456

Quoted 8 lines
xen/common/domain.c +5 -0
@@ -443,6 +449,12 @@ struct domain *domain_create(domid_t domid,
         d->nr_pirqs = min(d->nr_pirqs, nr_irqs);
 
         radix_tree_init(&d->pirq_tree);
+
+        if ( config->vmtrace_pt_order )
+        {
+            uint32_t shift_val = config->vmtrace_pt_order + PAGE_SHIFT;
+            d->vmtrace_pt_size = (1ULL << shift_val);

I don't think the vmtrace_pt_size domain field has been introduced yet?

Please check that each patch builds on it's own, or else we would break bisectability of the tree.

Also I would consider just storing this directly as an order, there's no reason to convert it back to a size?

Thanks, Roger.

#36 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

In reply to Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines · Michał Leszczyński
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Roger.

#37 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

How should I proceed?

Best regards,
Michał Leszczyński
CERT Polska

#38 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 21 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 6 lines
Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

There might be constrains on struct domain (the hypervisor internal domain tracking structure), but I think you are already using a size field there IIRC.

Quoted 2 lines

How should I proceed?

This is an unstable interface, so we could always change it. It seems like we might want to use a size parameter at some point to take advantage of non physically contiguous buffers, but if there are other blockers that prevent such field from being wider ATM I'm fine with it.

Thanks, Roger.

#39 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 24 lines · Roger Pau Monné
Quoted 23 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 20 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Roger.

Quoted 8 lines
Quoted 4 lines

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

Jan

#40 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 27 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
     uint32_t max_evtchn_port;
     int32_t max_grant_frames;
     int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

Quoted 6 lines
Quoted 3 lines

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

That could work, not sure if it needs to state however that those will be 4K pages, since Arm can have a different minimum page size IIRC? (or that's already the assumption for all number of frames fields) vmtrace_nr_frames seems fine to me.

Roger.

#41 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Hi,

Quoted 31 lines · Roger Pau Monné
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 27 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
      uint32_t max_evtchn_port;
      int32_t max_grant_frames;
      int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

Looking at the rest of the code, the toolstack accepts a 64-bit value. So this would lead to truncation of the buffer if it is bigger than 2^44 bytes.

I agree such buffer is unlikely, yet I still think we want to harden the code whenever we can. So the solution is to either prevent check truncation in libxl or directly use 64-bit in the domctl.

My preference is the latter.

Quoted 5 lines

That could work, not sure if it needs to state however that those will be 4K pages, since Arm can have a different minimum page size IIRC? (or that's already the assumption for all number of frames fields) vmtrace_nr_frames seems fine to me.

The hypercalls interface is using the same page granularity as the hypervisor (i.e 4KB).

While we already support guest using 64KB page granularity, it is impossible to have a 64KB Arm hypervisor in the current state. You are going to either break existing guest (if you switch to 64KB page granularity for the hypercall ABI) or render them insecure (the mimimum mapping in the P2M would be 64KB).

DOMCTLs are not stable yet, so using a number of pages is OK. However, I would strongly suggest to use a number of bytes for any xl/libxl/stable libraries interfaces as this avoids confusion and also make more futureproof.

Cheers,

Signature · 1 line
Julien Grall
#42 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 65 lines · Julien Grall

Hi,

Quoted 31 lines · Roger Pau Monné
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 27 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
      uint32_t max_evtchn_port;
      int32_t max_grant_frames;
      int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

Looking at the rest of the code, the toolstack accepts a 64-bit value. So this would lead to truncation of the buffer if it is bigger than 2^44 bytes.

I agree such buffer is unlikely, yet I still think we want to harden the code whenever we can. So the solution is to either prevent check truncation in libxl or directly use 64-bit in the domctl.

My preference is the latter.

Quoted 5 lines

That could work, not sure if it needs to state however that those will be 4K pages, since Arm can have a different minimum page size IIRC? (or that's already the assumption for all number of frames fields) vmtrace_nr_frames seems fine to me.

The hypercalls interface is using the same page granularity as the hypervisor (i.e 4KB).

While we already support guest using 64KB page granularity, it is impossible to have a 64KB Arm hypervisor in the current state. You are going to either break existing guest (if you switch to 64KB page granularity for the hypercall ABI) or render them insecure (the mimimum mapping in the P2M would be 64KB).

Quoted 4 lines

DOMCTLs are not stable yet, so using a number of pages is OK. However, I would strongly suggest to use a number of bytes for any xl/libxl/stable libraries interfaces as this avoids confusion and also make more futureproof.

If we can't settle on what "page size" means in the public interface (which imo is embarrassing), then how about going with number of kb, like other memory libxl controls do? (I guess using Mb, in line with other config file controls, may end up being too coarse here.) This would likely still allow for a 32-bit field to be wide enough.

Jan

#43 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Hi,

Quoted 71 lines · Jan Beulich
Quoted 65 lines · Julien Grall

Hi,

Quoted 31 lines · Roger Pau Monné
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 27 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
       uint32_t max_evtchn_port;
       int32_t max_grant_frames;
       int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

Looking at the rest of the code, the toolstack accepts a 64-bit value. So this would lead to truncation of the buffer if it is bigger than 2^44 bytes.

I agree such buffer is unlikely, yet I still think we want to harden the code whenever we can. So the solution is to either prevent check truncation in libxl or directly use 64-bit in the domctl.

My preference is the latter.

Quoted 5 lines

That could work, not sure if it needs to state however that those will be 4K pages, since Arm can have a different minimum page size IIRC? (or that's already the assumption for all number of frames fields) vmtrace_nr_frames seems fine to me.

The hypercalls interface is using the same page granularity as the hypervisor (i.e 4KB).

While we already support guest using 64KB page granularity, it is impossible to have a 64KB Arm hypervisor in the current state. You are going to either break existing guest (if you switch to 64KB page granularity for the hypercall ABI) or render them insecure (the mimimum mapping in the P2M would be 64KB).

Quoted 4 lines

DOMCTLs are not stable yet, so using a number of pages is OK. However, I would strongly suggest to use a number of bytes for any xl/libxl/stable libraries interfaces as this avoids confusion and also make more futureproof.

Quoted 5 lines

If we can't settle on what "page size" means in the public interface (which imo is embarrassing), then how about going with number of kb, like other memory libxl controls do? (I guess using Mb, in line with other config file controls, may end up being too coarse here.) This would likely still allow for a 32-bit field to be wide enough.

A 32-bit field would definitely not be able to cover a full address space. So do you mind to explain what is the upper bound you expect here?

Cheers,

Signature · 1 line
Julien Grall
#44 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 74 lines · Julien Grall

Hi,

Quoted 71 lines · Jan Beulich
Quoted 69 lines · Julien Grall

Hi,

Quoted 31 lines · Roger Pau Monné
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 27 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
       uint32_t max_evtchn_port;
       int32_t max_grant_frames;
       int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

Looking at the rest of the code, the toolstack accepts a 64-bit value. So this would lead to truncation of the buffer if it is bigger than 2^44 bytes.

I agree such buffer is unlikely, yet I still think we want to harden the code whenever we can. So the solution is to either prevent check truncation in libxl or directly use 64-bit in the domctl.

My preference is the latter.

Quoted 5 lines

That could work, not sure if it needs to state however that those will be 4K pages, since Arm can have a different minimum page size IIRC? (or that's already the assumption for all number of frames fields) vmtrace_nr_frames seems fine to me.

The hypercalls interface is using the same page granularity as the hypervisor (i.e 4KB).

While we already support guest using 64KB page granularity, it is impossible to have a 64KB Arm hypervisor in the current state. You are going to either break existing guest (if you switch to 64KB page granularity for the hypercall ABI) or render them insecure (the mimimum mapping in the P2M would be 64KB).

DOMCTLs are not stable yet, so using a number of pages is OK. However, I would strongly suggest to use a number of bytes for any xl/libxl/stable libraries interfaces as this avoids confusion and also make more futureproof.

Quoted 8 lines
Quoted 5 lines

If we can't settle on what "page size" means in the public interface (which imo is embarrassing), then how about going with number of kb, like other memory libxl controls do? (I guess using Mb, in line with other config file controls, may end up being too coarse here.) This would likely still allow for a 32-bit field to be wide enough.

A 32-bit field would definitely not be able to cover a full address space. So do you mind to explain what is the upper bound you expect here?

Do you foresee a need for buffer sizes of 4Tb and up?

Jan

#45 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 81 lines · Jan Beulich
Quoted 80 lines · Julien Grall

Hi,

Quoted 76 lines · Jan Beulich
Quoted 65 lines · Julien Grall

Hi,

Quoted 31 lines · Roger Pau Monné
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 27 lines · Michał Leszczyński

----- 2 lip 2020 o 11:00, Roger Pau Monné [email protected] napisał(a):

Quoted 19 lines · Michał Leszczyński

xen/include/public/domctl.h struct xen_domctl_createdomain { · line 95

Quoted 9 lines
xen/include/public/domctl.h +1 -0
index 59bdc28c89..7b8289d436 100644
@@ -92,6 +92,7 @@ struct xen_domctl_createdomain {
        uint32_t max_evtchn_port;
        int32_t max_grant_frames;
        int32_t max_maptrack_frames;
+    uint8_t vmtrace_pt_order;

I've been thinking about this, and even though this is a domctl (so not a stable interface) we might want to consider using a size (or a number of pages) here rather than an order. IPT also supports TOPA mode (kind of a linked list of buffers) that would allow for sizes not rounded to order boundaries to be used, since then only each item in the linked list needs to be rounded to an order boundary, so you could for example use three 4K pages in TOPA mode AFAICT.

Quoted 1 line

Roger.

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

Looking at the rest of the code, the toolstack accepts a 64-bit value. So this would lead to truncation of the buffer if it is bigger than 2^44 bytes.

I agree such buffer is unlikely, yet I still think we want to harden the code whenever we can. So the solution is to either prevent check truncation in libxl or directly use 64-bit in the domctl.

My preference is the latter.

Quoted 5 lines

That could work, not sure if it needs to state however that those will be 4K pages, since Arm can have a different minimum page size IIRC? (or that's already the assumption for all number of frames fields) vmtrace_nr_frames seems fine to me.

The hypercalls interface is using the same page granularity as the hypervisor (i.e 4KB).

While we already support guest using 64KB page granularity, it is impossible to have a 64KB Arm hypervisor in the current state. You are going to either break existing guest (if you switch to 64KB page granularity for the hypercall ABI) or render them insecure (the mimimum mapping in the P2M would be 64KB).

Quoted 4 lines

DOMCTLs are not stable yet, so using a number of pages is OK. However, I would strongly suggest to use a number of bytes for any xl/libxl/stable libraries interfaces as this avoids confusion and also make more futureproof.

If we can't settle on what "page size" means in the public interface (which imo is embarrassing), then how about going with number of kb, like other memory libxl controls do? (I guess using Mb, in line with other config file controls, may end up being too coarse here.) This would likely still allow for a 32-bit field to be wide enough.

Quoted 4 lines
Quoted 2 lines

A 32-bit field would definitely not be able to cover a full address space. So do you mind to explain what is the upper bound you expect here?

Do you foresee a need for buffer sizes of 4Tb and up?

Not I am aware of... However, I think the question was worth it given that "wide enough" can mean anything.

Cheers,

Signature · 1 line
Julien Grall
#46 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

----- 7 lip 2020 o 11:16, Julien Grall [email protected] napisał(a):

Quoted 64 lines · Jan Beulich
Quoted 58 lines · Julien Grall
Quoted 57 lines

Hi,

Quoted 53 lines · Jan Beulich
Quoted 42 lines · Julien Grall

Hi,

Quoted 8 lines · Roger Pau Monné
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Roger Pau Monné
Quoted 4 lines · Michał Leszczyński

In previous versions it was "size" but it was requested to change it to "order" in order to shrink the variable size from uint64_t to uint8_t, because there is limited space for xen_domctl_createdomain structure.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

It's likely I'm missing something here, but I wasn't aware xen_domctl_createdomain had any constrains regarding it's size. It's currently 48bytes which seems fairly small.

Additionally I would guess a uint32_t could do here, if the value passed was "number of pages" rather than "number of bytes"?

Looking at the rest of the code, the toolstack accepts a 64-bit value. So this would lead to truncation of the buffer if it is bigger than 2^44 bytes.

I agree such buffer is unlikely, yet I still think we want to harden the code whenever we can. So the solution is to either prevent check truncation in libxl or directly use 64-bit in the domctl.

My preference is the latter.

Quoted 5 lines

That could work, not sure if it needs to state however that those will be 4K pages, since Arm can have a different minimum page size IIRC? (or that's already the assumption for all number of frames fields) vmtrace_nr_frames seems fine to me.

The hypercalls interface is using the same page granularity as the hypervisor (i.e 4KB).

While we already support guest using 64KB page granularity, it is impossible to have a 64KB Arm hypervisor in the current state. You are going to either break existing guest (if you switch to 64KB page granularity for the hypercall ABI) or render them insecure (the mimimum mapping in the P2M would be 64KB).

Quoted 4 lines

DOMCTLs are not stable yet, so using a number of pages is OK. However, I would strongly suggest to use a number of bytes for any xl/libxl/stable libraries interfaces as this avoids confusion and also make more futureproof.

If we can't settle on what "page size" means in the public interface (which imo is embarrassing), then how about going with number of kb, like other memory libxl controls do? (I guess using Mb, in line with other config file controls, may end up being too coarse here.) This would likely still allow for a 32-bit field to be wide enough.

Quoted 4 lines
Quoted 2 lines

A 32-bit field would definitely not be able to cover a full address space. So do you mind to explain what is the upper bound you expect here?

Do you foresee a need for buffer sizes of 4Tb and up?

Quoted 7 lines

Not I am aware of... However, I think the question was worth it given that "wide enough" can mean anything.

Cheers,

--
Julien Grall

So would it be OK to use uint32_t everywhere and to store the trace buffer size as number of kB? I think this is the most straightforward option.

I would also stick with the name "processor_trace_buf_size" everywhere, both in the hypervisor, ABI and the toolstack, with the respective comments that the size is in kB.

Best regards,
Michał Leszczyński
CERT Polska

#47 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

Quoted 6 lines · Michał Leszczyński

So would it be OK to use uint32_t everywhere and to store the trace buffer size as number of kB? I think this is the most straightforward option.

I would also stick with the name "processor_trace_buf_size" everywhere, both in the hypervisor, ABI and the toolstack, with the respective comments that the size is in kB.

Perhaps even more clearly "processor_trace_buf_kb" then?

Jan

#48 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

----- 7 lip 2020 o 13:21, Jan Beulich [email protected] napisał(a):

Quoted 11 lines · Michał Leszczyński
Quoted 6 lines

So would it be OK to use uint32_t everywhere and to store the trace buffer size as number of kB? I think this is the most straightforward option.

I would also stick with the name "processor_trace_buf_size" everywhere, both in the hypervisor, ABI and the toolstack, with the respective comments that the size is in kB.

Perhaps even more clearly "processor_trace_buf_kb" then?

Jan

Ok.

Best regards,
Michał Leszczyński
CERT Polska

#49 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

In reply to Michał Leszczyński

Hi Michał,

Quoted 13 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to specify the size of per-vCPU trace buffer upon domain creation. This is zero by default (meaning: not enabled).

---

docs/man/xl.cfg.5.pod.in +0 -0
index 0532739c1f..78f434b722 100644

docs/man/xl.cfg.5.pod.in memory=8096 will report significantly less memory available for use · line 281

Quoted 4 lines
docs/man/xl.cfg.5.pod.in +1 -0
@@ -278,6 +278,16 @@ memory=8096 will report significantly less memory available for use
 than a system with maxmem=8096 memory=8096 due to the memory overhead
 of having to track the unused pages.
 
+=item B<vmtrace_pt_size=BYTES>

I don't like much this new configuration name. To me, "pt" sound like passthrough, as in pci passthrough. But it seems to be for "processor trace" (or tracing), isn't it? So if it is, then we have "trace" twice in the name and I don't think that configuration is about tracing the processor tracing feature. (Also I don't think we need to state "vm" in the name easier as every configuration option should be about a vm.)

How about a name that is easier to understand without having to know all the possible abbreviations? Maybe "processor_trace_buffer_size" or similar?

docs/man/xl.cfg.5.pod.in memory=8096 will report significantly less memory available for use · line 289

Quoted 8 lines
docs/man/xl.cfg.5.pod.in +8 -0
+
+Specifies the size of processor trace buffer that would be allocated
+for each vCPU belonging to this domain. Disabled (i.e. B<vmtrace_pt_size=0>
+by default. This must be set to non-zero value in order to be able to
+use processor tracing features with this domain.
+
+B<NOTE>: The size value must be between 4 kB and 4 GB and it must
+be also a power of 2.

Maybe the configuration variable could take KBYTES for kilo-bytes instead of just BYTES since the min is 4kB?

Also that item seems to be in the "Memory Allocation" section, but I don't think that's a good place as the other options are for the size of guest RAM. I don't know in which section this would be better but maybe "Other Options" would be OK.

Quoted 8 lines
docs/man/xl.cfg.5.pod.in +0 -0
 =back
 
 =head3 Guest Virtual NUMA Configuration
tools/xl/xl_parse.c +0 -0
index 61b4ef7b7e..4eba224590 100644

tools/xl/xl_parse.c void parse_config_data(const char *config_source, · line 1869

Quoted 9 lines
tools/xl/xl_parse.c +6 -0
@@ -1861,6 +1861,26 @@ void parse_config_data(const char *config_source,
         }
     }
 
+    if (!xlu_cfg_get_long(config, "vmtrace_pt_size", &l, 1) && l) {
+        int32_t shift = 0;
+
+        if (l & (l - 1))
+        {
+            fprintf(stderr, "ERROR: pt buffer size must be a power of 2\n");

It would be better to state the option name in the error message.

xen/include/public/domctl.h

Quoted 25 lines
tools/xl/xl_parse.c +14 -0
+            exit(1);
+        }
+
+        while (l >>= 1) ++shift;
+
+        if (shift <= XEN_PAGE_SHIFT)
+        {
+            fprintf(stderr, "ERROR: too small pt buffer\n");
+            exit(1);
+        }
+
+        b_info->vmtrace_pt_order = shift - XEN_PAGE_SHIFT;
+    }
+
     if (!xlu_cfg_get_list(config, "ioports", &ioports, &num_ioports, 0)) {
         b_info->num_ioports = num_ioports;
         b_info->ioports = calloc(num_ioports, sizeof(*b_info->ioports));
xen/common/domain.c +0 -0
index 0a33e0dfd6..27dcfbac8c 100644
xen/include/public/domctl.h +0 -0
index 59bdc28c89..7b8289d436 100644

I don't think it's wise to modify the toolstack, the hypervisor, and the hypercall ABI in the same patch. Can you change this last two files in a separate patch?

Thank you,

Signature · 1 line
Anthony PERARD
#50 lore raw

Re: [PATCH v4 03/10] tools/libxl: add vmtrace_pt_size parameter

In reply to Michał Leszczyński

Hi,

Quoted 36 lines · Michał Leszczyński
tools/libxl/libxl.h +8 -0
index 71709dc585..891e8e28d6 100644
@@ -438,6 +438,14 @@
   */
  #define LIBXL_HAVE_CREATEINFO_PASSTHROUGH 1
 
+/*
+ * LIBXL_HAVE_VMTRACE_PT_ORDER indicates that
+ * libxl_domain_create_info has a vmtrace_pt_order parameter, which
+ * allows to enable pre-allocation of processor tracing buffers
+ * with the given order of size.
+ */
+#define LIBXL_HAVE_VMTRACE_PT_ORDER 1
+
  /*
   * libxl ABI compatibility
   *
tools/libxl/libxl_create.c +1 -0
index 75862dc6ed..651d1f4c0f 100644
@@ -608,6 +608,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
              .max_evtchn_port = b_info->event_channels,
              .max_grant_frames = b_info->max_grant_frames,
              .max_maptrack_frames = b_info->max_maptrack_frames,
+            .vmtrace_pt_order = b_info->vmtrace_pt_order,
          };
 
          if (info->type != LIBXL_DOMAIN_TYPE_PV) {
tools/libxl/libxl_types.idl +0 -0
index 9d3f05f399..1c5dd43e4d 100644

tools/libxl/libxl_types.idl libxl_domain_build_info = Struct("domain_build_info",[ · line 648

Quoted 4 lines
tools/libxl/libxl_types.idl +1 -0
@@ -645,6 +645,8 @@ libxl_domain_build_info = Struct("domain_build_info",[
      # supported by x86 HVM and ARM support is planned.
      ("altp2m", libxl_altp2m_mode),
 
+    ("vmtrace_pt_order", integer),

libxl can be used by external projects (such libvirt) for implementing their own toolstack.

While on x86 you always have the same granularity, on Arm the hypervisor and each guest may have a different page granularity (e.g 4KB, 16KB, 64KB). So it is unclear what order one would have to use.

I think it would be best if the external user only specify the number of bytes. You can then sanity check the value and convert to an order (or number of pages) in libxl before passing the value to the hypervisor.

Cheers,

Signature · 1 line
Julien Grall
#51 lore raw

PATCH v4 4/10 x86/vmx: implement processor tracing for VMX

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Use Intel Processor Trace feature in order to
provision vmtrace_pt_* features.

---
 xen/arch/x86/hvm/vmx/vmx.c         | 89 ++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/hvm.h      | 38 +++++++++++++
 xen/include/asm-x86/hvm/vmx/vmcs.h |  3 +
 xen/include/asm-x86/hvm/vmx/vmx.h  | 14 +++++
 4 files changed, 144 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +89 -0
index ab19d9424e..db3f051b40 100644
@@ -508,11 +508,24 @@ static void vmx_restore_host_msrs(void)
 
 static void vmx_save_guest_msrs(struct vcpu *v)
 {
+    uint64_t rtit_ctl;
+
     /*
      * We cannot cache SHADOW_GS_BASE while the VCPU runs, as it can
      * be updated at any time via SWAPGS, which we cannot trap.
      */
     v->arch.hvm.vmx.shadow_gs = rdgsshadow();
+
+    if ( unlikely(v->arch.hvm.vmx.pt_state &&
+                  v->arch.hvm.vmx.pt_state->active) )
+    {
+        rdmsrl(MSR_RTIT_CTL, rtit_ctl);
+        BUG_ON(rtit_ctl & RTIT_CTL_TRACEEN);
+
+        rdmsrl(MSR_RTIT_STATUS, v->arch.hvm.vmx.pt_state->status);
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.pt_state->output_mask.raw);
+    }
 }
 
 static void vmx_restore_guest_msrs(struct vcpu *v)
@@ -524,6 +537,17 @@ static void vmx_restore_guest_msrs(struct vcpu *v)
 
     if ( cpu_has_msr_tsc_aux )
         wrmsr_tsc_aux(v->arch.msrs->tsc_aux);
+
+    if ( unlikely(v->arch.hvm.vmx.pt_state &&
+                  v->arch.hvm.vmx.pt_state->active) )
+    {
+        wrmsrl(MSR_RTIT_OUTPUT_BASE,
+               v->arch.hvm.vmx.pt_state->output_base);
+        wrmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.pt_state->output_mask.raw);
+        wrmsrl(MSR_RTIT_STATUS,
+               v->arch.hvm.vmx.pt_state->status);
+    }
 }
 
 void vmx_update_cpu_exec_control(struct vcpu *v)
@@ -2240,6 +2264,60 @@ static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info)
     return true;
 }
 
+static int vmx_init_pt(struct vcpu *v)
+{
+    v->arch.hvm.vmx.pt_state = xzalloc(struct pt_state);
+
+    if ( !v->arch.hvm.vmx.pt_state )
+        return -EFAULT;
+
+    if ( !v->arch.vmtrace.pt_buf )
+        return -EINVAL;
+
+    if ( !v->domain->vmtrace_pt_size )
+	return -EINVAL;
+
+    v->arch.hvm.vmx.pt_state->output_base = page_to_maddr(v->arch.vmtrace.pt_buf);
+    v->arch.hvm.vmx.pt_state->output_mask.raw = v->domain->vmtrace_pt_size - 1;
+
+    if ( vmx_add_host_load_msr(v, MSR_RTIT_CTL, 0) )
+        return -EFAULT;
+
+    if ( vmx_add_guest_msr(v, MSR_RTIT_CTL,
+                              RTIT_CTL_TRACEEN | RTIT_CTL_OS |
+                              RTIT_CTL_USR | RTIT_CTL_BRANCH_EN) )
+        return -EFAULT;
+
+    return 0;
+}
+
+static int vmx_destroy_pt(struct vcpu* v)
+{
+    if ( v->arch.hvm.vmx.pt_state )
+        xfree(v->arch.hvm.vmx.pt_state);
+
+    v->arch.hvm.vmx.pt_state = NULL;
+    return 0;
+}
+
+static int vmx_control_pt(struct vcpu *v, bool_t enable)
+{
+    if ( !v->arch.hvm.vmx.pt_state )
+        return -EINVAL;
+
+    v->arch.hvm.vmx.pt_state->active = enable;
+    return 0;
+}
+
+static int vmx_get_pt_offset(struct vcpu *v, uint64_t *offset)
+{
+    if ( !v->arch.hvm.vmx.pt_state )
+        return -EINVAL;
+
+    *offset = v->arch.hvm.vmx.pt_state->output_mask.offset;
+    return 0;
+}
+
 static struct hvm_function_table __initdata vmx_function_table = {
     .name                 = "VMX",
     .cpu_up_prepare       = vmx_cpu_up_prepare,
@@ -2295,6 +2373,10 @@ static struct hvm_function_table __initdata vmx_function_table = {
     .altp2m_vcpu_update_vmfunc_ve = vmx_vcpu_update_vmfunc_ve,
     .altp2m_vcpu_emulate_ve = vmx_vcpu_emulate_ve,
     .altp2m_vcpu_emulate_vmfunc = vmx_vcpu_emulate_vmfunc,
+    .vmtrace_init_pt = vmx_init_pt,
+    .vmtrace_destroy_pt = vmx_destroy_pt,
+    .vmtrace_control_pt = vmx_control_pt,
+    .vmtrace_get_pt_offset = vmx_get_pt_offset,
     .tsc_scaling = {
         .max_ratio = VMX_TSC_MULTIPLIER_MAX,
     },
@@ -3674,6 +3756,13 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
 
     hvm_invalidate_regs_fields(regs);
 
+    if ( unlikely(v->arch.hvm.vmx.pt_state &&
+                  v->arch.hvm.vmx.pt_state->active) )
+    {
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.pt_state->output_mask.raw);
+    }
+
     if ( paging_mode_hap(v->domain) )
     {
         /*
xen/include/asm-x86/hvm/hvm.h +38 -0
index 1eb377dd82..8f194889e5 100644
@@ -214,6 +214,12 @@ struct hvm_function_table {
     bool_t (*altp2m_vcpu_emulate_ve)(struct vcpu *v);
     int (*altp2m_vcpu_emulate_vmfunc)(const struct cpu_user_regs *regs);
 
+    /* vmtrace */
+    int (*vmtrace_init_pt)(struct vcpu *v);
+    int (*vmtrace_destroy_pt)(struct vcpu *v);
+    int (*vmtrace_control_pt)(struct vcpu *v, bool_t enable);
+    int (*vmtrace_get_pt_offset)(struct vcpu *v, uint64_t *offset);
+
     /*
      * Parameters and callbacks for hardware-assisted TSC scaling,
      * which are valid only when the hardware feature is available.
@@ -655,6 +661,38 @@ static inline bool altp2m_vcpu_emulate_ve(struct vcpu *v)
     return false;
 }
 
+static inline int vmtrace_init_pt(struct vcpu *v)
+{
+    if ( hvm_funcs.vmtrace_init_pt )
+        return hvm_funcs.vmtrace_init_pt(v);
+
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_destroy_pt(struct vcpu *v)
+{
+    if ( hvm_funcs.vmtrace_destroy_pt )
+        return hvm_funcs.vmtrace_destroy_pt(v);
+
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_control_pt(struct vcpu *v, bool_t enable)
+{
+    if ( hvm_funcs.vmtrace_control_pt )
+        return hvm_funcs.vmtrace_control_pt(v, enable);
+
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_get_pt_offset(struct vcpu *v, uint64_t *offset)
+{
+    if ( hvm_funcs.vmtrace_get_pt_offset )
+        return hvm_funcs.vmtrace_get_pt_offset(v, offset);
+
+    return -EOPNOTSUPP;
+}
+
 /*
  * This must be defined as a macro instead of an inline function,
  * because it uses 'struct vcpu' and 'struct domain' which have
xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0
index 0e9a0b8de6..64c0d82614 100644
@@ -186,6 +186,9 @@ struct vmx_vcpu {
      * pCPU and wakeup the related vCPU.
      */
     struct pi_blocking_vcpu pi_blocking;
+
+    /* State of processor trace feature */
+    struct pt_state      *pt_state;
 };
 
 int vmx_create_vmcs(struct vcpu *v);
xen/include/asm-x86/hvm/vmx/vmx.h +14 -0
index 111ccd7e61..be7213d3c0 100644
@@ -689,4 +689,18 @@ typedef union ldt_or_tr_instr_info {
     };
 } ldt_or_tr_instr_info_t;
 
+/* Processor Trace state per vCPU */
+struct pt_state {
+    bool_t active;
+    uint64_t status;
+    uint64_t output_base;
+    union {
+        uint64_t raw;
+        struct {
+            uint32_t size;
+            uint32_t offset;
+        };
+    } output_mask;
+};
+
 #endif /* __ASM_X86_HVM_VMX_VMX_H__ */
Signature · 1 line
2.20.1
#52 lore raw

Re: [PATCH v4 04/10] x86/vmx: implement processor tracing for VMX

Quoted 22 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Use Intel Processor Trace feature in order to
provision vmtrace_pt_* features.

---
 xen/arch/x86/hvm/vmx/vmx.c         | 89 ++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/hvm.h      | 38 +++++++++++++
 xen/include/asm-x86/hvm/vmx/vmcs.h |  3 +
 xen/include/asm-x86/hvm/vmx/vmx.h  | 14 +++++
 4 files changed, 144 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +1 -0
index ab19d9424e..db3f051b40 100644
@@ -508,11 +508,24 @@ static void vmx_restore_host_msrs(void)
 
 static void vmx_save_guest_msrs(struct vcpu *v)
 {
+    uint64_t rtit_ctl;

xen/arch/x86/hvm/vmx/vmx.c static void vmx_restore_host_msrs(void) · line 521

Quoted 10 lines
xen/arch/x86/hvm/vmx/vmx.c +5 -0
+
     /*
      * We cannot cache SHADOW_GS_BASE while the VCPU runs, as it can
      * be updated at any time via SWAPGS, which we cannot trap.
      */
     v->arch.hvm.vmx.shadow_gs = rdgsshadow();
+
+    if ( unlikely(v->arch.hvm.vmx.pt_state &&
+                  v->arch.hvm.vmx.pt_state->active) )
+    {

Nit: define rtit_ctl here to reduce the scope.

Quoted 29 lines
xen/arch/x86/hvm/vmx/vmx.c +18 -0
+        rdmsrl(MSR_RTIT_CTL, rtit_ctl);
+        BUG_ON(rtit_ctl & RTIT_CTL_TRACEEN);
+
+        rdmsrl(MSR_RTIT_STATUS, v->arch.hvm.vmx.pt_state->status);
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.pt_state->output_mask.raw);
+    }
 }
 
 static void vmx_restore_guest_msrs(struct vcpu *v)
@@ -524,6 +537,17 @@ static void vmx_restore_guest_msrs(struct vcpu *v)
 
     if ( cpu_has_msr_tsc_aux )
         wrmsr_tsc_aux(v->arch.msrs->tsc_aux);
+
+    if ( unlikely(v->arch.hvm.vmx.pt_state &&
+                  v->arch.hvm.vmx.pt_state->active) )
+    {
+        wrmsrl(MSR_RTIT_OUTPUT_BASE,
+               v->arch.hvm.vmx.pt_state->output_base);
+        wrmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.pt_state->output_mask.raw);
+        wrmsrl(MSR_RTIT_STATUS,
+               v->arch.hvm.vmx.pt_state->status);
+    }
 }
 
 void vmx_update_cpu_exec_control(struct vcpu *v)

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2272

Quoted 9 lines
xen/arch/x86/hvm/vmx/vmx.c +6 -0
@@ -2240,6 +2264,60 @@ static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info)
     return true;
 }
 
+static int vmx_init_pt(struct vcpu *v)
+{
+    v->arch.hvm.vmx.pt_state = xzalloc(struct pt_state);
+
+    if ( !v->arch.hvm.vmx.pt_state )
+        return -EFAULT;

-ENOMEM

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2274

Quoted 2 lines
xen/arch/x86/hvm/vmx/vmx.c +2 -0
+
+    if ( !v->arch.vmtrace.pt_buf )

Agian, I'm quite sure this doesn't build, since pt_buf is introduced in patch 5.

I will try to continue to review, but it's quite hard when fields not yet introduced are used in the code, as I have no idea what that is.

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2278

Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c +4 -0
+        return -EINVAL;
+
+    if ( !v->domain->vmtrace_pt_size )
+	return -EINVAL;

Indentation (hard tab), and could be joined with the previous check, since both return -EINVAL.

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2289

Quoted 11 lines
xen/arch/x86/hvm/vmx/vmx.c +11 -0
+
+    v->arch.hvm.vmx.pt_state->output_base = page_to_maddr(v->arch.vmtrace.pt_buf);
+    v->arch.hvm.vmx.pt_state->output_mask.raw = v->domain->vmtrace_pt_size - 1;
+
+    if ( vmx_add_host_load_msr(v, MSR_RTIT_CTL, 0) )
+        return -EFAULT;
+
+    if ( vmx_add_guest_msr(v, MSR_RTIT_CTL,
+                              RTIT_CTL_TRACEEN | RTIT_CTL_OS |
+                              RTIT_CTL_USR | RTIT_CTL_BRANCH_EN) )
+        return -EFAULT;

I think I've already pointed this out before (in v2), but please don't drop the returned error codes from vmx_add_host_load_msr and vmx_add_guest_msr. Please store them in a local variable and return those if != 0.

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2301

Quoted 12 lines
xen/arch/x86/hvm/vmx/vmx.c +12 -0
+
+    return 0;
+}
+
+static int vmx_destroy_pt(struct vcpu* v)
+{
+    if ( v->arch.hvm.vmx.pt_state )
+        xfree(v->arch.hvm.vmx.pt_state);
+
+    v->arch.hvm.vmx.pt_state = NULL;
+    return 0;
+}

I think those should be port of vmx_vcpu_{initialise/destroy}, there's no need to introduce new hooks for it? As the allocation size will be known at domain creation already.

xen/arch/x86/hvm/vmx/vmx.c static bool vmx_get_pending_event(struct vcpu *v, struct x86_event *info) · line 2303

Quoted 1 line
xen/arch/x86/hvm/vmx/vmx.c +1 -0
+static int vmx_control_pt(struct vcpu *v, bool_t enable)

Plain bool.

Quoted 21 lines
xen/arch/x86/hvm/vmx/vmx.c +17 -0
+{
+    if ( !v->arch.hvm.vmx.pt_state )
+        return -EINVAL;
+
+    v->arch.hvm.vmx.pt_state->active = enable;
+    return 0;
+}
+
+static int vmx_get_pt_offset(struct vcpu *v, uint64_t *offset)
+{
+    if ( !v->arch.hvm.vmx.pt_state )
+        return -EINVAL;
+
+    *offset = v->arch.hvm.vmx.pt_state->output_mask.offset;
+    return 0;
+}
+
 static struct hvm_function_table __initdata vmx_function_table = {
     .name                 = "VMX",
     .cpu_up_prepare       = vmx_cpu_up_prepare,

xen/arch/x86/hvm/vmx/vmx.c static struct hvm_function_table __initdata vmx_function_table = { · line 2379

Quoted 7 lines
xen/arch/x86/hvm/vmx/vmx.c +4 -0
@@ -2295,6 +2373,10 @@ static struct hvm_function_table __initdata vmx_function_table = {
     .altp2m_vcpu_update_vmfunc_ve = vmx_vcpu_update_vmfunc_ve,
     .altp2m_vcpu_emulate_ve = vmx_vcpu_emulate_ve,
     .altp2m_vcpu_emulate_vmfunc = vmx_vcpu_emulate_vmfunc,
+    .vmtrace_init_pt = vmx_init_pt,
+    .vmtrace_destroy_pt = vmx_destroy_pt,
+    .vmtrace_control_pt = vmx_control_pt,
+    .vmtrace_get_pt_offset = vmx_get_pt_offset,

As pointed out above, vmtrace_init_pt and vmtrace_destroy_pt should IMO be dropped and instead done in vmx_vcpu_{initialise/destroy}.

Quoted 78 lines
xen/arch/x86/hvm/vmx/vmx.c +7 -0
     .tsc_scaling = {
         .max_ratio = VMX_TSC_MULTIPLIER_MAX,
     },
@@ -3674,6 +3756,13 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
 
     hvm_invalidate_regs_fields(regs);
 
+    if ( unlikely(v->arch.hvm.vmx.pt_state &&
+                  v->arch.hvm.vmx.pt_state->active) )
+    {
+        rdmsrl(MSR_RTIT_OUTPUT_MASK,
+               v->arch.hvm.vmx.pt_state->output_mask.raw);
+    }
+
     if ( paging_mode_hap(v->domain) )
     {
         /*
xen/include/asm-x86/hvm/hvm.h +38 -0
index 1eb377dd82..8f194889e5 100644
@@ -214,6 +214,12 @@ struct hvm_function_table {
     bool_t (*altp2m_vcpu_emulate_ve)(struct vcpu *v);
     int (*altp2m_vcpu_emulate_vmfunc)(const struct cpu_user_regs *regs);
 
+    /* vmtrace */
+    int (*vmtrace_init_pt)(struct vcpu *v);
+    int (*vmtrace_destroy_pt)(struct vcpu *v);
+    int (*vmtrace_control_pt)(struct vcpu *v, bool_t enable);
+    int (*vmtrace_get_pt_offset)(struct vcpu *v, uint64_t *offset);
+
     /*
      * Parameters and callbacks for hardware-assisted TSC scaling,
      * which are valid only when the hardware feature is available.
@@ -655,6 +661,38 @@ static inline bool altp2m_vcpu_emulate_ve(struct vcpu *v)
     return false;
 }
 
+static inline int vmtrace_init_pt(struct vcpu *v)
+{
+    if ( hvm_funcs.vmtrace_init_pt )
+        return hvm_funcs.vmtrace_init_pt(v);
+
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_destroy_pt(struct vcpu *v)
+{
+    if ( hvm_funcs.vmtrace_destroy_pt )
+        return hvm_funcs.vmtrace_destroy_pt(v);
+
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_control_pt(struct vcpu *v, bool_t enable)
+{
+    if ( hvm_funcs.vmtrace_control_pt )
+        return hvm_funcs.vmtrace_control_pt(v, enable);
+
+    return -EOPNOTSUPP;
+}
+
+static inline int vmtrace_get_pt_offset(struct vcpu *v, uint64_t *offset)
+{
+    if ( hvm_funcs.vmtrace_get_pt_offset )
+        return hvm_funcs.vmtrace_get_pt_offset(v, offset);
+
+    return -EOPNOTSUPP;
+}
+
 /*
  * This must be defined as a macro instead of an inline function,
  * because it uses 'struct vcpu' and 'struct domain' which have
xen/include/asm-x86/hvm/vmx/vmcs.h +0 -0
index 0e9a0b8de6..64c0d82614 100644

xen/include/asm-x86/hvm/vmx/vmcs.h struct vmx_vcpu { · line 191

Quoted 6 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0
@@ -186,6 +186,9 @@ struct vmx_vcpu {
      * pCPU and wakeup the related vCPU.
      */
     struct pi_blocking_vcpu pi_blocking;
+
+    /* State of processor trace feature */
+    struct pt_state      *pt_state;

I think it's fine to add this here for now, but we might also consider putting it outside of a HVM specific structure if it's to be used by PV guests. Since all this is HVM specific I'm fine with adding it here.

xen/include/asm-x86/hvm/vmx/vmx.h typedef union ldt_or_tr_instr_info { · line 693

Quoted 13 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +0 -0
 };
 
 int vmx_create_vmcs(struct vcpu *v);
xen/include/asm-x86/hvm/vmx/vmx.h +2 -0
index 111ccd7e61..be7213d3c0 100644
@@ -689,4 +689,18 @@ typedef union ldt_or_tr_instr_info {
     };
 } ldt_or_tr_instr_info_t;
 
+/* Processor Trace state per vCPU */
+struct pt_state {

Please use ipt_state here, since this is an Intel specific structure.

xen/include/asm-x86/hvm/vmx/vmx.h typedef union ldt_or_tr_instr_info { · line 694

Quoted 1 line
xen/include/asm-x86/hvm/vmx/vmx.h +1 -0
+    bool_t active;

Plain bool.

Thanks, Roger.

#53 lore raw

PATCH v4 5/10 common/domain: allocate vmtrace_pt_buffer

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allocate processor trace buffer for each vCPU when the domain is created, deallocate trace buffers on domain destruction.

---
 xen/arch/x86/domain.c        | 11 +++++++++++
 xen/common/domain.c          | 32 ++++++++++++++++++++++++++++++++
 xen/include/asm-x86/domain.h |  4 ++++
 xen/include/xen/sched.h      |  4 ++++
 4 files changed, 51 insertions(+)
xen/arch/x86/domain.c +11 -0
index fee6c3931a..0d79fd390c 100644
@@ -2199,6 +2199,17 @@ int domain_relinquish_resources(struct domain *d)
                 altp2m_vcpu_disable_ve(v);
         }
 
+        for_each_vcpu ( d, v )
+        {
+            if ( !v->arch.vmtrace.pt_buf )
+                continue;
+
+            vmtrace_destroy_pt(v);
+
+            free_domheap_pages(v->arch.vmtrace.pt_buf,
+                get_order_from_bytes(v->domain->vmtrace_pt_size));
+        }
+
         if ( is_pv_domain(d) )
         {
             for_each_vcpu ( d, v )
xen/common/domain.c +32 -0
index 27dcfbac8c..8513659ef8 100644
@@ -137,6 +137,31 @@ static void vcpu_destroy(struct vcpu *v)
     free_vcpu_struct(v);
 }
 
+static int vmtrace_alloc_buffers(struct vcpu *v)
+{
+    struct page_info *pg;
+    uint64_t size = v->domain->vmtrace_pt_size;
+
+    if ( size < PAGE_SIZE || size > GB(4) || (size & (size - 1)) )
+    {
+        /*
+         * We don't accept trace buffer size smaller than single page
+         * and the upper bound is defined as 4GB in the specification.
+         * The buffer size must be also a power of 2.
+         */
+        return -EINVAL;
+    }
+
+    pg = alloc_domheap_pages(v->domain, get_order_from_bytes(size),
+                             MEMF_no_refcount);
+
+    if ( !pg )
+        return -ENOMEM;
+
+    v->arch.vmtrace.pt_buf = pg;
+    return 0;
+}
+
 struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
 {
     struct vcpu *v;
@@ -162,6 +187,9 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
     v->vcpu_id = vcpu_id;
     v->dirty_cpu = VCPU_CPU_CLEAN;
 
+    if ( d->vmtrace_pt_size && vmtrace_alloc_buffers(v) != 0 )
+        return NULL;
+
     spin_lock_init(&v->virq_lock);
 
     tasklet_init(&v->continue_hypercall_tasklet, NULL, NULL);
@@ -188,6 +216,9 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
     if ( arch_vcpu_create(v) != 0 )
         goto fail_sched;
 
+    if ( d->vmtrace_pt_size && vmtrace_init_pt(v) != 0 )
+        goto fail_sched;
+
     d->vcpu[vcpu_id] = v;
     if ( vcpu_id != 0 )
     {
@@ -422,6 +453,7 @@ struct domain *domain_create(domid_t domid,
     d->shutdown_code = SHUTDOWN_CODE_INVALID;
 
     spin_lock_init(&d->pbuf_lock);
+    spin_lock_init(&d->vmtrace_lock);
 
     rwlock_init(&d->vnuma_rwlock);
 
xen/include/asm-x86/domain.h +4 -0
index 6fd94c2e14..b01c107f5c 100644
@@ -627,6 +627,10 @@ struct arch_vcpu
     struct {
         bool next_interrupt_enabled;
     } monitor;
+
+    struct {
+        struct page_info *pt_buf;
+    } vmtrace;
 };
 
 struct guest_memory_policy
xen/include/xen/sched.h +4 -0
index ac53519d7f..48f0a61bbd 100644
@@ -457,6 +457,10 @@ struct domain
     unsigned    pbuf_idx;
     spinlock_t  pbuf_lock;
 
+    /* Used by vmtrace features */
+    spinlock_t  vmtrace_lock;
+    uint64_t    vmtrace_pt_size;
+
     /* OProfile support. */
     struct xenoprof *xenoprof;
 
Signature · 1 line
2.20.1
#54 lore raw

Re: [PATCH v4 05/10] common/domain: allocate vmtrace_pt_buffer

Quoted 40 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allocate processor trace buffer for each vCPU when the domain is created, deallocate trace buffers on domain destruction.

---
 xen/arch/x86/domain.c        | 11 +++++++++++
 xen/common/domain.c          | 32 ++++++++++++++++++++++++++++++++
 xen/include/asm-x86/domain.h |  4 ++++
 xen/include/xen/sched.h      |  4 ++++
 4 files changed, 51 insertions(+)
xen/arch/x86/domain.c +11 -0
index fee6c3931a..0d79fd390c 100644
@@ -2199,6 +2199,17 @@ int domain_relinquish_resources(struct domain *d)
                 altp2m_vcpu_disable_ve(v);
         }
 
+        for_each_vcpu ( d, v )
+        {
+            if ( !v->arch.vmtrace.pt_buf )
+                continue;
+
+            vmtrace_destroy_pt(v);
+
+            free_domheap_pages(v->arch.vmtrace.pt_buf,
+                get_order_from_bytes(v->domain->vmtrace_pt_size));
+        }
+
         if ( is_pv_domain(d) )
         {
             for_each_vcpu ( d, v )
xen/common/domain.c +0 -0
index 27dcfbac8c..8513659ef8 100644

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 143

Quoted 7 lines
xen/common/domain.c +4 -0
@@ -137,6 +137,31 @@ static void vcpu_destroy(struct vcpu *v)
     free_vcpu_struct(v);
 }
 
+static int vmtrace_alloc_buffers(struct vcpu *v)
+{
+    struct page_info *pg;
+    uint64_t size = v->domain->vmtrace_pt_size;

IMO you would be better by just storing an order here (like it's passed from the toolstack), that would avoid the checks and conversion to an order. Also vmtrace_pt_size could be of type unsigned int instead of uint64_t.

Quoted 8 lines
xen/common/domain.c +8 -0
+
+    if ( size < PAGE_SIZE || size > GB(4) || (size & (size - 1)) )
+    {
+        /*
+         * We don't accept trace buffer size smaller than single page
+         * and the upper bound is defined as 4GB in the specification.
+         * The buffer size must be also a power of 2.
+         */

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 161

Quoted 10 lines
xen/common/domain.c +10 -0
+        return -EINVAL;
+    }
+
+    pg = alloc_domheap_pages(v->domain, get_order_from_bytes(size),
+                             MEMF_no_refcount);
+
+    if ( !pg )
+        return -ENOMEM;
+
+    v->arch.vmtrace.pt_buf = pg;

You can assign to pt_buf directly IMO, no need for the pg local variable.

xen/common/domain.c struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id) · line 191

Quoted 12 lines
xen/common/domain.c +5 -0
+    return 0;
+}
+
 struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
 {
     struct vcpu *v;
@@ -162,6 +187,9 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
     v->vcpu_id = vcpu_id;
     v->dirty_cpu = VCPU_CPU_CLEAN;
 
+    if ( d->vmtrace_pt_size && vmtrace_alloc_buffers(v) != 0 )
+        return NULL;

You are leaking the allocated v here, see other error paths below in the function.

Quoted 42 lines
xen/common/domain.c +5 -0
+
     spin_lock_init(&v->virq_lock);
 
     tasklet_init(&v->continue_hypercall_tasklet, NULL, NULL);
@@ -188,6 +216,9 @@ struct vcpu *vcpu_create(struct domain *d, unsigned int vcpu_id)
     if ( arch_vcpu_create(v) != 0 )
         goto fail_sched;
 
+    if ( d->vmtrace_pt_size && vmtrace_init_pt(v) != 0 )
+        goto fail_sched;
+
     d->vcpu[vcpu_id] = v;
     if ( vcpu_id != 0 )
     {
@@ -422,6 +453,7 @@ struct domain *domain_create(domid_t domid,
     d->shutdown_code = SHUTDOWN_CODE_INVALID;
 
     spin_lock_init(&d->pbuf_lock);
+    spin_lock_init(&d->vmtrace_lock);
 
     rwlock_init(&d->vnuma_rwlock);
 
xen/include/asm-x86/domain.h +4 -0
index 6fd94c2e14..b01c107f5c 100644
@@ -627,6 +627,10 @@ struct arch_vcpu
     struct {
         bool next_interrupt_enabled;
     } monitor;
+
+    struct {
+        struct page_info *pt_buf;
+    } vmtrace;
 };
 
 struct guest_memory_policy
xen/include/xen/sched.h +0 -0
index ac53519d7f..48f0a61bbd 100644

xen/include/xen/sched.h struct domain · line 461

Quoted 5 lines
xen/include/xen/sched.h +2 -0
@@ -457,6 +457,10 @@ struct domain
     unsigned    pbuf_idx;
     spinlock_t  pbuf_lock;
 
+    /* Used by vmtrace features */
+    spinlock_t  vmtrace_lock;

Does this need to be per domain or rather per-vcpu? It's hard to tell because there's no user of it in the patch.

Thanks, Roger.

#55 lore raw

Re: [PATCH v4 05/10] common/domain: allocate vmtrace_pt_buffer

In reply to Michał Leszczyński

Hi,

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 149

Quoted 10 lines · Michał Leszczyński
xen/common/domain.c +10 -0
+static int vmtrace_alloc_buffers(struct vcpu *v)
+{
+    struct page_info *pg;
+    uint64_t size = v->domain->vmtrace_pt_size;
+
+    if ( size < PAGE_SIZE || size > GB(4) || (size & (size - 1)) )
+    {
+        /*
+         * We don't accept trace buffer size smaller than single page
+         * and the upper bound is defined as 4GB in the specification.

This is common code, so what specification are you talking about?

I am guessing this is an Intel one, but I don't think Intel should dictate the common code implementation.

xen/common/domain.c static void vcpu_destroy(struct vcpu *v) · line 161

Quoted 12 lines
xen/common/domain.c +12 -0
+         * The buffer size must be also a power of 2.
+         */
+        return -EINVAL;
+    }
+
+    pg = alloc_domheap_pages(v->domain, get_order_from_bytes(size),
+                             MEMF_no_refcount);
+
+    if ( !pg )
+        return -ENOMEM;
+
+    v->arch.vmtrace.pt_buf = pg;

v->arch.vmtrace.pt_buf is not defined on Arm. Please make sure common code build on all arch.

Cheers,

Signature · 1 line
Julien Grall
#56 lore raw

PATCH v4 6/10 memory: batch processing in acquire_resource()

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

---
 xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +29 -3
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
 }
 
 static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
 {
     struct domain *d, *currd = current->domain;
     xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
     /*
      * The mfn_list and gfn_list (below) arrays are ok on stack for the
      * moment since they are small, but if they need to grow in future
@@ -1077,8 +1079,17 @@ static int acquire_resource(
         return 0;
     }
 
+    total_frames = xmar.nr_frames;
+
+    if ( *start_extent )
+    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
     if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);
 
     rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
     if ( rc )
@@ -1135,6 +1146,14 @@ static int acquire_resource(
         }
     }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
  out:
     rcu_unlock_domain(d);
 
@@ -1600,7 +1619,14 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
     case XENMEM_acquire_resource:
         rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+            guest_handle_cast(arg, xen_mem_acquire_resource_t),
+            &start_extent);
+
+        if ( rc == -ERESTART )
+            return hypercall_create_continuation(
+                __HYPERVISOR_memory_op, "lh",
+                op | (start_extent << MEMOP_EXTENT_SHIFT), arg);
+
         break;
 
     default:
Signature · 1 line
2.20.1
#57 lore raw

Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

Quoted 4 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

This patch should be the first of thew series IMO, since it can go in independently of the rest, as it's a general improvement to XENMEM_acquire_resource.

Quoted 43 lines
---
 xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +13 -2
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
 }
 
 static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
 {
     struct domain *d, *currd = current->domain;
     xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
     /*
      * The mfn_list and gfn_list (below) arrays are ok on stack for the
      * moment since they are small, but if they need to grow in future
@@ -1077,8 +1079,17 @@ static int acquire_resource(
         return 0;
     }
 
+    total_frames = xmar.nr_frames;
+
+    if ( *start_extent )
+    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
     if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);
 
     rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
     if ( rc )

xen/common/memory.c static int acquire_resource( · line 1155

Quoted 10 lines
xen/common/memory.c +7 -0
@@ -1135,6 +1146,14 @@ static int acquire_resource(
         }
     }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }

I think you should add some kind of loop here, processing just 32 frames and preempting might be too low. You generally want to loop doing batches of 32 entries until hypercall_preempt_check() returns true.

Thanks, Roger.

#58 lore raw

Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

In reply to Michał Leszczyński

(+ Paul as the author XENMEM_acquire_resource)

Hi,

Quoted 29 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

---
  xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
  1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +3 -1
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
  }
 
  static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
  {
      struct domain *d, *currd = current->domain;
      xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
      /*
       * The mfn_list and gfn_list (below) arrays are ok on stack for the
       * moment since they are small, but if they need to grow in future

xen/common/memory.c static int acquire_resource( · line 1082

Quoted 4 lines
xen/common/memory.c +1 -0
@@ -1077,8 +1079,17 @@ static int acquire_resource(
          return 0;
      }
 
+    total_frames = xmar.nr_frames;

On 32-bit, the start_extent would be 26-bits wide which is not enough to cover all the xmar.nr_frames. Therefore, you want that check that it is possible to encode a continuation. Something like:

/* Is the size too large for us to encode a continuation? */ if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )

xen/common/memory.c static int acquire_resource( · line 1087

Quoted 4 lines
xen/common/memory.c +4 -0
+
+    if ( *start_extent ) > +    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;

As start_extent is exposed to the guest, you want to check if it is not bigger than xmar.nr_frames.

xen/common/memory.c static int acquire_resource( · line 1092

Quoted 6 lines
xen/common/memory.c +4 -1
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
      if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);

The documentation of the hypercall suggests that if you pass NULL, then it will return the maximum number value for nr_frames supported by the implementation. So technically a domain cannot use more than ARRAY_SIZE(mfn_list).

However, you new addition conflict with the documentation. Can you clarify how a domain will know that it can use more than ARRAY_SIZE(mfn_list)?

Quoted 19 lines
xen/common/memory.c +8 -0
      rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
      if ( rc )
@@ -1135,6 +1146,14 @@ static int acquire_resource(
          }
      }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
   out:
      rcu_unlock_domain(d);
 

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1623

Quoted 6 lines
xen/common/memory.c +2 -1
@@ -1600,7 +1619,14 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
      case XENMEM_acquire_resource:
          rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+            guest_handle_cast(arg, xen_mem_acquire_resource_t),
+            &start_extent);

Hmmm... it looks like we forgot to check that start_extent is always 0 when the hypercall was added.

As this is exposed to the guest, it technically means that there no guarantee that start_extent will always be 0.

However, in practice, this was likely the intention and should be the case. So it may just be enough to mention the potential breakage in the commit message.

@All: what do you think?

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1632

Quoted 10 lines
xen/common/memory.c +6 -0
+
+        if ( rc == -ERESTART )
+            return hypercall_create_continuation(
+                __HYPERVISOR_memory_op, "lh",
+                op | (start_extent << MEMOP_EXTENT_SHIFT), arg);
+
          break;
 
      default:

Cheers,

Signature · 1 line
Julien Grall
#59 lore raw

RE: [PATCH v4 06/10] memory: batch processing in acquire_resource()

Quoted 70 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 03 July 2020 11:36
To: Michał Leszczyński <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; [email protected]
Subject: Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

(+ Paul as the author XENMEM_acquire_resource)

Hi,

Quoted 29 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

---
  xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
  1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +3 -1
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
  }
 
  static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
  {
      struct domain *d, *currd = current->domain;
      xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
      /*
       * The mfn_list and gfn_list (below) arrays are ok on stack for the
       * moment since they are small, but if they need to grow in future

xen/common/memory.c static int acquire_resource( · line 1082

Quoted 4 lines
xen/common/memory.c +1 -0
@@ -1077,8 +1079,17 @@ static int acquire_resource(
          return 0;
      }
 
+    total_frames = xmar.nr_frames;

On 32-bit, the start_extent would be 26-bits wide which is not enough to cover all the xmar.nr_frames. Therefore, you want that check that it is possible to encode a continuation. Something like:

/* Is the size too large for us to encode a continuation? */ if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )

xen/common/memory.c static int acquire_resource( · line 1087

Quoted 4 lines
xen/common/memory.c +4 -0
+
+    if ( *start_extent ) > +    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;

As start_extent is exposed to the guest, you want to check if it is not bigger than xmar.nr_frames.

xen/common/memory.c static int acquire_resource( · line 1092

Quoted 6 lines
xen/common/memory.c +4 -1
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
      if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);
Quoted 8 lines

The documentation of the hypercall suggests that if you pass NULL, then it will return the maximum number value for nr_frames supported by the implementation. So technically a domain cannot use more than ARRAY_SIZE(mfn_list).

However, you new addition conflict with the documentation. Can you clarify how a domain will know that it can use more than ARRAY_SIZE(mfn_list)?

The domain should not need to know. It should be told the maximum number of frames of the type it wants. If we have to carve that up into batches inside Xen then the caller should not need to care, right?

Quoted 30 lines
Quoted 20 lines
xen/common/memory.c +8 -0
      rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
      if ( rc )
@@ -1135,6 +1146,14 @@ static int acquire_resource(
          }
      }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
   out:
      rcu_unlock_domain(d);
 

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1623

Quoted 6 lines
xen/common/memory.c +2 -1
@@ -1600,7 +1619,14 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
      case XENMEM_acquire_resource:
          rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+            guest_handle_cast(arg, xen_mem_acquire_resource_t),
+            &start_extent);

Hmmm... it looks like we forgot to check that start_extent is always 0 when the hypercall was added.

Quoted 3 lines

As this is exposed to the guest, it technically means that there no guarantee that start_extent will always be 0.

I don't follow. A start extent != 0 means you are in a continuation. How can you check for 0 without breaking continuations?

Paul

Quoted 21 lines

However, in practice, this was likely the intention and should be the case. So it may just be enough to mention the potential breakage in the commit message.

@All: what do you think?

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1632

Quoted 10 lines
xen/common/memory.c +6 -0
+
+        if ( rc == -ERESTART )
+            return hypercall_create_continuation(
+                __HYPERVISOR_memory_op, "lh",
+                op | (start_extent << MEMOP_EXTENT_SHIFT), arg);
+
          break;
 
      default:

Cheers,

--
Julien Grall

#60 lore raw

Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

Hi,

Quoted 75 lines · Paul Durrant
Quoted 75 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 03 July 2020 11:36
To: Michał Leszczyński <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; [email protected]
Subject: Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

(+ Paul as the author XENMEM_acquire_resource)

Hi,

Quoted 29 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

---
   xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
   1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +3 -1
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
   }
 
   static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
   {
       struct domain *d, *currd = current->domain;
       xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
       /*
        * The mfn_list and gfn_list (below) arrays are ok on stack for the
        * moment since they are small, but if they need to grow in future

xen/common/memory.c static int acquire_resource( · line 1082

Quoted 4 lines
xen/common/memory.c +1 -0
@@ -1077,8 +1079,17 @@ static int acquire_resource(
           return 0;
       }
 
+    total_frames = xmar.nr_frames;

On 32-bit, the start_extent would be 26-bits wide which is not enough to cover all the xmar.nr_frames. Therefore, you want that check that it is possible to encode a continuation. Something like:

/* Is the size too large for us to encode a continuation? */ if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )

xen/common/memory.c static int acquire_resource( · line 1087

Quoted 4 lines
xen/common/memory.c +4 -0
+
+    if ( *start_extent ) > +    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;

As start_extent is exposed to the guest, you want to check if it is not bigger than xmar.nr_frames.

xen/common/memory.c static int acquire_resource( · line 1092

Quoted 6 lines
xen/common/memory.c +4 -1
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
       if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);

The documentation of the hypercall suggests that if you pass NULL, then it will return the maximum number value for nr_frames supported by the implementation. So technically a domain cannot use more than ARRAY_SIZE(mfn_list).

Quoted 5 lines
Quoted 3 lines

However, you new addition conflict with the documentation. Can you clarify how a domain will know that it can use more than ARRAY_SIZE(mfn_list)?

The domain should not need to know. It should be told the maximum number of frames of the type it wants. If we have to carve that up into batches inside Xen then the caller should not need to care, right?

In the current implementation, we tell the guest how many frames it can request in a batch. This number may be much smaller that the maximum number of frames of the type.

Furthermore this value is not tie to the xmar.type. Therefore, it is valid for a guest to call this hypercall only once at boot to figure out the maximum batch.

So while the change you suggest looks a good idea, I don't think it is possible to do that with the current hypercall.

Quoted 31 lines
Quoted 31 lines
Quoted 21 lines
xen/common/memory.c +8 -0
       rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
       if ( rc )
@@ -1135,6 +1146,14 @@ static int acquire_resource(
           }
       }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
    out:
       rcu_unlock_domain(d);
 

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1623

Quoted 6 lines
xen/common/memory.c +2 -1
@@ -1600,7 +1619,14 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
       case XENMEM_acquire_resource:
           rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+            guest_handle_cast(arg, xen_mem_acquire_resource_t),
+            &start_extent);

Hmmm... it looks like we forgot to check that start_extent is always 0 when the hypercall was added.

Quoted 5 lines
Quoted 3 lines

As this is exposed to the guest, it technically means that there no guarantee that start_extent will always be 0.

I don't follow. A start extent != 0 means you are in a continuation. How can you check for 0 without breaking continuations?

I think you misundertood my point. My point is we never checked that start_extent was 0. So a guest could validly pass a non-zero value to start_extent and not break on older Xen release.

When this patch will be merged, such guest would behave differently. Or did I miss any check/documentation for the start_extent value?

Cheers,

Signature · 1 line
Julien Grall
#61 lore raw

Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

Quoted 88 lines · Julien Grall

Hi,

Quoted 75 lines · Paul Durrant
Quoted 75 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 03 July 2020 11:36
To: Michał Leszczyński <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; [email protected]
Subject: Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

(+ Paul as the author XENMEM_acquire_resource)

Hi,

Quoted 29 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

---
   xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
   1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +3 -1
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
   }
 
   static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
   {
       struct domain *d, *currd = current->domain;
       xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
       /*
        * The mfn_list and gfn_list (below) arrays are ok on stack for the
        * moment since they are small, but if they need to grow in future

xen/common/memory.c static int acquire_resource( · line 1082

Quoted 4 lines
xen/common/memory.c +1 -0
@@ -1077,8 +1079,17 @@ static int acquire_resource(
           return 0;
       }
 
+    total_frames = xmar.nr_frames;

On 32-bit, the start_extent would be 26-bits wide which is not enough to cover all the xmar.nr_frames. Therefore, you want that check that it is possible to encode a continuation. Something like:

/* Is the size too large for us to encode a continuation? */ if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )

xen/common/memory.c static int acquire_resource( · line 1087

Quoted 4 lines
xen/common/memory.c +4 -0
+
+    if ( *start_extent ) > +    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;

As start_extent is exposed to the guest, you want to check if it is not bigger than xmar.nr_frames.

xen/common/memory.c static int acquire_resource( · line 1092

Quoted 6 lines
xen/common/memory.c +4 -1
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
       if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);

The documentation of the hypercall suggests that if you pass NULL, then it will return the maximum number value for nr_frames supported by the implementation. So technically a domain cannot use more than ARRAY_SIZE(mfn_list).

Quoted 5 lines
Quoted 3 lines

However, you new addition conflict with the documentation. Can you clarify how a domain will know that it can use more than ARRAY_SIZE(mfn_list)?

The domain should not need to know. It should be told the maximum number of frames of the type it wants. If we have to carve that up into batches inside Xen then the caller should not need to care, right?

In the current implementation, we tell the guest how many frames it can request in a batch. This number may be much smaller that the maximum number of frames of the type.

Quoted 6 lines

Furthermore this value is not tie to the xmar.type. Therefore, it is valid for a guest to call this hypercall only once at boot to figure out the maximum batch.

So while the change you suggest looks a good idea, I don't think it is possible to do that with the current hypercall.

Doesn't the limit simply change to UINT_MAX >> MEMOP_EXTENT_SHIFT, which then is what should be reported?

Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
Quoted 16 lines
xen/common/memory.c +8 -0
@@ -1135,6 +1146,14 @@ static int acquire_resource(
           }
       }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
    out:
       rcu_unlock_domain(d);
 

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1623

Quoted 6 lines
xen/common/memory.c +2 -1
@@ -1600,7 +1619,14 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
       case XENMEM_acquire_resource:
           rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+            guest_handle_cast(arg, xen_mem_acquire_resource_t),
+            &start_extent);

Hmmm... it looks like we forgot to check that start_extent is always 0 when the hypercall was added.

As this is exposed to the guest, it technically means that there no guarantee that start_extent will always be 0.

Quoted 8 lines
Quoted 1 line

I don't follow. A start extent != 0 means you are in a continuation. How can you check for 0 without breaking continuations?

I think you misundertood my point. My point is we never checked that start_extent was 0. So a guest could validly pass a non-zero value to start_extent and not break on older Xen release.

When this patch will be merged, such guest would behave differently. Or did I miss any check/documentation for the start_extent value?

I think we may have done the same in the past already when enabling sub-ops for use of continuations. A guest specifying a non-zero start_extent itself is effectively a request for an undefined sub-op. With, as a result, undefined behavior.

Jan

#62 lore raw

Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

Hi,

Quoted 91 lines · Jan Beulich
Quoted 90 lines · Julien Grall
Quoted 75 lines · Paul Durrant
Quoted 75 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 03 July 2020 11:36
To: Michał Leszczyński <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; [email protected]
Subject: Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

(+ Paul as the author XENMEM_acquire_resource)

Hi,

Quoted 29 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

---
    xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
    1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +3 -1
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
    }
 
    static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
    {
        struct domain *d, *currd = current->domain;
        xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
        /*
         * The mfn_list and gfn_list (below) arrays are ok on stack for the
         * moment since they are small, but if they need to grow in future

xen/common/memory.c static int acquire_resource( · line 1082

Quoted 4 lines
xen/common/memory.c +1 -0
@@ -1077,8 +1079,17 @@ static int acquire_resource(
            return 0;
        }
 
+    total_frames = xmar.nr_frames;

On 32-bit, the start_extent would be 26-bits wide which is not enough to cover all the xmar.nr_frames. Therefore, you want that check that it is possible to encode a continuation. Something like:

/* Is the size too large for us to encode a continuation? */ if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )

xen/common/memory.c static int acquire_resource( · line 1087

Quoted 4 lines
xen/common/memory.c +4 -0
+
+    if ( *start_extent ) > +    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;

As start_extent is exposed to the guest, you want to check if it is not bigger than xmar.nr_frames.

xen/common/memory.c static int acquire_resource( · line 1092

Quoted 6 lines
xen/common/memory.c +4 -1
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
        if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);

The documentation of the hypercall suggests that if you pass NULL, then it will return the maximum number value for nr_frames supported by the implementation. So technically a domain cannot use more than ARRAY_SIZE(mfn_list).

Quoted 5 lines
Quoted 3 lines

However, you new addition conflict with the documentation. Can you clarify how a domain will know that it can use more than ARRAY_SIZE(mfn_list)?

The domain should not need to know. It should be told the maximum number of frames of the type it wants. If we have to carve that up into batches inside Xen then the caller should not need to care, right?

In the current implementation, we tell the guest how many frames it can request in a batch. This number may be much smaller that the maximum number of frames of the type.

Furthermore this value is not tie to the xmar.type. Therefore, it is valid for a guest to call this hypercall only once at boot to figure out the maximum batch.

Quoted 5 lines
Quoted 2 lines

So while the change you suggest looks a good idea, I don't think it is possible to do that with the current hypercall.

Doesn't the limit simply change to UINT_MAX >> MEMOP_EXTENT_SHIFT, which then is what should be reported?

Hmmm... Can you remind me whether we support migration to an older release?

But it may stilln't be a concern as this can only be used by Dom0 or a PV domain targeting another domain.

Quoted 37 lines
Quoted 37 lines
Quoted 27 lines
Quoted 27 lines
Quoted 17 lines
xen/common/memory.c +8 -0
@@ -1135,6 +1146,14 @@ static int acquire_resource(
            }
        }
 
+    if ( !rc )
+    {
+        *start_extent += xmar.nr_frames;
+
+        if ( *start_extent != total_frames )
+            rc = -ERESTART;
+    }
+
     out:
        rcu_unlock_domain(d);
 

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1623

Quoted 6 lines
xen/common/memory.c +2 -1
@@ -1600,7 +1619,14 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
        case XENMEM_acquire_resource:
            rc = acquire_resource(
-            guest_handle_cast(arg, xen_mem_acquire_resource_t));
+            guest_handle_cast(arg, xen_mem_acquire_resource_t),
+            &start_extent);

Hmmm... it looks like we forgot to check that start_extent is always 0 when the hypercall was added.

Quoted 5 lines
Quoted 3 lines

As this is exposed to the guest, it technically means that there no guarantee that start_extent will always be 0.

I don't follow. A start extent != 0 means you are in a continuation. How can you check for 0 without breaking continuations?

I think you misundertood my point. My point is we never checked that start_extent was 0. So a guest could validly pass a non-zero value to start_extent and not break on older Xen release.

Quoted 7 lines
Quoted 2 lines

When this patch will be merged, such guest would behave differently. Or did I miss any check/documentation for the start_extent value?

I think we may have done the same in the past already when enabling sub-ops for use of continuations. A guest specifying a non-zero start_extent itself is effectively a request for an undefined sub-op. With, as a result, undefined behavior.

Ok. So just mentioning the change in the commit message should be fine then.

Cheers,

Signature · 1 line
Julien Grall
#63 lore raw

Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

Quoted 10 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Julien Grall

In the current implementation, we tell the guest how many frames it can request in a batch. This number may be much smaller that the maximum number of frames of the type.

Furthermore this value is not tie to the xmar.type. Therefore, it is valid for a guest to call this hypercall only once at boot to figure out the maximum batch.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

So while the change you suggest looks a good idea, I don't think it is possible to do that with the current hypercall.

Doesn't the limit simply change to UINT_MAX >> MEMOP_EXTENT_SHIFT, which then is what should be reported?

Hmmm... Can you remind me whether we support migration to an older release?

I'm pretty sure we say "N -> N+1 only" somewhere, but this "somewhere" clearly isn't SUPPORT.md.

Jan

#64 lore raw

RE: [PATCH v4 06/10] memory: batch processing in acquire_resource()

In reply to Julien Grall
Quoted 101 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 03 July 2020 12:18
To: [email protected]; 'Michał Leszczyński' <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; 'Andrew Cooper' <[email protected]>; 'George Dunlap' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Jan Beulich' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

Hi,

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 03 July 2020 11:36
To: Michał Leszczyński <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Andrew Cooper <[email protected]>;

George

Quoted 2 lines
Quoted 2 lines

Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>;

[email protected]

Quoted 68 lines
Quoted 68 lines

Subject: Re: [PATCH v4 06/10] memory: batch processing in acquire_resource()

(+ Paul as the author XENMEM_acquire_resource)

Hi,

Quoted 29 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to acquire large resources by allowing acquire_resource() to process items in batches, using hypercall continuation.

---
   xen/common/memory.c | 32 +++++++++++++++++++++++++++++---
   1 file changed, 29 insertions(+), 3 deletions(-)
xen/common/memory.c +3 -1
index 714077c1e5..3ab06581a2 100644
@@ -1046,10 +1046,12 @@ static int acquire_grant_table(struct domain *d, unsigned int id,
   }
 
   static int acquire_resource(
-    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
+    XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg,
+    unsigned long *start_extent)
   {
       struct domain *d, *currd = current->domain;
       xen_mem_acquire_resource_t xmar;
+    uint32_t total_frames;
       /*
        * The mfn_list and gfn_list (below) arrays are ok on stack for the
        * moment since they are small, but if they need to grow in future

xen/common/memory.c static int acquire_resource( · line 1082

Quoted 4 lines
xen/common/memory.c +1 -0
@@ -1077,8 +1079,17 @@ static int acquire_resource(
           return 0;
       }
 
+    total_frames = xmar.nr_frames;

On 32-bit, the start_extent would be 26-bits wide which is not enough to cover all the xmar.nr_frames. Therefore, you want that check that it is possible to encode a continuation. Something like:

/* Is the size too large for us to encode a continuation? */ if ( unlikely(xmar.nr_frames > (UINT_MAX >> MEMOP_EXTENT_SHIFT)) )

xen/common/memory.c static int acquire_resource( · line 1087

Quoted 4 lines
xen/common/memory.c +4 -0
+
+    if ( *start_extent ) > +    {
+        xmar.frame += *start_extent;
+        xmar.nr_frames -= *start_extent;

As start_extent is exposed to the guest, you want to check if it is not bigger than xmar.nr_frames.

xen/common/memory.c static int acquire_resource( · line 1092

Quoted 6 lines
xen/common/memory.c +4 -1
+        guest_handle_add_offset(xmar.frame_list, *start_extent);
+    }
+
       if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
-        return -E2BIG;
+        xmar.nr_frames = ARRAY_SIZE(mfn_list);

The documentation of the hypercall suggests that if you pass NULL, then it will return the maximum number value for nr_frames supported by the implementation. So technically a domain cannot use more than ARRAY_SIZE(mfn_list).

Quoted 5 lines
Quoted 3 lines

However, you new addition conflict with the documentation. Can you clarify how a domain will know that it can use more than ARRAY_SIZE(mfn_list)?

The domain should not need to know. It should be told the maximum number of frames of the type it

wants. If we have to carve that up into batches inside Xen then the caller should not need to care, right?

In the current implementation, we tell the guest how many frames it can request in a batch. This number may be much smaller that the maximum number of frames of the type.

Quoted 7 lines

Furthermore this value is not tie to the xmar.type. Therefore, it is valid for a guest to call this hypercall only once at boot to figure out the maximum batch.

So while the change you suggest looks a good idea, I don't think it is possible to do that with the current hypercall.

Oh, I was clearly misremembering what the semantic was; I thought it was implementation max for the given type but indeed we do just return the array size, so we expect the caller to know the individual resource type limitations. So, as Jan says, passing back UINT_MAX >> MEMOP_EXTENT_SHIFT seems to be what we need.

Paul

#65 lore raw

PATCH v4 7/10 x86/mm: add vmtrace_buf resource type

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to map processor trace buffer using
acquire_resource().

---
 xen/arch/x86/mm.c           | 25 +++++++++++++++++++++++++
 xen/include/public/memory.h |  1 +
 2 files changed, 26 insertions(+)
xen/arch/x86/mm.c +25 -0
index e376fc7e8f..bb781bd90c 100644
@@ -4624,6 +4624,31 @@ int arch_acquire_resource(struct domain *d, unsigned int type,
         }
         break;
     }
+
+    case XENMEM_resource_vmtrace_buf:
+    {
+        mfn_t mfn;
+        unsigned int i;
+        struct vcpu *v = domain_vcpu(d, id);
+        rc = -EINVAL;
+
+        if ( !v )
+            break;
+
+        if ( !v->arch.vmtrace.pt_buf )
+            break;
+
+        mfn = page_to_mfn(v->arch.vmtrace.pt_buf);
+
+        if ( frame + nr_frames > (v->domain->vmtrace_pt_size >> PAGE_SHIFT) )
+            break;
+
+        rc = 0;
+        for ( i = 0; i < nr_frames; i++ )
+            mfn_list[i] = mfn_x(mfn_add(mfn, frame + i));
+
+        break;
+    }
 #endif
 
     default:
xen/include/public/memory.h +1 -0
index dbd35305df..f823c784c3 100644
@@ -620,6 +620,7 @@ struct xen_mem_acquire_resource {
 
 #define XENMEM_resource_ioreq_server 0
 #define XENMEM_resource_grant_table 1
+#define XENMEM_resource_vmtrace_buf 2
 
     /*
      * IN - a type-specific resource identifier, which must be zero
Signature · 1 line
2.20.1
#66 lore raw

Re: [PATCH v4 07/10] x86/mm: add vmtrace_buf resource type

Quoted 16 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Allow to map processor trace buffer using
acquire_resource().

---
 xen/arch/x86/mm.c           | 25 +++++++++++++++++++++++++
 xen/include/public/memory.h |  1 +
 2 files changed, 26 insertions(+)
xen/arch/x86/mm.c +0 -0
index e376fc7e8f..bb781bd90c 100644

xen/arch/x86/mm.c int arch_acquire_resource(struct domain *d, unsigned int type, · line 4632

Quoted 9 lines
xen/arch/x86/mm.c +6 -0
@@ -4624,6 +4624,31 @@ int arch_acquire_resource(struct domain *d, unsigned int type,
         }
         break;
     }
+
+    case XENMEM_resource_vmtrace_buf:
+    {
+        mfn_t mfn;
+        unsigned int i;
+        struct vcpu *v = domain_vcpu(d, id);

Missing blank newline between variable definitions and code.

xen/arch/x86/mm.c int arch_acquire_resource(struct domain *d, unsigned int type, · line 4644

Quoted 12 lines
xen/arch/x86/mm.c +12 -0
+        rc = -EINVAL;
+
+        if ( !v )
+            break;
+
+        if ( !v->arch.vmtrace.pt_buf )
+            break;
+
+        mfn = page_to_mfn(v->arch.vmtrace.pt_buf);
+
+        if ( frame + nr_frames > (v->domain->vmtrace_pt_size >> PAGE_SHIFT) )
+            break;

You can place all the checks done above in a single if.

Thanks, Roger.

#67 lore raw

PATCH v4 8/10 x86/domctl: add XEN_DOMCTL_vmtrace_op

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Implement domctl to manage the runtime state of
processor trace feature.

---
 xen/arch/x86/domctl.c       | 48 +++++++++++++++++++++++++++++++++++++
 xen/include/public/domctl.h | 26 ++++++++++++++++++++
 2 files changed, 74 insertions(+)
xen/arch/x86/domctl.c +48 -0
index 6f2c69788d..a041b724d8 100644
@@ -322,6 +322,48 @@ void arch_get_domain_info(const struct domain *d,
     info->arch_config.emulation_flags = d->arch.emulation_flags;
 }
 
+static int do_vmtrace_op(struct domain *d, struct xen_domctl_vmtrace_op *op,
+                         XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
+{
+    int rc;
+    struct vcpu *v;
+
+    if ( !vmtrace_supported )
+        return -EOPNOTSUPP;
+
+    if ( !is_hvm_domain(d) )
+        return -EOPNOTSUPP;
+
+    if ( op->vcpu >= d->max_vcpus )
+        return -EINVAL;
+
+    v = domain_vcpu(d, op->vcpu);
+    rc = 0;
+
+    switch ( op->cmd )
+    {
+    case XEN_DOMCTL_vmtrace_pt_enable:
+    case XEN_DOMCTL_vmtrace_pt_disable:
+        vcpu_pause(v);
+        spin_lock(&d->vmtrace_lock);
+
+        rc = vmtrace_control_pt(v, op->cmd == XEN_DOMCTL_vmtrace_pt_enable);
+
+        spin_unlock(&d->vmtrace_lock);
+        vcpu_unpause(v);
+        break;
+
+    case XEN_DOMCTL_vmtrace_pt_get_offset:
+        rc = vmtrace_get_pt_offset(v, &op->offset);
+        break;
+
+    default:
+        rc = -EOPNOTSUPP;
+    }
+
+    return rc;
+}
+
 #define MAX_IOPORTS 0x10000
 
 long arch_do_domctl(
@@ -337,6 +379,12 @@ long arch_do_domctl(
     switch ( domctl->cmd )
     {
 
+    case XEN_DOMCTL_vmtrace_op:
+        ret = do_vmtrace_op(d, &domctl->u.vmtrace_op, u_domctl);
+        if ( !ret )
+            copyback = true;
+	break;
+
     case XEN_DOMCTL_shadow_op:
         ret = paging_domctl(d, &domctl->u.shadow_op, u_domctl, 0);
         if ( ret == -ERESTART )
xen/include/public/domctl.h +26 -0
index 7b8289d436..f836cb5970 100644
@@ -1136,6 +1136,28 @@ struct xen_domctl_vuart_op {
                                  */
 };
 
+/* XEN_DOMCTL_vmtrace_op: Perform VM tracing related operation */
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+struct xen_domctl_vmtrace_op {
+    /* IN variable */
+    uint32_t cmd;
+/* Enable/disable external vmtrace for given domain */
+#define XEN_DOMCTL_vmtrace_pt_enable      1
+#define XEN_DOMCTL_vmtrace_pt_disable     2
+#define XEN_DOMCTL_vmtrace_pt_get_offset  3
+    domid_t domain;
+    uint32_t vcpu;
+    uint64_aligned_t size;
+
+    /* OUT variable */
+    uint64_aligned_t offset;
+};
+typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
+
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 struct xen_domctl {
     uint32_t cmd;
 #define XEN_DOMCTL_createdomain                   1
@@ -1217,6 +1239,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_vuart_op                      81
 #define XEN_DOMCTL_get_cpu_policy                82
 #define XEN_DOMCTL_set_cpu_policy                83
+#define XEN_DOMCTL_vmtrace_op                    84
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002
@@ -1277,6 +1300,9 @@ struct xen_domctl {
         struct xen_domctl_monitor_op        monitor_op;
         struct xen_domctl_psr_alloc         psr_alloc;
         struct xen_domctl_vuart_op          vuart_op;
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+        struct xen_domctl_vmtrace_op        vmtrace_op;
+#endif
         uint8_t                             pad[128];
     } u;
 };
Signature · 1 line
2.20.1
#68 lore raw

Re: [PATCH v4 08/10] x86/domctl: add XEN_DOMCTL_vmtrace_op

Quoted 20 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Implement domctl to manage the runtime state of
processor trace feature.

---
 xen/arch/x86/domctl.c       | 48 +++++++++++++++++++++++++++++++++++++
 xen/include/public/domctl.h | 26 ++++++++++++++++++++
 2 files changed, 74 insertions(+)
xen/arch/x86/domctl.c +1 -0
index 6f2c69788d..a041b724d8 100644
@@ -322,6 +322,48 @@ void arch_get_domain_info(const struct domain *d,
     info->arch_config.emulation_flags = d->arch.emulation_flags;
 }
 
+static int do_vmtrace_op(struct domain *d, struct xen_domctl_vmtrace_op *op,

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 335

Quoted 10 lines
xen/arch/x86/domctl.c +10 -0
+                         XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
+{
+    int rc;
+    struct vcpu *v;
+
+    if ( !vmtrace_supported )
+        return -EOPNOTSUPP;
+
+    if ( !is_hvm_domain(d) )
+        return -EOPNOTSUPP;

You can join both checks.

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 341

Quoted 6 lines
xen/arch/x86/domctl.c +6 -0
+
+    if ( op->vcpu >= d->max_vcpus )
+        return -EINVAL;
+
+    v = domain_vcpu(d, op->vcpu);
+    rc = 0;

No need to init rc to zero, after the switch below it will always be initialized.

Quoted 6 lines
xen/arch/x86/domctl.c +6 -0
+
+    switch ( op->cmd )
+    {
+    case XEN_DOMCTL_vmtrace_pt_enable:
+    case XEN_DOMCTL_vmtrace_pt_disable:
+        vcpu_pause(v);

xen/arch/x86/domctl.c void arch_get_domain_info(const struct domain *d, · line 357

Quoted 10 lines
xen/arch/x86/domctl.c +10 -0
+        spin_lock(&d->vmtrace_lock);
+
+        rc = vmtrace_control_pt(v, op->cmd == XEN_DOMCTL_vmtrace_pt_enable);
+
+        spin_unlock(&d->vmtrace_lock);
+        vcpu_unpause(v);
+        break;
+
+    case XEN_DOMCTL_vmtrace_pt_get_offset:
+        rc = vmtrace_get_pt_offset(v, &op->offset);

Since you don't pause the vcpu here, I think you want to use atomic operations to update v->arch.hvm.vmx.pt_state->output_mask.raw, or else you could see inconsistent results if a vmexit is updating it in parallel? (since you don't pause the target vcpu).

Thanks, Roger.

#69 lore raw

PATCH v4 9/10 tools/libxc: add xc_vmtrace_* functions

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add functions in libxc that use the new XEN_DOMCTL_vmtrace interface.

---
 tools/libxc/Makefile          |  1 +
 tools/libxc/include/xenctrl.h | 39 +++++++++++++++++++
 tools/libxc/xc_vmtrace.c      | 73 +++++++++++++++++++++++++++++++++++
 3 files changed, 113 insertions(+)
 create mode 100644 tools/libxc/xc_vmtrace.c
tools/libxc/Makefile +1 -0
index fae5969a73..605e44501d 100644
@@ -27,6 +27,7 @@ CTRL_SRCS-y       += xc_csched2.c
 CTRL_SRCS-y       += xc_arinc653.c
 CTRL_SRCS-y       += xc_rt.c
 CTRL_SRCS-y       += xc_tbuf.c
+CTRL_SRCS-y       += xc_vmtrace.c
 CTRL_SRCS-y       += xc_pm.c
 CTRL_SRCS-y       += xc_cpu_hotplug.c
 CTRL_SRCS-y       += xc_resume.c
tools/libxc/include/xenctrl.h +39 -0
index 113ddd935d..66966f6c17 100644
@@ -1585,6 +1585,45 @@ int xc_tbuf_set_cpu_mask(xc_interface *xch, xc_cpumap_t mask);
 
 int xc_tbuf_set_evt_mask(xc_interface *xch, uint32_t mask);
 
+/**
+ * Enable processor trace for given vCPU in given DomU.
+ * Allocate the trace ringbuffer with given size.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_enable(xc_interface *xch, uint32_t domid,
+                         uint32_t vcpu);
+
+/**
+ * Disable processor trace for given vCPU in given DomU.
+ * Deallocate the trace ringbuffer.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_disable(xc_interface *xch, uint32_t domid,
+                          uint32_t vcpu);
+
+/**
+ * Get current offset inside the trace ringbuffer.
+ * This allows to determine how much data was written into the buffer.
+ * Once buffer overflows, the offset will reset to 0 and the previous
+ * data will be overriden.
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid domain identifier
+ * @parm vcpu vcpu identifier
+ * @parm offset current offset inside trace buffer will be written there
+ * @return 0 on success, -1 on failure
+ */
+int xc_vmtrace_pt_get_offset(xc_interface *xch, uint32_t domid,
+                             uint32_t vcpu, uint64_t *offset);
+
 int xc_domctl(xc_interface *xch, struct xen_domctl *domctl);
 int xc_sysctl(xc_interface *xch, struct xen_sysctl *sysctl);
 
tools/libxc/xc_vmtrace.c new file +73 -0

73 lines not on this page · raw message

Signature · 1 line
2.20.1
#71 lore raw

PATCH v4 10/10 tools/proctrace: add proctrace tool

In reply to Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Add an demonstration tool that uses xc_vmtrace_* calls in order to manage external IPT monitoring for DomU.

---
 tools/proctrace/COPYING     | 339 ++++++++++++++++++++++++++++++++++++
 tools/proctrace/Makefile    |  48 +++++
 tools/proctrace/proctrace.c | 163 +++++++++++++++++
 3 files changed, 550 insertions(+)
 create mode 100644 tools/proctrace/COPYING
 create mode 100644 tools/proctrace/Makefile
 create mode 100644 tools/proctrace/proctrace.c
tools/proctrace/COPYING new file +339 -0
new file mode 100644
index 0000000000..c0a841112c
@@ -0,0 +1,339 @@
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+                       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; If not, see <http://www.gnu.org/licenses/>.
+
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Library General
+Public License instead of this License.
tools/proctrace/Makefile new file +48 -0

48 lines not on this page · raw message

tools/proctrace/proctrace.c new file +163 -0

163 lines not on this page · raw message

Signature · 1 line
2.20.1
#72 lore raw

Re: [PATCH v4 10/10] tools/proctrace: add proctrace tool

tools/proctrace/COPYING

Quoted 5 lines · Michał Leszczyński
tools/proctrace/COPYING new file +0 -0
new file mode 100644
index 0000000000..c0a841112c

The top-level COPYING file is GPL2.  There shouldn't be any need to include a second copy here.

Quoted 36 lines
tools/proctrace/Makefile new file +30 -0
new file mode 100644
index 0000000000..2983c477fe
@@ -0,0 +1,48 @@
+# Copyright (C) CERT Polska - NASK PIB
+# Author: Michał Leszczyński <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; under version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+XEN_ROOT=$(CURDIR)/../..
+include $(XEN_ROOT)/tools/Rules.mk
+
+CFLAGS  += -Werror
+CFLAGS  += $(CFLAGS_libxenevtchn)
+CFLAGS  += $(CFLAGS_libxenctrl)
+LDLIBS  += $(LDLIBS_libxenctrl)
+LDLIBS  += $(LDLIBS_libxenevtchn)
+LDLIBS  += $(LDLIBS_libxenforeignmemory)
+
+.PHONY: all
+all: build
+
+.PHONY: build
+build: proctrace
+
+.PHONY: install
+install: build

tools/proctrace/Makefile line 40

Quoted 10 lines
tools/proctrace/Makefile new file +10 -0
+	$(INSTALL_DIR) $(DESTDIR)$(sbindir)
+	$(INSTALL_PROG) proctrace $(DESTDIR)$(sbindir)/proctrace
+
+.PHONY: uninstall
+uninstall:
+	rm -f $(DESTDIR)$(sbindir)/proctrace
+
+.PHONY: clean
+clean:
+	$(RM) -f $(DEPS_RM)

You need to remove proctrace as well, for make clean to have the intended semantics.

tools/proctrace/Makefile line 46

Quoted 6 lines
tools/proctrace/Makefile new file +6 -0
+
+.PHONY: distclean
+distclean: clean
+
+iptlive: iptlive.o Makefile
+	$(CC) $(LDFLAGS) $< -o $@ $(LDLIBS) $(APPEND_LDFLAGS)

This rule looks to be totally unused?

tools/proctrace/Makefile

Quoted 10 lines
tools/proctrace/Makefile new file +10 -0
+#include <stdlib.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <signal.h>
+
+#include <xenctrl.h>
+#include <xen/xen.h>
+#include <xenforeignmemory.h>
+
+#define BUF_SIZE (16384 * XC_PAGE_SIZE)

This hardcodes the size of the buffer which is configurable per VM. Mapping the buffer fails when it is smaller than this.

It appears there is still outstanding bug from the acquire_resource work which never got fixed.  The guest_handle_is_null(xmar.frame_list) path in Xen is supposed to report the size of the resource, not the size of Xen's local buffer, so userspace can ask "how large is this resource".

I'll try and find some time to fix this and arrange for backports, but the current behaviour is nonsense, and problematic for new users.

~Andrew

#73 lore raw

Re: [PATCH v4 10/10] tools/proctrace: add proctrace tool

On Thu, Jul 02, 2020 at 04:10:57PM +0100, Andrew Cooper wrote:
[...]

Quoted 15 lines

tools/proctrace/proctrace.c line 34

Quoted 11 lines
tools/proctrace/proctrace.c new file +10 -0
+#include <stdlib.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <signal.h>
+
+#include <xenctrl.h>
+#include <xen/xen.h>
+#include <xenforeignmemory.h>
+
+#define BUF_SIZE (16384 * XC_PAGE_SIZE)

This hardcodes the size of the buffer which is configurable per VM. Mapping the buffer fails when it is smaller than this.

Quoted 7 lines

It appears there is still outstanding bug from the acquire_resource work which never got fixed.  The guest_handle_is_null(xmar.frame_list) path in Xen is supposed to report the size of the resource, not the size of Xen's local buffer, so userspace can ask "how large is this resource".

I'll try and find some time to fix this and arrange for backports, but the current behaviour is nonsense, and problematic for new users.

I can't quite figure out if this is a blocking comment of this tool to be accepted. Can you clarify?

Wei.

Quoted 2 lines

~Andrew

#74 lore raw
In reply to Michał Leszczyński
Quoted 86 lines · Michał Leszczyński

From: Michal Leszczynski <[email protected]>

Intel Processor Trace is an architectural extension available in modern Intel family CPUs. It allows recording the detailed trace of activity while the processor executes the code. One might use the recorded trace to reconstruct the code flow. It means, to find out the executed code paths, determine branches taken, and so forth.

The abovementioned feature is described in Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3C: System Programming Guide, Part 3, Chapter 36: "Intel Processor Trace."

This patch series implements an interface that Dom0 could use in order to enable IPT for particular vCPUs in DomU, allowing for external monitoring. Such a feature has numerous applications like malware monitoring, fuzzing, or performance testing.

Also thanks to Tamas K Lengyel for a few preliminary hints before first version of this patch was submitted to xen-devel.

Changed since v1:
* MSR_RTIT_CTL is managed using MSR load lists
* other PT-related MSRs are modified only when vCPU goes out of context
* trace buffer is now acquired as a resource
* added vmtrace_pt_size parameter in xl.cfg, the size of trace buffer must be specified in the moment of domain creation
* trace buffers are allocated on domain creation, destructed on domain destruction
* HVMOP_vmtrace_ipt_enable/disable is limited to enabling/disabling PT these calls don't manage buffer memory anymore
* lifted 32 MFN/GFN array limit when acquiring resources
* minor code style changes according to review

Changed since v2:
* trace buffer is now allocated on domain creation (in v2 it was allocated when hvm param was set)
* restored 32-item limit in mfn/gfn arrays in acquire_resource and instead implemented hypercall continuations
* code changes according to Jan's and Roger's review

Changed since v3:
* vmtrace HVMOPs are not implemented as DOMCTLs
* patches splitted up according to Andrew's comments
* code changes according to v3 review on the mailing list

Michal Leszczynski (10):
x86/vmx: add Intel PT MSR definitions
x86/vmx: add IPT cpu feature
tools/libxl: add vmtrace_pt_size parameter
x86/vmx: implement processor tracing for VMX
common/domain: allocate vmtrace_pt_buffer
memory: batch processing in acquire_resource()
x86/mm: add vmtrace_buf resource type
x86/domctl: add XEN_DOMCTL_vmtrace_op
tools/libxc: add xc_vmtrace_* functions
tools/proctrace: add proctrace tool

 docs/man/xl.cfg.5.pod.in                    |  10 +
 tools/golang/xenlight/helpers.gen.go        |   2 +
 tools/golang/xenlight/types.gen.go          |   1 +
 tools/libxc/Makefile                        |   1 +
 tools/libxc/include/xenctrl.h               |  39 +++
 tools/libxc/xc_vmtrace.c                    |  73 +++++
 tools/libxl/libxl.h                         |   8 +
 tools/libxl/libxl_create.c                  |   1 +
 tools/libxl/libxl_types.idl                 |   2 +
 tools/proctrace/COPYING                     | 339 ++++++++++++++++++++
 tools/proctrace/Makefile                    |  48 +++
 tools/proctrace/proctrace.c                 | 163 ++++++++++
 tools/xl/xl_parse.c                         |  20 ++
 xen/arch/x86/domain.c                       |  11 +
 xen/arch/x86/domctl.c                       |  48 +++
 xen/arch/x86/hvm/vmx/vmcs.c                 |   7 +-
 xen/arch/x86/hvm/vmx/vmx.c                  |  89 +++++
 xen/arch/x86/mm.c                           |  25 ++
 xen/common/domain.c                         |  46 +++
 xen/common/memory.c                         |  32 +-
 xen/include/asm-x86/cpufeature.h            |   1 +
 xen/include/asm-x86/domain.h                |   4 +
 xen/include/asm-x86/hvm/hvm.h               |  38 +++
 xen/include/asm-x86/hvm/vmx/vmcs.h          |   4 +
 xen/include/asm-x86/hvm/vmx/vmx.h           |  14 +
 xen/include/asm-x86/msr-index.h             |  37 +++
 xen/include/public/arch-x86/cpufeatureset.h |   1 +
 xen/include/public/domctl.h                 |  27 ++
Quoted 9 lines
 xen/include/public/memory.h                 |   1 +
 xen/include/xen/domain.h                    |   2 +
 xen/include/xen/sched.h                     |   4 +
 31 files changed, 1094 insertions(+), 4 deletions(-)
 create mode 100644 tools/libxc/xc_vmtrace.c
 create mode 100644 tools/proctrace/COPYING
 create mode 100644 tools/proctrace/Makefile
 create mode 100644 tools/proctrace/proctrace.c

FYI, this patchset is also available at:
https://github.com/icedevml/xen/tree/ipt-patch-v4

Hubert Jasudowicz