Support SIMD/eGPRs/SSP registers sampling for perf
Subsystems (13)
v10 cover Support SIMD/eGPRs/SSP registers sampling for perf
The v10 patch series requires the prerequisite bug-fix patchset:
https://lore.kernel.org/all/[email protected]/
Patch layout:
- Patches 1-5: Bug fixes and cleanup needed before enabling XSAVES-based sampling in NMI context.
- Patches 6-7: FPU-related preparation, including xsaves_nmi() and related cleanup/optimization.
- Patches 8-11: PMI-based XMM sampling support through the existing sample_regs_intr/sample_regs_user interfaces for both PERF_SAMPLE_REGS_INTR and PERF_SAMPLE_REGS_USER.
- Patches 12-19: New SIMD register interface and support for XMM/YMM/ZMM/OPMASK, APX eGPRs, and SSP through that interface.
- Patch 20: Extend arch PEBS to support YMM/ZMM/OPMASK, APX eGPRs, and SSP with the new interface.
- Patch 21: Enable new interface based sampling.
- Patches 22-23: arch PEBS bug fix and sanity check.
Changes since v9:
- Patch 07/23: Update fpu_swap_kvm_fpstate() to clear fpu->__task_fpstate only after switching back to host state, ensuring that when non-NULL it points to host FPU state.
- Patch 11/23: Fix an issue where guest FPU state can be sampled during host user-regs sampling.
- Patch 13/23: Improve and streamline perf_simd_reg_validate() to enforce full-register-only sampling for SIMD registers (XMM/YMM/ZMM/OPMASK).
- Patch 21/23: Simplify and unify PERF_PMU_CAP_SIMD_REGS capability handling for both PEBS and non-PEBS events.
- Refine changelog text across the full patch series.
Changes since v8:
- Patch 01/24: Fix missing PMI handler unregister on PMU initialization failure (Sashiko).
- Patch 11/24: Skip kernel-space SIMD sampling when exclude_kernel is set, to account for PMI skid (Sashiko).
- Patch 12/24: Skip guest kernel-space SIMD sampling for user register sampling.
- Patch 14/24: Strengthen SIMD register configuration validation in perf_simd_reg_validate() (Sashiko).
- Patch 17/24: Strengthen PRED (OPMASK) register configuration validation in perf_simd_reg_validate() (Sashiko).
- Patch 21/24: Enhance large-PEBS checks for SSP and APX eGPR registers (Sashiko).
- Patch 22/24: Refine PERF_PMU_CAP_SIMD_REGS setup: advertise the capability only when both XSAVES and Arch PEBS are supported (Peter).
- Refine patch changelogs across the series.
Changes since V7:
- Validate the return value of intel_pmu_init_hybrid() (Patch 01/23).
- Replace pt_regs with x86_perf_regs in xen_pmu_irq_handler() (Patch 06/23).
- Improve event_has_extended_regs() (Patch 09/23).
- Explicitly ensure the allocated XSAVE area is 64-byte aligned (Patch 10/23, Sashiko).
- Clear the SIMD register pointers in x86_user_regs to avoid exposing stale register data to user space (Patch 11/23, Sashiko).
- Refine the SIMD register interface and sample data layout, and add the missing SIMD data reservation in perf_prepare_sample() for non-x86 architectures (Patch 12/23, Sashiko).
- Improve perf_simd_reg_validate() for x86 (Patch 13/23, Sashiko).
- Refine SSP sampling and ensure the GPR sub-group flag is set for PEBS (Patch 19/23, Sashiko).
- Fix the incorrect large-PEBS check for XMM (Patch 20/23, Sashiko).
- Fix missing handling in x86_pmu_handle_guest_pebs() for back-to-back PMI detection (Patch 22/23, Sashiko).
- Strengthen the PEBS record header sanity checks to prevent invalid memory access (Patch 23/23, Sashiko).
Changes since V6:
- Fix potential overwritten issue in hybrid PMU structure (patch 01/24)
- Restrict PEBS events work on GP counters if no PEBS baseline suggested (patch 02/24)
- Use per-cpu x86_intr_regs for perf_event_nmi_handler() instead of temporary variable (patch 06/24)
- Add helper update_fpu_state_and_flag() to ensure TIF_NEED_FPU_LOAD is set after save_fpregs_to_fpstate() call (patch 09/24)
- Optimize and simplify x86_pmu_sample_xregs(), etc. (patch 11/24)
- Add macro word_for_each_set_bit() to simplify u64 set-bit iteration (patch 13/24)
- Add sanity check for PEBS fragment size (patch 24/24)
Changes since V5:
- Introduce 3 commits to fix newly found PEBS issues (Patch 01~03/19)
- Address Peter comments, including,
* Fully support user-regs sampling of the SIMD/eGPRs/SSP registers
* Adjust newly added fields in perf_event_attr to avoid holes
* Fix the endian issue introduced by for_each_set_bit() in event/core.c
* Remove some unnecessary macros from UAPI header perf_regs.h
* Enhance b2b NMI detection for all PEBS handlers to ensure identical behaviors of all PEBS handlers
- Split perf-tools patches which would be posted in a separate patchset later
Changes since V4:
- Rewrite some functions comments and commit messages (Dave)
- Add arch-PEBS based SIMD/eGPRs/SSP sampling support (Patch 15/19)
- Fix "suspecious NMI" warnning observed on PTL/NVL P-core and DMR by activating back-to-back NMI detection mechanism (Patch 16/19)
- Fix some minor issues on perf-tool patches (Patch 18/19)
Changes since V3:
- Drop the SIMD registers if an NMI hits kernel mode for REGS_USER.
- Only dump the available regs, rather than zero and dump the unavailable regs. It's possible that the dumped registers are a subset of the requested registers.
- Some minor updates to address Dapeng's comments in V3.
Changes since V2:
- Use the FPU format for the x86_pmu.ext_regs_mask as well
- Add a check before invoking xsaves_nmi()
- Add perf_simd_reg_check() to retrieve the number of available registers. If the kernel fails to get the requested registers, e.g., XSAVES fails, nothing dumps to the userspace (the V2 dumps all 0s).
- Add POC perf tool patches
Changes since V1:
- Apply the new interfaces to configure and dump the SIMD registers
- Utilize the existing FPU functions, e.g., xstate_calculate_size, get_xsave_addr().
This series adds support on x86 for sampling SIMD registers, APX eGPRs, and SSP with both PMI-based and PEBS-based sampling.
Starting with Intel Ice Lake, PEBS can sample XMM registers, but PMI-based XMM sampling is still not available. On newer Intel platforms with architectural PEBS support, such as Clearwater Forest and Diamond Rapids, the hardware also gains support for sampling additional SIMD state (XMM/YMM/ZMM/OPMASK), APX extended GPRs, and SSP.
To support these registers consistently across both PMI and PEBS, this series makes the following changes:
1. Adds a new perf_event_attr interface for SIMD register selection. The existing sample_regs_user/sample_regs_intr bitmaps do not have enough space to represent the full SIMD register set, so this series introduces dedicated fields for SIMD and predicate register masks and element widths.
2. Introduces a new sample data layout for SIMD register data. SIMD register payload is appended after the GPR payload, and a new ABI flag, PERF_SAMPLE_REGS_ABI_SIMD, indicates its presence.
3. Adds xsaves_nmi() to allow SIMD/eGPR/SSP sampling from PMI handlers in NMI context.
4. Extends the arch PEBS path to support YMM/ZMM/OPMASK, APX eGPRs, and SSP sampling.
New perf_event_attr fields
--------------------------
This series adds the following fields to perf_event_attr:
/*
* Defines the sampling SIMD/PRED(predicate) register bitmaps and
* qword (8-byte) lengths.
*
* sample_simd_regs_enabled != 0 indicates SIMD/PRED registers are
* requested. The register bitmaps and element sizes are described by:
*
* sample_simd_{vec,pred}_reg_{intr,user}
* sample_simd_{vec,pred}_reg_qwords
*
* sample_simd_regs_enabled == 0 indicates no SIMD/PRED registers are
* requested.
*/
__u16 sample_simd_regs_enabled;
__u16 sample_simd_pred_reg_qwords;
__u16 sample_simd_vec_reg_qwords;
__u16 __reserved_4;
__u32 sample_simd_pred_reg_intr;
__u32 sample_simd_pred_reg_user;
__u64 sample_simd_vec_reg_intr;
__u64 sample_simd_vec_reg_user;Field semantics:
- sample_simd_vec_reg_qwords: qword count for regular SIMD registers
- sample_simd_pred_reg_qwords: qword count for predicate registers
- sample_simd_vec_reg_{intr,user}: SIMD register masks for PERF_SAMPLE_REGS_INTR and PERF_SAMPLE_REGS_USER
- sample_simd_pred_reg_{intr,user}: predicate register masks for PERF_SAMPLE_REGS_INTR and PERF_SAMPLE_REGS_USER
- sample_simd_regs_enabled: indicates whether the new SIMD fields are in use
Examples:
To sample ZMM registers for PERF_SAMPLE_REGS_INTR:
sample_simd_regs_enabled = 1
sample_simd_vec_reg_qwords = 8 // 512 bits = 8 qwords
sample_simd_vec_reg_intr = 0xffffffff // zmm0-zmm31
To sample OPMASK registers for PERF_SAMPLE_REGS_USER:
sample_simd_regs_enabled = 1
sample_simd_pred_reg_qwords = 1 // 64 bits = 1 qword
sample_simd_pred_reg_user = 0xff // opmask0-opmask7
After introducing these fields, bits [63:32] in sample_regs_user and sample_regs_intr are reclaimed for APX eGPRs and SSP instead of the previous XMM0-XMM15 encoding.
Discussion of the new SIMD register interface is available at:
https://lore.kernel.org/lkml/[email protected]/
Sample data layout
------------------
SIMD register data is appended after the GPR data.
For PERF_SAMPLE_REGS_USER:
{ u64 abi; // enum perf_sample_regs_abi
u64 regs[weight(mask)];
struct {
u64 nr_vectors; // 0 ... weight(sample_simd_vec_reg_user)
u64 vector_qwords; // 0 ... sample_simd_vec_reg_qwords
u64 nr_pred; // 0 ... weight(sample_simd_pred_reg_user)
u64 pred_qwords; // 0 ... sample_simd_pred_reg_qwords
u64 data[nr_vectors * vector_qwords +
nr_pred * pred_qwords];
} && (abi & PERF_SAMPLE_REGS_ABI_SIMD)
}
For PERF_SAMPLE_REGS_INTR:
{ u64 abi; // enum perf_sample_regs_abi
u64 regs[weight(mask)];
struct {
u64 nr_vectors; // 0 ... weight(sample_simd_vec_reg_intr)
u64 vector_qwords; // 0 ... sample_simd_vec_reg_qwords
u64 nr_pred; // 0 ... weight(sample_simd_pred_reg_intr)
u64 pred_qwords; // 0 ... sample_simd_pred_reg_qwords
u64 data[nr_vectors * vector_qwords +
nr_pred * pred_qwords];
} && (abi & PERF_SAMPLE_REGS_ABI_SIMD)
}
PERF_SAMPLE_REGS_ABI_SIMD indicates that SIMD register data is present.
The metadata fields are encoded as u64 to keep perf tool parsing and cross-endian support straightforward.
Example
-------
$ perf record -I? available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 SSP XMM0-15 YMM0-15 ZMM0-31 OPMASK0-7
$ perf record --user-regs=? available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 SSP XMM0-15 YMM0-15 ZMM0-31 OPMASK0-7
$ perf record -e branches:p \
-Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \
-c 100000 ./test
$ perf report -D ...
14027761992115 0xcf30 [0x8a8]: PERF_RECORD_SAMPLE(IP, 0x1): 29964/29964:
0xffffffff9f085e24 period: 100000 addr: 0
... intr regs: mask 0x18001010003 ABI 64-bit
.... AX 0xdffffc0000000000 .... BX 0xffff8882297685e8 .... R8 0x0000000000000000 .... R16 0x0000000000000000 .... R31 0x0000000000000000 .... SSP 0x0000000000000000
... SIMD ABI nr_vectors 32 vector_qwords 8 nr_pred 8 pred_qwords 1 .... ZMM[0][0] 0x616c2f656d6f682f
.... ZMM[0][1] 0x696c2f7265737562
...
.... ZMM[31][7] 0x0000000000000000
.... OPMASK[0] 0x00000000fffffe00
....
.... OPMASK[7] 0x0000000000000000
...
Testing
-------
The following intr-regs, user-regs, and combined sampling tests were run on DMR and NVL. The sampled register data was reported correctly and no issues were observed.
$ ./perf record -e branches:p \
-Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask -b -c 10000 sleep 1 $ ./perf record -e branches \
-Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask -b -c 10000 sleep 1 $ ./perf record -e branches:p \
--user-regs=ax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \
-b -c 10000 sleep 1 $ ./perf record -e branches \
--user-regs=ax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \
-b -c 10000 sleep 1 $ ./perf record -e branches:p \
-Ixmm,ymm,zmm,opmask \
--user-regs=ax,bx,r8,r16,r31,ssp \
-b -c 10000 sleep 1 $ ./perf record -e branches:p \
--user-regs=xmm,ymm,zmm,opmask \
-Iax,bx,r8,r16,r31,ssp \
-b -c 10000 sleep 1 $ ./perf record -e branches:p \
-Iax,bx,r9,r17,r30,ssp \
--user-regs=ax,bx,r8,r16,r31,ssp \
-b -c 10000 sleep 1 $ ./perf record -e branches:p \
-Ixmm,opmask --user-regs=zmm \
-b -c 10000 taskset -c 0 sleep 1History:
v9: https://lore.kernel.org/all/[email protected]/
v8: https://lore.kernel.org/all/[email protected]/
v7: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/
Dapeng Mi (20):
perf/x86: Move hybrid PMU initialization before x86_pmu_starting_cpu() perf/x86/intel: Enable large PEBS sampling for XMMs perf/x86/intel: Convert x86_perf_regs to per-cpu variables
perf: Eliminate duplicate arch-specific function definitions perf/x86: Use x86_perf_regs in NMI handlers
x86/fpu: Add update_fpu_state_and_flag() helper perf/x86/intel: Centralize PERF_PMU_CAP_EXTENDED_REGS updates perf/x86: Enable XMM register sampling for non-PEBS events perf/x86: Enable XMM register sampling for REGS_USER case perf/x86: Support XMM sampling using sample_simd_vec_reg_* fields perf/x86: Support YMM sampling using sample_simd_vec_reg_* fields perf/x86: Support ZMM sampling using sample_simd_vec_reg_* fields perf/x86: Support OPMASK sampling using sample_simd_pred_reg_* fields
perf: Enhance perf_reg_validate() with simd_enabled argument perf/x86: Support eGPRs sampling using sample_regs_* fields perf/x86: Support SSP sampling using sample_regs_* fields perf/x86/intel: Support arch-PEBS based SIMD/eGPRs sampling perf/x86/intel: Advertise PERF_PMU_CAP_SIMD_REGS capability perf/x86: Activate back-to-back NMI detection for arch-PEBS induced
NMIs
perf/x86/intel: Add sanity check for PEBS record/fragment size
Kan Liang (3):
x86/fpu/xstate: Add xsaves_nmi() helper
perf: Move and enhance has_extended_regs() for arch-specific use
perf: Add sampling support for SIMD registers
arch/arm/kernel/perf_regs.c | 8 +- arch/arm64/kernel/perf_regs.c | 8 +- arch/csky/kernel/perf_regs.c | 8 +- arch/loongarch/kernel/perf_regs.c | 8 +- arch/mips/kernel/perf_regs.c | 8 +- arch/parisc/kernel/perf_regs.c | 8 +- arch/powerpc/perf/perf_regs.c | 2 +- arch/riscv/kernel/perf_regs.c | 8 +- arch/s390/kernel/perf_regs.c | 2 +- arch/x86/events/core.c | 483 +++++++++++++++++++++++++- arch/x86/events/intel/core.c | 108 ++++-- arch/x86/events/intel/ds.c | 232 ++++++++++--- arch/x86/events/perf_event.h | 208 ++++++++++- arch/x86/include/asm/fpu/sched.h | 5 +- arch/x86/include/asm/fpu/xstate.h | 3 + arch/x86/include/asm/msr-index.h | 7 + arch/x86/include/asm/perf_event.h | 35 +- arch/x86/include/uapi/asm/perf_regs.h | 53 +++ arch/x86/kernel/fpu/core.c | 33 +- arch/x86/kernel/fpu/xstate.c | 25 +- arch/x86/kernel/perf_regs.c | 174 ++++++++-- arch/x86/xen/pmu.c | 5 +- include/linux/perf_event.h | 23 ++ include/linux/perf_regs.h | 36 +- include/uapi/linux/perf_event.h | 49 ++- kernel/events/core.c | 177 +++++++++- 26 files changed, 1490 insertions(+), 226 deletions(-)
Signature · 1 line
2.34.1
PATCH v10 1/23 perf/x86: Move hybrid PMU initialization before x86_pmu_starting_cpu()
The current approach initializes hybrid PMU structures immediately before registering them. This is risky as it can lead to key fields, such as 'capabilities', being inadvertently overwritten.
Although no issues have arisen so far, this method is not ideal. It makes the PMU structure fields susceptible to being overwritten, especially with future changes that might initialize fields like 'capabilities' within init_hybrid_pmu() called by x86_pmu_starting_cpu().
To mitigate this potential problem, move the default hybrid structure initialization before calling x86_pmu_starting_cpu().
--- arch/x86/events/core.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
arch/x86/events/core.c +13 -2
@@ -2218,8 +2218,20 @@ static int __init init_hw_perf_events(void) pmu.attr_update = x86_pmu.attr_update; -if (!is_hybrid ()) +if (!is_hybrid ()) {x86_pmu_show_pmu_cap (NULL ); + }else { +int i; + +/* +* Init default ops. +* Must be called before registering x86_pmu_starting_cpu(), +* otherwise some key PMU fields, e.g., capabilities +* initialized in x86_pmu_starting_cpu(), would be overwritten. +*/ +for (i =0 ; i < x86_pmu.num_hybrid_pmus; i++) + x86_pmu.hybrid_pmu[i].pmu = pmu; + }if (!x86_pmu.read) x86_pmu.read = _x86_pmu_read;
@@ -2266,7 +2278,6 @@ static int __init init_hw_perf_events(void)for (i =0 ; i < x86_pmu.num_hybrid_pmus; i++) { hybrid_pmu = &x86_pmu.hybrid_pmu[i]; - hybrid_pmu->pmu = pmu; hybrid_pmu->pmu.type =-1 ; hybrid_pmu->pmu.attr_update = x86_pmu.attr_update; hybrid_pmu->pmu.capabilities |=PERF_PMU_CAP_EXTENDED_HW_TYPE ;
Signature · 1 line
2.34.1
PATCH v10 2/23 perf/x86/intel: Enable large PEBS sampling for XMMs
In reply to Dapeng MiModern PEBS hardware supports directly sampling XMM registers, then large PEBS can be enabled for XMM registers just like other GPRs.
--- arch/x86/events/intel/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/events/intel/core.c +2 -1
@@ -4723,7 +4723,8 @@ static unsigned long intel_pmu_large_pebs_flags(struct perf_event *event) flags &= ~PERF_SAMPLE_REGS_USER ;if (event->attr.sample_regs_user & ~PEBS_GP_REGS ) flags &= ~PERF_SAMPLE_REGS_USER ; -if (event->attr.sample_regs_intr & ~PEBS_GP_REGS ) +if (event->attr.sample_regs_intr & + ~(PEBS_GP_REGS |PERF_REG_EXTENDED_MASK )) flags &= ~PERF_SAMPLE_REGS_INTR ;return flags; }
Signature · 1 line
2.34.1
PATCH v10 3/23 perf/x86/intel: Convert x86_perf_regs to per-cpu variables
In reply to Dapeng MiCurrently, the drain_pebs() helpers, e.g., intel_pmu_drain_arch_pebs() define an on-stack x86_perf_regs. Upcoming patches will add new fields like *ymm_regs and *zmm_regs to the x86_perf_regs structure to support sampling for these SIMD registers. This would increase the stack size consumed by these helpers, potentially triggering the warning:
"the frame size of 1048 bytes is larger than 1024 bytes
[-Wframe-larger-than=]".
To eliminate this warning, convert x86_perf_regs to per-cpu variables.
Please note drain_pebs() can't be interrupted by other NMIs since either it's already in NMI context or PMU is already disabled.
--- arch/x86/events/intel/ds.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
arch/x86/events/intel/ds.c +8 -6
@@ -2910,6 +2910,8 @@ __intel_pmu_pebs_last_event(struct perf_event *event, } } +static DEFINE_PER_CPU(struct x86_perf_regs, x86_pebs_regs); + static __always_inline void __intel_pmu_pebs_events(struct perf_event *event, struct pt_regs *iregs,
@@ -2919,8 +2921,8 @@ __intel_pmu_pebs_events(struct perf_event *event, setup_fn setup_sample) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -struct x86_perf_regs perf_regs; -struct pt_regs *regs = &perf_regs.regs; +struct x86_perf_regs *perf_regs =this_cpu_ptr (&x86_pebs_regs); +struct pt_regs *regs = &perf_regs->regs;void *at =get_next_pebs_record_by_bit (base, top, bit);int cnt = count;
@@ -3168,8 +3170,8 @@ static void intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_dvoid *last[INTEL_PMC_IDX_FIXED +MAX_FIXED_PEBS_EVENTS ];struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct debug_store *ds = cpuc->ds; -struct x86_perf_regs perf_regs; -struct pt_regs *regs = &perf_regs.regs; +struct x86_perf_regs *perf_regs =this_cpu_ptr (&x86_pebs_regs); +struct pt_regs *regs = &perf_regs->regs;struct pebs_basic *basic;void *base, *at, *top;u64 mask;
@@ -3219,8 +3221,8 @@ static void intel_pmu_drain_arch_pebs(struct pt_regs *iregs,void *last[INTEL_PMC_IDX_FIXED +MAX_FIXED_PEBS_EVENTS ];struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);union arch_pebs_index index; -struct x86_perf_regs perf_regs; -struct pt_regs *regs = &perf_regs.regs; +struct x86_perf_regs *perf_regs =this_cpu_ptr (&x86_pebs_regs); +struct pt_regs *regs = &perf_regs->regs;void *base, *at, *top;u64 mask;
Signature · 1 line
2.34.1
PATCH v10 4/23 perf: Eliminate duplicate arch-specific function definitions
In reply to Dapeng MiDefine default common __weak functions for perf_reg_value(), perf_reg_validate(), perf_reg_abi() and perf_get_regs_user(). This helps to eliminate the duplicated arch-specific definitions.
No function changes intended.
--- arch/arm/kernel/perf_regs.c | 6 ------ arch/arm64/kernel/perf_regs.c | 6 ------ arch/csky/kernel/perf_regs.c | 6 ------ arch/loongarch/kernel/perf_regs.c | 6 ------ arch/mips/kernel/perf_regs.c | 6 ------ arch/parisc/kernel/perf_regs.c | 6 ------ arch/riscv/kernel/perf_regs.c | 6 ------ arch/x86/kernel/perf_regs.c | 6 ------ include/linux/perf_regs.h | 32 ++++++------------------------- kernel/events/core.c | 22 +++++++++++++++++++++ 10 files changed, 28 insertions(+), 74 deletions(-)
arch/arm/kernel/perf_regs.c +0 -6
@@ -31,9 +31,3 @@ u64 perf_reg_abi(struct task_struct *task)return PERF_SAMPLE_REGS_ABI_32 ; } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}
arch/arm64/kernel/perf_regs.c +0 -6
@@ -98,9 +98,3 @@ u64 perf_reg_abi(struct task_struct *task)return PERF_SAMPLE_REGS_ABI_64 ; } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}
arch/csky/kernel/perf_regs.c +0 -6
@@ -31,9 +31,3 @@ u64 perf_reg_abi(struct task_struct *task)return PERF_SAMPLE_REGS_ABI_32 ; } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}
arch/loongarch/kernel/perf_regs.c +0 -6
@@ -45,9 +45,3 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)return regs->regs[idx]; } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}
arch/mips/kernel/perf_regs.c +0 -6
@@ -60,9 +60,3 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)return (s64 )v;/* Sign extend if 32-bit. */ } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}
arch/parisc/kernel/perf_regs.c +0 -6
@@ -53,9 +53,3 @@ u64 perf_reg_abi(struct task_struct *task)return PERF_SAMPLE_REGS_ABI_64 ; } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}
arch/riscv/kernel/perf_regs.c +0 -6
@@ -35,9 +35,3 @@ u64 perf_reg_abi(struct task_struct *task)#endif } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}
arch/x86/kernel/perf_regs.c +0 -6
@@ -100,12 +100,6 @@ u64 perf_reg_abi(struct task_struct *task) return PERF_SAMPLE_REGS_ABI_32; } -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}#else /* CONFIG_X86_64 */ #define REG_NOSUPPORT ((1ULL << PERF_REG_X86_DS) | \ (1ULL << PERF_REG_X86_ES) | \
include/linux/perf_regs.h +6 -26
@@ -9,6 +9,12 @@ struct perf_regs {struct pt_regs *regs; }; +u64 perf_reg_value (struct pt_regs *regs,int idx); +int perf_reg_validate (u64 mask); +u64 perf_reg_abi (struct task_struct *task); +void perf_get_regs_user (struct perf_regs *regs_user, +struct pt_regs *regs); +#ifdef CONFIG_HAVE_PERF_REGS #include <asm/perf_regs.h>
@@ -16,35 +22,9 @@ struct perf_regs {#define PERF_REG_EXTENDED_MASK 0#endif -u64 perf_reg_value (struct pt_regs *regs,int idx); -int perf_reg_validate (u64 mask); -u64 perf_reg_abi (struct task_struct *task); -void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs);#else #define PERF_REG_EXTENDED_MASK 0 -static inline u64 perf_reg_value (struct pt_regs *regs,int idx) -{ -return 0 ; -} - -static inline int perf_reg_validate (u64 mask) -{ -return mask ? -ENOSYS :0 ; -} - -static inline u64 perf_reg_abi (struct task_struct *task) -{ -return PERF_SAMPLE_REGS_ABI_NONE ; -} - -static inline void perf_get_regs_user (struct perf_regs *regs_user, -struct pt_regs *regs) -{ - regs_user->regs =task_pt_regs (current); - regs_user->abi =perf_reg_abi (current); -}#endif /* CONFIG_HAVE_PERF_REGS */ #endif /* _LINUX_PERF_REGS_H */
kernel/events/core.c +22 -0
@@ -7807,6 +7807,28 @@ unsigned long perf_instruction_pointer(struct perf_event *event, 0 : perf_arch_instruction_pointer(regs); } +u64 __weak perf_reg_value (struct pt_regs *regs,int idx) +{ +return 0 ; +} + +int __weak perf_reg_validate (u64 mask) +{ +return mask ? -ENOSYS :0 ; +} + +u64 __weak perf_reg_abi (struct task_struct *task) +{ +return PERF_SAMPLE_REGS_ABI_NONE ; +} + +void __weak perf_get_regs_user (struct perf_regs *regs_user, +struct pt_regs *regs) +{ + regs_user->regs =task_pt_regs (current); + regs_user->abi =perf_reg_abi (current); +} +static void perf_output_sample_regs (struct perf_output_handle *handle,struct pt_regs *regs,u64 mask)
Signature · 1 line
2.34.1
PATCH v10 5/23 perf/x86: Use x86_perf_regs in NMI handlers
In reply to Dapeng MiSupport for sampling additional register state in NMI context (e.g. vector registers and SSP) requires an x86-specific register container. The generic pt_regs structure cannot represent all of the required x86 register data, so switch x86 NMI handlers to x86_perf_regs.
pt_regs is still passed to x86_pmu_handle_irq(), so there is no functional change to existing handling.
AMD IBS NMI handling does not use x86_pmu_handle_irq(), so this conversion does not apply to IBS. IBS support for extended register sampling can be added separately in follow-up patches.
--- arch/x86/events/core.c | 5 ++++- arch/x86/xen/pmu.c | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/x86/events/core.c +4 -1
@@ -1800,9 +1800,11 @@ void perf_put_guest_lvtpc(void)EXPORT_SYMBOL_FOR_KVM (perf_put_guest_lvtpc);#endif /* CONFIG_PERF_GUEST_MEDIATED_PMU */ +static DEFINE_PER_CPU(struct x86_perf_regs, x86_intr_regs);static int perf_event_nmi_handler (unsigned int cmd,struct pt_regs *regs) { +struct x86_perf_regs *x86_regs =this_cpu_ptr (&x86_intr_regs);u64 start_clock;u64 finish_clock;int ret;
@@ -1826,7 +1828,8 @@ perf_event_nmi_handler(unsigned int cmd, struct pt_regs *regs)return NMI_DONE ; start_clock =sched_clock (); - ret =static_call (x86_pmu_handle_irq)(regs); + x86_regs->regs = *regs; + ret =static_call (x86_pmu_handle_irq)(&x86_regs->regs); finish_clock =sched_clock ();perf_sample_event_took (finish_clock - start_clock);
arch/x86/xen/pmu.c +4 -1
@@ -456,12 +456,14 @@ static void xen_convert_regs(const struct xen_pmu_regs *xen_regs, } } +static DEFINE_PER_CPU(struct x86_perf_regs, x86_xen_intr_regs);irqreturn_t xen_pmu_irq_handler (int irq,void *dev_id) {int err, ret =IRQ_NONE ;struct pt_regs regs = {0 };const struct xen_pmu_data *xenpmu_data =get_xenpmu_data ();uint8_t xenpmu_flags =get_xenpmu_flags (); +struct x86_perf_regs *x86_regs =this_cpu_ptr (&x86_xen_intr_regs);if (!xenpmu_data) {pr_warn_once ("%s: pmudata not initialized\n" , __func__);
@@ -472,7 +474,8 @@ irqreturn_t xen_pmu_irq_handler(int irq, void *dev_id) xenpmu_flags |XENPMU_IRQ_PROCESSING ;xen_convert_regs (&xenpmu_data->pmu.r.regs, ®s, xenpmu_data->pmu.pmu_flags); -if (x86_pmu.handle_irq (®s)) + x86_regs->regs = regs; +if (x86_pmu.handle_irq (&x86_regs->regs)) ret =IRQ_HANDLED ;/* Write out cached context to HW */
Signature · 1 line
2.34.1
PATCH v10 6/23 x86/fpu/xstate: Add xsaves_nmi() helper
In reply to Dapeng MiFrom: Kan Liang <[email protected]>
Add xsaves_nmi() to save supported xsave states in NMI handler.
This function is similar to xsaves(), but should only be called within the NMI handler. This function returns the actual register contents at the moment the NMI occurs.
Currently the perf subsystem is the sole user of this helper. It uses this function to snapshot SIMD (XMM/YMM/ZMM) and APX eGPRs registers which would be added in subsequent patches.
--- arch/x86/include/asm/fpu/xstate.h | 1 + arch/x86/kernel/fpu/xstate.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+)
arch/x86/include/asm/fpu/xstate.h +1 -0
@@ -110,6 +110,7 @@ int xfeature_size(int xfeature_nr);void xsaves (struct xregs_state *xsave,u64 mask);void xrstors (struct xregs_state *xsave,u64 mask); +void xsaves_nmi (struct xregs_state *xsave,u64 mask);int xfd_enable_feature (u64 xfd_err);
arch/x86/kernel/fpu/xstate.c +23 -0
@@ -1474,6 +1474,29 @@ void xrstors(struct xregs_state *xstate, u64 mask)WARN_ON_ONCE (err); } +/** +* xsaves_nmi - Save selected components to a kernel xstate buffer in NMI +* @xstate: Pointer to the buffer +* @mask: Feature mask to select the components to save +* +* This function is similar to xsaves(), but should only be called within +* the NMI handler. This function returns the actual register contents at +* the moment the NMI occurs. +* +* Currently, the perf subsystem is the sole user of this helper. It uses +* the function to snapshot SIMD (XMM/YMM/ZMM) and APX eGPRs registers. +*/ +void xsaves_nmi (struct xregs_state *xstate,u64 mask) +{ +int err; + +if (!in_nmi ()) +return ; + +XSTATE_OP (XSAVES , xstate, (u32 )mask, (u32)(mask >>32 ), err); +WARN_ON_ONCE (err); +} +#if IS_ENABLED (CONFIG_KVM )void fpstate_clear_xstate_component (struct fpstate *fpstate,unsigned int xfeature) {
Signature · 1 line
2.34.1
PATCH v10 7/23 x86/fpu: Add update_fpu_state_and_flag() helper
In reply to Dapeng MiAdd update_fpu_state_and_flag() as suggested by Peter and Dave. The helper saves user FPU state and then sets TIF_NEED_FPU_LOAD, ensuring the task FPU state is saved whenever the flag is set.
Subsequent patches will use this guarantee in NMI context by checking TIF_NEED_FPU_LOAD before retrieving user FPU state from the saved task FPU state.
Also add barrier() in the host/guest FPU state switch path and move clearing of fpu->__task_fpstate after switching back to host state. So fpu->__task_fpstate is always observed as host FPU state when non-NULL.
--- arch/x86/include/asm/fpu/sched.h | 5 +++-- arch/x86/kernel/fpu/core.c | 33 ++++++++++++++++++++++++++------ 2 files changed, 30 insertions(+), 8 deletions(-)
arch/x86/include/asm/fpu/sched.h +3 -2
@@ -10,6 +10,8 @@#include <asm/trace/fpu.h> extern void save_fpregs_to_fpstate (struct fpu *fpu); +extern void update_fpu_state_and_flag (struct fpu *fpu, +struct task_struct *task);extern void fpu__drop (struct task_struct *tsk);extern int fpu_clone (struct task_struct *dst,u64 clone_flags,bool minimal,unsigned long shstk_addr);
@@ -36,8 +38,7 @@ static inline void switch_fpu(struct task_struct *old, int cpu) !(old->flags & (PF_KTHREAD | PF_USER_WORKER))) {struct fpu *old_fpu =x86_task_fpu (old); -set_tsk_thread_flag (old,TIF_NEED_FPU_LOAD ); -save_fpregs_to_fpstate (old_fpu); +update_fpu_state_and_flag (old_fpu, old);/* * The save operation preserved register state, so the * fpu_fpregs_owner_ctx is still @old_fpu. Store the
arch/x86/kernel/fpu/core.c +27 -6
@@ -213,6 +213,19 @@ void restore_fpregs_from_fpstate(struct fpstate *fpstate, u64 mask) } } +/* +* Save the FPU register state in fpu->fpstate->regs and set +* TIF_NEED_FPU_LOAD subsequently. +* +* Must be called with fpregs_lock() held, ensuring flag +* TIF_NEED_FPU_LOAD is set last. +*/ +void update_fpu_state_and_flag (struct fpu *fpu,struct task_struct *task) +{ +save_fpregs_to_fpstate (fpu); +set_tsk_thread_flag (task,TIF_NEED_FPU_LOAD ); +} +void fpu_reset_from_exception_fixup (void ) {restore_fpregs_from_fpstate (&init_fpstate,XFEATURE_MASK_FPSTATE );
@@ -383,13 +396,13 @@ int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)/* Swap fpstate */ if (enter_guest) { - fpu->__task_fpstate = cur_fps; +WRITE_ONCE (fpu->__task_fpstate, cur_fps); +barrier (); fpu->fpstate = guest_fps; guest_fps->in_use = true; }else { guest_fps->in_use = false; fpu->fpstate = fpu->__task_fpstate; - fpu->__task_fpstate =NULL ; } cur_fps = fpu->fpstate;
@@ -406,6 +419,16 @@ int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)xfd_update_state (cur_fps); } +/* +* Clear fpu->__task_fpstate after switching back to host state. +* A non-NULL __task_fpstate means guest state is still resident in +* hardware; reset it only once host state has been restored. +*/ +if (!enter_guest) { +barrier (); +WRITE_ONCE (fpu->__task_fpstate,NULL ); + } +fpregs_mark_activate ();fpregs_unlock ();return 0 ;
@@ -481,10 +504,8 @@ void kernel_fpu_begin_mask(unsigned int kfpu_mask)this_cpu_write (kernel_fpu_allowed, false);if (!(current->flags & (PF_KTHREAD |PF_USER_WORKER )) && - !test_thread_flag (TIF_NEED_FPU_LOAD )) { -set_thread_flag (TIF_NEED_FPU_LOAD ); -save_fpregs_to_fpstate (x86_task_fpu (current)); - } + !test_thread_flag (TIF_NEED_FPU_LOAD )) +update_fpu_state_and_flag (x86_task_fpu (current), current);__cpu_invalidate_fpregs_state ();/* Put sane initial values into the control registers. */
Signature · 1 line
2.34.1
PATCH v10 8/23 perf: Move and enhance has_extended_regs() for arch-specific use
In reply to Dapeng MiFrom: Kan Liang <[email protected]>
Move has_extended_regs() to include/linux/perf_event.h so it can be used by arch-specific code.
While moving it, enhance the check logic and rename it to event_has_extended_regs() to match existing perf event helper naming.
--- include/linux/perf_event.h | 10 ++++++++++ kernel/events/core.c | 8 +------- 2 files changed, 11 insertions(+), 7 deletions(-)
include/linux/perf_event.h +10 -0
@@ -1534,6 +1534,16 @@ perf_event__output_id_sample(struct perf_event *event,extern void perf_log_lost_samples (struct perf_event *event,u64 lost); +static inline bool event_has_extended_regs (struct perf_event *event) +{ +struct perf_event_attr *attr = &event->attr; + +return ((attr->sample_type &PERF_SAMPLE_REGS_USER ) && + (attr->sample_regs_user &PERF_REG_EXTENDED_MASK )) || + ((attr->sample_type &PERF_SAMPLE_REGS_INTR ) && + (attr->sample_regs_intr &PERF_REG_EXTENDED_MASK )); +} +static inline bool event_has_any_exclude_flag (struct perf_event *event) {struct perf_event_attr *attr = &event->attr;
kernel/events/core.c +1 -7
@@ -13116,12 +13116,6 @@ int perf_pmu_unregister(struct pmu *pmu) }EXPORT_SYMBOL_GPL (perf_pmu_unregister); -static inline bool has_extended_regs (struct perf_event *event) -{ -return (event->attr.sample_regs_user &PERF_REG_EXTENDED_MASK ) || - (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK ); -} -static int perf_try_init_event (struct pmu *pmu,struct perf_event *event) {struct perf_event_context *ctx =NULL ;
@@ -13156,7 +13150,7 @@ static int perf_try_init_event(struct pmu *pmu, struct perf_event *event) goto err_pmu;if (!(pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS ) && -has_extended_regs (event)) { +event_has_extended_regs (event)) { ret = -EOPNOTSUPP ; goto err_destroy; }
Signature · 1 line
2.34.1
PATCH v10 9/23 perf/x86/intel: Centralize PERF_PMU_CAP_EXTENDED_REGS updates
In reply to Dapeng MiPERF_PMU_CAP_EXTENDED_REGS is currently updated in multiple paths:intel_ds_pebs_init() and __intel_update_pmu_caps() for adaptive and architectural PEBS. This split makes the capability flow harder to reason about and is awkward for upcoming XSAVES-based SIMD register sampling, which also needs consistent capability setup and validation.
Refactor the update logic into a single helper, __intel_update_pmu_xregs_caps(), so adaptive and architectural PEBS share one implementation.
Call this helper from intel_update_pmu_caps() to keep PMU capability handling centralized in one place and avoid duplicated or conflicting updates.
--- arch/x86/events/intel/core.c | 16 ++++++++-------- arch/x86/events/intel/ds.c | 1 - 2 files changed, 8 insertions(+), 9 deletions(-)
arch/x86/events/intel/core.c +8 -8
@@ -6187,11 +6187,14 @@ static inline bool intel_pmu_broken_perf_cap(void)return false; } -static inline void __intel_update_pmu_caps (struct pmu *pmu) +static inline void __intel_update_pmu_xregs_caps (struct pmu *pmu) {struct pmu *dest_pmu = pmu ? pmu :x86_get_pmu (smp_processor_id ()); +u64 caps =hybrid (pmu, arch_pebs_cap).caps; -if (hybrid (pmu, arch_pebs_cap).caps &ARCH_PEBS_VECR_XMM ) +if ((x86_pmu.arch_pebs && (caps &ARCH_PEBS_VECR_XMM )) || + (!x86_pmu.arch_pebs && x86_pmu.intel_cap.pebs_format >=4 && + x86_pmu.intel_cap.pebs_baseline)) dest_pmu->capabilities |=PERF_PMU_CAP_EXTENDED_REGS ; }
@@ -6263,12 +6266,10 @@ static void update_pmu_cap_from_perfmonext(struct pmu *pmu)hybrid (pmu, arch_pebs_cap).counters = pebs_mask;hybrid (pmu, arch_pebs_cap).pdists = pdists_mask; -if (WARN_ON ((pebs_mask | pdists_mask) & ~cntrs_mask)) { +if (WARN_ON ((pebs_mask | pdists_mask) & ~cntrs_mask)) x86_pmu.arch_pebs =0 ; - }else { -__intel_update_pmu_caps (pmu); +else __intel_update_large_pebs_flags (pmu); - } }else {WARN_ON (x86_pmu.arch_pebs ==1 ); x86_pmu.arch_pebs =0 ;
@@ -6292,6 +6293,7 @@ static void intel_update_pmu_caps(struct pmu *pmu) hybrid_pmu(pmu)->pmu_type == hybrid_big)hybrid (pmu, intel_cap).perf_metrics =1 ; } +__intel_update_pmu_xregs_caps (pmu); }static void intel_pmu_check_hybrid_pmus (struct x86_hybrid_pmu *pmu)
@@ -6445,8 +6447,6 @@ static void intel_pmu_cpu_starting(int cpu) } } -__intel_update_pmu_caps (cpuc->pmu); -if (!cpuc->shared_regs)return ;
arch/x86/events/intel/ds.c +0 -1
@@ -3389,7 +3389,6 @@ static void __init intel_ds_pebs_init(void) x86_pmu.flags |=PMU_FL_PEBS_ALL ; x86_pmu.pebs_capable = ~0ULL ; pebs_qual ="-baseline" ; -x86_get_pmu (smp_processor_id ())->capabilities |=PERF_PMU_CAP_EXTENDED_REGS ; }else {/* Only basic record supported */ x86_pmu.large_pebs_flags &=
Signature · 1 line
2.34.1
PATCH v10 10/23 perf/x86: Enable XMM register sampling for non-PEBS events
In reply to Dapeng MiPreviously, XMM register sampling was only available for PEBS events starting from Icelake. Extend the support to non-PEBS events using the xsaves instruction, thereby completing the feature set.
To implement this, a 64-byte aligned buffer is required. A per-CPU ext_regs_buf is introduced to store SIMD and other registers, with an approximate size of 2K. The buffer is allocated using kzalloc_node(), ensuring natural and 64-byte alignment for all kmalloc() allocations with powers of 2.
XMM sampling for non-PEBS events is supported in the REGS_INTR case. Support for REGS_USER will be added in a subsequent patch. For PEBS events, XMM register sampling data is directly retrieved from PEBS records.
Future support for additional vector registers (YMM/ZMM/OPMASK) is planned. An ext_regs_mask is added to track the supported vector register groups.
--- arch/x86/events/core.c | 185 ++++++++++++++++++++++++++++-- arch/x86/events/intel/core.c | 24 +++- arch/x86/events/intel/ds.c | 8 +- arch/x86/events/perf_event.h | 13 +++ arch/x86/include/asm/fpu/xstate.h | 2 + arch/x86/include/asm/perf_event.h | 5 +- arch/x86/kernel/fpu/xstate.c | 2 +- 7 files changed, 219 insertions(+), 20 deletions(-)
arch/x86/events/core.c +174 -11
@@ -408,6 +408,53 @@ set_ext_hw_attr(struct hw_perf_event *hwc, struct perf_event *event)return x86_pmu_extra_regs (val, event); } +static DEFINE_PER_CPU(struct xregs_state *, ext_regs_buf); + +static void release_ext_regs_buffers (void ) +{ +int cpu; + +if (!x86_pmu.ext_regs_mask) +return ; + +for_each_possible_cpu (cpu) { +kfree (per_cpu (ext_regs_buf, cpu)); +per_cpu (ext_regs_buf, cpu) =NULL ; + } +} + +static void reserve_ext_regs_buffers (void ) +{ +bool compacted =cpu_feature_enabled (X86_FEATURE_XCOMPACTED ); +unsigned int size; +int cpu; + +if (!x86_pmu.ext_regs_mask) +return ; + +/* Add 64 bytes to satisfy the XSAVE area's 64-byte alignment. */ + size =xstate_calculate_size (x86_pmu.ext_regs_mask, compacted) +64 ; + +for_each_possible_cpu (cpu) { +per_cpu (ext_regs_buf, cpu) =kzalloc_node (size,GFP_KERNEL , +cpu_to_node (cpu)); +if (WARN_ON_ONCE (!per_cpu (ext_regs_buf, cpu))) + goto err; + } + +return ; + +err: +release_ext_regs_buffers (); +} + +static inline struct xregs_state *get_ext_regs_buf (int cpu) +{ +void *buf =per_cpu (ext_regs_buf, cpu); + +return buf ?PTR_ALIGN (buf,64 ) :NULL ; +} +int x86_reserve_hardware (void ) {int err =0 ;
@@ -420,6 +467,7 @@ int x86_reserve_hardware(void) }else {reserve_ds_buffers ();reserve_lbr_buffers (); +reserve_ext_regs_buffers (); } }if (!err)
@@ -436,6 +484,7 @@ void x86_release_hardware(void)release_pmc_hardware ();release_ds_buffers ();release_lbr_buffers (); +release_ext_regs_buffers ();mutex_unlock (&pmc_reserve_mutex); } }
@@ -653,18 +702,31 @@ int x86_pmu_hw_config(struct perf_event *event)return -EINVAL ; } -/* sample_regs_user never support XMM registers */ -if (unlikely (event->attr.sample_regs_user &PERF_REG_EXTENDED_MASK )) -return -EINVAL ; -/* -* Besides the general purpose registers, XMM registers may -* be collected in PEBS on some platforms, e.g. Icelake -*/ -if (unlikely (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK )) { -if (!(event->pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS )) -return -EINVAL ; +if (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) { +/* +* Besides the general purpose registers, XMM registers may +* be collected as well. +*/ +if (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK ) { +if (!(event->pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS )) +return -EINVAL ; + +if (event->attr.precise_ip) { +u64 caps =hybrid (event->pmu, arch_pebs_cap).caps; -if (!event->attr.precise_ip) +if (x86_pmu.arch_pebs && !(caps &ARCH_PEBS_VECR_XMM )) +return -EINVAL ; +if (!x86_pmu.arch_pebs && !x86_pmu.intel_cap.pebs_baseline) +return -EINVAL ; + } +if (!get_ext_regs_buf (raw_smp_processor_id ())) +return -ENOMEM ; + } + } + +if (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) { +/* XMM registers sampling for REGS_USER is not supported yet. */ +if (event->attr.sample_regs_user &PERF_REG_EXTENDED_MASK )return -EINVAL ; }
@@ -1717,6 +1779,107 @@ static void x86_pmu_del(struct perf_event *event, int flags)static_call_cond (x86_pmu_del)(event); } +void x86_pmu_clear_perf_regs (struct pt_regs *regs) +{ + struct x86_perf_regs *perf_regs = container_of(regs, struct x86_perf_regs, regs); + + perf_regs->xmm_regs =NULL ; +} + +static void update_perf_regs (struct x86_perf_regs *perf_regs, +struct xregs_state *xsave,u64 bitmap) +{ +u64 mask; + +if (!xsave) +return ; + +/* Restrict to features actually saved by XSAVES */ + mask = bitmap & xsave->header.xfeatures; + +if (mask &XFEATURE_MASK_SSE ) + perf_regs->xmm_space = xsave->i387.xmm_space; +} + +/* +* The x86 specific variant of perf_sample_regs_intr(). +* It would be extended to add more SIMD registers sampling support +* in later patches. +*/ +static void x86_pmu_update_regs_intr (struct perf_event *event, +struct perf_sample_data *data, +struct pt_regs *regs, +bool exclude_kernel) +{ +if (exclude_kernel && !user_mode (regs)) { + data->regs_intr.regs =NULL ; + data->regs_intr.abi =PERF_SAMPLE_REGS_ABI_NONE ; + }else { + data->regs_intr.regs = regs; + data->regs_intr.abi =perf_reg_abi (current); + } + + data->dyn_size +=sizeof (u64); +if (data->regs_intr.regs) { + data->dyn_size +=hweight64 (event->attr.sample_regs_intr) * +sizeof (u64); + } + +/* +* Set PERF_SAMPLE_REGS_INTR to bypass perf_sample_regs_intr() call +* in perf_prepare_sample() function. +*/ + data->sample_flags |=PERF_SAMPLE_REGS_INTR ; +} + +static void x86_pmu_sample_xregs (struct perf_event *event, +struct perf_sample_data *data, +bool from_pebs) +{ +struct xregs_state *xsave =get_ext_regs_buf (smp_processor_id ()); +u64 sample_type = event->attr.sample_type; +struct x86_perf_regs *perf_regs; +u64 intr_mask =0 ; + +if (WARN_ON_ONCE (!xsave) || !in_nmi ()) +return ; + +if ((sample_type &PERF_SAMPLE_REGS_INTR ) && data->regs_intr.regs) { +if (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK ) + intr_mask |=XFEATURE_MASK_SSE ; + + intr_mask &= x86_pmu.ext_regs_mask; + intr_mask = from_pebs ?0 : intr_mask; + } + +if (intr_mask) { + perf_regs =container_of (data->regs_intr.regs, + struct x86_perf_regs, regs); + xsave->header.xfeatures =0 ; +xsaves_nmi (xsave, intr_mask); +update_perf_regs (perf_regs, xsave, intr_mask); + } +} + +void x86_pmu_update_perf_regs (struct perf_event *event, +struct perf_sample_data *data, +struct pt_regs *regs, +bool from_pebs) +{ +u64 sample_type = event->attr.sample_type; + +if (!((sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK ))) +return ; + +if (sample_type &PERF_SAMPLE_REGS_INTR ) { +x86_pmu_update_regs_intr (event, data, regs, + event->attr.exclude_kernel); + } + +x86_pmu_sample_xregs (event, data, from_pebs); +} +int x86_pmu_handle_irq (struct pt_regs *regs) {struct perf_sample_data data;
arch/x86/events/intel/core.c +19 -5
@@ -3949,6 +3949,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (has_branch_stack (event))intel_pmu_lbr_save_brstack (&data, cpuc, event); +x86_pmu_clear_perf_regs (regs); +x86_pmu_update_perf_regs (event, &data, regs, false); +perf_event_overflow (event, &data, regs); }
@@ -6190,12 +6193,23 @@ static inline bool intel_pmu_broken_perf_cap(void)static inline void __intel_update_pmu_xregs_caps (struct pmu *pmu) {struct pmu *dest_pmu = pmu ? pmu :x86_get_pmu (smp_processor_id ()); -u64 caps =hybrid (pmu, arch_pebs_cap).caps; -if ((x86_pmu.arch_pebs && (caps &ARCH_PEBS_VECR_XMM )) || - (!x86_pmu.arch_pebs && x86_pmu.intel_cap.pebs_format >=4 && - x86_pmu.intel_cap.pebs_baseline)) - dest_pmu->capabilities |=PERF_PMU_CAP_EXTENDED_REGS ; +/* Only support the extension when XSAVES is available. */ +if (!boot_cpu_has (X86_FEATURE_XSAVES )) +return ; + +if (!boot_cpu_has (X86_FEATURE_XMM ) || + !cpu_has_xfeatures (XFEATURE_MASK_SSE ,NULL )) +return ; + +/* +* On current hybrid platforms, P-cores and E-cores expose the same +* XSAVE feature set. Therefore, using the global x86_pmu.ext_regs_mask +* is sufficient to represent the hardware-supported XSAVE features. +*/ + x86_pmu.ext_regs_mask |=XFEATURE_MASK_SSE ; + + dest_pmu->capabilities |=PERF_PMU_CAP_EXTENDED_REGS ; }static inline void __intel_update_large_pebs_flags (struct pmu *pmu)
arch/x86/events/intel/ds.c +6 -2
@@ -2508,7 +2508,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,return ; perf_regs = container_of(regs, struct x86_perf_regs, regs); - perf_regs->xmm_regs =NULL ; +x86_pmu_clear_perf_regs (regs); format_group = basic->format_group;
@@ -2593,6 +2593,8 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event, next_record += nr *sizeof (u64); } +x86_pmu_update_perf_regs (event, data, regs, true); + WARN_ONCE(next_record != __pebs + basic->format_size, "PEBS record size %u, expected %llu, config %llx\n", basic->format_size,
@@ -2625,7 +2627,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,return ; perf_regs = container_of(regs, struct x86_perf_regs, regs); - perf_regs->xmm_regs =NULL ; +x86_pmu_clear_perf_regs (regs);__setup_perf_sample_data (event, iregs, data);
@@ -2727,6 +2729,8 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, at = at + header->size; goto again; } + +x86_pmu_update_perf_regs (event, data, regs, true); } static inline void *
arch/x86/events/perf_event.h +13 -0
@@ -1028,6 +1028,12 @@ struct x86_pmu {struct extra_reg *extra_regs;unsigned int flags; +/* +* Extended regs, e.g., vector registers +* Utilize the same format as the XFEATURE_MASK_* +*/ +u64 ext_regs_mask; +/* * Intel host/guest support (KVM) */
@@ -1315,6 +1321,13 @@ void x86_pmu_enable_event(struct perf_event *event);int x86_pmu_handle_irq (struct pt_regs *regs); +void x86_pmu_clear_perf_regs (struct pt_regs *regs); + +void x86_pmu_update_perf_regs (struct perf_event *event, +struct perf_sample_data *data, +struct pt_regs *regs, +bool from_pebs); +void x86_pmu_show_pmu_cap (struct pmu *pmu);static inline int x86_pmu_num_counters (struct pmu *pmu)
arch/x86/include/asm/fpu/xstate.h +2 -0
@@ -112,6 +112,8 @@ void xsaves(struct xregs_state *xsave, u64 mask);void xrstors (struct xregs_state *xsave,u64 mask);void xsaves_nmi (struct xregs_state *xsave,u64 mask); +unsigned int xstate_calculate_size (u64 xfeatures,bool compacted); +int xfd_enable_feature (u64 xfd_err);#ifdef CONFIG_X86_64
arch/x86/include/asm/perf_event.h +4 -1
@@ -728,7 +728,10 @@ extern void perf_events_lapic_init(void);struct pt_regs ;struct x86_perf_regs {struct pt_regs regs; -u64 *xmm_regs; +union { +u64 *xmm_regs; +u32 *xmm_space;/* for xsaves */ + }; };extern unsigned long perf_arch_instruction_pointer (struct pt_regs *regs);
arch/x86/kernel/fpu/xstate.c +1 -1
@@ -587,7 +587,7 @@ static bool __init check_xstate_against_struct(int nr)return true; } -static unsigned int xstate_calculate_size (u64 xfeatures,bool compacted) +unsigned int xstate_calculate_size (u64 xfeatures,bool compacted) {unsigned int topmost =fls64 (xfeatures) -1 ;unsigned int offset, i;
Signature · 1 line
2.34.1
PATCH v10 11/23 perf/x86: Enable XMM register sampling for REGS_USER case
In reply to Dapeng MiSupport XMM register sampling for the REGS_USER case.
To handle simultaneous sampling of XMM registers for both REGS_INTR and REGS_USER cases, a per-CPU x86_user_regs is introduced to store REGS_USER-specific XMM registers. This prevents REGS_USER-specific XMM register data from being overwritten by REGS_INTR-specific data if they share the same x86_perf_regs structure.
To sample user-space XMM registers, the x86_pmu_update_user_xregs() helper function is added. It checks if the user-space FPU state has been cached into task FPU state. If so, the user-space XMM register data can be directly retrieved from the cached task FPU state, as the corresponding hardware registers have been cleared or switched to kernel-space state or guest state. Otherwise, the data could be read from the hardware registers using the xsaves instruction.
For PEBS events, x86_pmu_update_user_xregs() checks if the PEBS-sampled XMM register data belongs to user-space. If so, no further action is needed. Otherwise, the user-space XMM register data needs to be re-sampled using the same method as for non-PEBS events.
--- arch/x86/events/core.c | 154 ++++++++++++++++++++++++++++++++--- arch/x86/events/intel/core.c | 6 +- arch/x86/events/intel/ds.c | 5 +- 3 files changed, 147 insertions(+), 18 deletions(-)
arch/x86/events/core.c +141 -13
@@ -702,12 +702,12 @@ int x86_pmu_hw_config(struct perf_event *event)return -EINVAL ; } -if (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) { +if (event->attr.sample_type & (PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER )) {/* * Besides the general purpose registers, XMM registers may * be collected as well. */ -if (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK ) { +if (event_has_extended_regs (event)) {if (!(event->pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS ))return -EINVAL ;
@@ -724,12 +724,6 @@ int x86_pmu_hw_config(struct perf_event *event) } } -if (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) { -/* XMM registers sampling for REGS_USER is not supported yet. */ -if (event->attr.sample_regs_user &PERF_REG_EXTENDED_MASK ) -return -EINVAL ; - } -return x86_setup_perfctr (event); }
@@ -1832,14 +1826,129 @@ static void x86_pmu_update_regs_intr(struct perf_event *event, data->sample_flags |=PERF_SAMPLE_REGS_INTR ; } +static DEFINE_PER_CPU(struct x86_perf_regs, x86_user_regs); + +static void x86_pmu_get_regs_user (struct perf_sample_data *data, +struct pt_regs *regs) +{ +struct x86_perf_regs *x86_regs_user =this_cpu_ptr (&x86_user_regs); +struct perf_regs regs_user; + +x86_pmu_clear_perf_regs (&x86_regs_user->regs); + +perf_get_regs_user (®s_user, regs); + data->regs_user.abi = regs_user.abi; +if (regs_user.regs) { + x86_regs_user->regs = *regs_user.regs; + data->regs_user.regs = &x86_regs_user->regs; + }else { + data->regs_user.regs =NULL ; + } +} + +/* +* The x86 specific variant of perf_sample_regs_user(). +* Update data->regs_user fields for extended registers (e.g., SIMD). +*/ +static void x86_pmu_update_regs_user (struct perf_event *event, +struct perf_sample_data *data, +struct pt_regs *regs) +{ +struct x86_perf_regs *x86_regs_user =this_cpu_ptr (&x86_user_regs); +struct perf_event_attr *attr = &event->attr; +struct x86_perf_regs *perf_regs; + +/* +* PERF_SAMPLE_REGS_INTR and PERF_SAMPLE_REGS_USER can both be +* requested for one event. Keep user regs in a separate x86_perf_regs +* instance, so intr-reg collection does not overwrite user-reg data. +*/ +if (user_mode (regs)) { +x86_pmu_clear_perf_regs (&x86_regs_user->regs); + perf_regs = container_of(regs, struct x86_perf_regs, regs); +/* Copy all sampled regs data to x86_regs_user. */ + *x86_regs_user = *perf_regs; + data->regs_user.regs = &x86_regs_user->regs; + data->regs_user.abi =perf_reg_abi (current); + }else if (is_user_task (current)) { +x86_pmu_get_regs_user (data, regs); + }else { + data->regs_user.abi =PERF_SAMPLE_REGS_ABI_NONE ; + data->regs_user.regs =NULL ; + } + + data->dyn_size +=sizeof (u64); +if (data->regs_user.regs) + data->dyn_size +=hweight64 (attr->sample_regs_user) *sizeof (u64); + +/* +* Set PERF_SAMPLE_REGS_USER to bypass perf_sample_regs_user() call +* in perf_prepare_sample() function. +*/ + data->sample_flags |=PERF_SAMPLE_REGS_USER ; +} + +/* +* This function retrieves cached user-space fpu registers (XMM/YMM/ZMM). +* If TIF_NEED_FPU_LOAD is set or PMI hits into guest, it indicates that +* the user-space FPU state is cached. Otherwise, the data should be read +* directly from the hardware registers. +*/ +static inline u64 x86_pmu_update_user_xregs (struct perf_sample_data *data, +struct pt_regs *regs, +u64 mask,bool from_pebs) +{ +struct x86_perf_regs *perf_regs; +struct xregs_state *xsave; +struct fpu *fpu; +struct fpstate *fps; +u64 user_mask = mask; + +if (!is_user_task (current)) +return 0 ; + +if (data->regs_user.abi ==PERF_SAMPLE_REGS_ABI_NONE ) +return 0 ; + +/* +* If PEBS hits kernel space, need to re-sample extended +* registers for user space. +*/ +if (user_mode (regs)) + user_mask = from_pebs ?0 : mask; + + fpu =x86_task_fpu (current); + fps =READ_ONCE (fpu->__task_fpstate); +/* +* If fpu->__task_fpstate is set, it points to the cached user +* FPU state (e.g. with KVM guest-state swapping). Otherwise, +* when TIF_NEED_FPU_LOAD is set, fpu->fpstate holds the cached +* user state. If neither is true, the user state is live in hardware. +*/ +if (user_mask && (test_thread_flag (TIF_NEED_FPU_LOAD ) || fps)) { + perf_regs =container_of (data->regs_user.regs, + struct x86_perf_regs, regs); +if (!fps) + fps = fpu->fpstate; + xsave = &fps->regs.xsave; + +update_perf_regs (perf_regs, xsave, user_mask); +return 0 ; + } + +return user_mask; +} +static void x86_pmu_sample_xregs (struct perf_event *event,struct perf_sample_data *data, +struct pt_regs *regs,bool from_pebs) {struct xregs_state *xsave =get_ext_regs_buf (smp_processor_id ());u64 sample_type = event->attr.sample_type;struct x86_perf_regs *perf_regs;u64 intr_mask =0 ; +u64 user_mask =0 ;if (WARN_ON_ONCE (!xsave) || !in_nmi ())return ;
@@ -1852,13 +1961,31 @@ static void x86_pmu_sample_xregs(struct perf_event *event, intr_mask = from_pebs ?0 : intr_mask; } +if ((sample_type &PERF_SAMPLE_REGS_USER ) && data->regs_user.regs) { +if (event->attr.sample_regs_user &PERF_REG_EXTENDED_MASK ) + user_mask |=XFEATURE_MASK_SSE ; + + user_mask &= x86_pmu.ext_regs_mask; + user_mask =x86_pmu_update_user_xregs (data, regs, + user_mask, from_pebs); + } + +if (user_mask | intr_mask) { + xsave->header.xfeatures =0 ; +xsaves_nmi (xsave, user_mask | intr_mask); + } +if (intr_mask) { perf_regs =container_of (data->regs_intr.regs, struct x86_perf_regs, regs); - xsave->header.xfeatures =0 ; -xsaves_nmi (xsave, intr_mask);update_perf_regs (perf_regs, xsave, intr_mask); } + +if (user_mask) { + perf_regs =container_of (data->regs_user.regs, + struct x86_perf_regs, regs); +update_perf_regs (perf_regs, xsave, user_mask); + } } void x86_pmu_update_perf_regs(struct perf_event *event,
@@ -1868,16 +1995,17 @@ void x86_pmu_update_perf_regs(struct perf_event *event, {u64 sample_type = event->attr.sample_type; -if (!((sample_type &PERF_SAMPLE_REGS_INTR ) && - (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK ))) +if (!event_has_extended_regs (event))return ;if (sample_type &PERF_SAMPLE_REGS_INTR ) {x86_pmu_update_regs_intr (event, data, regs, event->attr.exclude_kernel); } +if (sample_type &PERF_SAMPLE_REGS_USER ) +x86_pmu_update_regs_user (event, data, regs); -x86_pmu_sample_xregs (event, data, from_pebs); +x86_pmu_sample_xregs (event, data, regs, from_pebs); }int x86_pmu_handle_irq (struct pt_regs *regs)
arch/x86/events/intel/core.c +3 -3
@@ -4719,15 +4719,15 @@ static void intel_pebs_aliases_skl(struct perf_event *event)static unsigned long intel_pmu_large_pebs_flags (struct perf_event *event) {unsigned long flags = x86_pmu.large_pebs_flags; +u64 gprs_mask =PEBS_GP_REGS |PERF_REG_EXTENDED_MASK ;if (event->attr.use_clockid) flags &= ~PERF_SAMPLE_TIME ;if (!event->attr.exclude_kernel) flags &= ~PERF_SAMPLE_REGS_USER ; -if (event->attr.sample_regs_user & ~PEBS_GP_REGS ) +if (event->attr.sample_regs_user & ~gprs_mask) flags &= ~PERF_SAMPLE_REGS_USER ; -if (event->attr.sample_regs_intr & - ~(PEBS_GP_REGS |PERF_REG_EXTENDED_MASK )) +if (event->attr.sample_regs_intr & ~gprs_mask) flags &= ~PERF_SAMPLE_REGS_INTR ;return flags; }
arch/x86/events/intel/ds.c +3 -2
@@ -1726,8 +1726,7 @@ static u64 pebs_update_adaptive_cfg(struct perf_event *event)if (gprs || (attr->precise_ip <2 ) || tsx_weight) pebs_data_cfg |=PEBS_DATACFG_GP ; -if ((sample_type &PERF_SAMPLE_REGS_INTR ) && - (attr->sample_regs_intr &PERF_REG_EXTENDED_MASK )) +if (event_has_extended_regs (event)) pebs_data_cfg |=PEBS_DATACFG_XMMS ;if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {
@@ -2930,6 +2929,8 @@ __intel_pmu_pebs_events(struct perf_event *event,void *at =get_next_pebs_record_by_bit (base, top, bit);int cnt = count; +x86_pmu_clear_perf_regs (regs); +if (!iregs) iregs = &dummy_iregs;
Signature · 1 line
2.34.1
PATCH v10 12/23 perf: Add sampling support for SIMD registers
In reply to Dapeng MiFrom: Kan Liang <[email protected]>
Users may be interested in sampling SIMD registers during profiling. The current sample_regs_* structure does not have sufficient space for all SIMD registers.
To address this, new attribute fields sample_simd_{pred,vec}_reg_* are added to struct perf_event_attr to represent the SIMD registers that are expected to be sampled.
Currently, the perf/x86 code supports XMM registers in sample_regs_*. To unify the configuration of SIMD registers and ensure a consistent method for configuring XMM and other SIMD registers, a new event attribute field, sample_simd_regs_enabled, is introduced. When sample_simd_regs_enabled is set, it indicates that all SIMD registers, including XMM, will be represented by the newly introduced sample_simd_{pred|vec}_reg_* fields. The original XMM space in sample_regs_* is reserved for future uses.
Since SIMD registers are wider than 64 bits, a new output format is introduced. The number and width of SIMD registers are dumped first, followed by the register values. The number and width are based on the user's configuration.
A new ABI, PERF_SAMPLE_REGS_ABI_SIMD, is added to indicate the new format. The enum perf_sample_regs_abi is now a bitmap. This change should not impact existing tools, as the version and bitmap remain the same for values 1 and 2.
Additionally, two new __weak functions are introduced:
- perf_simd_reg_value(): Retrieves the value of the requested SIMD register.
- perf_simd_reg_validate(): Validates the configuration of the SIMD registers.
A new flag, PERF_PMU_CAP_SIMD_REGS, is added to indicate that the PMU supports SIMD register dumping. An error is generated if sample_simd_{pred|vec}_reg_* is mistakenly set for a PMU that does not support this capability.
--- include/linux/perf_event.h | 12 +++ include/linux/perf_regs.h | 4 + include/uapi/linux/perf_event.h | 49 ++++++++++- kernel/events/core.c | 141 +++++++++++++++++++++++++++++--- 4 files changed, 192 insertions(+), 14 deletions(-)
include/linux/perf_event.h +12 -0
@@ -306,6 +306,7 @@ struct perf_event_pmu_context;#define PERF_PMU_CAP_AUX_PAUSE 0x0200#define PERF_PMU_CAP_AUX_PREFER_LARGE 0x0400#define PERF_PMU_CAP_MEDIATED_VPMU 0x0800 +#define PERF_PMU_CAP_SIMD_REGS 0x1000/** * pmu::scope
@@ -1534,6 +1535,17 @@ perf_event__output_id_sample(struct perf_event *event,extern void perf_log_lost_samples (struct perf_event *event,u64 lost); +static inline bool event_has_simd_regs (struct perf_event *event) +{ +struct perf_event_attr *attr = &event->attr; + +if (!(event->attr.sample_type & + (PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ))) +return false; + +return attr->sample_simd_regs_enabled !=0 ; +} +static inline bool event_has_extended_regs (struct perf_event *event) {struct perf_event_attr *attr = &event->attr;
include/linux/perf_regs.h +4 -0
@@ -14,6 +14,10 @@ int perf_reg_validate(u64 mask);u64 perf_reg_abi (struct task_struct *task);void perf_get_regs_user (struct perf_regs *regs_user,struct pt_regs *regs); +int perf_simd_reg_validate (u16 vec_qwords,u64 vec_mask, +u16 pred_qwords,u32 pred_mask); +u64 perf_simd_reg_value (struct pt_regs *regs,int idx, +u16 qwords_idx,bool pred);#ifdef CONFIG_HAVE_PERF_REGS #include <asm/perf_regs.h>
include/uapi/linux/perf_event.h +45 -4
79 lines not on this page · raw message
kernel/events/core.c +131 -10
226 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 13/23 perf/x86: Support XMM sampling using sample_simd_vec_reg_* fields
In reply to Dapeng MiSupport sampling of XMM registers using the sample_simd_vec_reg_* fields.
When sample_simd_regs_enabled is set, the original XMM space in the sample_regs_* field is treated as reserved. An INVAL error will be reported to user space if any bit is set in the original XMM space while sample_simd_regs_enabled is set.
The perf_reg_value function requires ABI information to understand the layout of sample_regs. To accommodate this, a new abi field is introduced in the struct x86_perf_regs to represent ABI information.
Additionally, the x86 specific perf_simd_reg_value() function is implemented to retrieve the XMM register values.
XMM sampling will be enabled in a subsequent patch that sets PERF_PMU_CAP_SIMD_REGS.
--- arch/x86/events/core.c | 84 +++++++++++++++++++++++---- arch/x86/events/intel/ds.c | 6 +- arch/x86/events/perf_event.h | 36 ++++++++++++ arch/x86/include/asm/perf_event.h | 1 + arch/x86/include/uapi/asm/perf_regs.h | 15 +++++ arch/x86/kernel/perf_regs.c | 75 +++++++++++++++++++++++- include/linux/perf_event.h | 1 + kernel/events/core.c | 2 +- 8 files changed, 204 insertions(+), 16 deletions(-)
arch/x86/events/core.c +73 -11
@@ -632,6 +632,34 @@ int x86_pmu_max_precise(struct pmu *pmu)return precise; } +static int pebs_simd_regs_validate (struct perf_event *event) +{ +u64 caps =hybrid (event->pmu, arch_pebs_cap).caps; + +if (event_needs_xmm (event) && + !x86_pmu.arch_pebs && !x86_pmu.intel_cap.pebs_baseline) +return -EINVAL ; +if (event_needs_xmm (event) && + x86_pmu.arch_pebs && !(caps &ARCH_PEBS_VECR_XMM )) +return -EINVAL ; + +return 0 ; +} + +static int event_simd_regs_validate (struct perf_event *event) +{ +if (!get_ext_regs_buf (raw_smp_processor_id ())) +return -ENOMEM ; +/* sample_simd_regs_enabled repurposes legacy XMM reg-mask slots. */ +if (event_has_extended_regs (event)) +return -EINVAL ; +if (event_needs_xmm (event) && + !(x86_pmu.ext_regs_mask &XFEATURE_MASK_SSE )) +return -EINVAL ; + +return 0 ; +} +int x86_pmu_hw_config (struct perf_event *event) {if (event->attr.precise_ip) {
@@ -703,11 +731,21 @@ int x86_pmu_hw_config(struct perf_event *event) }if (event->attr.sample_type & (PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER )) { -/* -* Besides the general purpose registers, XMM registers may -* be collected as well. -*/ -if (event_has_extended_regs (event)) { +int ret; + +if (event->attr.sample_simd_regs_enabled) { +if (!(event->pmu->capabilities &PERF_PMU_CAP_SIMD_REGS )) +return -EINVAL ; + +if (event->attr.precise_ip) { + ret =pebs_simd_regs_validate (event); +if (ret) +return ret; + } + ret =event_simd_regs_validate (event); +if (ret) +return ret; + }else if (event_has_extended_regs (event)) {if (!(event->pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS ))return -EINVAL ;
@@ -721,6 +759,8 @@ int x86_pmu_hw_config(struct perf_event *event) }if (!get_ext_regs_buf (raw_smp_processor_id ()))return -ENOMEM ; +if (!(x86_pmu.ext_regs_mask &XFEATURE_MASK_SSE )) +return -EINVAL ; } }
@@ -1777,6 +1817,7 @@ void x86_pmu_clear_perf_regs(struct pt_regs *regs) { struct x86_perf_regs *perf_regs = container_of(regs, struct x86_perf_regs, regs); + perf_regs->abi =PERF_SAMPLE_REGS_ABI_NONE ; perf_regs->xmm_regs =NULL ; }
@@ -1797,14 +1838,15 @@ static void update_perf_regs(struct x86_perf_regs *perf_regs,/* * The x86 specific variant of perf_sample_regs_intr(). -* It would be extended to add more SIMD registers sampling support -* in later patches. +* Update data->regs_intr fields for extended registers (e.g., SIMD). */ static void x86_pmu_update_regs_intr (struct perf_event *event,struct perf_sample_data *data,struct pt_regs *regs,bool exclude_kernel) { +struct x86_perf_regs *perf_regs; +if (exclude_kernel && !user_mode (regs)) { data->regs_intr.regs =NULL ; data->regs_intr.abi =PERF_SAMPLE_REGS_ABI_NONE ;
@@ -1817,6 +1859,14 @@ static void x86_pmu_update_regs_intr(struct perf_event *event,if (data->regs_intr.regs) { data->dyn_size +=hweight64 (event->attr.sample_regs_intr) *sizeof (u64); +if (event_has_simd_regs (event)) { + data->dyn_size +=perf_update_xregs_size (event, true); + data->regs_intr.abi |=PERF_SAMPLE_REGS_ABI_SIMD ; + } + + perf_regs =container_of (data->regs_intr.regs, + struct x86_perf_regs, regs); + perf_regs->abi = data->regs_intr.abi; }/*
@@ -1878,8 +1928,15 @@ static void x86_pmu_update_regs_user(struct perf_event *event, } data->dyn_size +=sizeof (u64); -if (data->regs_user.regs) +if (data->regs_user.regs) { data->dyn_size +=hweight64 (attr->sample_regs_user) *sizeof (u64); +if (event_has_simd_regs (event)) { + data->dyn_size +=perf_update_xregs_size (event, false); + data->regs_user.abi |=PERF_SAMPLE_REGS_ABI_SIMD ; + } + + x86_regs_user->abi = data->regs_user.abi; + }/* * Set PERF_SAMPLE_REGS_USER to bypass perf_sample_regs_user() call
@@ -1954,7 +2011,7 @@ static void x86_pmu_sample_xregs(struct perf_event *event,return ;if ((sample_type &PERF_SAMPLE_REGS_INTR ) && data->regs_intr.regs) { -if (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK ) +if (__event_needs_xmm (event,PERF_SAMPLE_REGS_INTR )) intr_mask |=XFEATURE_MASK_SSE ; intr_mask &= x86_pmu.ext_regs_mask;
@@ -1962,7 +2019,7 @@ static void x86_pmu_sample_xregs(struct perf_event *event, }if ((sample_type &PERF_SAMPLE_REGS_USER ) && data->regs_user.regs) { -if (event->attr.sample_regs_user &PERF_REG_EXTENDED_MASK ) +if (__event_needs_xmm (event,PERF_SAMPLE_REGS_USER )) user_mask |=XFEATURE_MASK_SSE ; user_mask &= x86_pmu.ext_regs_mask;
@@ -1995,7 +2052,12 @@ void x86_pmu_update_perf_regs(struct perf_event *event, {u64 sample_type = event->attr.sample_type; -if (!event_has_extended_regs (event)) +if (!(sample_type & + (PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ))) +return ; + +if (!event_needs_xmm (event) && + !event_has_simd_regs (event))return ;if (sample_type &PERF_SAMPLE_REGS_INTR ) {
arch/x86/events/intel/ds.c +4 -2
@@ -1726,8 +1726,10 @@ static u64 pebs_update_adaptive_cfg(struct perf_event *event)if (gprs || (attr->precise_ip <2 ) || tsx_weight) pebs_data_cfg |=PEBS_DATACFG_GP ; -if (event_has_extended_regs (event)) - pebs_data_cfg |=PEBS_DATACFG_XMMS ; +if (sample_type & (PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER )) { +if (event_needs_xmm (event)) + pebs_data_cfg |=PEBS_DATACFG_XMMS ; + }if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {/*
arch/x86/events/perf_event.h +36 -0
@@ -147,6 +147,42 @@ static inline bool is_acr_self_reload_event(struct perf_event *event)return test_bit (hwc->idx, (unsigned long *)&hwc->config1); } +static inline bool __event_needs_xmm (struct perf_event *event,u64 sample_type) +{ +if (event->attr.sample_simd_regs_enabled) { +if (event->attr.sample_simd_vec_reg_qwords <PERF_X86_XMM_QWORDS ) +return false; + +if ((sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_simd_vec_reg_user >0 )) +return true; + +if ((sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_simd_vec_reg_intr >0 )) +return true; + }else { +if ((sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_regs_user &PERF_REG_EXTENDED_MASK )) +return true; + +if ((sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_regs_intr &PERF_REG_EXTENDED_MASK )) +return true; + } + +return false; +} + +static inline bool event_needs_xmm (struct perf_event *event) +{ +return __event_needs_xmm (event, +PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ); +} +struct amd_nb {int nb_id;/* NorthBridge id */ int refcnt;/* reference count */
arch/x86/include/asm/perf_event.h +1 -0
@@ -728,6 +728,7 @@ extern void perf_events_lapic_init(void);struct pt_regs ;struct x86_perf_regs {struct pt_regs regs; +u64 abi;union {u64 *xmm_regs;u32 *xmm_space;/* for xsaves */
arch/x86/include/uapi/asm/perf_regs.h +15 -0
@@ -2,6 +2,8 @@#ifndef _ASM_X86_PERF_REGS_H#define _ASM_X86_PERF_REGS_H +#include <linux/bits.h> +enum perf_event_x86_regs {PERF_REG_X86_AX , PERF_REG_X86_BX,
@@ -55,4 +57,17 @@ enum perf_event_x86_regs {#define PERF_REG_EXTENDED_MASK (~((1ULL << PERF_REG_X86_XMM0) - 1)) +enum { +PERF_X86_SIMD_XMM_REGS =16 , +PERF_X86_SIMD_VEC_REGS_MAX =PERF_X86_SIMD_XMM_REGS , +}; + +#define PERF_X86_SIMD_VEC_MASK __GENMASK_ULL(PERF_X86_SIMD_VEC_REGS_MAX - 1, 0) + +enum { +/* 1 qword = 8 bytes */ +PERF_X86_XMM_QWORDS =2 , +PERF_X86_SIMD_QWORDS_MAX =PERF_X86_XMM_QWORDS , +}; +#endif /* _ASM_X86_PERF_REGS_H */
arch/x86/kernel/perf_regs.c +73 -2
99 lines not on this page · raw message
include/linux/perf_event.h +1 -0
@@ -1485,6 +1485,7 @@ static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *b br->reserved =0 ; } +extern u64 perf_update_xregs_size (struct perf_event *event,bool intr); extern void perf_output_sample(struct perf_output_handle *handle, struct perf_event_header *header, struct perf_sample_data *data,
kernel/events/core.c +1 -1
@@ -8709,7 +8709,7 @@ static __always_inline u64 __cond_set(u64 flags, u64 s, u64 d)return d * !!(flags & s); } -static u64 perf_update_xregs_size (struct perf_event *event,bool intr) +u64 perf_update_xregs_size (struct perf_event *event,bool intr) {u16 pred_qwords = event->attr.sample_simd_pred_reg_qwords;u16 vec_qwords = event->attr.sample_simd_vec_reg_qwords;
Signature · 1 line
2.34.1
PATCH v10 14/23 perf/x86: Support YMM sampling using sample_simd_vec_reg_* fields
In reply to Dapeng MiSupport sampling of YMM registers via the sample_simd_vec_reg_* fields.
Each YMM register consists of 4 u64 words, assembled from two halves:
XMM (the lower 2 u64 words) and YMMH (the upper 2 u64 words). Although both XMM and YMMH data can be retrieved with a single xsaves instruction, they are stored in separate locations. The perf_simd_reg_value() function is responsible for assembling these halves into a complete YMM register for output to userspace.
Additionally, sample_simd_vec_reg_qwords should be set to 4 to indicate YMM sampling.
YMM sampling will be enabled in a subsequent patch that sets PERF_PMU_CAP_SIMD_REGS.
--- arch/x86/events/core.c | 29 +++++++++++++++++++++------ arch/x86/events/perf_event.h | 26 ++++++++++++++++++++++++ arch/x86/include/asm/perf_event.h | 4 ++++ arch/x86/include/uapi/asm/perf_regs.h | 6 ++++-- arch/x86/kernel/perf_regs.c | 13 +++++++++++- 5 files changed, 69 insertions(+), 9 deletions(-)
arch/x86/events/core.c +23 -6
@@ -642,6 +642,9 @@ static int pebs_simd_regs_validate(struct perf_event *event)if (event_needs_xmm (event) && x86_pmu.arch_pebs && !(caps &ARCH_PEBS_VECR_XMM ))return -EINVAL ; +/* PEBS does not support YMM registers sampling yet. */ +if (event_needs_ymm (event)) +return -EINVAL ;return 0 ; }
@@ -656,6 +659,9 @@ static int event_simd_regs_validate(struct perf_event *event)if (event_needs_xmm (event) && !(x86_pmu.ext_regs_mask &XFEATURE_MASK_SSE ))return -EINVAL ; +if (event_needs_ymm (event) && + !(x86_pmu.ext_regs_mask &XFEATURE_MASK_YMM )) +return -EINVAL ;return 0 ; }
@@ -1819,6 +1825,7 @@ void x86_pmu_clear_perf_regs(struct pt_regs *regs) perf_regs->abi =PERF_SAMPLE_REGS_ABI_NONE ; perf_regs->xmm_regs =NULL ; + perf_regs->ymmh_regs =NULL ; } static void update_perf_regs(struct x86_perf_regs *perf_regs,
@@ -1834,6 +1841,8 @@ static void update_perf_regs(struct x86_perf_regs *perf_regs,if (mask &XFEATURE_MASK_SSE ) perf_regs->xmm_space = xsave->i387.xmm_space; +if (mask &XFEATURE_MASK_YMM ) + perf_regs->ymmh =get_xsave_addr (xsave,XFEATURE_YMM ); }/*
@@ -1996,6 +2005,18 @@ static inline u64 x86_pmu_update_user_xregs(struct perf_sample_data *data,return user_mask; } +static u64 get_simd_sample_mask (struct perf_event *event,u64 sample_type) +{ +u64 mask =0 ; + +if (__event_needs_xmm (event, sample_type)) + mask |=XFEATURE_MASK_SSE ; +if (__event_needs_ymm (event, sample_type)) + mask |=XFEATURE_MASK_YMM ; + +return mask; +} + static void x86_pmu_sample_xregs(struct perf_event *event, struct perf_sample_data *data, struct pt_regs *regs,
@@ -2011,17 +2032,13 @@ static void x86_pmu_sample_xregs(struct perf_event *event,return ;if ((sample_type &PERF_SAMPLE_REGS_INTR ) && data->regs_intr.regs) { -if (__event_needs_xmm (event,PERF_SAMPLE_REGS_INTR )) - intr_mask |=XFEATURE_MASK_SSE ; - + intr_mask |=get_simd_sample_mask (event,PERF_SAMPLE_REGS_INTR ); intr_mask &= x86_pmu.ext_regs_mask; intr_mask = from_pebs ?0 : intr_mask; }if ((sample_type &PERF_SAMPLE_REGS_USER ) && data->regs_user.regs) { -if (__event_needs_xmm (event,PERF_SAMPLE_REGS_USER )) - user_mask |=XFEATURE_MASK_SSE ; - + user_mask |=get_simd_sample_mask (event,PERF_SAMPLE_REGS_USER ); user_mask &= x86_pmu.ext_regs_mask; user_mask =x86_pmu_update_user_xregs (data, regs, user_mask, from_pebs);
arch/x86/events/perf_event.h +26 -0
@@ -183,6 +183,32 @@ static inline bool event_needs_xmm(struct perf_event *event) PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER); } +static inline bool __event_needs_ymm (struct perf_event *event,u64 sample_type) +{ +if (!event->attr.sample_simd_regs_enabled) +return false; +if (event->attr.sample_simd_vec_reg_qwords <PERF_X86_YMM_QWORDS ) +return false; + +if ((sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_simd_vec_reg_user >0 )) +return true; + +if ((sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_simd_vec_reg_intr >0 )) +return true; + +return false; +} + +static inline bool event_needs_ymm (struct perf_event *event) +{ +return __event_needs_ymm (event, +PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ); +} +struct amd_nb {int nb_id;/* NorthBridge id */ int refcnt;/* reference count */
arch/x86/include/asm/perf_event.h +4 -0
@@ -733,6 +733,10 @@ struct x86_perf_regs {u64 *xmm_regs;u32 *xmm_space;/* for xsaves */ }; +union { +u64 *ymmh_regs; +struct ymmh_struct *ymmh; + }; };extern unsigned long perf_arch_instruction_pointer (struct pt_regs *regs);
arch/x86/include/uapi/asm/perf_regs.h +4 -2
@@ -59,7 +59,8 @@ enum perf_event_x86_regs {enum {PERF_X86_SIMD_XMM_REGS =16 , -PERF_X86_SIMD_VEC_REGS_MAX =PERF_X86_SIMD_XMM_REGS , +PERF_X86_SIMD_YMM_REGS =16 , +PERF_X86_SIMD_VEC_REGS_MAX =PERF_X86_SIMD_YMM_REGS , };#define PERF_X86_SIMD_VEC_MASK __GENMASK_ULL(PERF_X86_SIMD_VEC_REGS_MAX - 1, 0)
@@ -67,7 +68,8 @@ enum {enum {/* 1 qword = 8 bytes */ PERF_X86_XMM_QWORDS =2 , -PERF_X86_SIMD_QWORDS_MAX =PERF_X86_XMM_QWORDS , +PERF_X86_YMM_QWORDS =4 , +PERF_X86_SIMD_QWORDS_MAX =PERF_X86_YMM_QWORDS , };#endif /* _ASM_X86_PERF_REGS_H */
arch/x86/kernel/perf_regs.c +12 -1
@@ -77,6 +77,8 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)return regs_get_register (regs, pt_regs_offset[idx]); } +#define PERF_X86_YMMH_QWORDS (PERF_X86_YMM_QWORDS / 2) +u64 perf_simd_reg_value (struct pt_regs *regs,int idx,u16 qwords_idx,bool pred) {
@@ -98,6 +100,11 @@ u64 perf_simd_reg_value(struct pt_regs *regs, int idx,return 0 ;return perf_regs->xmm_regs[idx *PERF_X86_XMM_QWORDS + qwords_idx]; + }else if (qwords_idx <PERF_X86_YMM_QWORDS ) { +if (!perf_regs->ymmh_regs) +return 0 ; +return perf_regs->ymmh_regs[idx *PERF_X86_YMMH_QWORDS + + qwords_idx -PERF_X86_XMM_QWORDS ]; }return 0 ;
@@ -115,7 +122,8 @@ int perf_simd_reg_validate(u16 vec_qwords, u64 vec_mask, }if (vec_qwords) { -if (vec_qwords !=PERF_X86_XMM_QWORDS ) +if (vec_qwords !=PERF_X86_XMM_QWORDS && + vec_qwords !=PERF_X86_YMM_QWORDS )return -EINVAL ;if (vec_mask & ~PERF_X86_SIMD_VEC_MASK )return -EINVAL ;
@@ -124,6 +132,9 @@ int perf_simd_reg_validate(u16 vec_qwords, u64 vec_mask,if (vec_qwords ==PERF_X86_XMM_QWORDS && mask && !bitmap_full (&mask,PERF_X86_SIMD_XMM_REGS ))return -EINVAL ; +if (vec_qwords ==PERF_X86_YMM_QWORDS && mask && + !bitmap_full (&mask,PERF_X86_SIMD_YMM_REGS )) +return -EINVAL ; }/* PRED registers are not supported yet. */
Signature · 1 line
2.34.1
PATCH v10 15/23 perf/x86: Support ZMM sampling using sample_simd_vec_reg_* fields
In reply to Dapeng MiSupport sampling of ZMM registers via the sample_simd_vec_reg_* fields.
Each ZMM register consists of 8 u64 words. Current x86 hardware supports up to 32 ZMM registers. For ZMM registers from ZMM0 to ZMM15, they are assembled from three parts: XMM (the lower 2 u64 words), YMMH (the middle 2 u64 words), and ZMMH (the upper 4 u64 words). The perf_simd_reg_value() function is responsible for assembling these three parts into a complete ZMM register for output to userspace.
For ZMM registers ZMM16 to ZMM31, each register can be read as a whole and directly outputted to userspace.
Additionally, sample_simd_vec_reg_qwords should be set to 8 to indicate ZMM sampling.
ZMM sampling will be enabled in a subsequent patch that sets PERF_PMU_CAP_SIMD_REGS.
--- arch/x86/events/core.c | 22 ++++++++++- arch/x86/events/perf_event.h | 54 +++++++++++++++++++++++++++ arch/x86/include/asm/perf_event.h | 8 ++++ arch/x86/include/uapi/asm/perf_regs.h | 8 +++- arch/x86/kernel/perf_regs.c | 19 +++++++++- 5 files changed, 106 insertions(+), 5 deletions(-)
arch/x86/events/core.c +20 -2
@@ -642,8 +642,10 @@ static int pebs_simd_regs_validate(struct perf_event *event)if (event_needs_xmm (event) && x86_pmu.arch_pebs && !(caps &ARCH_PEBS_VECR_XMM ))return -EINVAL ; -/* PEBS does not support YMM registers sampling yet. */ -if (event_needs_ymm (event)) +/* PEBS does not support YMM/ZMM registers sampling yet. */ +if (event_needs_ymm (event) || +event_needs_low16_zmm (event) || +event_needs_high16_zmm (event))return -EINVAL ;return 0 ;
@@ -662,6 +664,12 @@ static int event_simd_regs_validate(struct perf_event *event)if (event_needs_ymm (event) && !(x86_pmu.ext_regs_mask &XFEATURE_MASK_YMM ))return -EINVAL ; +if (event_needs_low16_zmm (event) && + !(x86_pmu.ext_regs_mask & XFEATURE_MASK_ZMM_Hi256)) +return -EINVAL ; +if (event_needs_high16_zmm (event) && + !(x86_pmu.ext_regs_mask & XFEATURE_MASK_Hi16_ZMM)) +return -EINVAL ;return 0 ; }
@@ -1826,6 +1834,8 @@ void x86_pmu_clear_perf_regs(struct pt_regs *regs) perf_regs->abi =PERF_SAMPLE_REGS_ABI_NONE ; perf_regs->xmm_regs =NULL ; perf_regs->ymmh_regs =NULL ; + perf_regs->zmmh_regs =NULL ; + perf_regs->h16zmm_regs =NULL ; } static void update_perf_regs(struct x86_perf_regs *perf_regs,
@@ -1843,6 +1853,10 @@ static void update_perf_regs(struct x86_perf_regs *perf_regs, perf_regs->xmm_space = xsave->i387.xmm_space;if (mask &XFEATURE_MASK_YMM ) perf_regs->ymmh =get_xsave_addr (xsave,XFEATURE_YMM ); +if (mask & XFEATURE_MASK_ZMM_Hi256) + perf_regs->zmmh =get_xsave_addr (xsave, XFEATURE_ZMM_Hi256); +if (mask & XFEATURE_MASK_Hi16_ZMM) + perf_regs->h16zmm =get_xsave_addr (xsave, XFEATURE_Hi16_ZMM); }/*
@@ -2013,6 +2027,10 @@ static u64 get_simd_sample_mask(struct perf_event *event, u64 sample_type) mask |=XFEATURE_MASK_SSE ;if (__event_needs_ymm (event, sample_type)) mask |=XFEATURE_MASK_YMM ; +if (__event_needs_low16_zmm (event, sample_type)) + mask |= XFEATURE_MASK_ZMM_Hi256; +if (__event_needs_high16_zmm (event, sample_type)) + mask |= XFEATURE_MASK_Hi16_ZMM;return mask; }
arch/x86/events/perf_event.h +54 -0
60 lines not on this page · raw message
arch/x86/include/asm/perf_event.h +8 -0
@@ -737,6 +737,14 @@ struct x86_perf_regs {u64 *ymmh_regs;struct ymmh_struct *ymmh; }; +union { +u64 *zmmh_regs; +struct avx_512_zmm_uppers_state *zmmh; + }; +union { +u64 *h16zmm_regs; +struct avx_512_hi16_state *h16zmm; + }; };extern unsigned long perf_arch_instruction_pointer (struct pt_regs *regs);
arch/x86/include/uapi/asm/perf_regs.h +6 -2
@@ -60,16 +60,20 @@ enum perf_event_x86_regs {enum {PERF_X86_SIMD_XMM_REGS =16 ,PERF_X86_SIMD_YMM_REGS =16 , -PERF_X86_SIMD_VEC_REGS_MAX =PERF_X86_SIMD_YMM_REGS , +PERF_X86_SIMD_ZMM_REGS =32 , +PERF_X86_SIMD_VEC_REGS_MAX =PERF_X86_SIMD_ZMM_REGS , };#define PERF_X86_SIMD_VEC_MASK __GENMASK_ULL(PERF_X86_SIMD_VEC_REGS_MAX - 1, 0) +#define PERF_X86_H16ZMM_BASE 16 +enum {/* 1 qword = 8 bytes */ PERF_X86_XMM_QWORDS =2 ,PERF_X86_YMM_QWORDS =4 , -PERF_X86_SIMD_QWORDS_MAX =PERF_X86_YMM_QWORDS , +PERF_X86_ZMM_QWORDS =8 , +PERF_X86_SIMD_QWORDS_MAX =PERF_X86_ZMM_QWORDS , };#endif /* _ASM_X86_PERF_REGS_H */
arch/x86/kernel/perf_regs.c +18 -1
@@ -78,6 +78,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx) }#define PERF_X86_YMMH_QWORDS (PERF_X86_YMM_QWORDS / 2) +#define PERF_X86_ZMMH_QWORDS (PERF_X86_ZMM_QWORDS / 2)u64 perf_simd_reg_value (struct pt_regs *regs,int idx,u16 qwords_idx,bool pred)
@@ -95,6 +96,13 @@ u64 perf_simd_reg_value(struct pt_regs *regs, int idx, qwords_idx >= PERF_X86_SIMD_QWORDS_MAX))return 0 ; +if (idx >=PERF_X86_H16ZMM_BASE ) { +if (!perf_regs->h16zmm_regs) +return 0 ; +return perf_regs->h16zmm_regs[(idx -PERF_X86_H16ZMM_BASE ) * +PERF_X86_ZMM_QWORDS + qwords_idx]; + } +if (qwords_idx <PERF_X86_XMM_QWORDS ) {if (!perf_regs->xmm_regs)return 0 ;
@@ -105,6 +113,11 @@ u64 perf_simd_reg_value(struct pt_regs *regs, int idx,return 0 ;return perf_regs->ymmh_regs[idx *PERF_X86_YMMH_QWORDS + qwords_idx -PERF_X86_XMM_QWORDS ]; + }else if (qwords_idx <PERF_X86_ZMM_QWORDS ) { +if (!perf_regs->zmmh_regs) +return 0 ; +return perf_regs->zmmh_regs[idx *PERF_X86_ZMMH_QWORDS + + qwords_idx -PERF_X86_YMM_QWORDS ]; }return 0 ;
@@ -123,7 +136,8 @@ int perf_simd_reg_validate(u16 vec_qwords, u64 vec_mask,if (vec_qwords) {if (vec_qwords !=PERF_X86_XMM_QWORDS && - vec_qwords !=PERF_X86_YMM_QWORDS ) + vec_qwords !=PERF_X86_YMM_QWORDS && + vec_qwords !=PERF_X86_ZMM_QWORDS )return -EINVAL ;if (vec_mask & ~PERF_X86_SIMD_VEC_MASK )return -EINVAL ;
@@ -135,6 +149,9 @@ int perf_simd_reg_validate(u16 vec_qwords, u64 vec_mask,if (vec_qwords ==PERF_X86_YMM_QWORDS && mask && !bitmap_full (&mask,PERF_X86_SIMD_YMM_REGS ))return -EINVAL ; +if (vec_qwords ==PERF_X86_ZMM_QWORDS && mask && + !bitmap_full (&mask,PERF_X86_SIMD_ZMM_REGS )) +return -EINVAL ; }/* PRED registers are not supported yet. */
Signature · 1 line
2.34.1
PATCH v10 16/23 perf/x86: Support OPMASK sampling using sample_simd_pred_reg_* fields
In reply to Dapeng MiSupport sampling of OPMASK registers via the sample_simd_pred_reg_* fields.
Each OPMASK register consists of 1 u64 word. Current x86 hardware supports 8 OPMASK registers. The perf_simd_reg_value() function is responsible for outputting OPMASK value to userspace.
Additionally, sample_simd_pred_reg_qwords should be set to 1 to indicate OPMASK sampling.
OPMASK sampling will be enabled in a subsequent patch that sets PERF_PMU_CAP_SIMD_REGS.
--- arch/x86/events/core.c | 13 +++++++++++-- arch/x86/events/perf_event.h | 27 +++++++++++++++++++++++++++ arch/x86/include/asm/perf_event.h | 4 ++++ arch/x86/include/uapi/asm/perf_regs.h | 5 +++++ arch/x86/kernel/perf_regs.c | 24 +++++++++++++++++++----- 5 files changed, 66 insertions(+), 7 deletions(-)
arch/x86/events/core.c +11 -2
@@ -642,10 +642,11 @@ static int pebs_simd_regs_validate(struct perf_event *event)if (event_needs_xmm (event) && x86_pmu.arch_pebs && !(caps &ARCH_PEBS_VECR_XMM ))return -EINVAL ; -/* PEBS does not support YMM/ZMM registers sampling yet. */ +/* PEBS does not support YMM/ZMM/OPMASK registers sampling yet. */ if (event_needs_ymm (event) ||event_needs_low16_zmm (event) || -event_needs_high16_zmm (event)) +event_needs_high16_zmm (event) || +event_needs_opmask (event))return -EINVAL ;return 0 ;
@@ -670,6 +671,9 @@ static int event_simd_regs_validate(struct perf_event *event)if (event_needs_high16_zmm (event) && !(x86_pmu.ext_regs_mask & XFEATURE_MASK_Hi16_ZMM))return -EINVAL ; +if (event_needs_opmask (event) && + !(x86_pmu.ext_regs_mask &XFEATURE_MASK_OPMASK )) +return -EINVAL ;return 0 ; }
@@ -1836,6 +1840,7 @@ void x86_pmu_clear_perf_regs(struct pt_regs *regs) perf_regs->ymmh_regs =NULL ; perf_regs->zmmh_regs =NULL ; perf_regs->h16zmm_regs =NULL ; + perf_regs->opmask_regs =NULL ; } static void update_perf_regs(struct x86_perf_regs *perf_regs,
@@ -1857,6 +1862,8 @@ static void update_perf_regs(struct x86_perf_regs *perf_regs, perf_regs->zmmh =get_xsave_addr (xsave, XFEATURE_ZMM_Hi256);if (mask & XFEATURE_MASK_Hi16_ZMM) perf_regs->h16zmm =get_xsave_addr (xsave, XFEATURE_Hi16_ZMM); +if (mask &XFEATURE_MASK_OPMASK ) + perf_regs->opmask =get_xsave_addr (xsave,XFEATURE_OPMASK ); }/*
@@ -2031,6 +2038,8 @@ static u64 get_simd_sample_mask(struct perf_event *event, u64 sample_type) mask |= XFEATURE_MASK_ZMM_Hi256;if (__event_needs_high16_zmm (event, sample_type)) mask |= XFEATURE_MASK_Hi16_ZMM; +if (__event_needs_opmask (event, sample_type)) + mask |=XFEATURE_MASK_OPMASK ;return mask; }
arch/x86/events/perf_event.h +27 -0
@@ -263,6 +263,33 @@ static inline bool event_needs_high16_zmm(struct perf_event *event) PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER); } +static inline bool __event_needs_opmask (struct perf_event *event, +u64 sample_type) +{ +if (!event->attr.sample_simd_regs_enabled) +return false; +if (event->attr.sample_simd_pred_reg_qwords !=PERF_X86_OPMASK_QWORDS ) +return false; + +if ((sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_simd_pred_reg_user >0 )) +return true; + +if ((sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_simd_pred_reg_intr >0 )) +return true; + +return false; +} + +static inline bool event_needs_opmask (struct perf_event *event) +{ +return __event_needs_opmask (event, +PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ); +} +struct amd_nb {int nb_id;/* NorthBridge id */ int refcnt;/* reference count */
arch/x86/include/asm/perf_event.h +4 -0
@@ -745,6 +745,10 @@ struct x86_perf_regs {u64 *h16zmm_regs;struct avx_512_hi16_state *h16zmm; }; +union { +u64 *opmask_regs; +struct avx_512_opmask_state *opmask; + }; };extern unsigned long perf_arch_instruction_pointer (struct pt_regs *regs);
arch/x86/include/uapi/asm/perf_regs.h +5 -0
@@ -62,14 +62,19 @@ enum {PERF_X86_SIMD_YMM_REGS =16 ,PERF_X86_SIMD_ZMM_REGS =32 ,PERF_X86_SIMD_VEC_REGS_MAX =PERF_X86_SIMD_ZMM_REGS , + +PERF_X86_SIMD_OPMASK_REGS =8 , + PERF_X86_SIMD_PRED_REGS_MAX = PERF_X86_SIMD_OPMASK_REGS, }; +#define PERF_X86_SIMD_PRED_MASK __GENMASK(PERF_X86_SIMD_PRED_REGS_MAX - 1, 0)#define PERF_X86_SIMD_VEC_MASK __GENMASK_ULL(PERF_X86_SIMD_VEC_REGS_MAX - 1, 0)#define PERF_X86_H16ZMM_BASE 16enum {/* 1 qword = 8 bytes */ +PERF_X86_OPMASK_QWORDS =1 ,PERF_X86_XMM_QWORDS =2 ,PERF_X86_YMM_QWORDS =4 , PERF_X86_ZMM_QWORDS = 8,
arch/x86/kernel/perf_regs.c +19 -5
@@ -89,8 +89,14 @@ u64 perf_simd_reg_value(struct pt_regs *regs, int idx,if (!(perf_regs->abi &PERF_SAMPLE_REGS_ABI_SIMD ))return 0 ; -if (pred) -return 0 ; +if (pred) { +if (WARN_ON_ONCE (idx >=PERF_X86_SIMD_PRED_REGS_MAX || + qwords_idx >=PERF_X86_OPMASK_QWORDS )) +return 0 ; +if (!perf_regs->opmask_regs) +return 0 ; +return perf_regs->opmask_regs[idx]; + }if (WARN_ON_ONCE (idx >=PERF_X86_SIMD_VEC_REGS_MAX || qwords_idx >=PERF_X86_SIMD_QWORDS_MAX ))
@@ -154,9 +160,17 @@ int perf_simd_reg_validate(u16 vec_qwords, u64 vec_mask,return -EINVAL ; } -/* PRED registers are not supported yet. */ -if (pred_qwords) -return -EINVAL ; +if (pred_qwords) { +if (pred_qwords !=PERF_X86_OPMASK_QWORDS ) +return -EINVAL ; +if (pred_mask & ~PERF_X86_SIMD_PRED_MASK ) +return -EINVAL ; +/* Only full-register sampling is allowed. */ + mask = pred_mask; +if (pred_qwords ==PERF_X86_OPMASK_QWORDS && mask && + !bitmap_full (&mask,PERF_X86_SIMD_OPMASK_REGS )) +return -EINVAL ; + } size =sizeof (u64) *4 ; size += (hweight64(vec_mask) * vec_qwords +
Signature · 1 line
2.34.1
PATCH v10 17/23 perf: Enhance perf_reg_validate() with simd_enabled argument
In reply to Dapeng MiThe upcoming patch will support x86 APX eGPRs sampling by using the reclaimed XMM register space to represent eGPRs in sample_regs_* fields.
To differentiate between XMM and eGPRs in sample_regs_* fields, an additional argument, simd_enabled, is introduced to the perf_reg_validate() helper. If simd_enabled is set to 1, it indicates that eGPRs are represented in sample_regs_* fields for the x86 platform; otherwise, XMM registers are represented.
--- arch/arm/kernel/perf_regs.c | 2 +- arch/arm64/kernel/perf_regs.c | 2 +- arch/csky/kernel/perf_regs.c | 2 +- arch/loongarch/kernel/perf_regs.c | 2 +- arch/mips/kernel/perf_regs.c | 2 +- arch/parisc/kernel/perf_regs.c | 2 +- arch/powerpc/perf/perf_regs.c | 2 +- arch/riscv/kernel/perf_regs.c | 2 +- arch/s390/kernel/perf_regs.c | 2 +- arch/x86/kernel/perf_regs.c | 4 ++-- include/linux/perf_regs.h | 2 +- kernel/events/core.c | 8 +++++--- 12 files changed, 17 insertions(+), 15 deletions(-)
arch/arm/kernel/perf_regs.c +1 -1
@@ -18,7 +18,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)#define REG_RESERVED (~((1ULL << PERF_REG_ARM_MAX) - 1)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask || mask ®_RESERVED )return -EINVAL ;
arch/arm64/kernel/perf_regs.c +1 -1
@@ -77,7 +77,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)#define REG_RESERVED (~((1ULL << PERF_REG_ARM64_MAX) - 1)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {u64 reserved_mask =REG_RESERVED ;
arch/csky/kernel/perf_regs.c +1 -1
@@ -18,7 +18,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)#define REG_RESERVED (~((1ULL << PERF_REG_CSKY_MAX) - 1)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask || mask ®_RESERVED )return -EINVAL ;
arch/loongarch/kernel/perf_regs.c +1 -1
@@ -25,7 +25,7 @@ u64 perf_reg_abi(struct task_struct *tsk) }#endif /* CONFIG_32BIT */ -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask)return -EINVAL ;
arch/mips/kernel/perf_regs.c +1 -1
@@ -28,7 +28,7 @@ u64 perf_reg_abi(struct task_struct *tsk) }#endif /* CONFIG_32BIT */ -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask)return -EINVAL ;
arch/parisc/kernel/perf_regs.c +1 -1
@@ -34,7 +34,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)#define REG_RESERVED (~((1ULL << PERF_REG_PARISC_MAX) - 1)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask || mask ®_RESERVED )return -EINVAL ;
arch/powerpc/perf/perf_regs.c +1 -1
@@ -125,7 +125,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)return regs_get_register (regs, pt_regs_offset[idx]); } -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask || mask ®_RESERVED )return -EINVAL ;
arch/riscv/kernel/perf_regs.c +1 -1
@@ -18,7 +18,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)#define REG_RESERVED (~((1ULL << PERF_REG_RISCV_MAX) - 1)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask || mask ®_RESERVED )return -EINVAL ;
arch/s390/kernel/perf_regs.c +1 -1
@@ -34,7 +34,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)#define REG_RESERVED (~((1UL << PERF_REG_S390_MAX) - 1)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {if (!mask || mask ®_RESERVED )return -EINVAL ;
arch/x86/kernel/perf_regs.c +2 -2
@@ -198,7 +198,7 @@ int perf_simd_reg_validate(u16 vec_qwords, u64 vec_mask, (1ULL << PERF_REG_X86_R14) | \ (1ULL << PERF_REG_X86_R15)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {/* The mask could be 0 if only the SIMD registers are interested */ if (mask & (REG_NOSUPPORT |PERF_REG_X86_RESERVED ))
@@ -218,7 +218,7 @@ u64 perf_reg_abi(struct task_struct *task) (1ULL << PERF_REG_X86_FS) | \ (1ULL << PERF_REG_X86_GS)) -int perf_reg_validate (u64 mask) +int perf_reg_validate (u64 mask,bool simd_enabled) {/* The mask could be 0 if only the SIMD registers are interested */ if (mask & (REG_NOSUPPORT |PERF_REG_X86_RESERVED ))
include/linux/perf_regs.h +1 -1
@@ -10,7 +10,7 @@ struct perf_regs { };u64 perf_reg_value (struct pt_regs *regs,int idx); -int perf_reg_validate (u64 mask); +int perf_reg_validate (u64 mask,bool simd_enabled);u64 perf_reg_abi (struct task_struct *task);void perf_get_regs_user (struct perf_regs *regs_user,struct pt_regs *regs);
kernel/events/core.c +5 -3
@@ -7812,7 +7812,7 @@ u64 __weak perf_reg_value(struct pt_regs *regs, int idx)return 0 ; } -int __weak perf_reg_validate (u64 mask) +int __weak perf_reg_validate (u64 mask,bool simd_enabled) {return mask ? -ENOSYS :0 ; }
@@ -13793,7 +13793,8 @@ static int perf_copy_attr(struct perf_event_attr __user *uattr, }if (attr->sample_type &PERF_SAMPLE_REGS_USER ) { - ret =perf_reg_validate (attr->sample_regs_user); + ret =perf_reg_validate (attr->sample_regs_user, + attr->sample_simd_regs_enabled);if (ret)return ret;if (attr->sample_simd_regs_enabled) {
@@ -13825,7 +13826,8 @@ static int perf_copy_attr(struct perf_event_attr __user *uattr, attr->sample_max_stack = sysctl_perf_event_max_stack;if (attr->sample_type &PERF_SAMPLE_REGS_INTR ) { - ret =perf_reg_validate (attr->sample_regs_intr); + ret =perf_reg_validate (attr->sample_regs_intr, + attr->sample_simd_regs_enabled);if (ret)return ret;if (attr->sample_simd_regs_enabled) {
Signature · 1 line
2.34.1
PATCH v10 18/23 perf/x86: Support eGPRs sampling using sample_regs_* fields
In reply to Dapeng MiSupport sampling of APX eGPRs (R16 ~ R31) via the sample_regs_* fields.
To sample eGPRs, the sample_simd_regs_enabled field must be set. This allows the spare space (reclaimed from the original XMM space) in the sample_regs_* fields to be used for representing eGPRs.
The perf_reg_value() function needs to check if the PERF_SAMPLE_REGS_ABI_SIMD flag is set first, and then determine whether to output eGPRs or legacy XMM registers to userspace.
The perf_reg_validate() function first checks the simd_enabled argument to determine if the eGPRs bitmap is represented in sample_regs_* fields. It then validates the eGPRs bitmap accordingly.
Currently, eGPRs sampling is only supported on the x86_64 architecture, as APX is only available on x86_64 platforms.
APX eGPRs sampling will be enabled in a subsequent patch that sets PERF_PMU_CAP_SIMD_REGS.
--- arch/x86/events/core.c | 25 +++++++++++--- arch/x86/events/intel/core.c | 10 ++++-- arch/x86/events/perf_event.h | 25 ++++++++++++++ arch/x86/include/asm/perf_event.h | 4 +++ arch/x86/include/uapi/asm/perf_regs.h | 26 ++++++++++++++ arch/x86/kernel/perf_regs.c | 50 +++++++++++++++++---------- 6 files changed, 115 insertions(+), 25 deletions(-)
arch/x86/events/core.c +21 -4
@@ -642,11 +642,12 @@ static int pebs_simd_regs_validate(struct perf_event *event)if (event_needs_xmm (event) && x86_pmu.arch_pebs && !(caps &ARCH_PEBS_VECR_XMM ))return -EINVAL ; -/* PEBS does not support YMM/ZMM/OPMASK registers sampling yet. */ +/* PEBS does not support YMM/ZMM/OPMASK/eGPR registers sampling yet. */ if (event_needs_ymm (event) ||event_needs_low16_zmm (event) ||event_needs_high16_zmm (event) || -event_needs_opmask (event)) +event_needs_opmask (event) || +event_needs_egprs (event))return -EINVAL ;return 0 ;
@@ -654,10 +655,21 @@ static int pebs_simd_regs_validate(struct perf_event *event)static int event_simd_regs_validate (struct perf_event *event) { +u64 reserved = ~GENMASK_ULL (PERF_REG_MISC_MAX -1 ,0 ); +if (!get_ext_regs_buf (raw_smp_processor_id ()))return -ENOMEM ; -/* sample_simd_regs_enabled repurposes legacy XMM reg-mask slots. */ -if (event_has_extended_regs (event)) +/* +* The XMM space in the perf_event_x86_regs is reclaimed +* for eGPRs and other general registers. +*/ +if (((event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_regs_intr & reserved)) || + ((event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_regs_user & reserved))) +return -EINVAL ; +if (event_needs_egprs (event) && + !(x86_pmu.ext_regs_mask &XFEATURE_MASK_APX ))return -EINVAL ;if (event_needs_xmm (event) && !(x86_pmu.ext_regs_mask &XFEATURE_MASK_SSE ))
@@ -1841,6 +1853,7 @@ void x86_pmu_clear_perf_regs(struct pt_regs *regs) perf_regs->zmmh_regs =NULL ; perf_regs->h16zmm_regs =NULL ; perf_regs->opmask_regs =NULL ; + perf_regs->egpr_regs =NULL ; } static void update_perf_regs(struct x86_perf_regs *perf_regs,
@@ -1864,6 +1877,8 @@ static void update_perf_regs(struct x86_perf_regs *perf_regs, perf_regs->h16zmm =get_xsave_addr (xsave, XFEATURE_Hi16_ZMM);if (mask &XFEATURE_MASK_OPMASK ) perf_regs->opmask =get_xsave_addr (xsave,XFEATURE_OPMASK ); +if (mask &XFEATURE_MASK_APX ) + perf_regs->egpr =get_xsave_addr (xsave,XFEATURE_APX ); }/*
@@ -2040,6 +2055,8 @@ static u64 get_simd_sample_mask(struct perf_event *event, u64 sample_type) mask |= XFEATURE_MASK_Hi16_ZMM;if (__event_needs_opmask (event, sample_type)) mask |=XFEATURE_MASK_OPMASK ; +if (__event_needs_egprs (event, sample_type)) + mask |=XFEATURE_MASK_APX ;return mask; }
arch/x86/events/intel/core.c +7 -3
@@ -4719,15 +4719,19 @@ static void intel_pebs_aliases_skl(struct perf_event *event)static unsigned long intel_pmu_large_pebs_flags (struct perf_event *event) {unsigned long flags = x86_pmu.large_pebs_flags; -u64 gprs_mask =PEBS_GP_REGS |PERF_REG_EXTENDED_MASK ; +u64 gprs_mask = event->attr.sample_simd_regs_enabled ? +PEBS_GP_REGS |PERF_X86_EGPRS_MASK : +PEBS_GP_REGS |PERF_REG_EXTENDED_MASK ;if (event->attr.use_clockid) flags &= ~PERF_SAMPLE_TIME ;if (!event->attr.exclude_kernel) flags &= ~PERF_SAMPLE_REGS_USER ; -if (event->attr.sample_regs_user & ~gprs_mask) +if ((event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_regs_user & ~gprs_mask)) flags &= ~PERF_SAMPLE_REGS_USER ; -if (event->attr.sample_regs_intr & ~gprs_mask) +if ((event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_regs_intr & ~gprs_mask)) flags &= ~PERF_SAMPLE_REGS_INTR ;return flags; }
arch/x86/events/perf_event.h +25 -0
@@ -290,6 +290,31 @@ static inline bool event_needs_opmask(struct perf_event *event) PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER); } +static inline bool __event_needs_egprs (struct perf_event *event, +u64 sample_type) +{ +if (!event->attr.sample_simd_regs_enabled) +return false; + +if ((sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_regs_user &PERF_X86_EGPRS_MASK )) +return true; + +if ((sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_regs_intr &PERF_X86_EGPRS_MASK )) +return true; + +return false; +} + +static inline bool event_needs_egprs (struct perf_event *event) +{ +return __event_needs_egprs (event, +PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ); +} +struct amd_nb {int nb_id;/* NorthBridge id */ int refcnt;/* reference count */
arch/x86/include/asm/perf_event.h +4 -0
@@ -749,6 +749,10 @@ struct x86_perf_regs {u64 *opmask_regs;struct avx_512_opmask_state *opmask; }; +union { +u64 *egpr_regs; +struct apx_state *egpr; + }; };extern unsigned long perf_arch_instruction_pointer (struct pt_regs *regs);
arch/x86/include/uapi/asm/perf_regs.h +26 -0
@@ -29,9 +29,34 @@ enum perf_event_x86_regs {PERF_REG_X86_R13 ,PERF_REG_X86_R14 ,PERF_REG_X86_R15 , +/* +* The eGPRs and XMM have overlaps. Only one can be used +* at a time. The ABI PERF_SAMPLE_REGS_ABI_SIMD is used to +* distinguish which one is used. If PERF_SAMPLE_REGS_ABI_SIMD +* is set, then eGPRs is used, otherwise, XMM is used. +* +* Extended GPRs (eGPRs) +*/ +PERF_REG_X86_R16 , +PERF_REG_X86_R17 , +PERF_REG_X86_R18 , +PERF_REG_X86_R19 , +PERF_REG_X86_R20 , +PERF_REG_X86_R21 , +PERF_REG_X86_R22 , +PERF_REG_X86_R23 , +PERF_REG_X86_R24 , +PERF_REG_X86_R25 , +PERF_REG_X86_R26 , +PERF_REG_X86_R27 , +PERF_REG_X86_R28 , +PERF_REG_X86_R29 , +PERF_REG_X86_R30 , +PERF_REG_X86_R31 ,/* These are the limits for the GPRs. */ PERF_REG_X86_32_MAX =PERF_REG_X86_GS +1 ,PERF_REG_X86_64_MAX =PERF_REG_X86_R15 +1 , +PERF_REG_MISC_MAX =PERF_REG_X86_R31 +1 ,/* These all need two bits set because they are 128bit */ PERF_REG_X86_XMM0 = 32,
@@ -56,6 +81,7 @@ enum perf_event_x86_regs { };#define PERF_REG_EXTENDED_MASK (~((1ULL << PERF_REG_X86_XMM0) - 1)) +#define PERF_X86_EGPRS_MASK __GENMASK_ULL(PERF_REG_X86_R31, PERF_REG_X86_R16)enum {PERF_X86_SIMD_XMM_REGS =16 ,
arch/x86/kernel/perf_regs.c +32 -18
77 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 19/23 perf/x86: Support SSP sampling using sample_regs_* fields
In reply to Dapeng MiSupport sampling of CET SSP register via the sample_regs_* fields.
To sample SSP, the sample_simd_regs_enabled field must be set. This allows the spare space (reclaimed from the original XMM space) in the sample_regs_* fields to be used for representing SSP.
Similar to eGPRs sampling, the perf_reg_value() function needs to check if the PERF_SAMPLE_REGS_ABI_SIMD flag is set first, and then determine whether to output SSP or legacy XMM registers to userspace.
Additionally, arch-PEBS supports sampling SSP, which is placed into the GPRs group. Also enables arch-PEBS-based SSP sampling.
Currently, SSP sampling is only supported on the x86_64 architecture, as CET is only available on x86_64 platforms.
SSP sampling will be enabled in a subsequent patch that sets PERF_PMU_CAP_SIMD_REGS.
--- arch/x86/events/core.c | 15 +++++++++++++++ arch/x86/events/intel/core.c | 3 ++- arch/x86/events/intel/ds.c | 13 +++++++++++-- arch/x86/events/perf_event.h | 25 +++++++++++++++++++++++++ arch/x86/include/asm/perf_event.h | 1 + arch/x86/include/uapi/asm/perf_regs.h | 7 ++++--- arch/x86/kernel/perf_regs.c | 5 +++++ 7 files changed, 63 insertions(+), 6 deletions(-)
arch/x86/events/core.c +15 -0
@@ -642,6 +642,10 @@ static int pebs_simd_regs_validate(struct perf_event *event)if (event_needs_xmm (event) && x86_pmu.arch_pebs && !(caps &ARCH_PEBS_VECR_XMM ))return -EINVAL ; + +if (event_needs_ssp (event) && + !(x86_pmu.arch_pebs && (caps &ARCH_PEBS_GPR ))) +return -EINVAL ;/* PEBS does not support YMM/ZMM/OPMASK/eGPR registers sampling yet. */ if (event_needs_ymm(event) || event_needs_low16_zmm(event) ||
@@ -671,6 +675,9 @@ static int event_simd_regs_validate(struct perf_event *event)if (event_needs_egprs (event) && !(x86_pmu.ext_regs_mask &XFEATURE_MASK_APX ))return -EINVAL ; +if (event_needs_ssp (event) && + !(x86_pmu.ext_regs_mask &XFEATURE_MASK_CET_USER )) +return -EINVAL ;if (event_needs_xmm (event) && !(x86_pmu.ext_regs_mask &XFEATURE_MASK_SSE ))return -EINVAL ;
@@ -1854,11 +1861,13 @@ void x86_pmu_clear_perf_regs(struct pt_regs *regs) perf_regs->h16zmm_regs =NULL ; perf_regs->opmask_regs =NULL ; perf_regs->egpr_regs =NULL ; + perf_regs->ssp =NULL ; }static void update_perf_regs (struct x86_perf_regs *perf_regs,struct xregs_state *xsave,u64 bitmap) { +struct cet_user_state *cet;u64 mask;if (!xsave)
@@ -1879,6 +1888,10 @@ static void update_perf_regs(struct x86_perf_regs *perf_regs, perf_regs->opmask =get_xsave_addr (xsave,XFEATURE_OPMASK );if (mask &XFEATURE_MASK_APX ) perf_regs->egpr =get_xsave_addr (xsave,XFEATURE_APX ); +if (mask &XFEATURE_MASK_CET_USER ) { + cet =get_xsave_addr (xsave,XFEATURE_CET_USER ); + perf_regs->ssp = cet ? &cet->user_ssp :NULL ; + } }/*
@@ -2057,6 +2070,8 @@ static u64 get_simd_sample_mask(struct perf_event *event, u64 sample_type) mask |=XFEATURE_MASK_OPMASK ;if (__event_needs_egprs (event, sample_type)) mask |=XFEATURE_MASK_APX ; +if (__event_needs_ssp (event, sample_type)) + mask |=XFEATURE_MASK_CET_USER ;return mask; }
arch/x86/events/intel/core.c +2 -1
@@ -4720,7 +4720,8 @@ static unsigned long intel_pmu_large_pebs_flags(struct perf_event *event) {unsigned long flags = x86_pmu.large_pebs_flags;u64 gprs_mask = event->attr.sample_simd_regs_enabled ? -PEBS_GP_REGS |PERF_X86_EGPRS_MASK : +PEBS_GP_REGS |PERF_X86_EGPRS_MASK | +BIT_ULL (PERF_REG_X86_SSP ) :PEBS_GP_REGS |PERF_REG_EXTENDED_MASK ;if (event->attr.use_clockid)
arch/x86/events/intel/ds.c +11 -2
@@ -1700,6 +1700,7 @@ static u64 pebs_update_adaptive_cfg(struct perf_event *event)u64 sample_type = attr->sample_type;u64 pebs_data_cfg =0 ;bool gprs, tsx_weight; +u64 xgprs_mask;if (!(sample_type & ~(PERF_SAMPLE_IP |PERF_SAMPLE_TIME )) && attr->precise_ip >1 )
@@ -1714,10 +1715,13 @@ static u64 pebs_update_adaptive_cfg(struct perf_event *event)* + precise_ip < 2 for the non event IP * + For RTM TSX weight we need GPRs for the abort code. */ + xgprs_mask = event->attr.sample_simd_regs_enabled ? +PEBS_GP_REGS |BIT_ULL (PERF_REG_X86_SSP ) : +PEBS_GP_REGS ; gprs = ((sample_type &PERF_SAMPLE_REGS_INTR ) && - (attr->sample_regs_intr &PEBS_GP_REGS )) || + (attr->sample_regs_intr & xgprs_mask)) || ((sample_type &PERF_SAMPLE_REGS_USER ) && - (attr->sample_regs_user &PEBS_GP_REGS )); + (attr->sample_regs_user & xgprs_mask)); tsx_weight = (sample_type & PERF_SAMPLE_WEIGHT_TYPE) && ((attr->config & INTEL_ARCH_EVENT_MASK) ==
@@ -2666,6 +2670,11 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,__setup_pebs_gpr_group (event, regs, (struct pebs_gprs *)gprs, sample_type); + +/* Currently only user space mode enables SSP. */ +if (user_mode (regs) && (sample_type & + (PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ))) + perf_regs->ssp = &gprs->ssp; }if (header->aux) {
arch/x86/events/perf_event.h +25 -0
@@ -315,6 +315,31 @@ static inline bool event_needs_egprs(struct perf_event *event) PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER); } +static inline bool __event_needs_ssp (struct perf_event *event, +u64 sample_type) +{ +if (!event->attr.sample_simd_regs_enabled) +return false; + +if ((sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_USER ) && + (event->attr.sample_regs_user &BIT_ULL (PERF_REG_X86_SSP ))) +return true; + +if ((sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_type &PERF_SAMPLE_REGS_INTR ) && + (event->attr.sample_regs_intr &BIT_ULL (PERF_REG_X86_SSP ))) +return true; + +return false; +} + +static inline bool event_needs_ssp (struct perf_event *event) +{ +return __event_needs_ssp (event, +PERF_SAMPLE_REGS_INTR |PERF_SAMPLE_REGS_USER ); +} +struct amd_nb {int nb_id;/* NorthBridge id */ int refcnt;/* reference count */
arch/x86/include/asm/perf_event.h +1 -0
@@ -753,6 +753,7 @@ struct x86_perf_regs {u64 *egpr_regs;struct apx_state *egpr; }; +u64 *ssp; };extern unsigned long perf_arch_instruction_pointer (struct pt_regs *regs);
arch/x86/include/uapi/asm/perf_regs.h +4 -3
@@ -30,10 +30,10 @@ enum perf_event_x86_regs {PERF_REG_X86_R14 ,PERF_REG_X86_R15 ,/* -* The eGPRs and XMM have overlaps. Only one can be used +* The eGPRs/SSP and XMM have overlaps. Only one can be used * at a time. The ABI PERF_SAMPLE_REGS_ABI_SIMD is used to * distinguish which one is used. If PERF_SAMPLE_REGS_ABI_SIMD -* is set, then eGPRs is used, otherwise, XMM is used. +* is set, then eGPRs/SSP is used, otherwise, XMM is used. * * Extended GPRs (eGPRs) */
@@ -53,10 +53,11 @@ enum perf_event_x86_regs {PERF_REG_X86_R29 ,PERF_REG_X86_R30 ,PERF_REG_X86_R31 , +PERF_REG_X86_SSP ,/* These are the limits for the GPRs. */ PERF_REG_X86_32_MAX =PERF_REG_X86_GS +1 ,PERF_REG_X86_64_MAX =PERF_REG_X86_R15 +1 , -PERF_REG_MISC_MAX =PERF_REG_X86_R31 +1 , +PERF_REG_MISC_MAX =PERF_REG_X86_SSP +1 ,/* These all need two bits set because they are 128bit */ PERF_REG_X86_XMM0 = 32,
arch/x86/kernel/perf_regs.c +5 -0
@@ -72,6 +72,11 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)return 0 ;return perf_regs->egpr_regs[idx -PERF_REG_X86_R16 ]; } +if (idx ==PERF_REG_X86_SSP ) { +if (!perf_regs->ssp) +return 0 ; +return *perf_regs->ssp; + } }else {if (idx >=PERF_REG_X86_XMM0 && idx <PERF_REG_X86_XMM_MAX ) {if (!perf_regs->xmm_regs)
Signature · 1 line
2.34.1
PATCH v10 20/23 perf/x86/intel: Support arch-PEBS based SIMD/eGPRs sampling
In reply to Dapeng MiSupport arch-PEBS based SIMD/eGPRs/SSP registers sampling.
Arch-PEBS supports sampling all of these registers. SIMD and eGPR registers are placed into the XSAVE-Enabled Registers (XER) group with the layout described below.
Field Name Registers Used Size
XSTATE_BV XINUSE for groups 8 B
Reserved Reserved 8 B
SSER XMM0-XMM15 16 regs * 16 B = 256 B
YMMHIR Upper 128 bits of YMM0-YMM15 16 regs * 16 B = 256 B
EGPR R16-R31 16 regs * 8 B = 128 B
OPMASKR K0-K7 8 regs * 8 B = 64 B
ZMMHIR Upper 256 bits of ZMM0-ZMM15 16 regs * 32 B = 512 B
Hi16ZMMR ZMM16-ZMM31 16 regs * 64 B = 1024 B
Memory space in the output buffer is allocated for these sub-groups as long as the corresponding Format.XER[55:49] bits in the PEBS record header are set. However, the arch-PEBS hardware engine does not write the sub-group if it is not used (in INIT state). In such cases, the corresponding bit in the XSTATE_BV bitmap is set to 0. Therefore, the XSTATE_BV field is checked to determine if the register data is actually written for each PEBS record. If not, the register data is not outputted to userspace.
Additionally, the MSRs IA32_PMC_{GPn|FXm}_CFG_C.[55:49] bits are used to manage which types of these registers need to be sampled.
Arch-PEBS based SIMD/eGPRs sampling will be enabled in a subsequent patch that sets PERF_PMU_CAP_SIMD_REGS.
--- arch/x86/events/core.c | 20 ++++++--- arch/x86/events/intel/core.c | 15 +++++++ arch/x86/events/intel/ds.c | 67 +++++++++++++++++++++++++++++-- arch/x86/include/asm/msr-index.h | 7 ++++ arch/x86/include/asm/perf_event.h | 8 +++- 5 files changed, 106 insertions(+), 11 deletions(-)
arch/x86/events/core.c +14 -6
@@ -646,12 +646,20 @@ static int pebs_simd_regs_validate(struct perf_event *event)if (event_needs_ssp (event) && !(x86_pmu.arch_pebs && (caps &ARCH_PEBS_GPR )))return -EINVAL ; -/* PEBS does not support YMM/ZMM/OPMASK/eGPR registers sampling yet. */ -if (event_needs_ymm (event) || -event_needs_low16_zmm (event) || -event_needs_high16_zmm (event) || -event_needs_opmask (event) || -event_needs_egprs (event)) +if (event_needs_ymm (event) && + !(x86_pmu.arch_pebs && (caps &ARCH_PEBS_VECR_YMMH ))) +return -EINVAL ; +if (event_needs_egprs (event) && + !(x86_pmu.arch_pebs && (caps &ARCH_PEBS_VECR_EGPRS ))) +return -EINVAL ; +if (event_needs_opmask (event) && + !(x86_pmu.arch_pebs && (caps &ARCH_PEBS_VECR_OPMASK ))) +return -EINVAL ; +if (event_needs_low16_zmm (event) && + !(x86_pmu.arch_pebs && (caps &ARCH_PEBS_VECR_ZMMH ))) +return -EINVAL ; +if (event_needs_high16_zmm (event) && + !(x86_pmu.arch_pebs && (caps &ARCH_PEBS_VECR_H16ZMM )))return -EINVAL ;return 0 ;
arch/x86/events/intel/core.c +15 -0
@@ -3498,6 +3498,21 @@ static void intel_pmu_enable_event_ext(struct perf_event *event)if (pebs_data_cfg &PEBS_DATACFG_XMMS ) ext |=ARCH_PEBS_VECR_XMM & cap.caps; +if (pebs_data_cfg &PEBS_DATACFG_YMMHS ) + ext |=ARCH_PEBS_VECR_YMMH & cap.caps; + +if (pebs_data_cfg &PEBS_DATACFG_EGPRS ) + ext |=ARCH_PEBS_VECR_EGPRS & cap.caps; + +if (pebs_data_cfg &PEBS_DATACFG_OPMASKS ) + ext |=ARCH_PEBS_VECR_OPMASK & cap.caps; + +if (pebs_data_cfg &PEBS_DATACFG_ZMMHS ) + ext |=ARCH_PEBS_VECR_ZMMH & cap.caps; + +if (pebs_data_cfg &PEBS_DATACFG_H16ZMMS ) + ext |=ARCH_PEBS_VECR_H16ZMM & cap.caps; +if (pebs_data_cfg &PEBS_DATACFG_LBRS ) ext |=ARCH_PEBS_LBR & cap.caps;
arch/x86/events/intel/ds.c +63 -4
83 lines not on this page · raw message
arch/x86/include/asm/msr-index.h +7 -0
@@ -350,6 +350,13 @@#define ARCH_PEBS_LBR_SHIFT 40#define ARCH_PEBS_LBR (0x3ull << ARCH_PEBS_LBR_SHIFT)#define ARCH_PEBS_VECR_XMM BIT_ULL(49) +#define ARCH_PEBS_VECR_YMMH BIT_ULL(50) +#define ARCH_PEBS_VECR_EGPRS BIT_ULL(51) +#define ARCH_PEBS_VECR_OPMASK BIT_ULL(53) +#define ARCH_PEBS_VECR_ZMMH BIT_ULL(54) +#define ARCH_PEBS_VECR_H16ZMM BIT_ULL(55) +#define ARCH_PEBS_VECR_EXT_SHIFT 49 +#define ARCH_PEBS_VECR_EXT (0x7full << ARCH_PEBS_VECR_EXT_SHIFT)#define ARCH_PEBS_GPR BIT_ULL(61)#define ARCH_PEBS_AUX BIT_ULL(62)#define ARCH_PEBS_EN BIT_ULL(63)
arch/x86/include/asm/perf_event.h +7 -1
@@ -150,6 +150,11 @@#define PEBS_DATACFG_LBRS BIT_ULL(3)#define PEBS_DATACFG_CNTR BIT_ULL(4)#define PEBS_DATACFG_METRICS BIT_ULL(5) +#define PEBS_DATACFG_YMMHS BIT_ULL(6) +#define PEBS_DATACFG_OPMASKS BIT_ULL(7) +#define PEBS_DATACFG_ZMMHS BIT_ULL(8) +#define PEBS_DATACFG_H16ZMMS BIT_ULL(9) +#define PEBS_DATACFG_EGPRS BIT_ULL(10)#define PEBS_DATACFG_LBR_SHIFT 24#define PEBS_DATACFG_CNTR_SHIFT 32#define PEBS_DATACFG_CNTR_MASK GENMASK_ULL(15, 0)
@@ -547,7 +552,8 @@ struct arch_pebs_header { rsvd3:7, xmm:1, ymmh:1, - rsvd4:2, + egpr:1, + rsvd4:1, opmask:1, zmmh:1, h16zmm:1,
Signature · 1 line
2.34.1
PATCH v10 21/23 perf/x86/intel: Advertise PERF_PMU_CAP_SIMD_REGS capability
In reply to Dapeng MiSet PERF_PMU_CAP_SIMD_REGS once XSAVES-backed extended state beyond XMM is available for sampling, e.g. YMM, ZMM, OPMASK, eGPRs, or SSP.
Keep backward compatibility unchanged. When only XMM sampling is available, continue to expose only PERF_PMU_CAP_EXTENDED_REGS. Expose PERF_PMU_CAP_SIMD_REGS only when additional SIMD/predicate register state is supported.
--- arch/x86/events/intel/core.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
arch/x86/events/intel/core.c +20 -0
@@ -6229,7 +6229,27 @@ static inline void __intel_update_pmu_xregs_caps(struct pmu *pmu)*/ x86_pmu.ext_regs_mask |=XFEATURE_MASK_SSE ; +if (boot_cpu_has (X86_FEATURE_AVX ) && +cpu_has_xfeatures (XFEATURE_MASK_YMM ,NULL )) + x86_pmu.ext_regs_mask |=XFEATURE_MASK_YMM ; +if (boot_cpu_has (X86_FEATURE_APX ) && +cpu_has_xfeatures (XFEATURE_MASK_APX ,NULL )) + x86_pmu.ext_regs_mask |=XFEATURE_MASK_APX ; +if (boot_cpu_has (X86_FEATURE_AVX512F )) { +if (cpu_has_xfeatures (XFEATURE_MASK_OPMASK ,NULL )) + x86_pmu.ext_regs_mask |=XFEATURE_MASK_OPMASK ; +if (cpu_has_xfeatures (XFEATURE_MASK_ZMM_Hi256,NULL )) + x86_pmu.ext_regs_mask |= XFEATURE_MASK_ZMM_Hi256; +if (cpu_has_xfeatures (XFEATURE_MASK_Hi16_ZMM,NULL )) + x86_pmu.ext_regs_mask |= XFEATURE_MASK_Hi16_ZMM; + } +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && +cpu_has_xfeatures (XFEATURE_MASK_CET_USER ,NULL )) + x86_pmu.ext_regs_mask |=XFEATURE_MASK_CET_USER ; + dest_pmu->capabilities |=PERF_PMU_CAP_EXTENDED_REGS ; +if (x86_pmu.ext_regs_mask >XFEATURE_MASK_SSE ) + dest_pmu->capabilities |=PERF_PMU_CAP_SIMD_REGS ; }static inline void __intel_update_large_pebs_flags (struct pmu *pmu)
Signature · 1 line
2.34.1
PATCH v10 22/23 perf/x86: Activate back-to-back NMI detection for arch-PEBS induced NMIs
In reply to Dapeng MiWhen two or more identical PEBS events with the same sampling period are programmed on a mix of PDIST and non-PDIST counters, multiple back-to-back NMIs can be triggered.
The Linux PMI handler processes the first NMI and clears the GLOBAL_STATUS MSR. If a second NMI is triggered immediately after the first, it is recognized as a "suspicious NMI" because no bits are set in the GLOBAL_STATUS MSR (cleared by the first NMI).
This issue does not lead to PEBS data corruption or data loss, but it does result in an annoying warning message.
The current NMI handler supports back-to-back NMI detection, but it requires the PMI handler to return the count of actually processed events, which the PEBS handler does not currently do.
Thus, modify the PEBS handlers to return the count of actually processed events, thereby activating back-to-back NMI detection and avoiding the "suspicious NMI" warning.
--- arch/x86/events/core.c | 2 +- arch/x86/events/intel/core.c | 31 +++++++++++++++++--------- arch/x86/events/intel/ds.c | 43 ++++++++++++++++++++++++------------ arch/x86/events/perf_event.h | 2 +- 4 files changed, 51 insertions(+), 27 deletions(-)
arch/x86/events/core.c +1 -1
@@ -93,7 +93,7 @@ DEFINE_STATIC_CALL_NULL(x86_pmu_stop_scheduling, *x86_pmu.stop_scheduling);DEFINE_STATIC_CALL_NULL (x86_pmu_sched_task, *x86_pmu.sched_task); -DEFINE_STATIC_CALL_NULL (x86_pmu_drain_pebs, *x86_pmu.drain_pebs); +DEFINE_STATIC_CALL_RET0 (x86_pmu_drain_pebs, *x86_pmu.drain_pebs);DEFINE_STATIC_CALL_NULL (x86_pmu_pebs_aliases, *x86_pmu.pebs_aliases);DEFINE_STATIC_CALL_NULL (x86_pmu_filter, *x86_pmu.filter);
arch/x86/events/intel/core.c +20 -11
@@ -3784,8 +3784,8 @@ static void intel_pmu_reset(void)* * The contents and other behavior of the guest event do not matter. */ -static void x86_pmu_handle_guest_pebs (struct pt_regs *regs, -struct perf_sample_data *data) +static int x86_pmu_handle_guest_pebs (struct pt_regs *regs, +struct perf_sample_data *data) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 guest_pebs_idxs = cpuc->pebs_enabled & ~cpuc->intel_ctrl_host_mask;
@@ -3793,11 +3793,11 @@ static void x86_pmu_handle_guest_pebs(struct pt_regs *regs,int bit;if (!unlikely (perf_guest_state ())) -return ; +return 0 ;if (!x86_pmu.pebs_ept || !x86_pmu.pebs_active || !guest_pebs_idxs) -return ; +return 0 ;for_each_set_bit (bit, (unsigned long *)&guest_pebs_idxs,X86_PMC_IDX_MAX ) { event = cpuc->events[bit];
@@ -3807,9 +3807,14 @@ static void x86_pmu_handle_guest_pebs(struct pt_regs *regs,perf_sample_data_init (data,0 , event->hw.last_period);perf_event_overflow (event, data, regs); -/* Inject one fake event is enough. */ -break ; +/* +* Inject one fake event is enough. +* Returning 1 to inform PMI is handled. +*/ +return 1 ; } + +return 0 ; }static int handle_pmi_common (struct pt_regs *regs,u64 status)
@@ -3858,9 +3863,11 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (__test_and_clear_bit (GLOBAL_STATUS_BUFFER_OVF_BIT , (unsigned long *)&status)) {u64 pebs_enabled = cpuc->pebs_enabled; - handled++; -x86_pmu_handle_guest_pebs (regs, &data); -static_call (x86_pmu_drain_pebs)(regs, &data); + handled +=x86_pmu_handle_guest_pebs (regs, &data); + handled +=static_call (x86_pmu_drain_pebs)(regs, &data); +/* Ensure no "suspicious NMI" warning for empty PEBS buffer. */ +if (!handled) + handled++;/* * PMI throttle may be triggered, which stops the PEBS event.
@@ -3887,8 +3894,10 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)*/ if (__test_and_clear_bit (GLOBAL_STATUS_ARCH_PEBS_THRESHOLD_BIT , (unsigned long *)&status)) { - handled++; -static_call (x86_pmu_drain_pebs)(regs, &data); + handled +=static_call (x86_pmu_drain_pebs)(regs, &data); +/* Ensure no "suspicious NMI" warning for empty PEBS buffer. */ +if (!handled) + handled++;if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ]))
arch/x86/events/intel/ds.c +29 -14
167 lines not on this page · raw message
arch/x86/events/perf_event.h +1 -1
@@ -1145,7 +1145,7 @@ struct x86_pmu {int pebs_record_size;int pebs_buffer_size;u64 pebs_events_mask; -void (*drain_pebs)(struct pt_regs *regs,struct perf_sample_data *data); +int (*drain_pebs)(struct pt_regs *regs,struct perf_sample_data *data);struct event_constraint *pebs_constraints;void (*pebs_aliases)(struct perf_event *event);u64 (*pebs_latency_data)(struct perf_event *event,u64 status);
Signature · 1 line
2.34.1
PATCH v10 23/23 perf/x86/intel: Add sanity check for PEBS record/fragment size
In reply to Dapeng MiPrevent potential infinite loops by adding a sanity check for the corrupted PEBS record or fragment size which could happen in theory.
If a corrupted PEBS fragment is detected, the entire PEBS record including the corrupted fragment and all subsequent records will be dropped and a NULL PEBS record is reported to user space. This ensures the integrity of PEBS data and prevents infinite loops in setup_arch_pebs_sample_data() again.
Please note software has no way to figure out which events are impacted by the corrupted record, so the last record of each event would be discarded for all events if corrupted record is detected even though it may be a well-formed record for some events.
--- arch/x86/events/intel/ds.c | 77 ++++++++++++++++++++++++++++---------- 1 file changed, 58 insertions(+), 19 deletions(-)
arch/x86/events/intel/ds.c +58 -19
@@ -2650,6 +2650,9 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, again: header = at; +if (!header->size) +return ; + next_record = at +sizeof (struct arch_pebs_header );if (header->basic) {struct arch_pebs_basic *basic = next_record;
@@ -2923,13 +2926,21 @@ __intel_pmu_pebs_last_event(struct perf_event *event,struct pt_regs *iregs, struct pt_regs *regs, struct perf_sample_data *data, - void *at, - int count, + void *at, int count, bool corrupted, setup_fn setup_sample) { struct hw_perf_event *hwc = &event->hw; -setup_sample (event, iregs, at, data, regs); +/* Skip parsing corrupted PEBS record. */ +if (corrupted) { +/* Clear stale register states in previous records. */ +memset (regs,0 ,sizeof (*regs)); +x86_pmu_clear_perf_regs (regs); +perf_sample_data_init (data,0 , event->hw.last_period); + }else { +setup_sample (event, iregs, at, data, regs); + } +if (iregs == &dummy_iregs) {/* * The PEBS records may be drained in the non-overflow context,
@@ -2947,12 +2958,16 @@ __intel_pmu_pebs_last_event(struct perf_event *event, }if (hwc->flags &PERF_X86_EVENT_AUTO_RELOAD ) { -if ((is_pebs_counter_event_group (event))) { -/* -* The value of each sample has been updated when setup -* the corresponding sample data. -*/ -perf_event_update_userpage (event); +if (is_pebs_counter_event_group (event)) { +if (corrupted) { +intel_pmu_save_and_restart_reload (event,1 ); + }else { +/* +* The value of each sample has been updated +* when setup the corresponding sample data. +*/ +perf_event_update_userpage (event); + } }else {/* * Now, auto-reload is only enabled in fixed period mode.
@@ -2976,7 +2991,7 @@ __intel_pmu_pebs_last_event(struct perf_event *event,* counters-snapshotting record, only needs to set the new * period for the counter. */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event) && !corrupted)static_call (x86_pmu_set_period)(event);else intel_pmu_save_and_restart (event);
@@ -3005,13 +3020,15 @@ __intel_pmu_pebs_events(struct perf_event *event, iregs = &dummy_iregs;while (cnt >1 ) { -__intel_pmu_pebs_event (event, iregs, regs, data, at, setup_sample); +__intel_pmu_pebs_event (event, iregs, regs, data, + at, setup_sample); at += cpuc->pebs_record_size; at =get_next_pebs_record_by_bit (at, top, bit); cnt--; } -__intel_pmu_pebs_last_event (event, iregs, regs, data, at, count, setup_sample); +__intel_pmu_pebs_last_event (event, iregs, regs, data, at, + count, false, setup_sample); }static int intel_pmu_drain_pebs_core (struct pt_regs *iregs,struct perf_sample_data *data)
@@ -3227,23 +3244,28 @@ static __always_inline void __intel_pmu_handle_last_pebs_record(struct pt_regs *iregs, struct pt_regs *regs, struct perf_sample_data *data, - u64 mask, short *counts, void **last, + u64 mask, short *counts, + void **last, bool corrupted, setup_fn setup_sample) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);struct perf_event *event; +bool handled = false;int bit;for_each_set_bit (bit, (unsigned long *)&mask,X86_PMC_IDX_MAX ) {if (!counts[bit])continue ; + handled = true; event = cpuc->events[bit]; -__intel_pmu_pebs_last_event (event, iregs, regs, data, last[bit], - counts[bit], setup_sample); + counts[bit], corrupted, setup_sample); } +/* All records are corrupted, reset sampling period. */ +if (!handled) +intel_pmu_pebs_event_update_no_drain (cpuc, mask); }static int intel_pmu_drain_pebs_icl (struct pt_regs *iregs,struct perf_sample_data *data)
@@ -3257,6 +3279,7 @@ static int intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_dastruct pebs_basic *basic;void *base, *at, *top;u64 events_bitmap =0 ; +bool corrupted = false;u64 mask;if (!x86_pmu.pebs_active)
@@ -3284,6 +3307,10 @@ static int intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_dau64 pebs_status; basic = at; +if (WARN_ON_ONCE (!basic->format_size)) { + corrupted = true; +break ; + }if (basic->format_size != cpuc->pebs_record_size)continue ;
@@ -3295,7 +3322,7 @@ static int intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_da }__intel_pmu_handle_last_pebs_record (iregs, regs, data, mask, counts, last, - setup_pebs_adaptive_sample_data); + corrupted, setup_pebs_adaptive_sample_data);return hweight64 (events_bitmap); }
@@ -3311,6 +3338,7 @@ static int intel_pmu_drain_arch_pebs(struct pt_regs *iregs,struct pt_regs *regs = &perf_regs->regs;void *base, *at, *top;u64 events_bitmap =0 ; +bool corrupted = false;u64 mask;rdmsrq (MSR_IA32_PEBS_INDEX , index.whole);
@@ -3345,8 +3373,10 @@ static int intel_pmu_drain_arch_pebs(struct pt_regs *iregs, header = at; -if (WARN_ON_ONCE (!header->size)) -break ; +if (WARN_ON_ONCE (!header->size)) { + corrupted = true; + goto done; + }/* 1st fragment or single record must have basic group */ if (!header->basic) {
@@ -3366,15 +3396,24 @@ static int intel_pmu_drain_arch_pebs(struct pt_regs *iregs,if (!header->size)break ; at += header->size; +if (WARN_ON_ONCE (at >= top)) { + corrupted = true; + goto done; + } header = at; }/* Skip last fragment or the single record */ at += header->size; +if (WARN_ON_ONCE (at > top)) { + corrupted = true; + goto done; + } } +done:__intel_pmu_handle_last_pebs_record (iregs, regs, data, mask, - counts, last, + counts, last, corrupted, setup_arch_pebs_sample_data);return hweight64 (events_bitmap);
Signature · 1 line
2.34.1
Ian Rogers
reviewerQuoted 4 lines · Dapeng Mi
The v10 patch series requires the prerequisite bug-fix patchset:
https://lore.kernel.org/all/[email protected]/
Quoted 14 lines
Patch layout:
- Patches 1-5: Bug fixes and cleanup needed before enabling XSAVES-based sampling in NMI context.
- Patches 6-7: FPU-related preparation, includingxsaves_nmi()and related cleanup/optimization.
- Patches 8-11: PMI-based XMM sampling support through the existing sample_regs_intr/sample_regs_user interfaces for bothPERF_SAMPLE_REGS_INTRandPERF_SAMPLE_REGS_USER.
- Patches 12-19: New SIMD register interface and support for XMM/YMM/ZMM/OPMASK, APX eGPRs, and SSP through that interface.
- Patch 20: Extend arch PEBS to support YMM/ZMM/OPMASK, APX eGPRs, and SSP with the new interface.
- Patch 21: Enable new interface based sampling.
- Patches 22-23: arch PEBS bug fix and sanity check.
Hi Dapeng,
The patch series didn't apply in Sashiko:
https://sashiko.dev/#/patchset/20260721062506.3745816-1-dapeng1.mi%40linux.intel.com
I believe the patches should be against tip.git perf/core
https://sashiko.dev/#/log/baseline/47919/2
which was last updated 2 weeks ago:
https://web.git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=perf/core&id=edda9051e267b7390c7ce24b1b71434414ad156e
Since the change is quite large it would be interesting to read the Sashiko reviews.
Thanks,
Ian
Quoted 379 lines
Changes since v9:
- Patch 07/23: Updatefpu_swap_kvm_fpstate()to clearfpu->__task_fpstateonly after switching back to host state, ensuring that when non-NULL it points to host FPU state.
- Patch 11/23: Fix an issue where guest FPU state can be sampled during host user-regs sampling.
- Patch 13/23: Improve and streamlineperf_simd_reg_validate()to enforce full-register-only sampling for SIMD registers (XMM/YMM/ZMM/OPMASK).
- Patch 21/23: Simplify and unifyPERF_PMU_CAP_SIMD_REGScapability handling for both PEBS and non-PEBS events.
- Refine changelog text across the full patch series.Changes since v8:
- Patch 01/24: Fix missing PMI handler unregister on PMU initialization failure (Sashiko).
- Patch 11/24: Skip kernel-space SIMD sampling whenexclude_kernelis set, to account for PMI skid (Sashiko).
- Patch 12/24: Skip guest kernel-space SIMD sampling for user register sampling.
- Patch 14/24: Strengthen SIMD register configuration validation inperf_simd_reg_validate()(Sashiko).
- Patch 17/24: Strengthen PRED (OPMASK) register configuration validation inperf_simd_reg_validate()(Sashiko).
- Patch 21/24: Enhance large-PEBS checks for SSP and APX eGPR registers (Sashiko).
- Patch 22/24: RefinePERF_PMU_CAP_SIMD_REGSsetup: advertise the capability only when both XSAVES and Arch PEBS are supported (Peter).
- Refine patch changelogs across the series.Changes since V7:
- Validate the return value ofintel_pmu_init_hybrid()(Patch 01/23).
- Replacept_regswithx86_perf_regsinxen_pmu_irq_handler()(Patch 06/23).
- Improveevent_has_extended_regs()(Patch 09/23).
- Explicitly ensure the allocated XSAVE area is 64-byte aligned (Patch 10/23, Sashiko).
- Clear the SIMD register pointers inx86_user_regsto avoid exposing stale register data to user space (Patch 11/23, Sashiko).
- Refine the SIMD register interface and sample data layout, and add the missing SIMD data reservation inperf_prepare_sample()for non-x86 architectures (Patch 12/23, Sashiko).
- Improveperf_simd_reg_validate()for x86 (Patch 13/23, Sashiko).
- Refine SSP sampling and ensure the GPR sub-group flag is set for PEBS (Patch 19/23, Sashiko).
- Fix the incorrect large-PEBS check for XMM (Patch 20/23, Sashiko).
- Fix missing handling inx86_pmu_handle_guest_pebs()for back-to-back PMI detection (Patch 22/23, Sashiko).
- Strengthen the PEBS record header sanity checks to prevent invalid memory access (Patch 23/23, Sashiko).Changes since V6:
- Fix potential overwritten issue in hybrid PMU structure (patch 01/24)
- Restrict PEBS events work on GP counters if no PEBS baseline suggested (patch 02/24)
- Use per-cpux86_intr_regsforperf_event_nmi_handler()instead of temporary variable (patch 06/24)
- Add helperupdate_fpu_state_and_flag()to ensureTIF_NEED_FPU_LOADis set aftersave_fpregs_to_fpstate()call (patch 09/24)
- Optimize and simplifyx86_pmu_sample_xregs(), etc. (patch 11/24)
- Add macroword_for_each_set_bit()to simplify u64 set-bit iteration (patch 13/24)
- Add sanity check for PEBS fragment size (patch 24/24)Changes since V5:
- Introduce 3 commits to fix newly found PEBS issues (Patch 01~03/19)
- Address Peter comments, including,
* Fully support user-regs sampling of the SIMD/eGPRs/SSP registers
* Adjust newly added fields inperf_event_attrto avoid holes
* Fix the endian issue introduced byfor_each_set_bit()inevent/core.c
* Remove some unnecessary macros from UAPI headerperf_regs.h
* Enhance b2b NMI detection for all PEBS handlers to ensure identical behaviors of all PEBS handlers
- Split perf-tools patches which would be posted in a separate patchset laterChanges since V4:
- Rewrite some functions comments and commit messages (Dave)
- Add arch-PEBS based SIMD/eGPRs/SSP sampling support (Patch 15/19)
- Fix "suspecious NMI" warnning observed on PTL/NVL P-core and DMR by activating back-to-back NMI detection mechanism (Patch 16/19)
- Fix some minor issues on perf-tool patches (Patch 18/19)Changes since V3:
- Drop the SIMD registers if an NMI hits kernel mode forREGS_USER.
- Only dump the available regs, rather than zero and dump the unavailable regs. It's possible that the dumped registers are a subset of the requested registers.
- Some minor updates to address Dapeng's comments in V3.Changes since V2:
- Use the FPU format for the x86_pmu.ext_regs_mask as well
- Add a check before invokingxsaves_nmi()
- Addperf_simd_reg_check()to retrieve the number of available registers. If the kernel fails to get the requested registers, e.g., XSAVES fails, nothing dumps to the userspace (the V2 dumps all 0s).
- Add POC perf tool patchesChanges since V1:
- Apply the new interfaces to configure and dump the SIMD registers
- Utilize the existing FPU functions, e.g.,xstate_calculate_size,get_xsave_addr().This series adds support on x86 for sampling SIMD registers, APX eGPRs, and SSP with both PMI-based and PEBS-based sampling.
Starting with Intel Ice Lake, PEBS can sample XMM registers, but PMI-based XMM sampling is still not available. On newer Intel platforms with architectural PEBS support, such as Clearwater Forest and Diamond Rapids, the hardware also gains support for sampling additional SIMD state (XMM/YMM/ZMM/OPMASK), APX extended GPRs, and SSP.
To support these registers consistently across both PMI and PEBS, this series makes the following changes:
1. Adds a new
perf_event_attrinterface for SIMD register selection. The existing sample_regs_user/sample_regs_intr bitmaps do not have enough space to represent the full SIMD register set, so this series introduces dedicated fields for SIMD and predicate register masks and element widths.2. Introduces a new sample data layout for SIMD register data. SIMD register payload is appended after the GPR payload, and a new ABI flag,
PERF_SAMPLE_REGS_ABI_SIMD, indicates its presence.3. Adds
xsaves_nmi()to allow SIMD/eGPR/SSP sampling from PMI handlers in NMI context.4. Extends the arch PEBS path to support YMM/ZMM/OPMASK, APX eGPRs, and SSP sampling.
New
perf_event_attrfields
--------------------------This series adds the following fields to
perf_event_attr:/* * Defines the sampling SIMD/PRED(predicate) register bitmaps and * qword (8-byte) lengths. * * sample_simd_regs_enabled != 0 indicates SIMD/PRED registers are * requested. The register bitmaps and element sizes are described by: * * sample_simd_{vec,pred}_reg_{intr,user} * sample_simd_{vec,pred}_reg_qwords * * sample_simd_regs_enabled == 0 indicates no SIMD/PRED registers are * requested. */ __u16 sample_simd_regs_enabled; __u16 sample_simd_pred_reg_qwords; __u16 sample_simd_vec_reg_qwords; __u16 __reserved_4; __u32 sample_simd_pred_reg_intr; __u32 sample_simd_pred_reg_user; __u64 sample_simd_vec_reg_intr; __u64 sample_simd_vec_reg_user;Field semantics:
-sample_simd_vec_reg_qwords: qword count for regular SIMD registers
-sample_simd_pred_reg_qwords: qword count for predicate registers
- sample_simd_vec_reg_{intr,user}: SIMD register masks forPERF_SAMPLE_REGS_INTRandPERF_SAMPLE_REGS_USER
- sample_simd_pred_reg_{intr,user}: predicate register masks forPERF_SAMPLE_REGS_INTRandPERF_SAMPLE_REGS_USER
-sample_simd_regs_enabled: indicates whether the new SIMD fields are in useExamples:
To sample ZMM registers for
PERF_SAMPLE_REGS_INTR:
sample_simd_regs_enabled= 1
sample_simd_vec_reg_qwords= 8 // 512 bits = 8 qwords
sample_simd_vec_reg_intr=0xffffffff// zmm0-zmm31To sample OPMASK registers for
PERF_SAMPLE_REGS_USER:
sample_simd_regs_enabled= 1
sample_simd_pred_reg_qwords= 1 // 64 bits = 1 qword
sample_simd_pred_reg_user=0xff// opmask0-opmask7After introducing these fields, bits [63:32] in
sample_regs_userandsample_regs_intrare reclaimed for APX eGPRs and SSP instead of the previous XMM0-XMM15 encoding.Discussion of the new SIMD register interface is available at:
https://lore.kernel.org/lkml/[email protected]/Sample data layout
------------------SIMD register data is appended after the GPR data.
For
PERF_SAMPLE_REGS_USER:{ u64 abi; // enum
perf_sample_regs_abi
u64 regs[weight(mask)];
struct {
u64nr_vectors; // 0 ...weight(sample_simd_vec_reg_user)
u64vector_qwords; // 0 ...sample_simd_vec_reg_qwords
u64nr_pred; // 0 ...weight(sample_simd_pred_reg_user)
u64pred_qwords; // 0 ...sample_simd_pred_reg_qwords
u64 data[nr_vectors *vector_qwords+
nr_pred*pred_qwords];
} && (abi &PERF_SAMPLE_REGS_ABI_SIMD)
}For
PERF_SAMPLE_REGS_INTR:{ u64 abi; // enum
perf_sample_regs_abi
u64 regs[weight(mask)];
struct {
u64nr_vectors; // 0 ...weight(sample_simd_vec_reg_intr)
u64vector_qwords; // 0 ...sample_simd_vec_reg_qwords
u64nr_pred; // 0 ...weight(sample_simd_pred_reg_intr)
u64pred_qwords; // 0 ...sample_simd_pred_reg_qwords
u64 data[nr_vectors *vector_qwords+
nr_pred*pred_qwords];
} && (abi &PERF_SAMPLE_REGS_ABI_SIMD)
}
PERF_SAMPLE_REGS_ABI_SIMDindicates that SIMD register data is present.The metadata fields are encoded as u64 to keep perf tool parsing and cross-endian support straightforward.
Example
-------$ perf record -I? available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 SSP XMM0-15 YMM0-15 ZMM0-31 OPMASK0-7$ perf record --user-regs=? available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 SSP XMM0-15 YMM0-15 ZMM0-31 OPMASK0-7$ perf record -e branches:p \ -Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \ -c 100000 ./test $ perf report -D...
140277619921150xcf30[0x8a8]:PERF_RECORD_SAMPLE(IP, 0x1): 29964/29964:
0xffffffff9f085e24period: 100000 addr: 0
... intr regs: mask0x18001010003ABI 64-bit.... AX 0xdffffc0000000000 .... BX 0xffff8882297685e8 .... R8 0x0000000000000000 .... R16 0x0000000000000000 .... R31 0x0000000000000000 .... SSP 0x0000000000000000... SIMD ABI
nr_vectors32vector_qwords8nr_pred8pred_qwords1 .... ZMM[0][0]0x616c2f656d6f682f
.... ZMM[0][1]0x696c2f7265737562
...
.... ZMM[31][7]0x0000000000000000
.... OPMASK[0]0x00000000fffffe00
....
.... OPMASK[7]0x0000000000000000
...Testing
-------The following intr-regs, user-regs, and combined sampling tests were run on DMR and NVL. The sampled register data was reported correctly and no issues were observed.
$ ./perf record -e branches:p \ -Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask -b -c 10000 sleep 1$ ./perf record -e branches \ -Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask -b -c 10000 sleep 1$ ./perf record -e branches:p \ --user-regs=ax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \ -b -c 10000 sleep 1$ ./perf record -e branches \ --user-regs=ax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ -Ixmm,ymm,zmm,opmask \ --user-regs=ax,bx,r8,r16,r31,ssp \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ --user-regs=xmm,ymm,zmm,opmask \ -Iax,bx,r8,r16,r31,ssp \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ -Iax,bx,r9,r17,r30,ssp \ --user-regs=ax,bx,r8,r16,r31,ssp \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ -Ixmm,opmask --user-regs=zmm \ -b -c 10000 taskset -c 0 sleep 1History:
v9: https://lore.kernel.org/all/[email protected]/
v8: https://lore.kernel.org/all/[email protected]/
v7: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/Dapeng Mi (20):
perf/x86: Move hybrid PMU initialization beforex86_pmu_starting_cpu()perf/x86/intel: Enable large PEBS sampling for XMMs perf/x86/intel: Convertx86_perf_regsto per-cpu variables
perf: Eliminate duplicate arch-specific function definitions perf/x86: Usex86_perf_regsin NMI handlers
x86/fpu: Addupdate_fpu_state_and_flag()helper perf/x86/intel: CentralizePERF_PMU_CAP_EXTENDED_REGSupdates perf/x86: Enable XMM register sampling for non-PEBS events perf/x86: Enable XMM register sampling forREGS_USERcase perf/x86: Support XMM sampling using sample_simd_vec_reg_* fields perf/x86: Support YMM sampling using sample_simd_vec_reg_* fields perf/x86: Support ZMM sampling using sample_simd_vec_reg_* fields perf/x86: Support OPMASK sampling using sample_simd_pred_reg_* fields
perf: Enhanceperf_reg_validate()withsimd_enabledargument perf/x86: Support eGPRs sampling using sample_regs_* fields perf/x86: Support SSP sampling using sample_regs_* fields perf/x86/intel: Support arch-PEBS based SIMD/eGPRs sampling perf/x86/intel: AdvertisePERF_PMU_CAP_SIMD_REGScapability perf/x86: Activate back-to-back NMI detection for arch-PEBS induced
NMIs
perf/x86/intel: Add sanity check for PEBS record/fragment sizeKan Liang (3):
x86/fpu/xstate: Addxsaves_nmi()helper
perf: Move and enhancehas_extended_regs()for arch-specific use
perf: Add sampling support for SIMD registersarch/arm/kernel/perf_regs.c | 8 +- arch/arm64/kernel/perf_regs.c | 8 +- arch/csky/kernel/perf_regs.c | 8 +- arch/loongarch/kernel/perf_regs.c | 8 +- arch/mips/kernel/perf_regs.c | 8 +- arch/parisc/kernel/perf_regs.c | 8 +- arch/powerpc/perf/perf_regs.c | 2 +- arch/riscv/kernel/perf_regs.c | 8 +- arch/s390/kernel/perf_regs.c | 2 +- arch/x86/events/core.c | 483 +++++++++++++++++++++++++- arch/x86/events/intel/core.c | 108 ++++-- arch/x86/events/intel/ds.c | 232 ++++++++++--- arch/x86/events/perf_event.h | 208 ++++++++++- arch/x86/include/asm/fpu/sched.h | 5 +- arch/x86/include/asm/fpu/xstate.h | 3 + arch/x86/include/asm/msr-index.h | 7 + arch/x86/include/asm/perf_event.h | 35 +- arch/x86/include/uapi/asm/perf_regs.h | 53 +++ arch/x86/kernel/fpu/core.c | 33 +- arch/x86/kernel/fpu/xstate.c | 25 +- arch/x86/kernel/perf_regs.c | 174 ++++++++-- arch/x86/xen/pmu.c | 5 +- include/linux/perf_event.h | 23 ++ include/linux/perf_regs.h | 36 +- include/uapi/linux/perf_event.h | 49 ++- kernel/events/core.c | 177 +++++++++- 26 files changed, 1490 insertions(+), 226 deletions(-)--
2.34.1
Quoted 27 lines · Ian Rogers
Quoted 17 lines · Dapeng Mi
The v10 patch series requires the prerequisite bug-fix patchset:
https://lore.kernel.org/all/[email protected]/Patch layout:
- Patches 1-5: Bug fixes and cleanup needed before enabling XSAVES-based sampling in NMI context.
- Patches 6-7: FPU-related preparation, includingxsaves_nmi()and related cleanup/optimization.
- Patches 8-11: PMI-based XMM sampling support through the existing sample_regs_intr/sample_regs_user interfaces for bothPERF_SAMPLE_REGS_INTRandPERF_SAMPLE_REGS_USER.
- Patches 12-19: New SIMD register interface and support for XMM/YMM/ZMM/OPMASK, APX eGPRs, and SSP through that interface.
- Patch 20: Extend arch PEBS to support YMM/ZMM/OPMASK, APX eGPRs, and SSP with the new interface.
- Patch 21: Enable new interface based sampling.
- Patches 22-23: arch PEBS bug fix and sanity check.Hi Dapeng,
The patch series didn't apply in Sashiko:
https://sashiko.dev/#/patchset/20260721062506.3745816-1-dapeng1.mi%40linux.intel.com
I believe the patches should be against tip.git perf/core
https://sashiko.dev/#/log/baseline/47919/2
which was last updated 2 weeks ago:
https://web.git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=perf/core&id=edda9051e267b7390c7ce24b1b71434414ad156e
Quoted 2 lines
Since the change is quite large it would be interesting to read the Sashiko reviews.
Ian, thanks for reminding this. The reason that sashiko failed to apply this patchset is that this v10 patchset depends on the below prerequisite bug-fix patchset which is based on the latest perf/core tree.
https://lore.kernel.org/all/[email protected]/
We triggered several rounds local sashiko review for this patchset internally and no real issues are found except some false positives. But anyway, once the prerequisite patchset is merged. I would resend this patchset and trigger the sashiko review.
Thanks.
Quoted 385 lines
Thanks,
IanQuoted 379 lines
Changes since v9:
- Patch 07/23: Updatefpu_swap_kvm_fpstate()to clearfpu->__task_fpstateonly after switching back to host state, ensuring that when non-NULL it points to host FPU state.
- Patch 11/23: Fix an issue where guest FPU state can be sampled during host user-regs sampling.
- Patch 13/23: Improve and streamlineperf_simd_reg_validate()to enforce full-register-only sampling for SIMD registers (XMM/YMM/ZMM/OPMASK).
- Patch 21/23: Simplify and unifyPERF_PMU_CAP_SIMD_REGScapability handling for both PEBS and non-PEBS events.
- Refine changelog text across the full patch series.Changes since v8:
- Patch 01/24: Fix missing PMI handler unregister on PMU initialization failure (Sashiko).
- Patch 11/24: Skip kernel-space SIMD sampling whenexclude_kernelis set, to account for PMI skid (Sashiko).
- Patch 12/24: Skip guest kernel-space SIMD sampling for user register sampling.
- Patch 14/24: Strengthen SIMD register configuration validation inperf_simd_reg_validate()(Sashiko).
- Patch 17/24: Strengthen PRED (OPMASK) register configuration validation inperf_simd_reg_validate()(Sashiko).
- Patch 21/24: Enhance large-PEBS checks for SSP and APX eGPR registers (Sashiko).
- Patch 22/24: RefinePERF_PMU_CAP_SIMD_REGSsetup: advertise the capability only when both XSAVES and Arch PEBS are supported (Peter).
- Refine patch changelogs across the series.Changes since V7:
- Validate the return value ofintel_pmu_init_hybrid()(Patch 01/23).
- Replacept_regswithx86_perf_regsinxen_pmu_irq_handler()(Patch 06/23).
- Improveevent_has_extended_regs()(Patch 09/23).
- Explicitly ensure the allocated XSAVE area is 64-byte aligned (Patch 10/23, Sashiko).
- Clear the SIMD register pointers inx86_user_regsto avoid exposing stale register data to user space (Patch 11/23, Sashiko).
- Refine the SIMD register interface and sample data layout, and add the missing SIMD data reservation inperf_prepare_sample()for non-x86 architectures (Patch 12/23, Sashiko).
- Improveperf_simd_reg_validate()for x86 (Patch 13/23, Sashiko).
- Refine SSP sampling and ensure the GPR sub-group flag is set for PEBS (Patch 19/23, Sashiko).
- Fix the incorrect large-PEBS check for XMM (Patch 20/23, Sashiko).
- Fix missing handling inx86_pmu_handle_guest_pebs()for back-to-back PMI detection (Patch 22/23, Sashiko).
- Strengthen the PEBS record header sanity checks to prevent invalid memory access (Patch 23/23, Sashiko).Changes since V6:
- Fix potential overwritten issue in hybrid PMU structure (patch 01/24)
- Restrict PEBS events work on GP counters if no PEBS baseline suggested (patch 02/24)
- Use per-cpux86_intr_regsforperf_event_nmi_handler()instead of temporary variable (patch 06/24)
- Add helperupdate_fpu_state_and_flag()to ensureTIF_NEED_FPU_LOADis set aftersave_fpregs_to_fpstate()call (patch 09/24)
- Optimize and simplifyx86_pmu_sample_xregs(), etc. (patch 11/24)
- Add macroword_for_each_set_bit()to simplify u64 set-bit iteration (patch 13/24)
- Add sanity check for PEBS fragment size (patch 24/24)Changes since V5:
- Introduce 3 commits to fix newly found PEBS issues (Patch 01~03/19)
- Address Peter comments, including,
* Fully support user-regs sampling of the SIMD/eGPRs/SSP registers
* Adjust newly added fields inperf_event_attrto avoid holes
* Fix the endian issue introduced byfor_each_set_bit()inevent/core.c
* Remove some unnecessary macros from UAPI headerperf_regs.h
* Enhance b2b NMI detection for all PEBS handlers to ensure identical behaviors of all PEBS handlers
- Split perf-tools patches which would be posted in a separate patchset laterChanges since V4:
- Rewrite some functions comments and commit messages (Dave)
- Add arch-PEBS based SIMD/eGPRs/SSP sampling support (Patch 15/19)
- Fix "suspecious NMI" warnning observed on PTL/NVL P-core and DMR by activating back-to-back NMI detection mechanism (Patch 16/19)
- Fix some minor issues on perf-tool patches (Patch 18/19)Changes since V3:
- Drop the SIMD registers if an NMI hits kernel mode forREGS_USER.
- Only dump the available regs, rather than zero and dump the unavailable regs. It's possible that the dumped registers are a subset of the requested registers.
- Some minor updates to address Dapeng's comments in V3.Changes since V2:
- Use the FPU format for the x86_pmu.ext_regs_mask as well
- Add a check before invokingxsaves_nmi()
- Addperf_simd_reg_check()to retrieve the number of available registers. If the kernel fails to get the requested registers, e.g., XSAVES fails, nothing dumps to the userspace (the V2 dumps all 0s).
- Add POC perf tool patchesChanges since V1:
- Apply the new interfaces to configure and dump the SIMD registers
- Utilize the existing FPU functions, e.g.,xstate_calculate_size,get_xsave_addr().This series adds support on x86 for sampling SIMD registers, APX eGPRs, and SSP with both PMI-based and PEBS-based sampling.
Starting with Intel Ice Lake, PEBS can sample XMM registers, but PMI-based XMM sampling is still not available. On newer Intel platforms with architectural PEBS support, such as Clearwater Forest and Diamond Rapids, the hardware also gains support for sampling additional SIMD state (XMM/YMM/ZMM/OPMASK), APX extended GPRs, and SSP.
To support these registers consistently across both PMI and PEBS, this series makes the following changes:
1. Adds a new
perf_event_attrinterface for SIMD register selection. The existing sample_regs_user/sample_regs_intr bitmaps do not have enough space to represent the full SIMD register set, so this series introduces dedicated fields for SIMD and predicate register masks and element widths.2. Introduces a new sample data layout for SIMD register data. SIMD register payload is appended after the GPR payload, and a new ABI flag,
PERF_SAMPLE_REGS_ABI_SIMD, indicates its presence.3. Adds
xsaves_nmi()to allow SIMD/eGPR/SSP sampling from PMI handlers in NMI context.4. Extends the arch PEBS path to support YMM/ZMM/OPMASK, APX eGPRs, and SSP sampling.
New
perf_event_attrfields
--------------------------This series adds the following fields to
perf_event_attr:/* * Defines the sampling SIMD/PRED(predicate) register bitmaps and * qword (8-byte) lengths. * * sample_simd_regs_enabled != 0 indicates SIMD/PRED registers are * requested. The register bitmaps and element sizes are described by: * * sample_simd_{vec,pred}_reg_{intr,user} * sample_simd_{vec,pred}_reg_qwords * * sample_simd_regs_enabled == 0 indicates no SIMD/PRED registers are * requested. */ __u16 sample_simd_regs_enabled; __u16 sample_simd_pred_reg_qwords; __u16 sample_simd_vec_reg_qwords; __u16 __reserved_4; __u32 sample_simd_pred_reg_intr; __u32 sample_simd_pred_reg_user; __u64 sample_simd_vec_reg_intr; __u64 sample_simd_vec_reg_user;Field semantics:
-sample_simd_vec_reg_qwords: qword count for regular SIMD registers
-sample_simd_pred_reg_qwords: qword count for predicate registers
- sample_simd_vec_reg_{intr,user}: SIMD register masks forPERF_SAMPLE_REGS_INTRandPERF_SAMPLE_REGS_USER
- sample_simd_pred_reg_{intr,user}: predicate register masks forPERF_SAMPLE_REGS_INTRandPERF_SAMPLE_REGS_USER
-sample_simd_regs_enabled: indicates whether the new SIMD fields are in useExamples:
To sample ZMM registers for
PERF_SAMPLE_REGS_INTR:
sample_simd_regs_enabled= 1
sample_simd_vec_reg_qwords= 8 // 512 bits = 8 qwords
sample_simd_vec_reg_intr=0xffffffff// zmm0-zmm31To sample OPMASK registers for
PERF_SAMPLE_REGS_USER:
sample_simd_regs_enabled= 1
sample_simd_pred_reg_qwords= 1 // 64 bits = 1 qword
sample_simd_pred_reg_user=0xff// opmask0-opmask7After introducing these fields, bits [63:32] in
sample_regs_userandsample_regs_intrare reclaimed for APX eGPRs and SSP instead of the previous XMM0-XMM15 encoding.Discussion of the new SIMD register interface is available at:
https://lore.kernel.org/lkml/[email protected]/Sample data layout
------------------SIMD register data is appended after the GPR data.
For
PERF_SAMPLE_REGS_USER:{ u64 abi; // enum
perf_sample_regs_abi
u64 regs[weight(mask)];
struct {
u64nr_vectors; // 0 ...weight(sample_simd_vec_reg_user)
u64vector_qwords; // 0 ...sample_simd_vec_reg_qwords
u64nr_pred; // 0 ...weight(sample_simd_pred_reg_user)
u64pred_qwords; // 0 ...sample_simd_pred_reg_qwords
u64 data[nr_vectors *vector_qwords+
nr_pred*pred_qwords];
} && (abi &PERF_SAMPLE_REGS_ABI_SIMD)
}For
PERF_SAMPLE_REGS_INTR:{ u64 abi; // enum
perf_sample_regs_abi
u64 regs[weight(mask)];
struct {
u64nr_vectors; // 0 ...weight(sample_simd_vec_reg_intr)
u64vector_qwords; // 0 ...sample_simd_vec_reg_qwords
u64nr_pred; // 0 ...weight(sample_simd_pred_reg_intr)
u64pred_qwords; // 0 ...sample_simd_pred_reg_qwords
u64 data[nr_vectors *vector_qwords+
nr_pred*pred_qwords];
} && (abi &PERF_SAMPLE_REGS_ABI_SIMD)
}
PERF_SAMPLE_REGS_ABI_SIMDindicates that SIMD register data is present.The metadata fields are encoded as u64 to keep perf tool parsing and cross-endian support straightforward.
Example
-------$ perf record -I? available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 SSP XMM0-15 YMM0-15 ZMM0-31 OPMASK0-7$ perf record --user-regs=? available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 SSP XMM0-15 YMM0-15 ZMM0-31 OPMASK0-7$ perf record -e branches:p \ -Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \ -c 100000 ./test $ perf report -D...
140277619921150xcf30[0x8a8]:PERF_RECORD_SAMPLE(IP, 0x1): 29964/29964:
0xffffffff9f085e24period: 100000 addr: 0
... intr regs: mask0x18001010003ABI 64-bit.... AX 0xdffffc0000000000 .... BX 0xffff8882297685e8 .... R8 0x0000000000000000 .... R16 0x0000000000000000 .... R31 0x0000000000000000 .... SSP 0x0000000000000000... SIMD ABI
nr_vectors32vector_qwords8nr_pred8pred_qwords1 .... ZMM[0][0]0x616c2f656d6f682f
.... ZMM[0][1]0x696c2f7265737562
...
.... ZMM[31][7]0x0000000000000000
.... OPMASK[0]0x00000000fffffe00
....
.... OPMASK[7]0x0000000000000000
...Testing
-------The following intr-regs, user-regs, and combined sampling tests were run on DMR and NVL. The sampled register data was reported correctly and no issues were observed.
$ ./perf record -e branches:p \ -Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask -b -c 10000 sleep 1$ ./perf record -e branches \ -Iax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask -b -c 10000 sleep 1$ ./perf record -e branches:p \ --user-regs=ax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \ -b -c 10000 sleep 1$ ./perf record -e branches \ --user-regs=ax,bx,r8,r16,r31,ssp,xmm,ymm,zmm,opmask \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ -Ixmm,ymm,zmm,opmask \ --user-regs=ax,bx,r8,r16,r31,ssp \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ --user-regs=xmm,ymm,zmm,opmask \ -Iax,bx,r8,r16,r31,ssp \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ -Iax,bx,r9,r17,r30,ssp \ --user-regs=ax,bx,r8,r16,r31,ssp \ -b -c 10000 sleep 1$ ./perf record -e branches:p \ -Ixmm,opmask --user-regs=zmm \ -b -c 10000 taskset -c 0 sleep 1History:
v9: https://lore.kernel.org/all/[email protected]/
v8: https://lore.kernel.org/all/[email protected]/
v7: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/Dapeng Mi (20):
perf/x86: Move hybrid PMU initialization beforex86_pmu_starting_cpu()perf/x86/intel: Enable large PEBS sampling for XMMs perf/x86/intel: Convertx86_perf_regsto per-cpu variables
perf: Eliminate duplicate arch-specific function definitions perf/x86: Usex86_perf_regsin NMI handlers
x86/fpu: Addupdate_fpu_state_and_flag()helper perf/x86/intel: CentralizePERF_PMU_CAP_EXTENDED_REGSupdates perf/x86: Enable XMM register sampling for non-PEBS events perf/x86: Enable XMM register sampling forREGS_USERcase perf/x86: Support XMM sampling using sample_simd_vec_reg_* fields perf/x86: Support YMM sampling using sample_simd_vec_reg_* fields perf/x86: Support ZMM sampling using sample_simd_vec_reg_* fields perf/x86: Support OPMASK sampling using sample_simd_pred_reg_* fields
perf: Enhanceperf_reg_validate()withsimd_enabledargument perf/x86: Support eGPRs sampling using sample_regs_* fields perf/x86: Support SSP sampling using sample_regs_* fields perf/x86/intel: Support arch-PEBS based SIMD/eGPRs sampling perf/x86/intel: AdvertisePERF_PMU_CAP_SIMD_REGScapability perf/x86: Activate back-to-back NMI detection for arch-PEBS induced
NMIs
perf/x86/intel: Add sanity check for PEBS record/fragment sizeKan Liang (3):
x86/fpu/xstate: Addxsaves_nmi()helper
perf: Move and enhancehas_extended_regs()for arch-specific use
perf: Add sampling support for SIMD registersarch/arm/kernel/perf_regs.c | 8 +- arch/arm64/kernel/perf_regs.c | 8 +- arch/csky/kernel/perf_regs.c | 8 +- arch/loongarch/kernel/perf_regs.c | 8 +- arch/mips/kernel/perf_regs.c | 8 +- arch/parisc/kernel/perf_regs.c | 8 +- arch/powerpc/perf/perf_regs.c | 2 +- arch/riscv/kernel/perf_regs.c | 8 +- arch/s390/kernel/perf_regs.c | 2 +- arch/x86/events/core.c | 483 +++++++++++++++++++++++++- arch/x86/events/intel/core.c | 108 ++++-- arch/x86/events/intel/ds.c | 232 ++++++++++--- arch/x86/events/perf_event.h | 208 ++++++++++- arch/x86/include/asm/fpu/sched.h | 5 +- arch/x86/include/asm/fpu/xstate.h | 3 + arch/x86/include/asm/msr-index.h | 7 + arch/x86/include/asm/perf_event.h | 35 +- arch/x86/include/uapi/asm/perf_regs.h | 53 +++ arch/x86/kernel/fpu/core.c | 33 +- arch/x86/kernel/fpu/xstate.c | 25 +- arch/x86/kernel/perf_regs.c | 174 ++++++++-- arch/x86/xen/pmu.c | 5 +- include/linux/perf_event.h | 23 ++ include/linux/perf_regs.h | 36 +- include/uapi/linux/perf_event.h | 49 ++- kernel/events/core.c | 177 +++++++++- 26 files changed, 1490 insertions(+), 226 deletions(-)--
2.34.1