perf/amd/uncore: Set all slices and threads to restore perf stat -a behaviour
Subsystems (3)
PATCH RESEND 1/7 perf/amd/uncore: Set all slices and threads to restore perf stat -a behaviour
Commit 2f217d58a8a0 ("perf/x86/amd/uncore: Set the thread mask for F17h L3 PMCs") inadvertently changed the uncore driver's behaviour wrt perf tool invocations with or without a CPU list, specified with -C / --cpu=.
Change the behaviour of the driver to assume the former all-cpu (-a) case, which is the more commonly desired default. This fixes '-a -A' invocations without explicit cpu lists (-C) to not count L3 events only on behalf of the first thread of the first core in the L3 domain.
BEFORE:
Activity performed by the first thread of the last core (CPU#43) in CPU#40's L3 domain is not reported by CPU#40:
sudo perf stat -a -A -e l3_request_g1.caching_l3_cache_accesses taskset -c 43 perf bench mem memcpy -s 32mb -l 100 -f default
...
CPU36 21,835 l3_request_g1.caching_l3_cache_accesses CPU40 87,066 l3_request_g1.caching_l3_cache_accesses CPU44 17,360 l3_request_g1.caching_l3_cache_accesses
...
AFTER:
The L3 domain activity is now reported by CPU#40:
sudo perf stat -a -A -e l3_request_g1.caching_l3_cache_accesses taskset -c 43 perf bench mem memcpy -s 32mb -l 100 -f default
...
CPU36 354,891 l3_request_g1.caching_l3_cache_accesses CPU40 1,780,870 l3_request_g1.caching_l3_cache_accesses CPU44 315,062 l3_request_g1.caching_l3_cache_accesses
...
---
Original submission:
https://lore.kernel.org/lkml/[email protected]/
arch/x86/events/amd/uncore.c | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-)
arch/x86/events/amd/uncore.c +8 -20
@@ -181,28 +181,16 @@ static void amd_uncore_del(struct perf_event *event, int flags) }/* -* Convert logical CPU number to L3 PMC Config ThreadMask format +* Return a full thread and slice mask until per-CPU is +* properly supported. */ -static u64 l3_thread_slice_mask (int cpu) +static u64 l3_thread_slice_mask (void ) { -u64 thread_mask, core =topology_core_id (cpu); -unsigned int shift, thread =0 ; +if (boot_cpu_data.x86 <=0x18 ) +return AMD64_L3_SLICE_MASK |AMD64_L3_THREAD_MASK ; -if (topology_smt_supported () && !topology_is_primary_thread (cpu)) - thread =1 ; - -if (boot_cpu_data.x86 <=0x18 ) { - shift =AMD64_L3_THREAD_SHIFT +2 * (core %4 ) + thread; - thread_mask =BIT_ULL (shift); - -return AMD64_L3_SLICE_MASK | thread_mask; - } - - core = (core <<AMD64_L3_COREID_SHIFT ) &AMD64_L3_COREID_MASK ; - shift =AMD64_L3_THREAD_SHIFT + thread; - thread_mask =BIT_ULL (shift); - -return AMD64_L3_EN_ALL_SLICES | core | thread_mask; +return AMD64_L3_EN_ALL_SLICES |AMD64_L3_EN_ALL_CORES | +AMD64_L3_F19H_THREAD_MASK ; }static int amd_uncore_event_init (struct perf_event *event)
@@ -232,7 +220,7 @@ static int amd_uncore_event_init(struct perf_event *event)* For other events, the two fields do not affect the count. */ if (l3_mask &&is_llc_event (event)) - hwc->config |=l3_thread_slice_mask (event->cpu); + hwc->config |=l3_thread_slice_mask (); uncore =event_to_amd_uncore (event);if (!uncore)
Signature · 1 line
2.27.0
PATCH 2/7 perf/x86/amd: Fix sampling Large Increment per Cycle events
Commit 5738891229a2 ("perf/x86/amd: Add support for Large Increment per Cycle Events") mistakenly zeroes the upper 16 bits of the count in set_period(). That's fine for counting with perf stat, but not sampling with perf record when only Large Increment events are being sampled. To enable sampling, we sign extend the upper 16 bits of the merged counter pair as described in the Family 17h PPRs:
"Software wanting to preload a value to a merged counter pair writes the high-order 16-bit value to the low-order 16 bits of the odd counter and then writes the low-order 48-bit value to the even counter. Reading the even counter of the merged counter pair returns the full 64-bit value."
--- arch/x86/events/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/events/core.c +2 -2
@@ -1284,11 +1284,11 @@ int x86_perf_event_set_period(struct perf_event *event)wrmsrl (hwc->event_base, (u64)(-left) & x86_pmu.cntval_mask);/* -* Clear the Merge event counter's upper 16 bits since +* Sign extend the Merge event counter's upper 16 bits since * we currently declare a 48-bit counter width */ if (is_counter_pair (hwc)) -wrmsrl (x86_pmu_event_addr (idx +1 ),0 ); +wrmsrl (x86_pmu_event_addr (idx +1 ),0xffff );/* * Due to erratum on certan cpu we need
Signature · 1 line
2.27.0
PATCH 3/7 arch/x86/amd/ibs: Fix re-arming IBS Fetch
In reply to Kim PhillipsStephane Eranian found a bug in that IBS' current Fetch counter was not being reset when the driver would write the new value to clear it along with the enable bit set, and found that adding an MSR write that would first disable IBS Fetch would make IBS Fetch reset its current count.
Indeed, the PPR for AMD Family 17h Model 31h B0 55803 Rev 0.54 - Sep 12, 2019 states "The periodic fetch counter is set to IbsFetchCnt [...] when IbsFetchEn is changed from 0 to 1."
Explicitly set IbsFetchEn to 0 and then to 1 when re-enabling IBS Fetch, so the driver properly resets the internal counter to 0 and IBS Fetch starts counting again.
It is not clear what versions of IBS hardware need IbsFetchEn explicitly zeroed and which historically may not have, so now make the driver always do it.
--- arch/x86/events/amd/ibs.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
arch/x86/events/amd/ibs.c +8 -1
@@ -363,7 +363,14 @@ perf_ibs_event_update(struct perf_ibs *perf_ibs, struct perf_event *event,static inline void perf_ibs_enable_event (struct perf_ibs *perf_ibs,struct hw_perf_event *hwc,u64 config) { -wrmsrl (hwc->config_base, hwc->config | config | perf_ibs->enable_mask); +u64 _config = (hwc->config | config) & ~perf_ibs->enable_mask; + +/* The periodic fetch counter is set when IbsFetchEn is changed from 0 to 1 */ +if (perf_ibs == &perf_ibs_fetch) +wrmsrl (hwc->config_base, _config); + + _config |= perf_ibs->enable_mask; +wrmsrl (hwc->config_base, _config); }/*
Signature · 1 line
2.27.0
PATCH 4/7 perf/x86/amd/ibs: Don't include randomized bits in get_ibs_op_count()
In reply to Kim Phillipsget_ibs_op_count() adds hardware's current count (IbsOpCurCnt) bits to its count regardless of hardware's valid status.
According to the PPR for AMD Family 17h Model 31h B0 55803 Rev 0.54, if the counter rolls over, valid status is set, and the lower 7 bits of IbsOpCurCnt are randomized by hardware.
Don't include those bits in the driver's event count.
--- arch/x86/events/amd/ibs.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
arch/x86/events/amd/ibs.c +8 -4
@@ -334,11 +334,15 @@ static u64 get_ibs_op_count(u64 config) {u64 count =0 ; +/* +* If the internal 27-bit counter rolled over, the count is MaxCnt +* and the lower 7 bits of CurCnt are randomized. +* Otherwise CurCnt has the full 27-bit current counter value. +*/ if (config &IBS_OP_VAL ) - count += (config &IBS_OP_MAX_CNT ) <<4 ;/* cnt rolled over */ - -if (ibs_caps &IBS_CAPS_RDWROPCNT ) - count += (config &IBS_OP_CUR_CNT ) >>32 ; + count = (config &IBS_OP_MAX_CNT ) <<4 ; +else if (ibs_caps &IBS_CAPS_RDWROPCNT ) + count = (config &IBS_OP_CUR_CNT ) >>32 ;return count; }
Signature · 1 line
2.27.0
PATCH 5/7 perf/x86/amd/ibs: Fix raw sample data accumulation
In reply to Kim PhillipsNeither IbsBrTarget nor OPDATA4 are populated in IBS Fetch mode. Don't accumulate them into raw sample user data in that case.
Also, in Fetch mode, add saving the IBS Fetch Control Extended MSR.
Technically, there is an ABI change here with respect to the IBS raw sample data format. I don't see any perf driver version information being included in perf.data file headers, but, existing users can detect whether the size of the sample record has reduced by 8 bytes to determine whether the IBS driver has this fix.
--- arch/x86/events/amd/ibs.c | 26 ++++++++++++++++---------- arch/x86/include/asm/msr-index.h | 1 + 2 files changed, 17 insertions(+), 10 deletions(-)
arch/x86/events/amd/ibs.c +16 -10
@@ -637,18 +637,24 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs) perf_ibs->offset_max, offset + 1); }while (offset < offset_max); +/* +* Read IbsBrTarget, IbsOpData4, and IbsExtdCtl separately +* depending on their availability. +* Can't add to offset_max as they are staggered +*/ if (event->attr.sample_type &PERF_SAMPLE_RAW ) { -/* -* Read IbsBrTarget and IbsOpData4 separately -* depending on their availability. -* Can't add to offset_max as they are staggered -*/ -if (ibs_caps &IBS_CAPS_BRNTRGT ) { -rdmsrl (MSR_AMD64_IBSBRTARGET , *buf++); - size++; +if (perf_ibs == &perf_ibs_op) { +if (ibs_caps &IBS_CAPS_BRNTRGT ) { +rdmsrl (MSR_AMD64_IBSBRTARGET , *buf++); + size++; + } +if (ibs_caps &IBS_CAPS_OPDATA4 ) { +rdmsrl (MSR_AMD64_IBSOPDATA4 , *buf++); + size++; + } } -if (ibs_caps &IBS_CAPS_OPDATA4 ) { -rdmsrl (MSR_AMD64_IBSOPDATA4 , *buf++); +if (perf_ibs == &perf_ibs_fetch && (ibs_caps &IBS_CAPS_FETCHCTLEXTD )) { +rdmsrl (MSR_AMD64_ICIBSEXTDCTL , *buf++); size++; } }
arch/x86/include/asm/msr-index.h +1 -0
@@ -464,6 +464,7 @@#define MSR_AMD64_IBSOP_REG_MASK ((1UL<<MSR_AMD64_IBSOP_REG_COUNT)-1)#define MSR_AMD64_IBSCTL 0xc001103a#define MSR_AMD64_IBSBRTARGET 0xc001103b +#define MSR_AMD64_ICIBSEXTDCTL 0xc001103c#define MSR_AMD64_IBSOPDATA4 0xc001103d#define MSR_AMD64_IBS_REG_COUNT_MAX 8/* includes MSR_AMD64_IBSBRTARGET */ #define MSR_AMD64_SEV 0xc0010131
Signature · 1 line
2.27.0
PATCH 6/7 perf/x86/amd/ibs: Support 27-bit extended Op/cycle counter
In reply to Kim PhillipsIBS hardware with the OpCntExt feature gets a 7-bit wider internal counter. Both the maximum and current count bitfields in the IBS_OP_CTL register are extended to support reading and writing it.
No changes are necessary to the driver for handling the extra contiguous current count bits (IbsOpCurCnt), as the driver already passes through 32 bits of that field. However, the driver has to do some extra bit manipulation when converting from a period to the non-contiguous (although conveniently aligned) extra bits in the IbsOpMaxCnt bitfield.
This decreases IBS Op interrupt overhead when the period is over 1,048,560 (0xffff0), which would previously activate the driver's software counter. That threshold is now 134,217,712 (0x7fffff0).
--- arch/x86/events/amd/ibs.c | 42 +++++++++++++++++++++++-------- arch/x86/include/asm/perf_event.h | 1 + 2 files changed, 32 insertions(+), 11 deletions(-)
arch/x86/events/amd/ibs.c +31 -11
@@ -339,10 +339,13 @@ static u64 get_ibs_op_count(u64 config)* and the lower 7 bits of CurCnt are randomized. * Otherwise CurCnt has the full 27-bit current counter value. */ -if (config &IBS_OP_VAL ) +if (config &IBS_OP_VAL ) { count = (config &IBS_OP_MAX_CNT ) <<4 ; -else if (ibs_caps &IBS_CAPS_RDWROPCNT ) +if (ibs_caps &IBS_CAPS_OPCNTEXT ) + count += config &IBS_OP_MAX_CNT_EXT_MASK ; + }else if (ibs_caps &IBS_CAPS_RDWROPCNT ) { count = (config &IBS_OP_CUR_CNT ) >>32 ; + }return count; }
@@ -405,7 +408,7 @@ static void perf_ibs_start(struct perf_event *event, int flags)struct hw_perf_event *hwc = &event->hw; struct perf_ibs *perf_ibs = container_of(event->pmu, struct perf_ibs, pmu);struct cpu_perf_ibs *pcpu =this_cpu_ptr (perf_ibs->pcpu); -u64 period; +u64 period, config =0 ;if (WARN_ON_ONCE (!(hwc->state &PERF_HES_STOPPED )))return ;
@@ -414,13 +417,19 @@ static void perf_ibs_start(struct perf_event *event, int flags) hwc->state =0 ;perf_ibs_set_period (perf_ibs, hwc, &period); +if (perf_ibs == &perf_ibs_op && (ibs_caps &IBS_CAPS_OPCNTEXT )) { + config |= period &IBS_OP_MAX_CNT_EXT_MASK ; + period &= ~IBS_OP_MAX_CNT_EXT_MASK ; + } + config |= period >>4 ; +/* * Set STARTED before enabling the hardware, such that a subsequent NMI * must observe it. */ set_bit (IBS_STARTED , pcpu->state);clear_bit (IBS_STOPPING , pcpu->state); -perf_ibs_enable_event (perf_ibs, hwc, period >>4 ); +perf_ibs_enable_event (perf_ibs, hwc, config);perf_event_update_userpage (event); }
@@ -588,7 +597,7 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)struct perf_ibs_data ibs_data;int offset, size, check_rip, offset_max, throttle =0 ;unsigned int msr; -u64 *buf, *config, period; +u64 *buf, *config, period, new_config =0 ;if (!test_bit (IBS_STARTED , pcpu->state)) { fail:
@@ -683,13 +692,17 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)if (throttle) {perf_ibs_stop (event,0 ); }else { - period >>=4 ; - -if ((ibs_caps &IBS_CAPS_RDWROPCNT ) && - (*config &IBS_OP_CNT_CTL )) - period |= *config &IBS_OP_CUR_CNT_RAND ; +if (perf_ibs == &perf_ibs_op) { +if (ibs_caps &IBS_CAPS_OPCNTEXT ) { + new_config = period &IBS_OP_MAX_CNT_EXT_MASK ; + period &= ~IBS_OP_MAX_CNT_EXT_MASK ; + } +if ((ibs_caps &IBS_CAPS_RDWROPCNT ) && (*config &IBS_OP_CNT_CTL )) + new_config |= *config &IBS_OP_CUR_CNT_RAND ; + } + new_config |= period >>4 ; -perf_ibs_enable_event (perf_ibs, hwc, period); +perf_ibs_enable_event (perf_ibs, hwc, new_config); }perf_event_update_userpage (event);
@@ -756,6 +769,13 @@ static __init void perf_event_ibs_init(void) perf_ibs_op.config_mask |=IBS_OP_CNT_CTL ; *attr++ = &format_attr_cnt_ctl.attr; } + +if (ibs_caps &IBS_CAPS_OPCNTEXT ) { + perf_ibs_op.max_period |=IBS_OP_MAX_CNT_EXT_MASK ; + perf_ibs_op.config_mask |=IBS_OP_MAX_CNT_EXT_MASK ; + perf_ibs_op.cnt_mask |=IBS_OP_MAX_CNT_EXT_MASK ; + } +perf_ibs_pmu_init (&perf_ibs_op,"ibs_op" );register_nmi_handler (NMI_LOCAL , perf_ibs_nmi_handler,0 ,"perf_ibs" );
arch/x86/include/asm/perf_event.h +1 -0
@@ -334,6 +334,7 @@ struct pebs_xmm {#define IBS_OP_ENABLE (1ULL<<17)#define IBS_OP_MAX_CNT 0x0000FFFFULL#define IBS_OP_MAX_CNT_EXT 0x007FFFFFULL/* not a register bit mask */ +#define IBS_OP_MAX_CNT_EXT_MASK (0x7FULL<<20)/* separate upper 7 bits */ #define IBS_RIP_INVALID (1ULL<<38)#ifdef CONFIG_X86_LOCAL_APIC
Signature · 1 line
2.27.0
PATCH 7/7 perf/x86/rapl: Add AMD Fam19h RAPL support
In reply to Kim PhillipsFamily 19h RAPL support did not change from Family 17h; extend the existing Fam17h support to work on Family 19h too.
--- arch/x86/events/rapl.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/events/rapl.c +1 -0
@@ -815,6 +815,7 @@ static const struct x86_cpu_id rapl_model_match[] __initconst = {X86_MATCH_INTEL_FAM6_MODEL (SAPPHIRERAPIDS_X , &model_spr),X86_MATCH_VENDOR_FAM (AMD ,0x17 , &model_amd_fam17h),X86_MATCH_VENDOR_FAM (HYGON ,0x18 , &model_amd_fam17h), +X86_MATCH_VENDOR_FAM (AMD ,0x19 , &model_amd_fam17h), {}, };MODULE_DEVICE_TABLE (x86cpu, rapl_model_match);
Signature · 1 line
2.27.0
Sasha Levin
maintainerHi
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag
fixing commit: 2f217d58a8a0 ("perf/x86/amd/uncore: Set the thread mask for F17h L3 PMCs").
The bot has tested the following trees: v5.8.2, v5.7.16, v5.4.59, v4.19.140, v4.14.193.
v5.8.2: Build OK!
v5.7.16: Build OK!
v5.4.59: Failed to apply! Possible dependencies:
4dcc3df82573 ("perf/amd/uncore: Prepare L3 thread mask code for Family 19h") 9689dbbeaea8 ("perf/amd/uncore: Make L3 thread mask code more readable") e48667b86548 ("perf/amd/uncore: Add support for Family 19h L3 PMU")
v4.19.140: Failed to apply! Possible dependencies:
4dcc3df82573 ("perf/amd/uncore: Prepare L3 thread mask code for Family 19h") 6d0ef316b9f8 ("x86/events: Add Hygon Dhyana support to PMU infrastructure") 9689dbbeaea8 ("perf/amd/uncore: Make L3 thread mask code more readable") e48667b86548 ("perf/amd/uncore: Add support for Family 19h L3 PMU")
v4.14.193: Failed to apply! Possible dependencies:
4dcc3df82573 ("perf/amd/uncore: Prepare L3 thread mask code for Family 19h") 6d0ef316b9f8 ("x86/events: Add Hygon Dhyana support to PMU infrastructure") 9689dbbeaea8 ("perf/amd/uncore: Make L3 thread mask code more readable") e48667b86548 ("perf/amd/uncore: Add support for Family 19h L3 PMU")
NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
Signature · 2 lines
Thanks Sasha