arm64: Cache HW update of Access Flag support
RFC PATCH 1/1 arm64: Cache HW update of Access Flag support
Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].
Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.
This patch avoids the extra cost by caching the register value after the first read. It should be safe to do so, since this register mustn't change for a specific processor.
As a side note, we can't rely on the usual cpucap for this bit, because for some reason that I'm missing, it is always enabled for aarch64, even if the functionality is not supported by the processor.
--- arch/arm64/include/asm/cpufeature.h | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
arch/arm64/include/asm/cpufeature.h +13 -3
@@ -859,14 +859,24 @@ static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange)/* Check whether hardware update of the Access flag is supported */ static inline bool cpu_has_hw_af (void ) { -u64 mmfr1; +static unsigned int has_af =-1U ;if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false; - mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); -return cpuid_feature_extract_unsigned_field (mmfr1, +/* +* Cache the register to avoid repeatedly reading it, +* which is an emulated operation on KVM guests. +* +* Also do not rely on cpucap, since this bit is always set +* there, regardless of actual status. See has_hw_dbm() for +* details. +*/ +if (unlikely (has_af ==-1U )) + has_af =cpuid_feature_extract_unsigned_field ( +read_cpuid (ID_AA64MMFR1_EL1 ),ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); +return has_af; }static inline bool cpu_has_pan (void )
Signature · 7 lines
2.35.3 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Will Deacon
maintainerQuoted 49 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.This patch avoids the extra cost by caching the register value after the first read. It should be safe to do so, since this register mustn't change for a specific processor.
As a side note, we can't rely on the usual cpucap for this bit, because for some reason that I'm missing, it is always enabled for aarch64, even if the functionality is not supported by the processor.
Signed-off-by Gabriel Krisman Bertazi <[email protected]>--- arch/arm64/include/asm/cpufeature.h | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)arch/arm64/include/asm/cpufeature.h +4
-3@@ -859,14 +859,24 @@ static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange)/* Check whether hardware update of the Access flag is supported */ static inline bool cpu_has_hw_af (void ) { -u64 mmfr1; +static unsigned int has_af =-1U ;if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false; - mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); - return cpuid_feature_extract_unsigned_field(mmfr1, +/* +* Cache the register to avoid repeatedly reading it, +* which is an emulated operation on KVM guests.
arch/arm64/include/asm/cpufeature.h static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange) · line 879
Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +9
-0+* +* Also do not rely on cpucap, since this bit is always set +* there, regardless of actual status. See has_hw_dbm() for +* details. +*/ +if (unlikely (has_af ==-1U )) + has_af =cpuid_feature_extract_unsigned_field ( +read_cpuid (ID_AA64MMFR1_EL1 ),ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); +return has_af;
The intention here was to read the value for the _current_ CPU, since it might not be the same across the system when big.MISTAKE gets involved.
However, since this is really just a performance optimisation and I think that the access flag tends to be uniformly supported in practice anyway, the best bet is probably just to read the sanitised version of the field using read_sanitised_ftr_reg().
Can you give that a shot, please?
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 9 lines · Will Deacon
arch/arm64/include/asm/cpufeature.h static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange) · line 879
Quoted 4 lines · Gabriel Krisman Bertazi
arch/arm64/include/asm/cpufeature.h +3
-0+ has_af =cpuid_feature_extract_unsigned_field ( +read_cpuid (ID_AA64MMFR1_EL1 ),ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); +return has_af;The intention here was to read the value for the _current_ CPU, since it might not be the same across the system when big.MISTAKE gets involved.
Quoted 6 lines
However, since this is really just a performance optimisation and I think that the access flag tends to be uniformly supported in practice anyway, the best bet is probably just to read the sanitised version of the field using
read_sanitised_ftr_reg().Can you give that a shot, please?
Hey Will,
Thanks for the review.
I reran the benchmark over the weekend and the impact of the bsearch lookup seems to be <1% worse than my original patch, which is negligible, IMO. I will shortly follow up with a v2 applying your suggestion.
Thanks!
Signature · 6 lines
Gabriel Krisman Bertazi _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Will Deacon
maintainerQuoted 18 lines · Gabriel Krisman Bertazi
Quoted 9 lines · Will Deacon
Review
Quoted 4 lines · Gabriel Krisman Bertazi
(quoted diff) +3
-0+ has_af = cpuid_feature_extract_unsigned_field( + read_cpuid(ID_AA64MMFR1_EL1), ID_AA64MMFR1_EL1_HAFDBS_SHIFT); + return has_af;The intention here was to read the value for the _current_ CPU, since it might not be the same across the system when big.MISTAKE gets involved.
Quoted 6 lines
However, since this is really just a performance optimisation and I think that the access flag tends to be uniformly supported in practice anyway, the best bet is probably just to read the sanitised version of the field using
read_sanitised_ftr_reg().Can you give that a shot, please?
Quoted 8 lines
Hey Will,
Thanks for the review.
I reran the benchmark over the weekend and the impact of the bsearch lookup seems to be <1% worse than my original patch, which is negligible, IMO. I will shortly follow up with a v2 applying your suggestion.
FWIW, I had a hack kicking around to avoid the bsearch. It wasn't too fiddly to rebase, so I've included it below if you want to take it for a spin. The grotty part is having to maintain the extra enum, which is why I didn't merge it in the past.
Will
--->8
arch/arm64/include/asm/cpufeature.h +46 -0
@@ -636,7 +636,53 @@ static inline bool id_aa64pfr1_mte(u64 pfr1)void __init setup_cpu_features (void );void check_local_cpu_capabilities (void ); +#define ARM64_FTR_REG2IDX (id) id ## _IDX +enum arm64_ftr_reg_idx { + ARM64_FTR_REG2IDX(SYS_ID_PFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_PFR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_DFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_MMFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_MMFR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_MMFR2_EL1), + ARM64_FTR_REG2IDX(SYS_ID_MMFR3_EL1), + ARM64_FTR_REG2IDX(SYS_ID_ISAR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_ISAR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_ISAR2_EL1), + ARM64_FTR_REG2IDX(SYS_ID_ISAR3_EL1), + ARM64_FTR_REG2IDX(SYS_ID_ISAR4_EL1), + ARM64_FTR_REG2IDX(SYS_ID_ISAR5_EL1), + ARM64_FTR_REG2IDX(SYS_ID_MMFR4_EL1), + ARM64_FTR_REG2IDX(SYS_ID_ISAR6_EL1), + ARM64_FTR_REG2IDX(SYS_MVFR0_EL1), + ARM64_FTR_REG2IDX(SYS_MVFR1_EL1), + ARM64_FTR_REG2IDX(SYS_MVFR2_EL1), + ARM64_FTR_REG2IDX(SYS_ID_PFR2_EL1), + ARM64_FTR_REG2IDX(SYS_ID_DFR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_MMFR5_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64PFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64PFR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64ZFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64SMFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64DFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64DFR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64ISAR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64ISAR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64ISAR2_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64MMFR0_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64MMFR1_EL1), + ARM64_FTR_REG2IDX(SYS_ID_AA64MMFR2_EL1), + ARM64_FTR_REG2IDX(SYS_ZCR_EL1), + ARM64_FTR_REG2IDX(SYS_SMCR_EL1), + ARM64_FTR_REG2IDX(SYS_GMID_EL1), + ARM64_FTR_REG2IDX(SYS_CTR_EL0), + ARM64_FTR_REG2IDX(SYS_DCZID_EL0), + ARM64_FTR_REG2IDX(SYS_CNTFRQ_EL0), + +ARM64_FTR_REG_IDX_MAX , +}; +u64 read_sanitised_ftr_reg (u32 id); +u64 read_sanitised_ftr_reg_by_idx (enum arm64_ftr_reg_idx idx);u64 __read_sysreg_by_encoding (u32 sys_id);static inline bool cpu_supports_mixed_endian_el0 (void )
arch/arm64/kernel/cpufeature.c +25 -11
@@ -624,19 +624,21 @@ static const struct arm64_ftr_bits ftr_raz[] = { ARM64_FTR_END, }; -#define __ARM64_FTR_REG_OVERRIDE (id_str, id, table, ovr) { \ - .sys_id = id, \ - .reg = &(struct arm64_ftr_reg){ \ - .name = id_str, \ - .override = (ovr), \ - .ftr_bits = &((table)[0]), \ - }} +#define __ARM64_FTR_REG_OVERRIDE (id_str, id_idx, id, table, ovr) \ + [id_idx] = { \ + .sys_id = id, \ + .reg = &(struct arm64_ftr_reg) { \ + .name = id_str, \ + .override = (ovr), \ + .ftr_bits = &((table)[0]), \ + } \ + } \#define ARM64_FTR_REG_OVERRIDE (id, table, ovr) \ - __ARM64_FTR_REG_OVERRIDE(#id, id, table, ovr) + __ARM64_FTR_REG_OVERRIDE(#id, id ## _IDX, id, table, ovr)#define ARM64_FTR_REG (id, table) \ - __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) + __ARM64_FTR_REG_OVERRIDE(#id, id ## _IDX, id, table, &no_override)struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override;struct arm64_ftr_override __ro_after_init id_aa64pfr0_override;
@@ -648,7 +650,7 @@ struct arm64_ftr_override __ro_after_init id_aa64isar2_override;static const struct __ftr_reg_entry {u32 sys_id; -struct arm64_ftr_reg *reg; +struct arm64_ftr_reg *reg; } arm64_ftr_regs[] = {/* Op1 = 0, CRn = 0, CRm = 1 */
@@ -713,7 +715,7 @@ static const struct __ftr_reg_entry { ARM64_FTR_REG(SYS_GMID_EL1, ftr_gmid),/* Op1 = 3, CRn = 0, CRm = 0 */ - { SYS_CTR_EL0, &arm64_ftr_reg_ctrel0 }, + [ARM64_FTR_REG2IDX(SYS_CTR_EL0)] = { SYS_CTR_EL0, &arm64_ftr_reg_ctrel0 }, ARM64_FTR_REG(SYS_DCZID_EL0, ftr_dczid),/* Op1 = 3, CRn = 14, CRm = 0 */
@@ -1329,6 +1331,18 @@ u64 read_sanitised_ftr_reg(u32 id) }EXPORT_SYMBOL_GPL (read_sanitised_ftr_reg); +u64 read_sanitised_ftr_reg_by_idx (enum arm64_ftr_reg_idx idx) +{ +struct arm64_ftr_reg *regp; + +if (WARN_ON ((unsigned )idx >=ARM64_FTR_REG_IDX_MAX )) +return 0 ; + + regp = arm64_ftr_regs[idx].reg; +return regp->sys_val; +} +EXPORT_SYMBOL_GPL (read_sanitised_ftr_reg_by_idx); +#define read_sysreg_case (r) \ case r: val = read_sysreg_s(r); break;
Signature · 7 lines
2.39.0.314.g84b9a713c41-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 4 lines · Will Deacon
FWIW, I had a hack kicking around to avoid the bsearch. It wasn't too fiddly to rebase, so I've included it below if you want to take it for a spin. The grotty part is having to maintain the extra enum, which is why I didn't merge it in the past.
Hi Will,
I applied your patch, converted my fix to use it and re-ran the benchmark. For the record, the test is a mmtests workload that builds the kernel a few times with allmod config. If you want to reproduce it, I used the configuration below:
https://github.com/gormanm/mmtests/blob/master/configs/config-workload-kernbench-max
From my observation, the array patch doesn't seem to meaningfully impact this benchmark. Using my RFC as a baseline, and comparing read_sanitised_ftr_reg with bsearch and your patch (array), I got:
RFC bsearch array
Amean syst-20 2886.71 ( 0.00%) 2913.26 * -0.92%* 2911.01 * -0.84%*
I sent a v2 that uses read_sanitised_ftr_reg on another thread [1]. If you decide to merge your patch, I'm happy to re-spin it to use this new interface.
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
Thanks,
Signature · 6 lines
Gabriel Krisman Bertazi _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel