arm64: Defer the GMID_EL1 read to {init,update}_cpu_features()
Fuad Tabba
reviewerPATCH v2 1/1 arm64: Defer the GMID_EL1 read to {init,update}_cpu_features()
__cpuinfo_store_cpu() gates the GMID_EL1 read on the raw ID_AA64PFR1_EL1, so it reads the register on MTE hardware even when the kernel has disabled MTE (CONFIG_ARM64_MTE=n or arm64.nomte). KVM sets HCR_EL2.TID5 in that case, which traps the read to EL2 and injects an UNDEF:
Internal error: Oops - Undefined instruction: 0000000002000000 [#1] pc : __cpuinfo_store_cpu+0xf4/0x264 Call trace: __cpuinfo_store_cpu+0xf4/0x264 (P) secondary_start_kernel+0xc8/0x1d0 __secondary_switched+0xc0/0xc4 Kernel panic - not syncing: Attempted to kill the idle task!
Only pKVM is affected, and only for a CPU that is offlined and brought back online. Every CPU's first bring-up precedes KVM's initcall, and pKVM refuses CPU_ON for a CPU that was not online when KVM initialised, so hotplug is the only path to cpuinfo_store_cpu() with TID5 set. pKVM's PSCI CPU_ON relay sets the host HCR before the CPU enters EL1, whereas plain nVHE sets it at CPUHP_AP_KVM_ONLINE, after cpuinfo_store_cpu().
Defer the read to {init,update}_cpu_features() and gate it on the sanitised ID register, as MPAM already does. system_supports_mte() cannot serve as the gate, as update_cpu_features() also runs during initial SMP bring-up, before smp_cpus_done() calls setup_system_features(). The init path gains the CONFIG_ARM64_MTE test the update path already had, leaving SYS_GMID_EL1 uninitialised when MTE is compiled out, where its only other user (lib/mte.S) is not built.
---
Notes:
Changes since v1:
- Clarified that the trap only fires on a CPU that is offlined and brought back online, not a late first boot (Marc).
Tested on QEMU with -machine virt,mte=on, under pKVM.
Offline/online CPU1 with arm64.nomte on the host cmdline: unpatched panics in __cpuinfo_store_cpu(), patched does not.
arch/arm64/kernel/cpufeature.c | 19 ++++++++++++++++--- arch/arm64/kernel/cpuinfo.c | 8 +++++--- 2 files changed, 21 insertions(+), 6 deletions(-)
arch/arm64/kernel/cpufeature.c +16 -3
@@ -1176,6 +1176,17 @@ static bool detect_ftr_has_mpam(void)return id_aa64pfr0_mpam (pfr0) ||id_aa64pfr1_mpamfrac (pfr1); } +/* +* Mirrors system_supports_mte(), which cannot be used before the capabilities +* are finalised. KVM sets HCR_EL2.TID5 when it is false, trapping GMID_EL1. +*/ +static bool detect_has_mte (void ) +{ +u64 pfr1 =read_sanitised_ftr_reg (SYS_ID_AA64PFR1_EL1 ); + +return IS_ENABLED (CONFIG_ARM64_MTE ) &&id_aa64pfr1_mte (pfr1); +} +void __init init_cpu_features (struct cpuinfo_arm64 *info) {/* Before we start using the tables, make sure it is sorted */
@@ -1228,8 +1239,10 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info)init_cpu_ftr_reg (SYS_MPAMIDR_EL1 , info->reg_mpamidr); } -if (id_aa64pfr1_mte (info->reg_id_aa64pfr1)) +if (detect_has_mte ()) { + info->reg_gmid =read_cpuid (GMID_EL1 );init_cpu_ftr_reg (SYS_GMID_EL1 , info->reg_gmid); + } }static void update_cpu_ftr_reg (struct arm64_ftr_reg *reg,u64 new)
@@ -1490,8 +1503,8 @@ void update_cpu_features(int cpu,* they read/write depends on the GMID_EL1.BS field. Check that the * value is the same on all CPUs. */ -if (IS_ENABLED (CONFIG_ARM64_MTE ) && -id_aa64pfr1_mte (info->reg_id_aa64pfr1)) { +if (detect_has_mte ()) { + info->reg_gmid =read_cpuid (GMID_EL1 ); taint |=check_update_ftr_reg (SYS_GMID_EL1 , cpu, info->reg_gmid, boot->reg_gmid); }
arch/arm64/kernel/cpuinfo.c +5 -3
@@ -502,12 +502,14 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) info->reg_id_aa64smfr0 =read_cpuid (ID_AA64SMFR0_EL1 ); info->reg_id_aa64fpfr0 =read_cpuid (ID_AA64FPFR0_EL1 ); -if (id_aa64pfr1_mte (info->reg_id_aa64pfr1)) - info->reg_gmid =read_cpuid (GMID_EL1 ); -if (id_aa64pfr0_32bit_el0 (info->reg_id_aa64pfr0))__cpuinfo_store_cpu_32bit (&info->aarch32); +/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/ +/* * info->reg_mpamidr deferred to {init,update}_cpu_features because we * don't want to read it (and trigger a trap on buggy firmware) if
Signature · 1 line
2.39.5
Quoted 8 lines · Fuad Tabba
arch/arm64/kernel/cpuinfo.c +0
-0@@ -502,12 +502,14 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) info->reg_id_aa64smfr0 =read_cpuid (ID_AA64SMFR0_EL1 ); info->reg_id_aa64fpfr0 =read_cpuid (ID_AA64FPFR0_EL1 );
arch/arm64/kernel/cpuinfo.c static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · line 511
Quoted 10 lines
arch/arm64/kernel/cpuinfo.c +4
-3-if (id_aa64pfr1_mte (info->reg_id_aa64pfr1)) - info->reg_gmid =read_cpuid (GMID_EL1 ); -if (id_aa64pfr0_32bit_el0 (info->reg_id_aa64pfr0))__cpuinfo_store_cpu_32bit (&info->aarch32); +/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/
The only nit I have is that cpu_data[0] now won't have the GMID_EL1 set. It doesn't matter as we don't expose it via c_show() or sysfs, rather for consistency. We have this inconsistency already with mpamidr.
I guess we just need to call init_cpu_features() on the per-CPU data before the assignment to boot_cpu_data. Something like (untested):
arch/arm64/kernel/cpuinfo.c +1 -1
@@ -538,7 +538,7 @@ void __init cpuinfo_store_boot_cpu(void) {struct cpuinfo_arm64 *info = &per_cpu (cpu_data,0 );__cpuinfo_store_cpu (info); +init_cpu_features (info); boot_cpu_data = *info; -init_cpu_features (&boot_cpu_data); }
Either way:
Will Deacon
maintainerQuoted 34 lines · Fuad Tabba
__cpuinfo_store_cpu()gates theGMID_EL1read on the rawID_AA64PFR1_EL1, so it reads the register on MTE hardware even when the kernel has disabled MTE (CONFIG_ARM64_MTE=nor arm64.nomte). KVM sets HCR_EL2.TID5 in that case, which traps the read to EL2 and injects an UNDEF:Internal error: Oops - Undefined instruction: 0000000002000000 [#1] pc : __cpuinfo_store_cpu+0xf4/0x264 Call trace: __cpuinfo_store_cpu+0xf4/0x264 (P) secondary_start_kernel+0xc8/0x1d0 __secondary_switched+0xc0/0xc4 Kernel panic - not syncing: Attempted to kill the idle task!Only pKVM is affected, and only for a CPU that is offlined and brought back online. Every CPU's first bring-up precedes KVM's initcall, and pKVM refuses
CPU_ONfor a CPU that was not online when KVM initialised, so hotplug is the only path tocpuinfo_store_cpu()with TID5 set. pKVM's PSCICPU_ONrelay sets the host HCR before the CPU enters EL1, whereas plain nVHE sets it atCPUHP_AP_KVM_ONLINE, aftercpuinfo_store_cpu().Defer the read to {init,update}_cpu_features() and gate it on the sanitised ID register, as MPAM already does.
system_supports_mte()cannot serve as the gate, asupdate_cpu_features()also runs during initial SMP bring-up, beforesmp_cpus_done()callssetup_system_features(). The init path gains theCONFIG_ARM64_MTEtest the update path already had, leavingSYS_GMID_EL1uninitialised when MTE is compiled out, where its only other user (lib/mte.S) is not built.Fixes f35abcbb8a084 ("KVM: arm64: Trap MTE access and discovery when MTE is disabled")Signed-off-by Fuad Tabba <[email protected]>---
[...]
Quoted 8 lines
arch/arm64/kernel/cpuinfo.c +0
-0@@ -502,12 +502,14 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) info->reg_id_aa64smfr0 =read_cpuid (ID_AA64SMFR0_EL1 ); info->reg_id_aa64fpfr0 =read_cpuid (ID_AA64FPFR0_EL1 );
arch/arm64/kernel/cpuinfo.c static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · line 511
Quoted 10 lines
arch/arm64/kernel/cpuinfo.c +4
-3-if (id_aa64pfr1_mte (info->reg_id_aa64pfr1)) - info->reg_gmid =read_cpuid (GMID_EL1 ); -if (id_aa64pfr0_32bit_el0 (info->reg_id_aa64pfr0))__cpuinfo_store_cpu_32bit (&info->aarch32); +/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/
I don't think we should defer this, as I've been actively doing the opposite for parallel CPU onlining (where the ID registers can be read concurrently by incoming CPUs to amortise the cost of a trap) and also for the RNG traps during early boot:
https://lore.kernel.org/all/annJ0oDB2HObQC5j@willie-the-truck/
If you look at the diff I sent in the thread above (I didn't get a reply), the idea is that __read_sysreg_by_encoding() reads from the ID register values stashed by cpuinfo_store_cpu(). So you could use that to check id_aa64pfr1 before reading gmid, as it will give you the sanitised view.
The only snag is that I didn't convert all the registers over, so there's some work there.
WDYT?
Will
Fuad Tabba
reviewerHi Will,
Quoted 10 lines · Will Deacon
arch/arm64/kernel/cpuinfo.c static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · line 511
Quoted 4 lines
arch/arm64/kernel/cpuinfo.c +4
-0+/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/ I don't think we should defer this, as I've been actively doing the opposite for parallel CPU onlining (where the ID registers can be read concurrently by incoming CPUs to amortise the cost of a trap) and also for the RNG traps during early boot:
Quoted 7 lines
https://lore.kernel.org/all/annJ0oDB2HObQC5j@willie-the-truck/
If you look at the diff I sent in the thread above (I didn't get a reply), the idea is that
__read_sysreg_by_encoding()reads from the ID register values stashed bycpuinfo_store_cpu(). So you could use that to checkid_aa64pfr1before reading gmid, as it will give you the sanitised view.
I just did, but I don't think it'll work. The thing is, __read_sysreg_by_encoding() gives the override-applied local value, not the folded sys_val.
arm64.nomte is a command-line override, so that works. But CONFIG_ARM64_MTE=n sets none. Any gate would still need an explicit IS_ENABLED(CONFIG_ARM64_MTE).
That said, I don't have to defer it. I can keep the info->reg_gmid read in __cpuinfo_store_cpu() and gate that read on the state that arms the trap:
static inline bool gmid_el1_accessible(u64 pfr1)
{
if (!IS_ENABLED(CONFIG_ARM64_MTE))
return false;
if (system_capabilities_finalized())
return system_supports_mte();
return id_aa64pfr1_mte(pfr1);
}TID5 is set from system_supports_mte() and only once capabilities are finalised, so the local check is still correct before that. Gating on system_supports_mte() keeps the different-physical-CPU case you want to preserve working: if one re-onlines with MTE while the system view has it folded off, the gate stays false and it won't read GMID_EL1.
If you agree, I'll respin with that.
Cheers,
/fuad
Quoted 7 lines
The only snag is that I didn't convert all the registers over, so there's some work there.
WDYT?
Will
Catalin Marinas
maintainerQuoted 22 lines · Fuad Tabba
Hi Will,
Quoted 10 lines · Will Deacon
arch/arm64/kernel/cpuinfo.c static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · line 511
Quoted 4 lines
arch/arm64/kernel/cpuinfo.c +4
-0+/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/ I don't think we should defer this, as I've been actively doing the opposite for parallel CPU onlining (where the ID registers can be read concurrently by incoming CPUs to amortise the cost of a trap) and also for the RNG traps during early boot:
Quoted 7 lines
https://lore.kernel.org/all/annJ0oDB2HObQC5j@willie-the-truck/
If you look at the diff I sent in the thread above (I didn't get a reply), the idea is that
__read_sysreg_by_encoding()reads from the ID register values stashed bycpuinfo_store_cpu(). So you could use that to checkid_aa64pfr1before reading gmid, as it will give you the sanitised view.
Quoted 7 lines
I just did, but I don't think it'll work. The thing is,
__read_sysreg_by_encoding()gives the override-applied local value, not the foldedsys_val.arm64.nomte is a command-line override, so that works. But
CONFIG_ARM64_MTE=nsets none. Any gate would still need an explicitIS_ENABLED(CONFIG_ARM64_MTE).
We need to revive this series:
https://lore.kernel.org/all/[email protected]/
I don't remember where we left it but in principle config off or override should look similar to the kernel.
There's also the override making an absent feature present. Suzuki has attempted to fix this in a reply:
https://lore.kernel.org/all/[email protected]/
Quoted 14 lines
That said, I don't have to defer it. I can keep the
info->reg_gmidread in__cpuinfo_store_cpu()and gate that read on the state that arms the trap:static inline bool gmid_el1_accessible(u64 pfr1) { if (!IS_ENABLED(CONFIG_ARM64_MTE)) return false; if (system_capabilities_finalized()) return system_supports_mte(); return id_aa64pfr1_mte(pfr1); }
Quoted 5 lines
TID5 is set from
system_supports_mte()and only once capabilities are finalised, so the local check is still correct before that. Gating onsystem_supports_mte()keeps the different-physical-CPU case you want to preserve working: if one re-onlines with MTE while the system view has it folded off, the gate stays false and it won't readGMID_EL1.
I think this would be the easiest to backport and we can look at reworking this infrastructure in the future (Will's proposal, Marc's override fixes etc.)
Signature · 1 line
Catalin
Will Deacon
maintainerQuoted 42 lines · Fuad Tabba
Quoted 10 lines · Will Deacon
arch/arm64/kernel/cpuinfo.c static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · line 511
Quoted 4 lines
arch/arm64/kernel/cpuinfo.c +4
-0+/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/ I don't think we should defer this, as I've been actively doing the opposite for parallel CPU onlining (where the ID registers can be read concurrently by incoming CPUs to amortise the cost of a trap) and also for the RNG traps during early boot:
Quoted 7 lines
https://lore.kernel.org/all/annJ0oDB2HObQC5j@willie-the-truck/
If you look at the diff I sent in the thread above (I didn't get a reply), the idea is that
__read_sysreg_by_encoding()reads from the ID register values stashed bycpuinfo_store_cpu(). So you could use that to checkid_aa64pfr1before reading gmid, as it will give you the sanitised view.I just did, but I don't think it'll work. The thing is,
__read_sysreg_by_encoding()gives the override-applied local value, not the foldedsys_val.arm64.nomte is a command-line override, so that works. But
CONFIG_ARM64_MTE=nsets none. Any gate would still need an explicitIS_ENABLED(CONFIG_ARM64_MTE).That said, I don't have to defer it. I can keep the
info->reg_gmidread in__cpuinfo_store_cpu()and gate that read on the state that arms the trap:static inline bool gmid_el1_accessible(u64 pfr1) { if (!IS_ENABLED(CONFIG_ARM64_MTE)) return false; if (system_capabilities_finalized()) return system_supports_mte(); return id_aa64pfr1_mte(pfr1); }
Quoted 5 lines
TID5 is set from
system_supports_mte()and only once capabilities are finalised, so the local check is still correct before that. Gating onsystem_supports_mte()keeps the different-physical-CPU case you want to preserve working: if one re-onlines with MTE while the system view has it folded off, the gate stays false and it won't readGMID_EL1.
I still don't understand why you have system_supports_mte() here. If the CPU has MTE, the CONFIG option is enabled and the cmdline override isn't set, we can read GMID just fine on this CPU, regardless of the system capabilities.
Quoted 1 line
If you agree, I'll respin with that.
I can't tell who's calling gmid_el1_accessible() so it's hard to tell whether I agree or not. May as well send a v3 though, so we can see what you have in mind!
Will
Fuad Tabba
reviewerQuoted 50 lines · Will Deacon
Quoted 42 lines · Fuad Tabba
Quoted 10 lines · Will Deacon
arch/arm64/kernel/cpuinfo.c static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · line 511
Quoted 4 lines
arch/arm64/kernel/cpuinfo.c +4
-0+/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/ I don't think we should defer this, as I've been actively doing the opposite for parallel CPU onlining (where the ID registers can be read concurrently by incoming CPUs to amortise the cost of a trap) and also for the RNG traps during early boot:
Quoted 7 lines
https://lore.kernel.org/all/annJ0oDB2HObQC5j@willie-the-truck/
If you look at the diff I sent in the thread above (I didn't get a reply), the idea is that
__read_sysreg_by_encoding()reads from the ID register values stashed bycpuinfo_store_cpu(). So you could use that to checkid_aa64pfr1before reading gmid, as it will give you the sanitised view.I just did, but I don't think it'll work. The thing is,
__read_sysreg_by_encoding()gives the override-applied local value, not the foldedsys_val.arm64.nomte is a command-line override, so that works. But
CONFIG_ARM64_MTE=nsets none. Any gate would still need an explicitIS_ENABLED(CONFIG_ARM64_MTE).That said, I don't have to defer it. I can keep the
info->reg_gmidread in__cpuinfo_store_cpu()and gate that read on the state that arms the trap:static inline bool gmid_el1_accessible(u64 pfr1) { if (!IS_ENABLED(CONFIG_ARM64_MTE)) return false; if (system_capabilities_finalized()) return system_supports_mte(); return id_aa64pfr1_mte(pfr1); }Quoted 5 lines
TID5 is set from
system_supports_mte()and only once capabilities are finalised, so the local check is still correct before that. Gating onsystem_supports_mte()keeps the different-physical-CPU case you want to preserve working: if one re-onlines with MTE while the system view has it folded off, the gate stays false and it won't readGMID_EL1.
Quoted 4 lines
I still don't understand why you have
system_supports_mte()here. If the CPU has MTE, the CONFIG option is enabled and the cmdline override isn't set, we can read GMID just fine on this CPU, regardless of the system capabilities.
I could gate on the local value with the override applied instead.
Quoted 5 lines
Quoted 1 line
If you agree, I'll respin with that.
I can't tell who's calling
gmid_el1_accessible()so it's hard to tell whether I agree or not. May as well send a v3 though, so we can see what you have in mind!
Sending it now. Probably shorter than just describing it.
Cheers,
/fuad
Quoted 1 line
Will
Catalin Marinas
maintainerQuoted 62 lines · Will Deacon
Quoted 34 lines · Fuad Tabba
__cpuinfo_store_cpu()gates theGMID_EL1read on the rawID_AA64PFR1_EL1, so it reads the register on MTE hardware even when the kernel has disabled MTE (CONFIG_ARM64_MTE=nor arm64.nomte). KVM sets HCR_EL2.TID5 in that case, which traps the read to EL2 and injects an UNDEF:Internal error: Oops - Undefined instruction: 0000000002000000 [#1] pc : __cpuinfo_store_cpu+0xf4/0x264 Call trace: __cpuinfo_store_cpu+0xf4/0x264 (P) secondary_start_kernel+0xc8/0x1d0 __secondary_switched+0xc0/0xc4 Kernel panic - not syncing: Attempted to kill the idle task!Only pKVM is affected, and only for a CPU that is offlined and brought back online. Every CPU's first bring-up precedes KVM's initcall, and pKVM refuses
CPU_ONfor a CPU that was not online when KVM initialised, so hotplug is the only path tocpuinfo_store_cpu()with TID5 set. pKVM's PSCICPU_ONrelay sets the host HCR before the CPU enters EL1, whereas plain nVHE sets it atCPUHP_AP_KVM_ONLINE, aftercpuinfo_store_cpu().Defer the read to {init,update}_cpu_features() and gate it on the sanitised ID register, as MPAM already does.
system_supports_mte()cannot serve as the gate, asupdate_cpu_features()also runs during initial SMP bring-up, beforesmp_cpus_done()callssetup_system_features(). The init path gains theCONFIG_ARM64_MTEtest the update path already had, leavingSYS_GMID_EL1uninitialised when MTE is compiled out, where its only other user (lib/mte.S) is not built.Fixes f35abcbb8a084 ("KVM: arm64: Trap MTE access and discovery when MTE is disabled")Signed-off-by Fuad Tabba <[email protected]>---
[...]
Quoted 8 lines
arch/arm64/kernel/cpuinfo.c +0
-0@@ -502,12 +502,14 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) info->reg_id_aa64smfr0 =read_cpuid (ID_AA64SMFR0_EL1 ); info->reg_id_aa64fpfr0 =read_cpuid (ID_AA64FPFR0_EL1 );arch/arm64/kernel/cpuinfo.c static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · line 511
Quoted 10 lines
arch/arm64/kernel/cpuinfo.c +4
-3-if (id_aa64pfr1_mte (info->reg_id_aa64pfr1)) - info->reg_gmid =read_cpuid (GMID_EL1 ); -if (id_aa64pfr0_32bit_el0 (info->reg_id_aa64pfr0))__cpuinfo_store_cpu_32bit (&info->aarch32); +/* +* info->reg_gmid deferred to {init,update}_cpu_features because +* reading it traps to EL2 when MTE is disabled. +*/ I don't think we should defer this, as I've been actively doing the opposite for parallel CPU onlining (where the ID registers can be read concurrently by incoming CPUs to amortise the cost of a trap) and also for the RNG traps during early boot:
Quoted 7 lines
https://lore.kernel.org/all/annJ0oDB2HObQC5j@willie-the-truck/
If you look at the diff I sent in the thread above (I didn't get a reply), the idea is that
__read_sysreg_by_encoding()reads from the ID register values stashed bycpuinfo_store_cpu(). So you could use that to checkid_aa64pfr1before reading gmid, as it will give you the sanitised view.
Since cpu_data[] has the raw per-cpu regs, I think it makes a lot of sense to read the cached values in __read_sysreg_by_encoding() than issuing the MRS again.
Given Marc's earlier series to make config=n and override behave similarly, I wonder whether we should just store the overridden value in cpu_data (with the clamp Suzuki was proposing to avoid making up features on the command line). We can then get rid of the override applied in various places and it might be easier to take the config into account as well.
Signature · 1 line
Catalin