An alternative series for asymmetric AArch32 systems
Subsystems (4)
Will Deacon
maintainerv2 cover An alternative series for asymmetric AArch32 systems
Hello again,
This is version two of the ravingly popular series I previously posted here:
https://lore.kernel.org/r/[email protected]
Changes since v1 include:
* Fix setting of compat hwcaps
* Simplify sysfs code to use DEVICE_ATTR_RO()
* Allow onlining of late CPUs in face of mismatch
* Use a static key in the context-switch path
* Avoid printing that we detected 32-bit EL0 support when we didn't
This has unfortunately introduced more complexity than I would've liked, but it also seems to be free of any rough edges.
I haven't yet tackled the execve() problem raised by Suren. I've got some local hacks, but nothing I'm happy with yet. They will come as follow-up patches in any case.
As before, I don't think we should merge this stuff until we've figured out what's going on in Android, but hopefully we can reach some agreement on the basics before then.
Cheers,
Will
--->8
Will Deacon (6):
arm64: cpuinfo: Split AArch32 registers out into a separate struct
arm64: Allow mismatched 32-bit EL0 support
KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support
arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs
arm64: Advertise CPUs capable of running 32-bit applications in sysfs
arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0
.../ABI/testing/sysfs-devices-system-cpu | 9 + .../admin-guide/kernel-parameters.txt | 7 + arch/arm64/include/asm/cpu.h | 44 ++-- arch/arm64/include/asm/cpucaps.h | 2 +- arch/arm64/include/asm/cpufeature.h | 8 +- arch/arm64/kernel/cpufeature.c | 198 ++++++++++++++---- arch/arm64/kernel/cpuinfo.c | 53 ++--- arch/arm64/kernel/process.c | 19 +- arch/arm64/kernel/signal.c | 26 +++ arch/arm64/kvm/arm.c | 11 +- 10 files changed, 288 insertions(+), 89 deletions(-)
Signature · 1 line
2.29.2.222.g5d2a92d10f8-goog
Will Deacon
maintainerPATCH v2 1/6 arm64: cpuinfo: Split AArch32 registers out into a separate struct
In preparation for late initialisation of the "sanitised" AArch32 register state, move the AArch32 registers out of 'struct cpuinfo' and into their own struct definition.
--- arch/arm64/include/asm/cpu.h | 44 +++++++++++---------- arch/arm64/kernel/cpufeature.c | 71 ++++++++++++++++++---------------- arch/arm64/kernel/cpuinfo.c | 53 +++++++++++++------------ 3 files changed, 89 insertions(+), 79 deletions(-)
arch/arm64/include/asm/cpu.h +24 -20
@@ -12,26 +12,7 @@/* * Records attributes of an individual CPU. */ -struct cpuinfo_arm64 { -struct cpu cpu; -struct kobject kobj; -u32 reg_ctr; -u32 reg_cntfrq; -u32 reg_dczid; -u32 reg_midr; -u32 reg_revidr; - -u64 reg_id_aa64dfr0; -u64 reg_id_aa64dfr1; -u64 reg_id_aa64isar0; -u64 reg_id_aa64isar1; -u64 reg_id_aa64mmfr0; -u64 reg_id_aa64mmfr1; -u64 reg_id_aa64mmfr2; -u64 reg_id_aa64pfr0; -u64 reg_id_aa64pfr1; -u64 reg_id_aa64zfr0; - +struct cpuinfo_32bit {u32 reg_id_dfr0;u32 reg_id_dfr1;u32 reg_id_isar0;
@@ -54,6 +35,29 @@ struct cpuinfo_arm64 {u32 reg_mvfr0;u32 reg_mvfr1;u32 reg_mvfr2; +}; + +struct cpuinfo_arm64 { +struct cpu cpu; +struct kobject kobj; +u32 reg_ctr; +u32 reg_cntfrq; +u32 reg_dczid; +u32 reg_midr; +u32 reg_revidr; + +u64 reg_id_aa64dfr0; +u64 reg_id_aa64dfr1; +u64 reg_id_aa64isar0; +u64 reg_id_aa64isar1; +u64 reg_id_aa64mmfr0; +u64 reg_id_aa64mmfr1; +u64 reg_id_aa64mmfr2; +u64 reg_id_aa64pfr0; +u64 reg_id_aa64pfr1; +u64 reg_id_aa64zfr0; + +struct cpuinfo_32bit aarch32;/* pseudo-ZCR for recording maximum ZCR_EL1 LEN value: */ u64 reg_zcr;
arch/arm64/kernel/cpufeature.c +37 -34
102 lines not on this page · raw message
arch/arm64/kernel/cpuinfo.c +28 -25
65 lines not on this page · raw message
Signature · 1 line
2.29.2.222.g5d2a92d10f8-goog
Will Deacon
maintainerPATCH v2 2/6 arm64: Allow mismatched 32-bit EL0 support
In reply to Will DeaconWhen confronted with a mixture of CPUs, some of which support 32-bit applications and others which don't, we quite sensibly treat the system as 64-bit only for userspace and prevent execve() of 32-bit binaries.
Unfortunately, some crazy folks have decided to build systems like this with the intention of running 32-bit applications, so relax our sanitisation logic to continue to advertise 32-bit support to userspace on these systems and track the real 32-bit capable cores in a cpumask instead. For now, the default behaviour remains but will be tied to a command-line option in a later patch.
--- arch/arm64/include/asm/cpucaps.h | 2 +- arch/arm64/include/asm/cpufeature.h | 8 ++- arch/arm64/kernel/cpufeature.c | 103 ++++++++++++++++++++++++++-- 3 files changed, 104 insertions(+), 9 deletions(-)
arch/arm64/include/asm/cpucaps.h +1 -1
@@ -20,7 +20,7 @@#define ARM64_ALT_PAN_NOT_UAO 10#define ARM64_HAS_VIRT_HOST_EXTN 11#define ARM64_WORKAROUND_CAVIUM_27456 12 -#define ARM64_HAS_32BIT_EL0 13 +#define ARM64_HAS_32BIT_EL0_DO_NOT_USE 13#define ARM64_HARDEN_EL2_VECTORS 14#define ARM64_HAS_CNP 15#define ARM64_HAS_NO_FPSIMD 16
arch/arm64/include/asm/cpufeature.h +7 -1
@@ -604,9 +604,15 @@ static inline bool cpu_supports_mixed_endian_el0(void)return id_aa64mmfr0_mixed_endian_el0 (read_cpuid (ID_AA64MMFR0_EL1 )); } +const struct cpumask *system_32bit_el0_cpumask (void ); +DECLARE_STATIC_KEY_FALSE (arm64_mismatched_32bit_el0); +static inline bool system_supports_32bit_el0 (void ) { -return cpus_have_const_cap (ARM64_HAS_32BIT_EL0 ); +u64 pfr0 =read_sanitised_ftr_reg (SYS_ID_AA64PFR0_EL1 ); + +return id_aa64pfr0_32bit_el0 (pfr0) || +static_branch_unlikely (&arm64_mismatched_32bit_el0); }static inline bool system_supports_4kb_granule (void )
arch/arm64/kernel/cpufeature.c +96 -7
164 lines not on this page · raw message
Signature · 1 line
2.29.2.222.g5d2a92d10f8-goog
Will Deacon
maintainerPATCH v2 3/6 KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support
In reply to Will DeaconIf a vCPU tries to run 32-bit code on a system with mismatched support at EL0, then we should kill it.
--- arch/arm64/kvm/arm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +10 -1
@@ -633,6 +633,15 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu) } } +static bool vcpu_mode_is_bad_32bit (struct kvm_vcpu *vcpu) +{ +if (likely (!vcpu_mode_is_32bit (vcpu))) +return false; + +return !system_supports_32bit_el0 () || +static_branch_unlikely (&arm64_mismatched_32bit_el0); +} +/** * kvm_arch_vcpu_ioctl_run - the main VCPU run function to execute guest code * @vcpu: The VCPU pointer
@@ -816,7 +825,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)* with the asymmetric AArch32 case), return to userspace with * a fatal error. */ -if (!system_supports_32bit_el0 () &&vcpu_mode_is_32bit (vcpu)) { +if (vcpu_mode_is_bad_32bit (vcpu)) {/* * As we have caught the guest red-handed, decide that * it isn't fit for purpose anymore by making the vcpu
Signature · 1 line
2.29.2.222.g5d2a92d10f8-goog
Will Deacon
maintainerPATCH v2 4/6 arm64: Kill 32-bit applications scheduled on 64-bit-only CPUs
In reply to Will DeaconScheduling a 32-bit application on a 64-bit-only CPU is a bad idea.
Ensure that 32-bit applications always take the slow-path when returning to userspace on a system with mismatched support at EL0, so that we can avoid trying to run on a 64-bit-only CPU and force a SIGKILL instead.
--- arch/arm64/kernel/process.c | 19 ++++++++++++++++++- arch/arm64/kernel/signal.c | 26 ++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-)
arch/arm64/kernel/process.c +18 -1
@@ -542,6 +542,15 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,write_sysreg (val, cntkctl_el1); } +static void compat_thread_switch (struct task_struct *next) +{ +if (!is_compat_thread (task_thread_info (next))) +return ; + +if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) +set_tsk_thread_flag (next,TIF_NOTIFY_RESUME ); +} +/* * Thread switching. */
@@ -558,6 +567,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,uao_thread_switch (next);ssbs_thread_switch (next);erratum_1418040_thread_switch (prev, next); +compat_thread_switch (next);/* * Complete any pending TLB or cache maintenance on this CPU in case
@@ -620,8 +630,15 @@ unsigned long arch_align_stack(unsigned long sp)*/ void arch_setup_new_exec (void ) { - current->mm->context.flags =is_compat_task () ?MMCF_AARCH32 :0 ; +unsigned long mmflags =0 ; + +if (is_compat_task ()) { + mmflags =MMCF_AARCH32 ; +if (static_branch_unlikely (&arm64_mismatched_32bit_el0)) +set_tsk_thread_flag (current,TIF_NOTIFY_RESUME ); + } + current->mm->context.flags = mmflags;ptrauth_thread_init_user (current);if (task_spec_ssb_noexec (current)) {
arch/arm64/kernel/signal.c +26 -0
@@ -911,6 +911,19 @@ static void do_signal(struct pt_regs *regs)restore_saved_sigmask (); } +static bool cpu_affinity_invalid (struct pt_regs *regs) +{ +if (!compat_user_mode (regs)) +return false; + +/* +* We're preemptible, but a reschedule will cause us to check the +* affinity again. +*/ +return !cpumask_test_cpu (raw_smp_processor_id (), +system_32bit_el0_cpumask ()); +} + asmlinkage void do_notify_resume(struct pt_regs *regs,unsigned long thread_flags) {
@@ -948,6 +961,19 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,if (thread_flags & _TIF_NOTIFY_RESUME) {tracehook_notify_resume (regs);rseq_handle_notify_resume (NULL , regs); + +/* +* If we reschedule after checking the affinity +* then we must ensure that TIF_NOTIFY_RESUME +* is set so that we check the affinity again. +* Since tracehook_notify_resume() clears the +* flag, ensure that the compiler doesn't move +* it after the affinity check. +*/ +barrier (); + +if (cpu_affinity_invalid (regs)) +force_sig (SIGKILL ); }if (thread_flags & _TIF_FOREIGN_FPSTATE)
Signature · 1 line
2.29.2.222.g5d2a92d10f8-goog
Will Deacon
maintainerPATCH v2 5/6 arm64: Advertise CPUs capable of running 32-bit applications in sysfs
In reply to Will DeaconSince 32-bit applications will be killed if they are caught trying to execute on a 64-bit-only CPU in a mismatched system, advertise the set of 32-bit capable CPUs to userspace in sysfs.
--- .../ABI/testing/sysfs-devices-system-cpu | 9 +++++++++ arch/arm64/kernel/cpufeature.c | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +9 -0
@@ -493,6 +493,15 @@ Description: AArch64 CPU registers 'identification' directory exposes the CPU ID registers for identifying model and revision of the CPU. +What: /sys/devices/system/cpu/aarch32_el0 +Date: November 2020 +Contact: Linux ARM Kernel Mailing list <[email protected]> +Description: Identifies the subset of CPUs in the system that can execute + AArch32 (32-bit ARM) applications. If present, the same format as + /sys/devices/system/cpu/{offline,online,possible,present} is used. + If absent, then all or none of the CPUs can execute AArch32 + applications and execve() will behave accordingly. + What: /sys/devices/system/cpu/cpu#/cpu_capacity Date: December 2016 Contact: Linux kernel mailing list <[email protected]>
arch/arm64/kernel/cpufeature.c +19 -0
@@ -67,6 +67,7 @@#include <linux/crash_dump.h> #include <linux/sort.h> #include <linux/stop_machine.h> +#include <linux/sysfs.h> #include <linux/types.h> #include <linux/mm.h> #include <linux/cpu.h>
@@ -1269,6 +1270,24 @@ const struct cpumask *system_32bit_el0_cpumask(void)return cpu_present_mask; } +static ssize_t aarch32_el0_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +const struct cpumask *mask =system_32bit_el0_cpumask (); + +return sysfs_emit (buf,"%*pbl\n" ,cpumask_pr_args (mask)); +} +static const DEVICE_ATTR_RO (aarch32_el0); + +static int __init aarch32_el0_sysfs_init (void ) +{ +if (!allow_mismatched_32bit_el0) +return 0 ; + +return device_create_file (cpu_subsys.dev_root, &dev_attr_aarch32_el0); +} +device_initcall (aarch32_el0_sysfs_init); +static bool has_32bit_el0 (const struct arm64_cpu_capabilities *entry,int scope) {if (!has_cpuid_feature (entry, scope))
Signature · 1 line
2.29.2.222.g5d2a92d10f8-goog
Will Deacon
maintainerPATCH v2 6/6 arm64: Hook up cmdline parameter to allow mismatched 32-bit EL0
In reply to Will DeaconAllow systems with mismatched 32-bit support at EL0 to run 32-bit applications based on a new kernel parameter.
--- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ arch/arm64/kernel/cpufeature.c | 7 +++++++ 2 files changed, 14 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -289,6 +289,13 @@ do not want to use tracing_snapshot_alloc() as it needs to be done where GFP_KERNEL allocations are allowed. + allow_mismatched_32bit_el0 [ARM64] + Allow execve() of 32-bit applications and setting of the + PER_LINUX32 personality on systems where only a strict + subset of the CPUs support 32-bit EL0. When this + parameter is present, the set of CPUs supporting 32-bit + EL0 is indicated by /sys/devices/system/cpu/aarch32_el0. + amd_iommu= [HW,X86-64] Pass parameters to the AMD IOMMU driver in the system. Possible values are:
arch/arm64/kernel/cpufeature.c +7 -0
@@ -1270,6 +1270,13 @@ const struct cpumask *system_32bit_el0_cpumask(void)return cpu_present_mask; } +static int __init parse_32bit_el0_param (char *str) +{ + allow_mismatched_32bit_el0 = true; +return 0 ; +} +early_param ("allow_mismatched_32bit_el0" , parse_32bit_el0_param); +static ssize_t aarch32_el0_show (struct device *dev,struct device_attribute *attr,char *buf) {
Signature · 1 line
2.29.2.222.g5d2a92d10f8-goog
Re: [PATCH v2 5/6] arm64: Advertise CPUs capable of running 32-bit applications in sysfs
In reply to Will DeaconQuoted 9 lines · Will Deacon
Since 32-bit applications will be killed if they are caught trying to execute on a 64-bit-only CPU in a mismatched system, advertise the set of 32-bit capable CPUs to userspace in sysfs.
Signed-off-by Will Deacon <[email protected]>--- .../ABI/testing/sysfs-devices-system-cpu | 9 +++++++++ arch/arm64/kernel/cpufeature.c | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+)
I still think the "kill processes that can not run on this CPU" is crazy but that has nothing to do with this sysfs file patch, which looks good to me:
Catalin Marinas
maintainerRe: [PATCH v2 5/6] arm64: Advertise CPUs capable of running 32-bit applications in sysfs
Quoted 5 lines · Greg Kroah-Hartman
Quoted 4 lines · Will Deacon
Since 32-bit applications will be killed if they are caught trying to execute on a 64-bit-only CPU in a mismatched system, advertise the set of 32-bit capable CPUs to userspace in sysfs.
Quoted 7 lines
Quoted 5 lines
Signed-off-by Will Deacon <[email protected]>--- .../ABI/testing/sysfs-devices-system-cpu | 9 +++++++++ arch/arm64/kernel/cpufeature.c | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+)I still think the "kill processes that can not run on this CPU" is crazy
I agree it's crazy, though we try to keep the kernel support simple while making it a user-space problem. The alternative is to force-migrate such process to a more capable CPU, potentially against the desired user cpumask. In addition, we'd have to block CPU hot-unplug in case the last 32-bit capable CPU disappears.
The only sane thing is not to allow 32-bit processes at all on such hardware but I think we lost that battle ;).
Signature · 1 line
Catalin
Re: [PATCH v2 3/6] KVM: arm64: Kill 32-bit vCPUs on systems with mismatched EL0 support
In reply to Will DeaconQuoted 1 line · Will Deacon
If a vCPU tries to run 32-bit code on a system with mismatched support
nit: s/tries to run/is caught running/
Quoted 30 lines
at EL0, then we should kill it.
Signed-off-by Will Deacon <[email protected]>--- arch/arm64/kvm/arm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/arm64/kvm/arm.c +9
-0@@ -633,6 +633,15 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu) } } +static bool vcpu_mode_is_bad_32bit (struct kvm_vcpu *vcpu) +{ +if (likely (!vcpu_mode_is_32bit (vcpu))) +return false; + +return !system_supports_32bit_el0 () || +static_branch_unlikely (&arm64_mismatched_32bit_el0); +} +/** * kvm_arch_vcpu_ioctl_run - the main VCPU run function to execute guest code
* @vcpu: The VCPU pointer
arch/arm64/kvm/arm.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · line 831
Quoted 8 lines
arch/arm64/kvm/arm.c +1
-1@@ -816,7 +825,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)* with the asymmetric AArch32 case), return to userspace with * a fatal error. */ -if (!system_supports_32bit_el0 () &&vcpu_mode_is_32bit (vcpu)) { +if (vcpu_mode_is_bad_32bit (vcpu)) {/* * As we have caught the guest red-handed, decide that * it isn't fit for purpose anymore by making the vcpu
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 5/6] arm64: Advertise CPUs capable of running 32-bit applications in sysfs
In reply to Catalin MarinasQuoted 12 lines · Catalin Marinas
Quoted 11 lines · Greg Kroah-Hartman
Quoted 9 lines · Will Deacon
Since 32-bit applications will be killed if they are caught trying to execute on a 64-bit-only CPU in a mismatched system, advertise the set of 32-bit capable CPUs to userspace in sysfs.
Signed-off-by Will Deacon <[email protected]>--- .../ABI/testing/sysfs-devices-system-cpu | 9 +++++++++ arch/arm64/kernel/cpufeature.c | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+)
Quoted 7 lines
Quoted 1 line
I still think the "kill processes that can not run on this CPU" is crazy
I agree it's crazy, though we try to keep the kernel support simple while making it a user-space problem. The alternative is to force-migrate such process to a more capable CPU, potentially against the desired user cpumask. In addition, we'd have to block CPU hot-unplug in case the last 32-bit capable CPU disappears.
You should block CPU hot-unplug for the last 32bit capable CPU, why would you not want that if there are any active 32bit processes running?
And how is userspace going to know that it is creating a 32bit process? Are you now going to force all calls to exec() to be mediated somehow by putting an ELF parser in the init process?
Quoted 2 lines
The only sane thing is not to allow 32-bit processes at all on such hardware but I think we lost that battle ;).
That was a hardware decision that was made for some specific reason, so supporting it in the best way seems to be our best option given that people obviously must want this crazy type of system otherwise they wouldn't be paying for it!
While punting the logic out to userspace is simple for the kernel, and of course my first option, I think this isn't going to work in the long-run and the kernel will have to "know" what type of process it is scheduling in order to correctly deal with this nightmare as userspace can't do that well, if at all.
thanks,
greg k-h
Marc Zyngier
maintainerRe: [PATCH v2 5/6] arm64: Advertise CPUs capable of running 32-bit applications in sysfs
On 2020-11-10 09:36, Greg Kroah-Hartman wrote:
[...]
Quoted 5 lines
While punting the logic out to userspace is simple for the kernel, and of course my first option, I think this isn't going to work in the long-run and the kernel will have to "know" what type of process it is scheduling in order to correctly deal with this nightmare as userspace can't do that well, if at all.
For that to happen, we must first decide which part of the userspace ABI we are prepared to compromise on. The most obvious one would be to allow overriding the affinity on exec, but I'm pretty sure it has bad interactions with cgroups, on top of violating the existing ABI which mandates that the affinity is inherited across exec.
There may be other options (always make at least one 32bit-capable CPU part of the process affinity?), but they all imply some form of userspace visible regressions.
Pick your poison... :-/
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v2 5/6] arm64: Advertise CPUs capable of running 32-bit applications in sysfs
Quoted 10 lines · Marc Zyngier
On 2020-11-10 09:36, Greg Kroah-Hartman wrote:
[...]
Quoted 5 lines
While punting the logic out to userspace is simple for the kernel, and of course my first option, I think this isn't going to work in the long-run and the kernel will have to "know" what type of process it is scheduling in order to correctly deal with this nightmare as userspace can't do that well, if at all.
Quoted 5 lines
For that to happen, we must first decide which part of the userspace ABI we are prepared to compromise on. The most obvious one would be to allow overriding the affinity on exec, but I'm pretty sure it has bad interactions with cgroups, on top of violating the existing ABI which mandates that the affinity is inherited across exec.
So you are saying that you have to violate this today with this patch set? Or would have to violate that if the scheduler got involved?
How is userspace going to "know" how to do all of this properly? Who is going to write that code?
Quoted 5 lines
There may be other options (always make at least one 32bit-capable CPU part of the process affinity?), but they all imply some form of userspace visible regressions.
Pick your poison... :-/
What do the system designers and users of this hardware recommend? They are the ones that dreamed up this, and seem to somehow want this. What do they think the correct solution is here as obviously they must have thought this through when designing such a beast, right?
And if they didn't think any of this through then why are they wanting to run Linux on this thing?
thanks,
greg k-h
Marc Zyngier
maintainerRe: [PATCH v2 5/6] arm64: Advertise CPUs capable of running 32-bit applications in sysfs
Quoted 11 lines · Greg Kroah-Hartman
Quoted 10 lines · Marc Zyngier
On 2020-11-10 09:36, Greg Kroah-Hartman wrote:
[...]
Quoted 5 lines
While punting the logic out to userspace is simple for the kernel, and of course my first option, I think this isn't going to work in the long-run and the kernel will have to "know" what type of process it is scheduling in order to correctly deal with this nightmare as userspace can't do that well, if at all.
Quoted 8 lines
Quoted 5 lines
For that to happen, we must first decide which part of the userspace ABI we are prepared to compromise on. The most obvious one would be to allow overriding the affinity on exec, but I'm pretty sure it has bad interactions with cgroups, on top of violating the existing ABI which mandates that the affinity is inherited across exec.
So you are saying that you have to violate this today with this patch set? Or would have to violate that if the scheduler got involved?
Doing nothing (as with this series) doesn't result in an ABI breakage. It "only" results in an unreliable system. If you start making decisions behind userspace's back for the sake of making things reliable (which is a commendable goal), you break the ABI.
Rock, please meet A Hard Place.
And that's the real issue: there is no good solution to this problem. Only a different set of ugly compromises.
Quoted 9 lines
How is userspace going to "know" how to do all of this properly? Who is going to write that code?
Quoted 5 lines
There may be other options (always make at least one 32bit-capable CPU part of the process affinity?), but they all imply some form of userspace visible regressions.
Pick your poison... :-/
Quoted 7 lines
What do the system designers and users of this hardware recommend? They are the ones that dreamed up this, and seem to somehow want this. What do they think the correct solution is here as obviously they must have thought this through when designing such a beast, right?
And if they didn't think any of this through then why are they wanting to run Linux on this thing?
At this stage, I'll reach out for some Bombay mix (I dislike pop corn).
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Catalin Marinas
maintainerRe: [PATCH v2 5/6] arm64: Advertise CPUs capable of running 32-bit applications in sysfs
In reply to Greg Kroah-HartmanQuoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Catalin Marinas
Quoted 12 lines · Greg Kroah-Hartman
Quoted 9 lines · Will Deacon
Since 32-bit applications will be killed if they are caught trying to execute on a 64-bit-only CPU in a mismatched system, advertise the set of 32-bit capable CPUs to userspace in sysfs.
Signed-off-by Will Deacon <[email protected]>--- .../ABI/testing/sysfs-devices-system-cpu | 9 +++++++++ arch/arm64/kernel/cpufeature.c | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+)I still think the "kill processes that can not run on this CPU" is crazy
Quoted 8 lines
Quoted 5 lines
I agree it's crazy, though we try to keep the kernel support simple while making it a user-space problem. The alternative is to force-migrate such process to a more capable CPU, potentially against the desired user cpumask. In addition, we'd have to block CPU hot-unplug in case the last 32-bit capable CPU disappears.
You should block CPU hot-unplug for the last 32bit capable CPU, why would you not want that if there are any active 32bit processes running?
That's been done in one of the versions submitted to the Android kernel (which also handles automatic task placement, though by overriding the user cpumask):
https://android-review.googlesource.com/c/kernel/common/+/1437100/7
I think preventing CPU offlining makes sense only together with forcing 32-bit task placement from the kernel. If we decide to go for the SIGKILL approach with user-driven affinity, careful CPU offlining should also be handled by user-space.
Signature · 1 line
Catalin
Catalin Marinas
maintainerRe: [PATCH v2 2/6] arm64: Allow mismatched 32-bit EL0 support
In reply to Will DeaconHi Will,
Review
Quoted 4 lines · Will Deacon
(quoted diff) +4
-0+static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope) +{ + if (!has_cpuid_feature(entry, scope)) + return allow_mismatched_32bit_el0;
I still don't like overriding the cpufeature mechanism in this way. What about something like below? It still doesn't fit perfectly but at least the capability represents what was detected in the system. We then decide in system_supports_32bit_el0() whether to allow asymmetry. There is an extra trick to park a non-AArch32 capable CPU in has_32bit_el0() if it comes up late and the feature has already been advertised with !allow_mismatched_32bit_el0.
I find it clearer, though I probably stared at it more than at your patch ;).
arch/arm64/include/asm/cpufeature.h +5 -1
@@ -604,9 +604,13 @@ static inline bool cpu_supports_mixed_endian_el0(void)return id_aa64mmfr0_mixed_endian_el0 (read_cpuid (ID_AA64MMFR0_EL1 )); } +extern bool allow_mismatched_32bit_el0; +extern bool mismatched_32bit_el0; +static inline bool system_supports_32bit_el0 (void ) { -return cpus_have_const_cap (ARM64_HAS_32BIT_EL0 ); +return cpus_have_const_cap (ARM64_HAS_32BIT_EL0 ) && + (!mismatched_32bit_el0 || allow_mismatched_32bit_el0); }static inline bool system_supports_4kb_granule (void )
arch/arm64/kernel/cpufeature.c +38 -3
60 lines not on this page · raw message
Signature · 1 line
Catalin
Will Deacon
maintainerRe: [PATCH v2 2/6] arm64: Allow mismatched 32-bit EL0 support
Quoted 13 lines · Catalin Marinas
arch/arm64/kernel/cpufeature.c has_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope)
Quoted 4 lines · Will Deacon
arch/arm64/kernel/cpufeature.c +4
-0+static bool has_32bit_el0 (const struct arm64_cpu_capabilities *entry,int scope) +{ +if (!has_cpuid_feature (entry, scope)) +return allow_mismatched_32bit_el0;I still don't like overriding the cpufeature mechanism in this way. What about something like below? It still doesn't fit perfectly but at least the capability represents what was detected in the system. We then decide in
system_supports_32bit_el0()whether to allow asymmetry. There is an extra trick to park a non-AArch32 capable CPU inhas_32bit_el0()if it comes up late and the feature has already been advertised with !allow_mismatched_32bit_el0.
I deliberately allow late onlining of 64-bit-only cores and I don't think this is something we should forbid (although it's not clear from your patch when allow_mismatched_32bit_el0 gets set). Furthermore, killing CPUs from the matches callback feels _very_ dodgy to me, as it's invoked indirectly by things such as this_cpu_has_cap().
Quoted 2 lines
I find it clearer, though I probably stared at it more than at your patch ;).
Yeah, swings and roundabouts...
I think we're quibbling on implementation details a bit here whereas we should probably be focussing on what to do about execve() and CPU hotplug. Your patch doesn't apply on top of my series or replace this one, so there's not an awful lot I can do with it. I'm about to post a v3 with a tentative solution for execve(), so please could you demonstrate your idea on top of that so I can see how it fits together?
I'd like to move on from the "I don't like this" (none of us do) discussion and figure out the functional aspects, if possible. We can always paint it a different colour later on, but we don't even have a full solution yet.
Thanks,
Will
Catalin Marinas
maintainerRe: [PATCH v2 2/6] arm64: Allow mismatched 32-bit EL0 support
Quoted 15 lines · Will Deacon
Quoted 13 lines · Catalin Marinas
arch/arm64/kernel/cpufeature.c has_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope)
Quoted 4 lines · Will Deacon
arch/arm64/kernel/cpufeature.c +4
-0+static bool has_32bit_el0 (const struct arm64_cpu_capabilities *entry,int scope) +{ +if (!has_cpuid_feature (entry, scope)) +return allow_mismatched_32bit_el0;I still don't like overriding the cpufeature mechanism in this way. What about something like below? It still doesn't fit perfectly but at least the capability represents what was detected in the system. We then decide in
system_supports_32bit_el0()whether to allow asymmetry. There is an extra trick to park a non-AArch32 capable CPU inhas_32bit_el0()if it comes up late and the feature has already been advertised with !allow_mismatched_32bit_el0.
Quoted 2 lines
I deliberately allow late onlining of 64-bit-only cores and I don't think this is something we should forbid
I agree and my patch allows this. That's a property of WEAK_LOCAL_CPUFEATURE.
Quoted 2 lines
(although it's not clear from your patch when
allow_mismatched_32bit_el0gets set).
It doesn't, that was meant as a discussion point around the cpufeature framework but still relying on your other patches for cpumask, cmdline argument.
Quoted 3 lines
Furthermore, killing CPUs from the matches callback feels _very_ dodgy to me, as it's invoked indirectly by things such as
this_cpu_has_cap().
Yeah, this part is not nice. What we want here is for a late 64-bit only CPU to be parked if !allow_mismatched_32bit_el0 and we detected 32-bit already (symmetric).
Quoted 7 lines
Quoted 2 lines
I find it clearer, though I probably stared at it more than at your patch ;).
Yeah, swings and roundabouts...
I think we're quibbling on implementation details a bit here whereas we should probably be focussing on what to do about
execve()and CPU hotplug.
Do we need to solve the execve() problem? If yes, we have to get the scheduler involved. The hotplug case I think is simpler, just make sure we have a last standing 32-bit capable CPU, no ABI changes.
For execve(), arguably we don't necessarily break the execve() ABI as the affinity change may be done later when scheduling the task. But given that it's a user opt-in for this feature anyway, we just document the ABI changes.
Quoted 4 lines
Your patch doesn't apply on top of my series or replace this one, so there's not an awful lot I can do with it. I'm about to post a v3 with a tentative solution for
execve(), so please could you demonstrate your idea on top of that so I can see how it fits together?
I'll give it a go and if it looks any nicer, I'll post something. As you say, it's more like personal preference on the implementation details.
On the functional aspects, we must preserve the current behaviour like detecting symmetry and parking late CPUs if they don't comply in the !allow_mismatched_32bit_el0 case. In the allow_mismatched_32bit_el0 case, we relax this to the equivalent of a weak feature (either missing or present for early/late CPUs) but only report 32-bit if we found an early 32-bit capable CPU (I ended up with a truth table on a piece of paper).
Signature · 1 line
Catalin