arm64: Make Aarch32 compatibility enablement optional at boot

Major aspect of Aarch32 emulation is the ability to load 32bit processes.
That's currently decided (among others) by compat_elf_check_arch().

Make the macro use aarch32_enabled() to decide if Aarch32 compat is enabled before loading a 32bit process.

Signed-off-by Andrea della Porta <[email protected]>

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/arm64/kernel/process.c +1 -1

index 657ea273c0f9..96832f1ec3ee 100644
@@ -601,7 +601,7 @@ unsigned long arch_align_stack(unsigned long sp)
 #ifdef CONFIG_COMPAT
 int compat_elf_check_arch(const struct elf32_hdr *hdr)
 {
-	if (!system_supports_32bit_el0())
+	if (!system_supports_32bit_el0() || !aarch32_enabled())
 		return false;
 
 	if ((hdr)->e_machine != EM_ARM)

Signature · 1 line

2.35.3

Quoted 4 lines · Andrea della Porta

Major aspect of Aarch32 emulation is the ability to load 32bit processes.
That's currently decided (among others) by compat_elf_check_arch().

Quoted 4 lines

Make the macro use aarch32_enabled() to decide if Aarch32 compat is enabled before loading a 32bit process.

Signed-off-by Andrea della Porta <[email protected]>

Why can't you make system_supports_32bit_el0() take the option into account instead?

Quoted 5 lines · Mark Rutland

Quoted 4 lines · Andrea della Porta

Major aspect of Aarch32 emulation is the ability to load 32bit processes.
That's currently decided (among others) by compat_elf_check_arch().

Quoted 8 lines

Quoted 4 lines

Make the macro use aarch32_enabled() to decide if Aarch32 compat is enabled before loading a 32bit process.

Signed-off-by Andrea della Porta <[email protected]>

Why can't you make system_supports_32bit_el0() take the option into account instead?

I may be wrong here, but it seems to me that system_supports_32bit_el0() answers teh question "can this system supports compat execution?" rather than "do I want this system to run any compat execution?". That's the point of aarch32_enabled(), to state whether we want teh system to run A32 code or not, regardless of the system supporting it (of course, if the system does not support A32 in EL0, this is a no-no, but that's another story).

Quoted 15 lines · Andrea della Porta

Quoted 5 lines · Mark Rutland

Quoted 4 lines · Andrea della Porta

Major aspect of Aarch32 emulation is the ability to load 32bit processes.
That's currently decided (among others) by compat_elf_check_arch().

Quoted 8 lines

Quoted 4 lines

Make the macro use aarch32_enabled() to decide if Aarch32 compat is enabled before loading a 32bit process.

Signed-off-by Andrea della Porta <[email protected]>

Why can't you make system_supports_32bit_el0() take the option into account instead?

Quoted 6 lines

I may be wrong here, but it seems to me that system_supports_32bit_el0() answers teh question "can this system supports compat execution?" rather than "do I want this system to run any compat execution?". That's the point of aarch32_enabled(), to state whether we want teh system to run A32 code or not, regardless of the system supporting it (of course, if the system does not support A32 in EL0, this is a no-no, but that's another story).

That's what the implementation does today, but we're really using it as a "do we intend for 32-bit EL0 to work?" predicate, and generally the system_supports_${FEATURE}() helpers are affected by the combination of actual HW support, kernel config options, *and* kernel command line options. For example, system_supports_sve() is affected by both CONFIG_ARM64_SVE and the "arm64.nosve" command line option.

Thanks,
Mark.

Quoted 30 lines · Mark Rutland

Quoted 15 lines · Andrea della Porta

Quoted 5 lines · Mark Rutland

Quoted 4 lines · Andrea della Porta

Major aspect of Aarch32 emulation is the ability to load 32bit processes.
That's currently decided (among others) by compat_elf_check_arch().

Quoted 8 lines

Quoted 4 lines

Make the macro use aarch32_enabled() to decide if Aarch32 compat is enabled before loading a 32bit process.

Signed-off-by Andrea della Porta <[email protected]>

Why can't you make system_supports_32bit_el0() take the option into account instead?

Quoted 6 lines

I may be wrong here, but it seems to me that system_supports_32bit_el0() answers teh question "can this system supports compat execution?" rather than "do I want this system to run any compat execution?". That's the point of aarch32_enabled(), to state whether we want teh system to run A32 code or not, regardless of the system supporting it (of course, if the system does not support A32 in EL0, this is a no-no, but that's another story).

That's what the implementation does today, but we're really using it as a "do we intend for 32-bit EL0 to work?" predicate, and generally the system_supports_${FEATURE}() helpers are affected by the combination of actual HW support, kernel config options, *and* kernel command line options. For example, system_supports_sve() is affected by both CONFIG_ARM64_SVE and the "arm64.nosve" command line option.

Quoted 2 lines

Thanks,
Mark.

Many thanks for the explanation, then inserting aach32_enabled() in system_supports_32bit_el0() is the way to go.

Another major aspect of supporting running of 32bit processes is the ability to access 32bit syscalls. Such syscalls can be invoked by using the svc instruction.

If Aarch32 emulation is disabled ensure that calling svc results in the same behavior as if CONFIG_COMPAT has not been enabled (i.e. a kernel panic).

Signed-off-by Andrea della Porta <[email protected]>

---
 arch/arm64/kernel/entry-common.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

arch/arm64/kernel/entry-common.c +0 -0

index 69ff9b8c0bde..32761760d9dd 100644

arch/arm64/kernel/entry-common.c +13 -3

+
 	unsigned long esr = read_sysreg(esr_el1);
 
 	switch (ESR_ELx_EC(esr)) {

@@ -865,17 +875,26 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
 
 asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
 {
-	__el0_irq_handler_common(regs);
+	if (!aarch32_enabled())
+		el0t_32_irq_ni_handler(regs);
+	else
+		__el0_irq_handler_common(regs);
 }
 
 asmlinkage void noinstr el0t_32_fiq_handler(struct pt_regs *regs)
 {
-	__el0_fiq_handler_common(regs);
+	if (!aarch32_enabled())
+		el0t_32_fiq_ni_handler(regs);
+	else
+		__el0_fiq_handler_common(regs);
 }
 
 asmlinkage void noinstr el0t_32_error_handler(struct pt_regs *regs)
 {
-	__el0_error_handler_common(regs);
+	if (!aarch32_enabled())
+		el0t_32_error_ni_handler(regs);
+	else
+		__el0_error_handler_common(regs);
 }
 
 bool __aarch32_enabled __ro_after_init = true;

Signature · 1 line

2.35.3

Quoted 7 lines · Andrea della Porta

Another major aspect of supporting running of 32bit processes is the ability to access 32bit syscalls. Such syscalls can be invoked by using the svc instruction.

If Aarch32 emulation is disabled ensure that calling svc results in the same behavior as if CONFIG_COMPAT has not been enabled (i.e. a kernel panic).

It's not "emulation" it's directly supported by the hardware.

Quoted 14 lines

Signed-off-by Andrea della Porta <[email protected]>

---
 arch/arm64/kernel/entry-common.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

arch/arm64/kernel/entry-common.c +0 -0

index 69ff9b8c0bde..32761760d9dd 100644
@@ -802,6 +802,11 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
 }
 

arch/arm64/kernel/entry-common.c asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs) · line 808

Quoted 5 lines

arch/arm64/kernel/entry-common.c +4 -0

 #ifdef CONFIG_COMPAT
+UNHANDLED(el0t, 32, sync_ni)
+UNHANDLED(el0t, 32, irq_ni)
+UNHANDLED(el0t, 32, fiq_ni)
+UNHANDLED(el0t, 32, error_ni)

IRQ, FIQ, and SError are not syscalls, so the commit title is bad.

arch/arm64/kernel/entry-common.c static void noinstr el0_svc_compat(struct pt_regs *regs) · line 832

Quoted 13 lines

arch/arm64/kernel/entry-common.c +5 -0

+
 static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
 {
 	enter_from_user_mode(regs);

@@ -821,6 +826,11 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
 
 asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
 {
+	if (!aarch32_enabled()) {
+		el0t_32_sync_ni_handler(regs);
+		return;
+	}

Why do we have to do this at all?

If we don't have AArch32 tasks, these paths are unreachable. Why do we need to check that they aren't called?

Mark.

Distributions would like to reduce their attack surface as much as possible but at the same time they'd want to retain flexibility to cater to a variety of legacy software. This stems from the conjecture that compat layer is likely rarely tested and could have latent security bugs. Ideally distributions will set their default policy and also give users the ability to override it as appropriate.

To enable this use case, introduce CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED compile time option, which controls whether 32bit processes/syscalls should be allowed or not. This option is aimed mainly at distributions to set their preferred default behavior in their kernels.

To allow users to override the distro's policy, introduce the 'aarch32_emulation' parameter which allows overriding CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED state at boot time.

Signed-off-by Andrea della Porta <[email protected]>

---
 Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
 arch/arm64/Kconfig                              | 9 +++++++++
 arch/arm64/kernel/entry-common.c                | 8 +++++++-
 3 files changed, 23 insertions(+), 1 deletion(-)

Documentation/admin-guide/kernel-parameters.txt +0 -0

index 0a1731a0f0ef..a41c5e6f5d2e 100644

Documentation/admin-guide/kernel-parameters.txt +1 -0

+
 	acpi=		[HW,ACPI,X86,ARM64,RISCV64]
 			Advanced Configuration and Power Interface
 			Format: { force | on | off | strict | noirq | rsdt |

arch/arm64/Kconfig +9 -0

index b10515c0200b..66c4cb273550 100644
@@ -1725,6 +1725,15 @@ config SETEND_EMULATION
 	  If unsure, say Y
 endif # ARMV8_DEPRECATED
 
+config AARCH32_EMULATION_DEFAULT_DISABLED
+	bool "Aarch32 emulation disabled by default"
+	default n
+	depends on COMPAT
+	help
+	  Make Aarch32 emulation disabled by default. This prevents loading 32-bit
+	  processes and access to 32-bit syscalls.
+
+	  If unsure, leave it to its default value.
 endif # COMPAT
 
 menu "ARMv8.1 architectural features"

arch/arm64/kernel/entry-common.c +7 -1

index 32761760d9dd..07f2b4e632b8 100644
@@ -897,7 +897,13 @@ asmlinkage void noinstr el0t_32_error_handler(struct pt_regs *regs)
 		__el0_error_handler_common(regs);
 }
 
-bool __aarch32_enabled __ro_after_init = true;
+bool __aarch32_enabled __ro_after_init = !IS_ENABLED(CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED);
+
+static int aarch32_emulation_override_cmdline(char *arg)
+{
+	return kstrtobool(arg, &__aarch32_enabled);
+}
+early_param("aarch32_emulation", aarch32_emulation_override_cmdline);
 #else /* CONFIG_COMPAT */
 UNHANDLED(el0t, 32, sync)
 UNHANDLED(el0t, 32, irq)

Signature · 1 line

2.35.3

Quoted 28 lines · Andrea della Porta

Distributions would like to reduce their attack surface as much as possible but at the same time they'd want to retain flexibility to cater to a variety of legacy software. This stems from the conjecture that compat layer is likely rarely tested and could have latent security bugs. Ideally distributions will set their default policy and also give users the ability to override it as appropriate.

To enable this use case, introduce CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED compile time option, which controls whether 32bit processes/syscalls should be allowed or not. This option is aimed mainly at distributions to set their preferred default behavior in their kernels.

To allow users to override the distro's policy, introduce the 'aarch32_emulation' parameter which allows overriding CONFIG_AARCH32_EMULATION_DEFAULT_DISABLED state at boot time.

Signed-off-by Andrea della Porta <[email protected]>

---
 Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
 arch/arm64/Kconfig                              | 9 +++++++++
 arch/arm64/kernel/entry-common.c                | 8 +++++++-
 3 files changed, 23 insertions(+), 1 deletion(-)

Documentation/admin-guide/kernel-parameters.txt +0 -0

index 0a1731a0f0ef..a41c5e6f5d2e 100644

Documentation/admin-guide/kernel-parameters.txt line 6

Quoted 6 lines

Documentation/admin-guide/kernel-parameters.txt +6 -0

@@ -1,3 +1,10 @@
+	aarch32_emulation=	[ARM64]
+			Format: <bool>
+			When true, allows loading 32-bit programs and executing
+			32-bit syscalls, essentially overriding
+			AARCH32_EMULATION_DEFAULT_DISABLED at boot time. when false,
+			unconditionally disables AARCH32 emulation.

Can we please drop the 'emulation' part of the name? We don't use that terminology on arm64 for regular execution of compat tasks, and only use that to refer to true emulation of deprecated instructions.

We already have the 'allow_mismatched_32bit_el0' option; can we please us a name that aligns with that? e.g. 'allow_32bit_el0=false' to disable support.

Aarch32 compatibility mode is enabled at compile time through CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support (for both processes and syscalls) be enabled at boot time using a kernel parameter. Also, it provides a mean for distributions to set their own default without sacrificing compatibility support, that is users can override default behaviour through the kernel parameter.

Hi,

Quoted 7 lines · Andrea della Porta

Aarch32 compatibility mode is enabled at compile time through CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support (for both processes and syscalls) be enabled at boot time using a kernel parameter. Also, it provides a mean for distributions to set their own default without sacrificing compatibility support, that is users can override default behaviour through the kernel parameter.

I proposed something similar in the past:

https://lkml.kernel.org/linux-fsdevel/[email protected]/

bu the conclusion there (see the reply from Kees) was that it was better to either use existing seccomp mechanisms or add something to control which binfmts can be loaded.

Quoted 11 lines · Will Deacon

Hi,

Quoted 7 lines · Andrea della Porta

Aarch32 compatibility mode is enabled at compile time through CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support (for both processes and syscalls) be enabled at boot time using a kernel parameter. Also, it provides a mean for distributions to set their own default without sacrificing compatibility support, that is users can override default behaviour through the kernel parameter.

Quoted 7 lines

I proposed something similar in the past:

https://lkml.kernel.org/linux-fsdevel/[email protected]/

bu the conclusion there (see the reply from Kees) was that it was better to either use existing seccomp mechanisms or add something to control which binfmts can be loaded.

Right, I was going to reply along the same lines here: x86 is a bit of a special case that needs this, but I believe all the other architectures already guard the compat syscall execution on test_thread_flag(TIF_32BIT) that is only set by the compat binfmt loader.

Doing the reverse is something that has however come up in the past several times and that could be interesting: In order to run userspace emulation (qemu-user, fex, ...) we may want to allow calling syscalls and ioctls for foreign ABIs in a native task, and at that point having a mechanism to control this capability globally or per task would be useful as well.

The compat mode (arm32 on arm64) is the easiest case here, but the same thing could be done for emulating the very subtle architecture differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat, or any of the above on riscv or loongarch).

Arnd

Quoted 7 lines · Arnd Bergmann

On Wed, Oct 18, 2023, at 14:27, Will Deacon wrote:

Right, I was going to reply along the same lines here: x86 is a bit of a special case that needs this, but I believe all the other architectures already guard the compat syscall execution on test_thread_flag(TIF_32BIT) that is only set by the compat binfmt loader.

Are you referring to the fact that x86 can switch at will between 32- and 64- bit code?

Regarding the TIF_32BIT flag, thanks for the head-up. I still believe though that this mechanism can somehow break down in the future, since prohibiting 32 bit executable loading *and* blocking 32 bit compat syscall are two separate path of execution, held together by the architecture prohibiting to switch to A32 instructions by design. Breaking the first rule and embedding wisely crafted A32 instruction in an executable is easy, while the difficult part is finding some 'reentrancy' to be able to do the execution state switch, as pinted out in https://lore.kernel.org/lkml/ZTD0DAes-J-YQ2eu@apocalypse/.
I agree it's highly speculative and not something to be concerned right now, it's just a head up, should the need arise in the future.

Quoted 7 lines

Doing the reverse is something that has however come up in the past several times and that could be interesting: In order to run userspace emulation (qemu-user, fex, ...) we may want to allow calling syscalls and ioctls for foreign ABIs in a native task, and at that point having a mechanism to control this capability globally or per task would be useful as well.

Quoted 4 lines

The compat mode (arm32 on arm64) is the easiest case here, but the same thing could be done for emulating the very subtle architecture differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat, or any of the above on riscv or loongarch).

Really interesting, Since it's more related to emulation needs (my patch has another focus due to the fact that A64 can execute A32 natively), I'll take a look at this separately.

Hi,

Quoted 7 lines · Andrea della Porta

Aarch32 compatibility mode is enabled at compile time through CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support (for both processes and syscalls) be enabled at boot time using a kernel parameter. Also, it provides a mean for distributions to set their own default without sacrificing compatibility support, that is users can override default behaviour through the kernel parameter.

I proposed something similar in the past:

https://lkml.kernel.org/linux-fsdevel/[email protected]/

bu the conclusion there (see the reply from Kees) was that it was better to either use existing seccomp mechanisms or add something to control which binfmts can be loaded.

Will

Aarch32 compatibility mode is enabled at compile time through CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support (for both processes and syscalls) be enabled at boot time using a kernel parameter. Also, it provides a mean for distributions to set their own default without sacrificing compatibility support, that is users can override default behaviour through the kernel parameter.

*** Notes about syscall management ***
VBAR_EL1 register, which holds the exception table address, is setup very early in the boot process, before parse_early_param(). This means that it's not possible to access boot parameter before setting the register. Also, setting the aforementioned register for secondary cpus is done later in the boot flow. Several ways to work around this has been considered, among which:

* resetting VBAR_EL1 to point to one of two vector tables (the former with 32-bit exceptions handler enabled and the latter pointing to unhandled stub, just as if CONFIG_COMPAT is enabled) depending on the proposed boot parameter. This has the disadvantage to produce a somewhat messy patchset involving several lines, has higher cognitive load since there are at least three places where the register is getting changed (not near to each other), and have implications on other code segments (namely kpti, kvm and vdso), requiring special care.

* patching the vector table contents once the early param is available. This has most of the implications of the previous option (except maybe not impacting other code segments), plus it sounds a little 'hackish'.

The chosen approach involves conditional executing 32-bit syscalls depending on the parameter value.

This of course results in a little performance loss, but has the following advantages:

* all the cons from previously explained alternatives are solved
* users of 32-bit apps on 64-bit kernel are already suffering from performance losses due to 32-bit apps not fully leveraging the 64-bit processor, so they are already aware of this
* users of 32-bit apps on 64-bit kernel are believed
to be a minority and most of the time there are sources available to be recompiled for 64-bit as a workaround for better performance

It worth mentioning that users of 64-bit apps are, of course, unaffected.

Based on the work from Nikolay Borisov, see:

Link https://lkml.org/lkml/2023/6/23/387

Andrea della Porta (4):
arm64: Introduce aarch32_enabled() arm64/process: Make loading of 32bit processes depend on

    aarch32_enabled()
  arm64/entry-common: Make Aarch32 syscalls' availability depend on
    aarch32_enabled()

arm64: Make Aarch32 emulation boot time configurable

 .../admin-guide/kernel-parameters.txt         |  7 ++++
 arch/arm64/Kconfig                            |  9 +++++
 arch/arm64/include/asm/compat.h               | 12 +++++++
 arch/arm64/kernel/entry-common.c              | 33 +++++++++++++++++--
 arch/arm64/kernel/process.c                   |  2 +-
 5 files changed, 59 insertions(+), 4 deletions(-)

Signature · 1 line

2.35.3

Quoted 7 lines · Andrea della Porta

Aarch32 compatibility mode is enabled at compile time through CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support (for both processes and syscalls) be enabled at boot time using a kernel parameter. Also, it provides a mean for distributions to set their own default without sacrificing compatibility support, that is users can override default behaviour through the kernel parameter.

Can you elaborate on *why* people want such a policy?

Quoted 23 lines

*** Notes about syscall management ***
VBAR_EL1 register, which holds the exception table address, is setup very early in the boot process, before parse_early_param(). This means that it's not possible to access boot parameter before setting the register. Also, setting the aforementioned register for secondary cpus is done later in the boot flow. Several ways to work around this has been considered, among which:

* resetting VBAR_EL1 to point to one of two vector tables (the former with 32-bit exceptions handler enabled and the latter pointing to unhandled stub, just as if CONFIG_COMPAT is enabled) depending on the proposed boot parameter. This has the disadvantage to produce a somewhat messy patchset involving several lines, has higher cognitive load since there are at least three places where the register is getting changed (not near to each other), and have implications on other code segments (namely kpti, kvm and vdso), requiring special care.

* patching the vector table contents once the early param is available. This has most of the implications of the previous option (except maybe not impacting other code segments), plus it sounds a little 'hackish'.

Quoted 2 lines

The chosen approach involves conditional executing 32-bit syscalls depending on the parameter value.

Why does the compat syscall path need to do anything?

On arm64 it's not possible to issue compat syscalls from a native 64-bit task. If you prevent the loading of AArch32 binaries, none of the compat syscalls will be reachable at all.

That's the proper way to implement this, and we already have logic for that as part of the mismatched AArch32 support.

Quoted 2 lines

This of course results in a little performance loss, but has the following advantages:

A performance loss for what relative to what?

How much of a performance loss?