Skip to contentSkip to conversation

arm64: Enable BTI for the executable as well as the interpreter

PATCH Superseded Mark Brown 4 participants 18 messages Jan 24, 2022 – Feb 25, 2022

Subsystems (4)
VFS arm64 Generic include/asm header files exec and binfmt

v8 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Mark Brown

author
#0 lore raw

v8 cover arm64: Enable BTI for the executable as well as the interpreter

Deployments of BTI on arm64 have run into issues interacting with systemd's MemoryDenyWriteExecute feature. Currently for dynamically linked executables the kernel will only handle architecture specific properties like BTI for the interpreter, the expectation is that the interpreter will then handle any properties on the main executable. For BTI this means remapping the executable segments PROT_EXEC | PROT_BTI.

This interacts poorly with MemoryDenyWriteExecute since that is implemented using a seccomp filter which prevents setting PROT_EXEC on already mapped memory and lacks the context to be able to detect that memory is already mapped with PROT_EXEC. This series resolves this by handling the BTI property for both the interpreter and the main executable.

This does mean that we may get more code with BTI enabled if running on a system without BTI support in the dynamic linker, this is expected to be a safe configuration and testing seems to confirm that. It also reduces the flexibility userspace has to disable BTI but it is expected that for cases where there are problems which require BTI to be disabled it is more likely that it will need to be disabled on a system level.

v8:
- Rebase onto v5.17-rc1.
v7:
- Rebase onto v5.16-rc1.
v6:
- Rebase onto v5.15-rc1.
v5:
- Rebase onto v5.14-rc2.
- Tweak changelog on patch 1.
- Use the helper for interpreter/executable flag in elf.h as well.
v4:
- Rebase onto v5.14-rc1.
v3:
- Fix passing of properties for parsing by the main executable.
- Drop has_interp from arch_parse_elf_property().
- Coding style tweaks.
v2:
- Add a patch dropping has_interp from arch_adjust_elf_prot()
- Fix bisection issue with static executables on arm64 in the first patch.

Mark Brown (4):
elf: Allow architectures to parse properties on the main executable
arm64: Enable BTI for main executable as well as the interpreter
elf: Remove has_interp property from arch_adjust_elf_prot()
elf: Remove has_interp property from arch_parse_elf_property()

 arch/arm64/include/asm/elf.h | 14 ++++++++++++--
 arch/arm64/kernel/process.c  | 16 +++-------------
 fs/binfmt_elf.c              | 29 ++++++++++++++++++++---------
 include/linux/elf.h          |  8 +++++---
 4 files changed, 40 insertions(+), 27 deletions(-)

base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07

Signature · 1 line
2.30.2
author
#1 lore raw

PATCH v8 1/4 elf: Allow architectures to parse properties on the main executable

Currently the ELF code only attempts to parse properties on the image that will start execution, either the interpreter or for statically linked executables the main executable. The expectation is that any property handling for the main executable will be done by the interpreter. This is a bit inconsistent since we do map the executable and is causing problems for the arm64 BTI support when used in conjunction with systemd's use of seccomp to implement MemoryDenyWriteExecute which stops the dynamic linker adjusting the permissions of executable segments.

Allow architectures to handle properties for both the dynamic linker and main executable, adjusting arch_parse_elf_properties() to have a new flag is_interp flag as with arch_elf_adjust_prot() and calling it for both the main executable and any intepreter.

The user of this code, arm64, is adapted to ensure that there is no functional change.

Signed-off-by Mark Brown <[email protected]>
Tested-by Jeremy Linton <[email protected]>
Reviewed-by Dave Martin <[email protected]>
---
 arch/arm64/include/asm/elf.h |  3 ++-
 fs/binfmt_elf.c              | 27 +++++++++++++++++++--------
 include/linux/elf.h          |  4 +++-
 3 files changed, 24 insertions(+), 10 deletions(-)
arch/arm64/include/asm/elf.h +2 -1
index 97932fbf973d..5cc002376abe 100644
@@ -259,6 +259,7 @@ struct arch_elf_state {
 
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
+					  bool has_interp, bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	/* No known properties for AArch32 yet */

@@ -271,7 +272,7 @@ static inline int arch_parse_elf_property(u32 type, const void *data,
 		if (datasz != sizeof(*p))
 			return -ENOEXEC;
 
-		if (system_supports_bti() &&
+		if (system_supports_bti() && has_interp == is_interp &&
 		    (*p & GNU_PROPERTY_AARCH64_FEATURE_1_BTI))
 			arch->flags |= ARM64_ELF_BTI;
 	}
fs/binfmt_elf.c +19 -8
index 605017eb9349..eabd7e0f6592 100644
@@ -716,8 +716,9 @@ static unsigned long load_elf_interp(struct elfhdr *interp_elf_ex,
  */
 
 static int parse_elf_property(const char *data, size_t *off, size_t datasz,
-			      struct arch_elf_state *arch,
-			      bool have_prev_type, u32 *prev_type)
+			      struct arch_elf_state *arch, bool has_interp,
+			      bool is_interp, bool have_prev_type,
+			      u32 *prev_type)
 {
 	size_t o, step;
 	const struct gnu_property *pr;

@@ -751,7 +752,8 @@ static int parse_elf_property(const char *data, size_t *off, size_t datasz,
 	*prev_type = pr->pr_type;
 
 	ret = arch_parse_elf_property(pr->pr_type, data + o,
-				      pr->pr_datasz, ELF_COMPAT, arch);
+				      pr->pr_datasz, ELF_COMPAT,
+				      has_interp, is_interp, arch);
 	if (ret)
 		return ret;
 

@@ -764,6 +766,7 @@ static int parse_elf_property(const char *data, size_t *off, size_t datasz,
 #define NOTE_NAME_SZ (sizeof(GNU_PROPERTY_TYPE_0_NAME))
 
 static int parse_elf_properties(struct file *f, const struct elf_phdr *phdr,
+				bool has_interp, bool is_interp,
 				struct arch_elf_state *arch)
 {
 	union {

@@ -813,7 +816,8 @@ static int parse_elf_properties(struct file *f, const struct elf_phdr *phdr,
 	have_prev_type = false;
 	do {
 		ret = parse_elf_property(note.data, &off, datasz, arch,
-					 have_prev_type, &prev_type);
+					 has_interp, is_interp, have_prev_type,
+					 &prev_type);
 		have_prev_type = true;
 	} while (!ret);
 

@@ -828,6 +832,7 @@ static int load_elf_binary(struct linux_binprm *bprm)
 	unsigned long error;
 	struct elf_phdr *elf_ppnt, *elf_phdata, *interp_elf_phdata = NULL;
 	struct elf_phdr *elf_property_phdata = NULL;
+	struct elf_phdr *interp_elf_property_phdata = NULL;
 	unsigned long elf_bss, elf_brk;
 	int bss_prot = 0;
 	int retval, i;

@@ -963,12 +968,11 @@ static int load_elf_binary(struct linux_binprm *bprm)
 			goto out_free_dentry;
 
 		/* Pass PT_LOPROC..PT_HIPROC headers to arch code */
-		elf_property_phdata = NULL;
 		elf_ppnt = interp_elf_phdata;
 		for (i = 0; i < interp_elf_ex->e_phnum; i++, elf_ppnt++)
 			switch (elf_ppnt->p_type) {
 			case PT_GNU_PROPERTY:
-				elf_property_phdata = elf_ppnt;
+				interp_elf_property_phdata = elf_ppnt;
 				break;
 
 			case PT_LOPROC ... PT_HIPROC:

@@ -979,10 +983,17 @@ static int load_elf_binary(struct linux_binprm *bprm)
 					goto out_free_dentry;
 				break;
 			}
+
+		retval = parse_elf_properties(interpreter,
+					      interp_elf_property_phdata,
+					      true, true, &arch_state);
+		if (retval)
+			goto out_free_dentry;
+
 	}
 
-	retval = parse_elf_properties(interpreter ?: bprm->file,
-				      elf_property_phdata, &arch_state);
+	retval = parse_elf_properties(bprm->file, elf_property_phdata,
+				      interpreter, false, &arch_state);
 	if (retval)
 		goto out_free_dentry;
include/linux/elf.h +3 -1
index c9a46c4e183b..1c45ecf29147 100644
@@ -88,13 +88,15 @@ struct arch_elf_state;
 #ifndef CONFIG_ARCH_USE_GNU_PROPERTY
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
+					  bool has_interp, bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	return 0;
 }
 #else
 extern int arch_parse_elf_property(u32 type, const void *data, size_t datasz,
-				   bool compat, struct arch_elf_state *arch);
+				   bool compat, bool has_interp, bool is_interp,
+				   struct arch_elf_state *arch);
 #endif
 
 #ifdef CONFIG_ARCH_HAVE_ELF_PROT
Signature · 1 line
2.30.2
maintainer Reviewed-by on v8 1/4
#2 lore raw

Re: [PATCH v8 1/4] elf: Allow architectures to parse properties on the main executable

Quoted 14 lines · Mark Brown

Currently the ELF code only attempts to parse properties on the image that will start execution, either the interpreter or for statically linked executables the main executable. The expectation is that any property handling for the main executable will be done by the interpreter. This is a bit inconsistent since we do map the executable and is causing problems for the arm64 BTI support when used in conjunction with systemd's use of seccomp to implement MemoryDenyWriteExecute which stops the dynamic linker adjusting the permissions of executable segments.

Allow architectures to handle properties for both the dynamic linker and main executable, adjusting arch_parse_elf_properties() to have a new flag is_interp flag as with arch_elf_adjust_prot() and calling it for both the main executable and any intepreter.

Quoted 6 lines

The user of this code, arm64, is adapted to ensure that there is no functional change.

Signed-off-by Mark Brown <[email protected]>
Tested-by Jeremy Linton <[email protected]>
Reviewed-by Dave Martin <[email protected]>
Reviewed-by Catalin Marinas <[email protected]>
author
#3 lore raw

PATCH v8 2/4 arm64: Enable BTI for main executable as well as the interpreter

In reply to Mark Brown

Currently for dynamically linked ELF executables we only enable BTI for the interpreter, expecting the interpreter to do this for the main executable. This is a bit inconsistent since we do map main executable and is causing issues with systemd's MemoryDenyWriteExecute feature which is implemented using a seccomp filter which prevents setting PROT_EXEC on already mapped memory and lacks the context to be able to detect that memory is already mapped with PROT_EXEC.

Resolve this by checking the BTI property for the main executable and enabling BTI if it is present when doing the initial mapping. This does mean that we may get more code with BTI enabled if running on a system without BTI support in the dynamic linker, this is expected to be a safe configuration and testing seems to confirm that. It also reduces the flexibility userspace has to disable BTI but it is expected that for cases where there are problems which require BTI to be disabled it is more likely that it will need to be disabled on a system level.

Signed-off-by Mark Brown <[email protected]>
Reviewed-by Dave Martin <[email protected]>
Tested-by Jeremy Linton <[email protected]>
---
 arch/arm64/include/asm/elf.h | 15 ++++++++++++---
 arch/arm64/kernel/process.c  | 14 ++------------
 2 files changed, 14 insertions(+), 15 deletions(-)
arch/arm64/include/asm/elf.h +12 -3
index 5cc002376abe..c4aa60db76a4 100644
@@ -251,12 +251,21 @@ struct arch_elf_state {
 	int flags;
 };
 
-#define ARM64_ELF_BTI		(1 << 0)
+#define ARM64_ELF_INTERP_BTI		(1 << 0)
+#define ARM64_ELF_EXEC_BTI		(1 << 1)
 
 #define INIT_ARCH_ELF_STATE {			\
 	.flags = 0,				\
 }
 
+static inline int arm64_elf_bti_flag(bool is_interp)
+{
+	if (is_interp)
+		return ARM64_ELF_INTERP_BTI;
+	else
+		return ARM64_ELF_EXEC_BTI;
+}
+
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
 					  bool has_interp, bool is_interp,

@@ -272,9 +281,9 @@ static inline int arch_parse_elf_property(u32 type, const void *data,
 		if (datasz != sizeof(*p))
 			return -ENOEXEC;
 
-		if (system_supports_bti() && has_interp == is_interp &&
+		if (system_supports_bti() &&
 		    (*p & GNU_PROPERTY_AARCH64_FEATURE_1_BTI))
-			arch->flags |= ARM64_ELF_BTI;
+			arch->flags |= arm64_elf_bti_flag(is_interp);
 	}
 
 	return 0;
arch/arm64/kernel/process.c +2 -12
index 5369e649fa79..8157910a4181 100644
@@ -704,18 +704,8 @@ core_initcall(tagged_addr_init);
 int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,
 			 bool has_interp, bool is_interp)
 {
-	/*
-	 * For dynamically linked executables the interpreter is
-	 * responsible for setting PROT_BTI on everything except
-	 * itself.
-	 */
-	if (is_interp != has_interp)
-		return prot;
-
-	if (!(state->flags & ARM64_ELF_BTI))
-		return prot;
-
-	if (prot & PROT_EXEC)
+	if ((prot & PROT_EXEC) &&
+	    (state->flags & arm64_elf_bti_flag(is_interp)))
 		prot |= PROT_BTI;
 
 	return prot;
Signature · 1 line
2.30.2
maintainer Reviewed-by on v8 2/4
#4 lore raw

Re: [PATCH v8 2/4] arm64: Enable BTI for main executable as well as the interpreter

Quoted 8 lines · Mark Brown

Currently for dynamically linked ELF executables we only enable BTI for the interpreter, expecting the interpreter to do this for the main executable. This is a bit inconsistent since we do map main executable and is causing issues with systemd's MemoryDenyWriteExecute feature which is implemented using a seccomp filter which prevents setting PROT_EXEC on already mapped memory and lacks the context to be able to detect that memory is already mapped with PROT_EXEC.

Quoted 5 lines

Resolve this by checking the BTI property for the main executable and enabling BTI if it is present when doing the initial mapping. This does mean that we may get more code with BTI enabled if running on a system without BTI support in the dynamic linker, this is expected to be a safe configuration and testing seems to confirm that.

Alternatively we could only enable BTI on the main executable if the interpreter is also compiled with the BTI attributes. But it still feels a bit inconsistent as a BTI-capable interpreter doesn't necessarily mean it had to be compiled with BTI support.

Quoted 4 lines

It also reduces the
flexibility userspace has to disable BTI but it is expected that for cases where there are problems which require BTI to be disabled it is more likely that it will need to be disabled on a system level.

That's my expectation as well. I'm not aware of any plans to make a dynamic loader choose BTI based on some environment variable (not sure about Android zygote, I haven't heard anything either).

Quoted 3 lines
Signed-off-by Mark Brown <[email protected]>
Reviewed-by Dave Martin <[email protected]>
Tested-by Jeremy Linton <[email protected]>

For this patch:

Reviewed-by Catalin Marinas <[email protected]>
author
#5 lore raw

PATCH v8 3/4 elf: Remove has_interp property from arch_adjust_elf_prot()

In reply to Mark Brown

Since we have added an is_interp flag to arch_parse_elf_property() we can drop the has_interp flag from arch_elf_adjust_prot(), the only user was the arm64 code which no longer needs it and any future users will be able to use arch_parse_elf_properties() to determine if an interpreter is in use.

Signed-off-by Mark Brown <[email protected]>
Tested-by Jeremy Linton <[email protected]>
Reviewed-by Dave Martin <[email protected]>
---
 arch/arm64/kernel/process.c | 2 +-
 fs/binfmt_elf.c             | 2 +-
 include/linux/elf.h         | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)
arch/arm64/kernel/process.c +1 -1
index 8157910a4181..8017d963be3f 100644
@@ -702,7 +702,7 @@ core_initcall(tagged_addr_init);
 
 #ifdef CONFIG_BINFMT_ELF
 int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,
-			 bool has_interp, bool is_interp)
+			 bool is_interp)
 {
 	if ((prot & PROT_EXEC) &&
 	    (state->flags & arm64_elf_bti_flag(is_interp)))
fs/binfmt_elf.c +1 -1
index eabd7e0f6592..c1620358ab77 100644
@@ -580,7 +580,7 @@ static inline int make_prot(u32 p_flags, struct arch_elf_state *arch_state,
 	if (p_flags & PF_X)
 		prot |= PROT_EXEC;
 
-	return arch_elf_adjust_prot(prot, arch_state, has_interp, is_interp);
+	return arch_elf_adjust_prot(prot, arch_state, is_interp);
 }
 
 /* This is much more generalized than the library routine read function,
include/linux/elf.h +2 -2
index 1c45ecf29147..d8392531899d 100644
@@ -101,11 +101,11 @@ extern int arch_parse_elf_property(u32 type, const void *data, size_t datasz,
 
 #ifdef CONFIG_ARCH_HAVE_ELF_PROT
 int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,
-			 bool has_interp, bool is_interp);
+			 bool is_interp);
 #else
 static inline int arch_elf_adjust_prot(int prot,
 				       const struct arch_elf_state *state,
-				       bool has_interp, bool is_interp)
+				       bool is_interp)
 {
 	return prot;
 }
Signature · 1 line
2.30.2
maintainer Reviewed-by on v8 3/4
#6 lore raw

Re: [PATCH v8 3/4] elf: Remove has_interp property from arch_adjust_elf_prot()

Quoted 9 lines · Mark Brown

Since we have added an is_interp flag to arch_parse_elf_property() we can drop the has_interp flag from arch_elf_adjust_prot(), the only user was the arm64 code which no longer needs it and any future users will be able to use arch_parse_elf_properties() to determine if an interpreter is in use.

Signed-off-by Mark Brown <[email protected]>
Tested-by Jeremy Linton <[email protected]>
Reviewed-by Dave Martin <[email protected]>
Reviewed-by Catalin Marinas <[email protected]>
author
#7 lore raw

PATCH v8 4/4 elf: Remove has_interp property from arch_parse_elf_property()

In reply to Mark Brown

Since all current users of arch_parse_elf_property() are able to treat the interpreter and main executable orthogonaly the has_interp argument is now redundant so remove it.

Signed-off-by Mark Brown <[email protected]>
Tested-by Jeremy Linton <[email protected]>
Reviewed-by Dave Martin <[email protected]>
---
 arch/arm64/include/asm/elf.h | 2 +-
 fs/binfmt_elf.c              | 2 +-
 include/linux/elf.h          | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)
arch/arm64/include/asm/elf.h +1 -1
index c4aa60db76a4..8009e8f07f1e 100644
@@ -268,7 +268,7 @@ static inline int arm64_elf_bti_flag(bool is_interp)
 
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
-					  bool has_interp, bool is_interp,
+					  bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	/* No known properties for AArch32 yet */
fs/binfmt_elf.c +1 -1
index c1620358ab77..caedfdcead61 100644
@@ -753,7 +753,7 @@ static int parse_elf_property(const char *data, size_t *off, size_t datasz,
 
 	ret = arch_parse_elf_property(pr->pr_type, data + o,
 				      pr->pr_datasz, ELF_COMPAT,
-				      has_interp, is_interp, arch);
+				      is_interp, arch);
 	if (ret)
 		return ret;
include/linux/elf.h +2 -2
index d8392531899d..cdb080d4b34a 100644
@@ -88,14 +88,14 @@ struct arch_elf_state;
 #ifndef CONFIG_ARCH_USE_GNU_PROPERTY
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
-					  bool has_interp, bool is_interp,
+					  bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	return 0;
 }
 #else
 extern int arch_parse_elf_property(u32 type, const void *data, size_t datasz,
-				   bool compat, bool has_interp, bool is_interp,
+				   bool compat, bool is_interp,
 				   struct arch_elf_state *arch);
 #endif
Signature · 1 line
2.30.2
maintainer Reviewed-by on v8 4/4
#8 lore raw

Re: [PATCH v8 4/4] elf: Remove has_interp property from arch_parse_elf_property()

Quoted 7 lines · Mark Brown

Since all current users of arch_parse_elf_property() are able to treat the interpreter and main executable orthogonaly the has_interp argument is now redundant so remove it.

Signed-off-by Mark Brown <[email protected]>
Tested-by Jeremy Linton <[email protected]>
Reviewed-by Dave Martin <[email protected]>
Reviewed-by Catalin Marinas <[email protected]>
maintainer
#9 lore raw
In reply to Mark Brown
Quoted 8 lines · Mark Brown

Deployments of BTI on arm64 have run into issues interacting with systemd's MemoryDenyWriteExecute feature. Currently for dynamically linked executables the kernel will only handle architecture specific properties like BTI for the interpreter, the expectation is that the interpreter will then handle any properties on the main executable. For BTI this means remapping the executable segments PROT_EXEC | PROT_BTI.

Quoted 6 lines

This interacts poorly with MemoryDenyWriteExecute since that is implemented using a seccomp filter which prevents setting PROT_EXEC on already mapped memory and lacks the context to be able to detect that memory is already mapped with PROT_EXEC. This series resolves this by handling the BTI property for both the interpreter and the main executable.

This appears to be a user-visible change which cannot be detected or disabled from userspace. If there is code out there which does not work when BTI is enabled, won't that now explode when the kernel enables it? How are we supposed to handle such a regression?

Will

maintainer
#10 lore raw
Quoted 16 lines · Will Deacon
Quoted 8 lines · Mark Brown

Deployments of BTI on arm64 have run into issues interacting with systemd's MemoryDenyWriteExecute feature. Currently for dynamically linked executables the kernel will only handle architecture specific properties like BTI for the interpreter, the expectation is that the interpreter will then handle any properties on the main executable. For BTI this means remapping the executable segments PROT_EXEC | PROT_BTI.

Quoted 6 lines

This interacts poorly with MemoryDenyWriteExecute since that is implemented using a seccomp filter which prevents setting PROT_EXEC on already mapped memory and lacks the context to be able to detect that memory is already mapped with PROT_EXEC. This series resolves this by handling the BTI property for both the interpreter and the main executable.

Quoted 4 lines

This appears to be a user-visible change which cannot be detected or disabled from userspace. If there is code out there which does not work when BTI is enabled, won't that now explode when the kernel enables it? How are we supposed to handle such a regression?

If this ever happens, the only workaround is to disable BTI on the kernel command line. If we need a knob closer to user, we could add a sysctl option (as we did for the tagged address ABI, though I doubt people are even aware that exists). The dynamic loader doesn't do anything smart when deciding to map objects with PROT_BTI (like env variables), it simply relies on the ELF information.

I think that's very unlikely and feedback from Szabolcs in the past and additional testing by Mark and Jeremy was that it should be fine. The architecture allows interworking between BTI and non-BTI objects and on distros with both BTI and MDWE enabled, this is already the case: the main executable is mapped without PROT_BTI while the libraries will be mapped with PROT_BTI. The new behaviour allows both to be mapped with PROT_BTI, just as if MDWE was disabled.

I think the only difference would be with a BTI-unware dynamic loader (e.g. older distro). Here the main executable, if compiled with BTI, would be mapped as executable while the rest of the libraries are non-BTI. The interworking should be fine but we can't test everything since such BTI binaries would not normally be part of the distro.

If there are dodgy libraries out there that do tricks and branch into the middle of a function in the main executable, they will fail with this series but also fail if MDWE is disabled and the dynamic linker is BTI-aware. So this hardly counts as a use-case.

For consistency, I think whoever does the initial mapping should also set the correct attributes as we do for static binaries. If you think another knob is needed other than the cmdline, I'm fine with it.

Signature · 1 line
Catalin
#11 lore raw
Quoted 21 lines · Catalin Marinas via Libc-alpha
Quoted 4 lines · Will Deacon

This appears to be a user-visible change which cannot be detected or disabled from userspace. If there is code out there which does not work when BTI is enabled, won't that now explode when the kernel enables it? How are we supposed to handle such a regression?

If this ever happens, the only workaround is to disable BTI on the kernel command line. If we need a knob closer to user, we could add a sysctl option (as we did for the tagged address ABI, though I doubt people are even aware that exists). The dynamic loader doesn't do anything smart when deciding to map objects with PROT_BTI (like env variables), it simply relies on the ELF information.

I think that's very unlikely and feedback from Szabolcs in the past and additional testing by Mark and Jeremy was that it should be fine. The architecture allows interworking between BTI and non-BTI objects and on distros with both BTI and MDWE enabled, this is already the case: the main executable is mapped without PROT_BTI while the libraries will be mapped with PROT_BTI. The new behaviour allows both to be mapped with PROT_BTI, just as if MDWE was disabled.

Quoted 5 lines

I think the only difference would be with a BTI-unware dynamic loader (e.g. older distro). Here the main executable, if compiled with BTI, would be mapped as executable while the rest of the libraries are non-BTI. The interworking should be fine but we can't test everything since such BTI binaries would not normally be part of the distro.

note: a bti marked exe has to be built against a bti enabled runtime (so crt files etc are bti compatible) and any system that executes such an exe must be newer and thus bti aware (according to glibc abi compatibility rules).

so while it's possible that ld.so is not bti marked when an exe is, the ld.so will be at least bti aware, which means it will map all bti binaries (including the exe) with PROT_BTI.

so this doesn't really affect glibc based systems. and there are not many other systems that can produce bti marked exes. (i don't think this can cause problems on android either)

if we ever wanted to map bti marked binaries without PROT_BTI and introduced a knob to do that in ld.so, then this change would be problematic (we cannot easily remove PROT_BTI from the exe), but we don't have such plans.

Quoted 12 lines

If there are dodgy libraries out there that do tricks and branch into the middle of a function in the main executable, they will fail with this series but also fail if MDWE is disabled and the dynamic linker is BTI-aware. So this hardly counts as a use-case.

For consistency, I think whoever does the initial mapping should also set the correct attributes as we do for static binaries. If you think another knob is needed other than the cmdline, I'm fine with it.

Signature · 1 line
Catalin
author
#12 lore raw
Quoted 4 lines · Szabolcs Nagy

if we ever wanted to map bti marked binaries without PROT_BTI and introduced a knob to do that in ld.so, then this change would be problematic (we cannot easily remove PROT_BTI from the exe), but we don't have such plans.

In general or only in the case where MWDE is enabled (in which case it's the same problem as exists today trying to enable BTI in the presence of MWDE)?

#13 lore raw
Quoted 10 lines · Mark Brown
Quoted 4 lines · Szabolcs Nagy

if we ever wanted to map bti marked binaries without PROT_BTI and introduced a knob to do that in ld.so, then this change would be problematic (we cannot easily remove PROT_BTI from the exe), but we don't have such plans.

In general or only in the case where MWDE is enabled (in which case it's the same problem as exists today trying to enable BTI in the presence of MWDE)?

ah yes with mwde only.

author
#14 lore raw
In reply to Catalin Marinas
Quoted 7 lines · Catalin Marinas
Quoted 4 lines · Will Deacon

This appears to be a user-visible change which cannot be detected or disabled from userspace. If there is code out there which does not work when BTI is enabled, won't that now explode when the kernel enables it? How are we supposed to handle such a regression?

Quoted 6 lines

If this ever happens, the only workaround is to disable BTI on the kernel command line. If we need a knob closer to user, we could add a sysctl option (as we did for the tagged address ABI, though I doubt people are even aware that exists). The dynamic loader doesn't do anything smart when deciding to map objects with PROT_BTI (like env variables), it simply relies on the ELF information.

The dynamic loader is the place where I'd expect to do any per-executable workarounds, but currently that's not actually implemented anywhere. Someone could also make a tool to strip BTI annotations from executables.

Quoted 10 lines

I think the only difference would be with a BTI-unware dynamic loader (e.g. older distro). Here the main executable, if compiled with BTI, would be mapped as executable while the rest of the libraries are non-BTI. The interworking should be fine but we can't test everything since such BTI binaries would not normally be part of the distro.

If there are dodgy libraries out there that do tricks and branch into the middle of a function in the main executable, they will fail with this series but also fail if MDWE is disabled and the dynamic linker is BTI-aware. So this hardly counts as a use-case.

I'm not aware of any issues we've run into which are due to interworking between binaries rather than within a binary due to either miscompilation or doing something in hand coded assembler that needs updating for BTI. It doesn't mean it can't happen but it's hard to see what people might be doing.

Quoted 3 lines

For consistency, I think whoever does the initial mapping should also set the correct attributes as we do for static binaries. If you think another knob is needed other than the cmdline, I'm fine with it.

Might also be worth pointing out that we already map the vDSO with BTI enabled if it's built with BTI.

maintainer
#15 lore raw
In reply to Catalin Marinas
Quoted 43 lines · Catalin Marinas
Quoted 16 lines · Will Deacon
Quoted 8 lines · Mark Brown

Deployments of BTI on arm64 have run into issues interacting with systemd's MemoryDenyWriteExecute feature. Currently for dynamically linked executables the kernel will only handle architecture specific properties like BTI for the interpreter, the expectation is that the interpreter will then handle any properties on the main executable. For BTI this means remapping the executable segments PROT_EXEC | PROT_BTI.

Quoted 6 lines

This interacts poorly with MemoryDenyWriteExecute since that is implemented using a seccomp filter which prevents setting PROT_EXEC on already mapped memory and lacks the context to be able to detect that memory is already mapped with PROT_EXEC. This series resolves this by handling the BTI property for both the interpreter and the main executable.

Quoted 4 lines

This appears to be a user-visible change which cannot be detected or disabled from userspace. If there is code out there which does not work when BTI is enabled, won't that now explode when the kernel enables it? How are we supposed to handle such a regression?

If this ever happens, the only workaround is to disable BTI on the kernel command line. If we need a knob closer to user, we could add a sysctl option (as we did for the tagged address ABI, though I doubt people are even aware that exists). The dynamic loader doesn't do anything smart when deciding to map objects with PROT_BTI (like env variables), it simply relies on the ELF information.

I think that's very unlikely and feedback from Szabolcs in the past and additional testing by Mark and Jeremy was that it should be fine. The architecture allows interworking between BTI and non-BTI objects and on distros with both BTI and MDWE enabled, this is already the case: the main executable is mapped without PROT_BTI while the libraries will be mapped with PROT_BTI. The new behaviour allows both to be mapped with PROT_BTI, just as if MDWE was disabled.

I think the only difference would be with a BTI-unware dynamic loader (e.g. older distro). Here the main executable, if compiled with BTI, would be mapped as executable while the rest of the libraries are non-BTI. The interworking should be fine but we can't test everything since such BTI binaries would not normally be part of the distro.

Quoted 8 lines

If there are dodgy libraries out there that do tricks and branch into the middle of a function in the main executable, they will fail with this series but also fail if MDWE is disabled and the dynamic linker is BTI-aware. So this hardly counts as a use-case.

For consistency, I think whoever does the initial mapping should also set the correct attributes as we do for static binaries. If you think another knob is needed other than the cmdline, I'm fine with it.

I still think this new behaviour should be opt-in, so adding a sysctl for that would be my preference if we proceed with this approach.

Will

author
#16 lore raw
Quoted 2 lines · Will Deacon

I still think this new behaviour should be opt-in, so adding a sysctl for that would be my preference if we proceed with this approach.

I'm happy to have a sysctl but I'd rather it be opt out rather than opt in since it seems better to default to enabling the security feature when there is a strong expectation that it would seem better to enable it by default sine it's not expected to be disruptive and the sysctl is more of a "what if there's a problem" thing.

maintainer
#17 lore raw
Quoted 10 lines · Mark Brown
Quoted 2 lines · Will Deacon

I still think this new behaviour should be opt-in, so adding a sysctl for that would be my preference if we proceed with this approach.

I'm happy to have a sysctl but I'd rather it be opt out rather than opt in since it seems better to default to enabling the security feature when there is a strong expectation that it would seem better to enable it by default sine it's not expected to be disruptive and the sysctl is more of a "what if there's a problem" thing.

I think new behaviour has to be opt-in, so that if somebody takes a new kernel then we can guarantee it's not going to break them. Systemd can enable this unconditionally if it wants to.

Will