Skip to contentSkip to conversation

tools/nolibc: HACK!: Add basic self relocation for static PIE for m68k nommu FDPIC

RFC PATCH In review Daniel Palmer 3 participants 4 messages Jan 16 – Jan 19 Nolibc header file

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 tools/nolibc: HACK!: Add basic self relocation for static PIE for m68k nommu FDPIC

This is some very quick hacky code to test if this works and get some ideas..

- I'm messing with m68k nommu. Currently I use FLAT binaries and this is working with nolibc as-is mostly. Sometimes some relocations that elf2flat doesn't like get generated and the resulting FLAT binary doesn't have any relocation information and crashes which isn't good if you don't have an mmu.

- Since commit 1bde925d2354 ("fs/binfmt_elf_fdpic.c: provide NOMMU loader for regular ELF binaries") the FDPIC loader has apparently been able to load non-FDPIC binaries as long as they are PIE and can be relocated. I have been messing with this thinking that maybe I can stop using FLAT binaries.

- By default linking with -pie is trying to set ld.so as the interpreter to do the relocation. I don't think I have anything that can do that in my system. I am using uclibc but statically linked. Aside from my programs written with nolibc there is a busybox FLAT that is statically linked to uclibc and nothing else.

Eitherway, the plan is not to have any libc and have everything compiled with nolibc. I'm writing a small init, shell etc with nolibc that will replace busybox and not cause constant OOMs.

- So, I can generate PIE binaries but they can't work because I have no linker to relocate them but apparently static PIE is a thing and with a normal toolchain you'd get a crt that does the relocation before jumping to main().

- I thought it shouldn't be too hard to add something like that to crt.h in nolibc and then pass --no-dynamic-linker when linking to not set an interpreter.

- I got it working enough that a static pie "hello, world" loads and runs:

/ # /root/test.elf

[    9.970000] FDPIC ____ LOAD 23 ____
[    9.970000] FDPIC Mapped Object [executable]:
[    9.970000] FDPIC - elfhdr   : 6d8000
[    9.970000] FDPIC - entry    : 6d83e4
[    9.970000] FDPIC - PHDR[]   : 6d8034
[    9.970000] FDPIC - DYNAMIC[]: 6da7b0
[    9.970000] FDPIC - LOAD[0] : 006d8000-006d87ad [va=0 ms=7ae]
[    9.970000] FDPIC - LOAD[1] : 006da7b0-006da873 [va=27b0 ms=c4]
[    9.970000] FDPIC - start_code  6d8000
[    9.970000] FDPIC - end_code    6d87ae
[    9.970000] FDPIC - start_data  6da7b0
[    9.970000] FDPIC - end_data    6da874
[    9.970000] FDPIC - start_brk   6e0000
[    9.970000] FDPIC - brk         6e0000
[    9.970000] FDPIC - start_stack 6fff00
hello, world!
[    9.980000] test.elf (23) used greatest stack depth: 5348 bytes left

Questions:

- My use case is weird/niche but maybe there are uses for static pie nolibc binaries?
- If so what would be a cleaner way of implementing this?

- Right now the base address offset all of the relocations against is hardcoded. Maybe someone knows how I'm meant to get that properly?

---
 tools/include/nolibc/crt.h | 56 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
tools/include/nolibc/crt.h +56 -0
index d9262998dae9..0931915280d8 100644
@@ -10,6 +10,7 @@
 #ifndef NOLIBC_NO_RUNTIME
 
 #include "compiler.h"
+#include "elf.h"
 
 char **environ __attribute__((weak));
 const unsigned long *_auxv __attribute__((weak));
@@ -47,6 +48,61 @@ void _start_c(long *sp)
 	/* initialize stack protector */
 	__stack_chk_init();
 
+#ifdef NOLIBC_STATIC_PIE
+#define R_68K_RELATIVE	22
+{
+	void *base = (void *) 0x6d8000; // TODO: how to actually get this?
+	unsigned int rela_count = 0;
+	unsigned int rela_off = 0;
+	unsigned long dyn_addr;
+	Elf32_Rela *rela;
+	Elf32_Addr *addr;
+	Elf32_Dyn *dyn;
+	int i;
+
+	/* For m68k with the FDPIC loader d5 contains the offset to the DYNAMIC segment */
+	__asm__ volatile (
+		"move.l %%d5, %0\n"
+		: "=r" (dyn_addr)
+	);
+	dyn = (Elf32_Dyn *) dyn_addr;
+
+	/* Go through the DYNAMIC segment and get the offset to rela and the number of relocations */
+	for (; dyn->d_tag != DT_NULL; dyn++) {
+		switch (dyn->d_tag) {
+		case DT_RELA:
+			rela_off = dyn->d_un.d_ptr;
+			break;
+		case DT_RELACOUNT:
+			rela_count = dyn->d_un.d_val;
+			break;
+		}
+	}
+
+	if (!rela_off || !rela_count)
+		exit(42); //TODO nonsense error
+
+	rela = base + rela_off;
+
+	/* Do the relocations, only R_68K_RELATIVE for now */
+	for (i = 0; i < rela_count; i++) {
+		Elf32_Rela *entry = &rela[i];
+
+		switch (ELF32_R_TYPE(entry->r_info)) {
+		case R_68K_RELATIVE:
+		{
+			addr = (Elf32_Addr *)(base + entry->r_offset);
+			*addr = (Elf32_Addr) (base + entry->r_addend);
+		}
+			break;
+		default:
+			exit(43); //TODO nonsense error
+			break;
+		}
+	}
+}
+#endif
+
 	/*
 	 * sp  :    argc          <-- argument count, required by main()
 	 * argv:    argv[0]       <-- argument vector, required by main()
Signature · 1 line
2.51.0
#1 lore raw

Hi Daniel,

Quoted 12 lines · Daniel Palmer

This is some very quick hacky code to test if this works and get some ideas..

- I'm messing with m68k nommu. Currently I use FLAT binaries and this is working with nolibc as-is mostly. Sometimes some relocations that elf2flat doesn't like get generated and the resulting FLAT binary doesn't have any relocation information and crashes which isn't good if you don't have an mmu.

- Since commit 1bde925d2354 ("fs/binfmt_elf_fdpic.c: provide NOMMU loader for regular ELF binaries") the FDPIC loader has apparently been able to load non-FDPIC binaries as long as they are PIE and can be relocated. I have been messing with this thinking that maybe I can stop using FLAT binaries.

Quoted 4 lines

- By default linking with -pie is trying to set ld.so as the interpreter to do the relocation. I don't think I have anything that can do that in my system. I am using uclibc but statically linked. Aside from my programs written with nolibc there is a busybox FLAT that is statically linked to uclibc and nothing else.

If you are using uClibc-ng as the C library and you select this config:

UCLIBC_FORMAT_ELF=y

instead of the UCLIBC_FORMAT_FLAT* options then it will generate the usual libc.a static library, and also a run time linker ld-uClibc.so. That will do the usual link/load at application run time.

Here is a script I use to test it all, generating for the mcf5208 nommu qemu target:

https://github.com/gregungerer/simple-linux/blob/master/build-m68knommu-linux-uclibc-elf.sh

Quoted 6 lines

Eitherway, the plan is not to have any libc and have everything compiled with nolibc. I'm writing a small init, shell etc with nolibc that will replace busybox and not cause constant OOMs.

- So, I can generate PIE binaries but they can't work because I have no linker to relocate them but apparently static PIE is a thing and with a normal toolchain you'd get a crt that does the relocation before jumping to main().

I messed around a few years back with a really basic stand alone link loader that might be useful:

https://github.com/gregungerer/uldso

It probably doesn't work for all possible ELF files, but it worked good enough to load and run busybox for example. And I only used uClibc-ng as the C library too. I never tried with nolibc.

Quoted 2 lines

- I thought it shouldn't be too hard to add something like that to crt.h in nolibc and then pass --no-dynamic-linker when linking to not set an interpreter.

Something like the uldso above avoids adding anything extra to the start up code, since it runs as the interpreter. That feels like a better solution.

Regards
Greg

Quoted 108 lines

- I got it working enough that a static pie "hello, world" loads and runs:

/ # /root/test.elf

[    9.970000] FDPIC ____ LOAD 23 ____
[    9.970000] FDPIC Mapped Object [executable]:
[    9.970000] FDPIC - elfhdr   : 6d8000
[    9.970000] FDPIC - entry    : 6d83e4
[    9.970000] FDPIC - PHDR[]   : 6d8034
[    9.970000] FDPIC - DYNAMIC[]: 6da7b0
[    9.970000] FDPIC - LOAD[0] : 006d8000-006d87ad [va=0 ms=7ae]
[    9.970000] FDPIC - LOAD[1] : 006da7b0-006da873 [va=27b0 ms=c4]
[    9.970000] FDPIC - start_code  6d8000
[    9.970000] FDPIC - end_code    6d87ae
[    9.970000] FDPIC - start_data  6da7b0
[    9.970000] FDPIC - end_data    6da874
[    9.970000] FDPIC - start_brk   6e0000
[    9.970000] FDPIC - brk         6e0000
[    9.970000] FDPIC - start_stack 6fff00
hello, world!
[    9.980000] test.elf (23) used greatest stack depth: 5348 bytes left

Questions:

- My use case is weird/niche but maybe there are uses for static pie nolibc binaries?
- If so what would be a cleaner way of implementing this?

- Right now the base address offset all of the relocations against is hardcoded. Maybe someone knows how I'm meant to get that properly?

---
  tools/include/nolibc/crt.h | 56 ++++++++++++++++++++++++++++++++++++++
  1 file changed, 56 insertions(+)
tools/include/nolibc/crt.h +56 -0
index d9262998dae9..0931915280d8 100644
@@ -10,6 +10,7 @@
  #ifndef NOLIBC_NO_RUNTIME
 
  #include "compiler.h"
+#include "elf.h"
 
  char **environ __attribute__((weak));
  const unsigned long *_auxv __attribute__((weak));
@@ -47,6 +48,61 @@ void _start_c(long *sp)
  	/* initialize stack protector */
  	__stack_chk_init();
 
+#ifdef NOLIBC_STATIC_PIE
+#define R_68K_RELATIVE	22
+{
+	void *base = (void *) 0x6d8000; // TODO: how to actually get this?
+	unsigned int rela_count = 0;
+	unsigned int rela_off = 0;
+	unsigned long dyn_addr;
+	Elf32_Rela *rela;
+	Elf32_Addr *addr;
+	Elf32_Dyn *dyn;
+	int i;
+
+	/* For m68k with the FDPIC loader d5 contains the offset to the DYNAMIC segment */
+	__asm__ volatile (
+		"move.l %%d5, %0\n"
+		: "=r" (dyn_addr)
+	);
+	dyn = (Elf32_Dyn *) dyn_addr;
+
+	/* Go through the DYNAMIC segment and get the offset to rela and the number of relocations */
+	for (; dyn->d_tag != DT_NULL; dyn++) {
+		switch (dyn->d_tag) {
+		case DT_RELA:
+			rela_off = dyn->d_un.d_ptr;
+			break;
+		case DT_RELACOUNT:
+			rela_count = dyn->d_un.d_val;
+			break;
+		}
+	}
+
+	if (!rela_off || !rela_count)
+		exit(42); //TODO nonsense error
+
+	rela = base + rela_off;
+
+	/* Do the relocations, only R_68K_RELATIVE for now */
+	for (i = 0; i < rela_count; i++) {
+		Elf32_Rela *entry = &rela[i];
+
+		switch (ELF32_R_TYPE(entry->r_info)) {
+		case R_68K_RELATIVE:
+		{
+			addr = (Elf32_Addr *)(base + entry->r_offset);
+			*addr = (Elf32_Addr) (base + entry->r_addend);
+		}
+			break;
+		default:
+			exit(43); //TODO nonsense error
+			break;
+		}
+	}
+}
+#endif
+
  	/*
  	 * sp  :    argc          <-- argument count, required by main()
  	 * argv:    argv[0]       <-- argument vector, required by main()
#2 lore raw
In reply to Daniel Palmer

Hi Daniel,

Quoted 12 lines · Daniel Palmer

This is some very quick hacky code to test if this works and get some ideas..

- I'm messing with m68k nommu. Currently I use FLAT binaries and this is working with nolibc as-is mostly. Sometimes some relocations that elf2flat doesn't like get generated and the resulting FLAT binary doesn't have any relocation information and crashes which isn't good if you don't have an mmu.

- Since commit 1bde925d2354 ("fs/binfmt_elf_fdpic.c: provide NOMMU loader for regular ELF binaries") the FDPIC loader has apparently been able to load non-FDPIC binaries as long as they are PIE and can be relocated. I have been messing with this thinking that maybe I can stop using FLAT binaries.

Quoted 7 lines

- By default linking with -pie is trying to set ld.so as the interpreter to do the relocation. I don't think I have anything that can do that in my system. I am using uclibc but statically linked. Aside from my programs written with nolibc there is a busybox FLAT that is statically linked to uclibc and nothing else.

Eitherway, the plan is not to have any libc and have everything compiled with nolibc. I'm writing a small init, shell etc with nolibc that will replace busybox and not cause constant OOMs.

So far so good.

Quoted 19 lines

- So, I can generate PIE binaries but they can't work because I have no linker to relocate them but apparently static PIE is a thing and with a normal toolchain you'd get a crt that does the relocation before jumping to main().

- I thought it shouldn't be too hard to add something like that to crt.h in nolibc and then pass --no-dynamic-linker when linking to not set an interpreter.

- I got it working enough that a static pie "hello, world" loads and runs:

/ # /root/test.elf

[    9.970000] FDPIC ____ LOAD 23 ____
[    9.970000] FDPIC Mapped Object [executable]:
[    9.970000] FDPIC - elfhdr   : 6d8000
[    9.970000] FDPIC - entry    : 6d83e4
[    9.970000] FDPIC - PHDR[]   : 6d8034
[    9.970000] FDPIC - DYNAMIC[]: 6da7b0
[    9.970000] FDPIC - LOAD[0] : 006d8000-006d87ad [va=0 ms=7ae]
[    9.970000] FDPIC - LOAD[1] : 006da7b0-006da873 [va=27b0 ms=c4]
[    9.970000] FDPIC - start_code  6d8000
Quoted 8 lines
[    9.970000] FDPIC - end_code    6d87ae
[    9.970000] FDPIC - start_data  6da7b0
[    9.970000] FDPIC - end_data    6da874
[    9.970000] FDPIC - start_brk   6e0000
[    9.970000] FDPIC - brk         6e0000
[    9.970000] FDPIC - start_stack 6fff00
hello, world!
[    9.980000] test.elf (23) used greatest stack depth: 5348 bytes left

Nice! Is it sufficient for nolibc-test? If so, can you provide instructions for that?

Quoted 3 lines

Questions:

- My use case is weird/niche but maybe there are uses for static pie nolibc binaries?

I think it is a valid usecase to support nommu systems. Also from a security perspective, pie should be better. While I would like to defer to the 'uldso' tool from Greg, nolibc positions itself as usable for single-binary systems. Requiring a different tool would run a bit counter to that.
In my opinion it depends on the implementation complexity. If we can have a simple implementation shared by all architectures, why not.

Quoted 1 line

- If so what would be a cleaner way of implementing this?

Some comments inline.

Quoted 2 lines

- Right now the base address offset all of the relocations against is hardcoded. Maybe someone knows how I'm meant to get that properly?

That should be getauxval(AT_BASE), I think.

Quoted 18 lines
---
 tools/include/nolibc/crt.h | 56 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
tools/include/nolibc/crt.h +1 -0
index d9262998dae9..0931915280d8 100644
@@ -10,6 +10,7 @@
 #ifndef NOLIBC_NO_RUNTIME
 
 #include "compiler.h"
+#include "elf.h"
 
 char **environ __attribute__((weak));
 const unsigned long *_auxv __attribute__((weak));

tools/include/nolibc/crt.h void _start_c(long *sp) · line 51

Quoted 4 lines
tools/include/nolibc/crt.h +1 -0
@@ -47,6 +48,61 @@ void _start_c(long *sp)
 	/* initialize stack protector */
 	__stack_chk_init();
 
+#ifdef NOLIBC_STATIC_PIE

It might be possible to detect this automatically based on the preprocessor symbols __pie__/__PIE__.

tools/include/nolibc/crt.h void _start_c(long *sp) · line 52

Quoted 1 line
tools/include/nolibc/crt.h +1 -0
+#define R_68K_RELATIVE	22

These definition should be moved to the UAPI headers. See my related changes I did for some other architectures:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

It means that nolibc would require new UAPI headers for pie support, but given that pie didn't work before at all, that should be fine.

tools/include/nolibc/crt.h void _start_c(long *sp) · line 54

Quoted 2 lines
tools/include/nolibc/crt.h +2 -0
+{
+	void *base = (void *) 0x6d8000; // TODO: how to actually get this?

getauxval(AT_BASE)

tools/include/nolibc/crt.h void _start_c(long *sp) · line 67

Quoted 13 lines
tools/include/nolibc/crt.h +12 -0
+	unsigned int rela_count = 0;
+	unsigned int rela_off = 0;
+	unsigned long dyn_addr;
+	Elf32_Rela *rela;
+	Elf32_Addr *addr;
+	Elf32_Dyn *dyn;
+	int i;
 
+	/* For m68k with the FDPIC loader d5 contains the offset to the DYNAMIC segment */
+	__asm__ volatile (
+		"move.l %%d5, %0\n"
+		: "=r" (dyn_addr)
+	);

This should go into arch-m68k.h.
Is %d5 guaranteed not to have been clobbered by now? If not we need to pass it in from the asm __start().

Quoted 6 lines
tools/include/nolibc/crt.h +6 -0
+	dyn = (Elf32_Dyn *) dyn_addr;
+
+	/* Go through the DYNAMIC segment and get the offset to rela and the number of relocations */
+	for (; dyn->d_tag != DT_NULL; dyn++) {
+		switch (dyn->d_tag) {
+		case DT_RELA:

tools/include/nolibc/crt.h void _start_c(long *sp) · line 83

Quoted 10 lines
tools/include/nolibc/crt.h +10 -0
+			rela_off = dyn->d_un.d_ptr;
+			break;
+		case DT_RELACOUNT:
+			rela_count = dyn->d_un.d_val;
+			break;
+		}
+	}
+
+	if (!rela_off || !rela_count)
+		exit(42); //TODO nonsense error

Maybe __builtin_trap()/__nolibc_trap()?

tools/include/nolibc/crt.h void _start_c(long *sp) · line 87

Quoted 4 lines
tools/include/nolibc/crt.h +4 -0
+
+	rela = base + rela_off;
+
+	/* Do the relocations, only R_68K_RELATIVE for now */

Are there any more that would need to be supported? How many are there for other architectures? (My knowledge here is limited)

tools/include/nolibc/crt.h void _start_c(long *sp) · line 95

Quoted 8 lines
tools/include/nolibc/crt.h +8 -0
+	for (i = 0; i < rela_count; i++) {
+		Elf32_Rela *entry = &rela[i];
+
+		switch (ELF32_R_TYPE(entry->r_info)) {
+		case R_68K_RELATIVE:
+		{
+			addr = (Elf32_Addr *)(base + entry->r_offset);
+			*addr = (Elf32_Addr) (base + entry->r_addend);

Can these point into read-only memory?
Do we need to fiddle with mprotect()?

tools/include/nolibc/crt.h void _start_c(long *sp) · line 104

Quoted 9 lines
tools/include/nolibc/crt.h +9 -0
+		}
+			break;
+		default:
+			exit(43); //TODO nonsense error
+			break;
+		}
+	}
+}
+#endif

This whole block should go into a dedicated file/function.

tools/include/nolibc/crt.h void _start_c(long *sp) · line 108

Quoted 4 lines
tools/include/nolibc/crt.h +1 -0
+
 	/*
 	 * sp  :    argc          <-- argument count, required by main()
 	 * argv:    argv[0]       <-- argument vector, required by main()

All in all, if we can keep the complexity for a shared implementation at the level of the code above, then I am in favor of implementing this in nolibc.

Willy, your opinion?

Thomas

#3 lore raw

Hi Thomas,

Quoted 2 lines · Thomas Weißschuh

Nice! Is it sufficient for nolibc-test? If so, can you provide instructions for that?

Not yet. I will try it and get back to you.

Quoted 6 lines
Quoted 3 lines

Questions:

- My use case is weird/niche but maybe there are uses for static pie nolibc binaries?

While I would like to defer to the 'uldso' tool from Greg, nolibc positions itself as usable for single-binary systems. Requiring a different tool would run a bit counter to that.

I tried Greg's uldso and I couldn't get it working just yet but I think I can get that working.
But I think having something in the kernel source would also be handy. I worked out how to use it with the prebuilt m68k toolchain on kernel.org too so everything needed to build nommu binaries could be sourced from kernel.org.

Quoted 5 lines

In my opinion it depends on the implementation complexity. If we can have a simple implementation shared by all architectures, why not.

Quoted 1 line

- If so what would be a cleaner way of implementing this?

Some comments inline.

Thanks. I'll get this working on amd64 so we can see how much arch specific stuff there is, address the comments and send again maybe later this week.

Daniel