Skip to contentSkip to conversation

Link the relocatable x86 kernel as PIE

PATCH Applied Ard Biesheuvel 10 participants 54 messages Jan 8 – Mar 9

Subsystems (19)
x86 ACPI Library code HID +15 more subsystems
lore
Share & tools
lore

Ard Biesheuvel

author maintainer
#0 lore raw

cover Link the relocatable x86 kernel as PIE

This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].

The main sticking point is the fact that PIE linking on x86_64 requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.

This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)

So now that we have some actual numbers, I would like to try and revisit this discussion, and get a conclusion on whether this is really a non-starter. Note that only the KASLR kernel would rely on this, and disabling CONFIG_RANDOMIZE_BASE will revert to the current situation (provided that patch #4 is applied)

Some minor asm tweaks are needed too (patches #9 - #17), but those all seem uncontroversial to me.

The first 5 patches are general cleanup, and could be taken into consideration independently of the discussion around PIC codegen.

[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).

[2] EFI zboot is already used by various architectures that have no decompressor stage at all (arm64, RISC-V, LoongArch), and this format can be combined with an ELF payload too. EFI zboot accommodates non-EFI boot chains by describing the size, offset, payload type and compression type in its header, so that it can be extracted and booted by other means.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Dave Hansen <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Josh Poimboeuf <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kees Cook <[email protected]>
Cc Uros Bizjak <[email protected]>
Cc Brian Gerst <[email protected]>

Ard Biesheuvel (19):
x86/idt: Move idt_table to __ro_after_init section x86/sev: Don't emit BSS_DECRYPT section unless it is in use
x86: Combine .data with .bss in kernel mapping
x86: Make the 64-bit bzImage always physically relocatable x86/efistub: Simplify early remapping of kernel text
alloc_tag: Use __ prefixed ELF section names tools/objtool: Treat indirect ftrace calls as direct calls
x86: Use PIE codegen for the relocatable 64-bit kernel x86/pm-trace: Use RIP-relative accesses for .tracedata x86/kvm: Use RIP-relative addressing
x86/rethook: Use RIP-relative reference for fake return address x86/sync_core: Use RIP-relative addressing x86/entry_64: Use RIP-relative addressing x86/hibernate: Prefer RIP-relative accesses
x64/acpi: Use PIC-compatible references in wakeup_64.S x86/kexec: Use 64-bit wide absolute reference from relocated code x86/head64: Avoid absolute references in startup asm x86/boot: Implement support for RELA/RELR/REL runtime relocations x86/kernel: Switch to PIE linking for the relocatable kernel

 arch/x86/Kconfig                        |  45 ++++---
 arch/x86/Makefile                       |  24 +++-
 arch/x86/boot/Makefile                  |   1 +
 arch/x86/boot/compressed/Makefile       |   2 +-
 arch/x86/boot/compressed/head_64.S      |   4 -
 arch/x86/boot/compressed/misc.c         |  85 +++++++++++--
 arch/x86/boot/header.S                  |   8 +-
 arch/x86/entry/calling.h                |   9 +-
 arch/x86/entry/entry_64.S               |  14 +--
 arch/x86/entry/vdso/Makefile            |   1 +
 arch/x86/include/asm/boot.h             |   2 -
 arch/x86/include/asm/pm-trace.h         |   4 +-
 arch/x86/include/asm/sync_core.h        |   3 +-
 arch/x86/kernel/acpi/wakeup_64.S        |  11 +-
 arch/x86/kernel/head_64.S               |  17 +--
 arch/x86/kernel/idt.c                   |   5 +-
 arch/x86/kernel/kvm.c                   |   5 +-
 arch/x86/kernel/relocate_kernel_64.S    |   2 +-
 arch/x86/kernel/rethook.c               |   6 +-
 arch/x86/kernel/vmlinux.lds.S           | 132 ++++++++++++--------
 arch/x86/mm/init_64.c                   |   5 +-
 arch/x86/mm/pat/set_memory.c            |   2 +-
 arch/x86/power/hibernate_asm_64.S       |   4 +-
 arch/x86/realmode/rm/Makefile           |   1 +
 drivers/base/power/trace.c              |   6 +-
 drivers/firmware/efi/libstub/x86-stub.c |   4 +-
 include/asm-generic/codetag.lds.h       |  14 ++-
 include/asm-generic/vmlinux.lds.h       |   1 +
 include/linux/alloc_tag.h               |  11 +-
 include/linux/hidden.h                  |   2 +
 include/uapi/linux/elf.h                |   3 +
 lib/alloc_tag.c                         |   6 +-
 tools/objtool/check.c                   |  32 ++++-
 33 files changed, 315 insertions(+), 156 deletions(-)

base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8

Signature · 1 line
2.47.3
author maintainer
#1 lore raw

PATCH 1/19 x86/idt: Move idt_table to __ro_after_init section

Currently, idt_table is allocated as page-aligned .bss, and remapped read-only after init. This breaks a 2 MiB large page into 4k page mappings, which defeats some of the effort done at boot to map the kernel image using large pages, for improved TLB efficiency.

Mark this allocation as __ro_after_init instead, so it will be made read-only automatically after boot, without breaking up large page mappings.

This also fixes a latent bug on i386, where the size of idt_table is less than a page, and so remapping it read-only could potentially affect other read-write variables too, if those are not page-aligned as well.

---
 arch/x86/kernel/idt.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
arch/x86/kernel/idt.c +1 -4
index f445bec516a0..d6da25d7964f 100644
@@ -170,7 +170,7 @@ static const __initconst struct idt_data apic_idts[] = {
 };
 
 /* Must be page-aligned because the real IDT is used in the cpu entry area */
-static gate_desc idt_table[IDT_ENTRIES] __page_aligned_bss;
+static gate_desc idt_table[IDT_ENTRIES] __aligned(PAGE_SIZE) __ro_after_init;
 
 static struct desc_ptr idt_descr __ro_after_init = {
 	.size		= IDT_TABLE_SIZE - 1,
@@ -308,9 +308,6 @@ void __init idt_setup_apic_and_irq_gates(void)
 	idt_map_in_cea();
 	load_idt(&idt_descr);
 
-	/* Make the IDT table read only */
-	set_memory_ro((unsigned long)&idt_table, 1);
-
 	idt_setup_done = true;
 }
 
Signature · 1 line
2.47.3
author maintainer
#9 lore raw

PATCH 2/19 x86/sev: Don't emit BSS_DECRYPT section unless it is in use

In reply to Ard Biesheuvel

The BSS_DECRYPT section that gets emitted into .bss will be empty if CONFIG_AMD_MEM_ENCRYPT is not defined. However, due to the fact that it is injected into .bss rather than emitted as a separate section, the 2 MiB alignment that it specifies is still taken into account unconditionally, pushing .bss out to the next 2 MiB boundary, leaving a gap that is never freed.

So only emit a non-empty BSS_DECRYPT section if it is going to be used. In that case, it would still be nice to free the padding, but that is left for later.

---
 arch/x86/kernel/vmlinux.lds.S | 21 +++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
arch/x86/kernel/vmlinux.lds.S +12 -9
index d7af4a64c211..3a24a3fc55f5 100644
@@ -67,7 +67,18 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 
 #define ALIGN_ENTRY_TEXT_BEGIN	. = ALIGN(PMD_SIZE);
 #define ALIGN_ENTRY_TEXT_END	. = ALIGN(PMD_SIZE);
+#else
+
+#define X86_ALIGN_RODATA_BEGIN
+#define X86_ALIGN_RODATA_END					\
+		. = ALIGN(PAGE_SIZE);				\
+		__end_rodata_aligned = .;
 
+#define ALIGN_ENTRY_TEXT_BEGIN
+#define ALIGN_ENTRY_TEXT_END
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
 /*
  * This section contains data which will be mapped as decrypted. Memory
  * encryption operates on a page basis. Make this section PMD-aligned
@@ -88,17 +99,9 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 	__pi___end_bss_decrypted = .;				\
 
 #else
-
-#define X86_ALIGN_RODATA_BEGIN
-#define X86_ALIGN_RODATA_END					\
-		. = ALIGN(PAGE_SIZE);				\
-		__end_rodata_aligned = .;
-
-#define ALIGN_ENTRY_TEXT_BEGIN
-#define ALIGN_ENTRY_TEXT_END
 #define BSS_DECRYPTED
-
 #endif
+
 #if defined(CONFIG_X86_64) && defined(CONFIG_KEXEC_CORE)
 #define KEXEC_RELOCATE_KERNEL					\
 	. = ALIGN(0x100);					\
Signature · 1 line
2.47.3
author maintainer
#11 lore raw

PATCH 3/19 x86: Combine .data with .bss in kernel mapping

In reply to Ard Biesheuvel

The primary mapping of the kernel image is made using huge pages where possible, mostly to minimize TLB pressure (Only the entry text section requires alignment to 2 MiB). This involves some rounding and padding of the .text and .rodata sections, resulting in gaps. These gaps are smaller than a huge page, and are remapped using different permissions, resulting in fragmentation of the huge page mappings at the edges of those regions.

Similarly, there is a gap between .data and .bss, where the init text and data regions reside. This means that the end of the .data region and the start of the .bss region are not covered by huge page mappings either, even though both regions use the same permissions (RW+NX).

Improve the situation, by placing .data and .bss adjacently in the linker map, and putting the init text and data regions after .rodata, taking the place of the rodata/data gap. This results in one fewer gap, and a more efficient mapping of the .data and .bss regions.

To preserve the x86_64 ELF layout with PT_LOAD regions aligned to 2 MiB, start the second ELF segment at .init.data and align it to 2 MiB. The resulting padding will be covered by the init region and will be freed along with it after boot.

defconfig + Clang 19:

Before:

  0xffffffff81000000-0xffffffff82200000    18M  ro  PSE  GLB x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro       GLB x  pte
  0xffffffff8231c000-0xffffffff82400000   912K  RW       GLB NX pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE  GLB NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro       GLB NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K  RW       GLB NX pte
  0xffffffff82c00000-0xffffffff83400000     8M  RW  PSE  GLB NX pmd
  0xffffffff83400000-0xffffffff83800000     4M  RW       GLB NX pte

After:

  0xffffffff81000000-0xffffffff82200000    18M  ro  PSE  GLB x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro       GLB x  pte
  0xffffffff8231c000-0xffffffff82400000   912K  RW       GLB NX pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE  GLB NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro       GLB NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K  RW       GLB NX pte
  0xffffffff82c00000-0xffffffff82e00000     2M  RW  PSE  GLB NX pmd
  0xffffffff82e00000-0xffffffff83000000     2M  RW       GLB NX pte
  0xffffffff83000000-0xffffffff83800000     8M  RW  PSE  GLB NX pmd

With the gaps removed/unmapped (pti=on)

Before:

  0xffffffff81000000-0xffffffff81200000     2M  ro  PSE  GLB x  pmd
  0xffffffff81200000-0xffffffff82200000    16M  ro  PSE      x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro           x  pte
  0xffffffff8231c000-0xffffffff82400000   912K                  pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE      NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro           NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K                  pte
  0xffffffff82c00000-0xffffffff83400000     8M  RW  PSE      NX pmd
  0xffffffff83400000-0xffffffff8342a000   168K  RW           NX pte

0xffffffff8342a000-0xffffffff836f3000 2852K pte
0xffffffff836f3000-0xffffffff83800000 1076K RW NX pte

After:

  0xffffffff81000000-0xffffffff81200000     2M  ro  PSE  GLB x  pmd
  0xffffffff81200000-0xffffffff82200000    16M  ro  PSE      x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro           x  pte
  0xffffffff8231c000-0xffffffff82400000   912K                  pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE      NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro           NX pte
  0xffffffff82b40000-0xffffffff82e3d000  3060K                  pte
  0xffffffff82e3d000-0xffffffff83000000  1804K  RW           NX pte
  0xffffffff83000000-0xffffffff83800000     8M  RW  PSE      NX pmd
---
 arch/x86/kernel/vmlinux.lds.S | 91 +++++++++++---------
 arch/x86/mm/init_64.c         |  5 +-
 arch/x86/mm/pat/set_memory.c  |  2 +-
 3 files changed, 52 insertions(+), 46 deletions(-)
arch/x86/kernel/vmlinux.lds.S +50 -41
index 3a24a3fc55f5..1dee2987c42b 100644
@@ -61,12 +61,15 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 #define X86_ALIGN_RODATA_BEGIN	. = ALIGN(HPAGE_SIZE);
 
 #define X86_ALIGN_RODATA_END					\
-		. = ALIGN(HPAGE_SIZE);				\
-		__end_rodata_hpage_align = .;			\
-		__end_rodata_aligned = .;
+		. = ALIGN(PAGE_SIZE);				\
+		__end_rodata_aligned = ALIGN(HPAGE_SIZE);
 
 #define ALIGN_ENTRY_TEXT_BEGIN	. = ALIGN(PMD_SIZE);
 #define ALIGN_ENTRY_TEXT_END	. = ALIGN(PMD_SIZE);
+
+#define DATA_SEGMENT_START					\
+	. = ALIGN(HPAGE_SIZE);					\
+	__data_segment_start = .;
 #else
 
 #define X86_ALIGN_RODATA_BEGIN
@@ -76,9 +79,14 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 
 #define ALIGN_ENTRY_TEXT_BEGIN
 #define ALIGN_ENTRY_TEXT_END
+
+#define DATA_SEGMENT_START					\
+	. = ALIGN(PAGE_SIZE);					\
+	__data_segment_start = .;
 #endif
 
 #ifdef CONFIG_AMD_MEM_ENCRYPT
+
 /*
  * This section contains data which will be mapped as decrypted. Memory
  * encryption operates on a page basis. Make this section PMD-aligned
@@ -171,43 +179,6 @@ SECTIONS
 	RO_DATA(PAGE_SIZE)
 	X86_ALIGN_RODATA_END
 
-	/* Data */
-	.data : AT(ADDR(.data) - LOAD_OFFSET) {
-		/* Start of data section */
-		_sdata = .;
-
-		/* init_task */
-		INIT_TASK_DATA(THREAD_SIZE)
-
-		/* equivalent to task_pt_regs(&init_task) */
-		__top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE;
-
-#ifdef CONFIG_X86_32
-		/* 32 bit has nosave before _edata */
-		NOSAVE_DATA
-#endif
-
-		PAGE_ALIGNED_DATA(PAGE_SIZE)
-
-		CACHE_HOT_DATA(L1_CACHE_BYTES)
-
-		CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES)
-
-		DATA_DATA
-		CONSTRUCTORS
-		KEXEC_RELOCATE_KERNEL
-
-		/* rarely changed data like cpu maps */
-		READ_MOSTLY_DATA(INTERNODE_CACHE_BYTES)
-
-		/* End of data section */
-		_edata = .;
-	} :data
-
-	BUG_TABLE
-
-	ORC_UNWIND_TABLE
-
 	/* Init code and data - will be freed after init */
 	. = ALIGN(PAGE_SIZE);
 	.init.begin : AT(ADDR(.init.begin) - LOAD_OFFSET) {
@@ -229,7 +200,8 @@ SECTIONS
 		__inittext_end = .;
 	}
 
-	INIT_DATA_SECTION(16)
+	DATA_SEGMENT_START
+	INIT_DATA_SECTION(16) :data
 
 	.x86_cpu_dev.init : AT(ADDR(.x86_cpu_dev.init) - LOAD_OFFSET) {
 		__x86_cpu_dev_start = .;
@@ -358,6 +330,43 @@ SECTIONS
 		__smp_locks_end = .;
 	}
 
+	/* Data */
+	.data : AT(ADDR(.data) - LOAD_OFFSET) {
+		/* Start of data section */
+		_sdata = .;
+
+		/* init_task */
+		INIT_TASK_DATA(THREAD_SIZE)
+
+		/* equivalent to task_pt_regs(&init_task) */
+		__top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE;
+
+#ifdef CONFIG_X86_32
+		/* 32 bit has nosave before _edata */
+		NOSAVE_DATA
+#endif
+
+		PAGE_ALIGNED_DATA(PAGE_SIZE)
+
+		CACHE_HOT_DATA(L1_CACHE_BYTES)
+
+		CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES)
+
+		DATA_DATA
+		CONSTRUCTORS
+		KEXEC_RELOCATE_KERNEL
+
+		/* rarely changed data like cpu maps */
+		READ_MOSTLY_DATA(INTERNODE_CACHE_BYTES)
+
+		/* End of data section */
+		_edata = .;
+	}
+
+	BUG_TABLE
+
+	ORC_UNWIND_TABLE
+
 #ifdef CONFIG_X86_64
 	.data_nosave : AT(ADDR(.data_nosave) - LOAD_OFFSET) {
 		NOSAVE_DATA
arch/x86/mm/init_64.c +1 -4
index 9983017ecbe0..6c2120dd5607 100644
@@ -1397,9 +1397,8 @@ void mark_rodata_ro(void)
 {
 	unsigned long start = PFN_ALIGN(_text);
 	unsigned long rodata_start = PFN_ALIGN(__start_rodata);
-	unsigned long end = (unsigned long)__end_rodata_hpage_align;
+	unsigned long end = (unsigned long)__end_rodata;
 	unsigned long text_end = PFN_ALIGN(_etext);
-	unsigned long rodata_end = PFN_ALIGN(__end_rodata);
 	unsigned long all_end;
 
 	printk(KERN_INFO "Write protecting the kernel read-only data: %luk\n",
@@ -1435,8 +1434,6 @@ void mark_rodata_ro(void)
 
 	free_kernel_image_pages("unused kernel image (text/rodata gap)",
 				(void *)text_end, (void *)rodata_start);
-	free_kernel_image_pages("unused kernel image (rodata/data gap)",
-				(void *)rodata_end, (void *)_sdata);
 }
 
 /*
arch/x86/mm/pat/set_memory.c +1 -1
index 6c6eb486f7a6..ad4d55f2413b 100644
@@ -554,7 +554,7 @@ static pgprotval_t protect_kernel_text(unsigned long start, unsigned long end)
 static pgprotval_t protect_kernel_text_ro(unsigned long start,
 					  unsigned long end)
 {
-	unsigned long t_end = (unsigned long)__end_rodata_hpage_align - 1;
+	unsigned long t_end = (unsigned long)__end_rodata - 1;
 	unsigned long t_start = (unsigned long)_text;
 	unsigned int level;
 
Signature · 1 line
2.47.3
author maintainer
#14 lore raw

PATCH 4/19 x86: Make the 64-bit bzImage always physically relocatable

In reply to Ard Biesheuvel

On x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.

On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.

Clarify this in Kconfig and in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whether CONFIG_RELOCATABLE is set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite for EFI_STUB and RANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.

---
 arch/x86/Kconfig                   | 40 ++++++++++++--------
 arch/x86/boot/compressed/head_64.S |  4 --
 arch/x86/boot/compressed/misc.c    |  8 ++--
 arch/x86/boot/header.S             |  8 +---
 4 files changed, 29 insertions(+), 31 deletions(-)
arch/x86/Kconfig +24 -16
index 80527299f859..bf51e17d5813 100644
@@ -1931,7 +1931,7 @@ config EFI
 config EFI_STUB
 	bool "EFI stub support"
 	depends on EFI
-	select RELOCATABLE
+	select RELOCATABLE if X86_32
 	help
 	  This kernel feature allows a bzImage to be loaded directly
 	  by EFI firmware without the use of a bootloader.
@@ -2028,8 +2028,9 @@ config PHYSICAL_START
 	help
 	  This gives the physical address where the kernel is loaded.
 
-	  If the kernel is not relocatable (CONFIG_RELOCATABLE=n) then bzImage
-	  will decompress itself to above physical address and run from there.
+	  If the kernel is not relocatable (CONFIG_RELOCATABLE=n) and built for
+	  i386, then the bzImage will decompress itself to the above physical
+	  address and run from there.
 	  Otherwise, bzImage will run from the address where it has been loaded
 	  by the boot loader. The only exception is if it is loaded below the
 	  above physical address, in which case it will relocate itself there.
@@ -2064,16 +2065,22 @@ config PHYSICAL_START
 	  Don't change this unless you know what you are doing.
 
 config RELOCATABLE
-	bool "Build a relocatable kernel"
-	default y
+	bool "Build a relocatable kernel" if X86_32
+	default X86_32
 	help
-	  This builds a kernel image that retains relocation information
-	  so it can be loaded someplace besides the default 1MB.
+	  This builds a kernel image that retains relocation information so it
+	  can be placed someplace besides the default PAGE_OFFSET + 1MB. This
+	  is a prerequisite for KASLR.
 	  The relocations tend to make the kernel binary about 10% larger,
 	  but are discarded at runtime.
 
-	  One use is for the kexec on panic case where the recovery kernel
-	  must live at a different physical address than the primary
+	  On i386, where the virtual and physical load offset of the kernel
+	  must be equal, this also allows the kernel image to be placed at a
+	  physical load address that differs from the compile time default. On
+	  x86_64, this is always permitted.
+
+	  One use is for the kexec on panic case on i386, where the recovery
+	  kernel must live at a different physical address than the primary
 	  kernel.
 
 	  Note: If CONFIG_RELOCATABLE=y, then the kernel runs from the address
@@ -2082,7 +2089,7 @@ config RELOCATABLE
 
 config RANDOMIZE_BASE
 	bool "Randomize the address of the kernel image (KASLR)"
-	depends on RELOCATABLE
+	select RELOCATABLE
 	default y
 	help
 	  In support of Kernel Address Space Layout Randomization (KASLR),
@@ -2118,7 +2125,7 @@ config RANDOMIZE_BASE
 # Relocation on x86 needs some additional build support
 config X86_NEED_RELOCS
 	def_bool y
-	depends on RANDOMIZE_BASE || (X86_32 && RELOCATABLE)
+	depends on RELOCATABLE
 	select ARCH_VMLINUX_NEEDS_RELOCS
 
 config PHYSICAL_ALIGN
@@ -2131,12 +2138,13 @@ config PHYSICAL_ALIGN
 	  where kernel is loaded and run from. Kernel is compiled for an
 	  address which meets above alignment restriction.
 
-	  If bootloader loads the kernel at a non-aligned address and
-	  CONFIG_RELOCATABLE is set, kernel will move itself to nearest
-	  address aligned to above value and run from there.
+	  If the bootloader loads the kernel at a non-aligned address and it
+	  is built for x86_64 or CONFIG_RELOCATABLE is set, the kernel will
+	  move itself to the nearest address aligned to above value and run
+	  from there.
 
-	  If bootloader loads the kernel at a non-aligned address and
-	  CONFIG_RELOCATABLE is not set, kernel will ignore the run time
+	  If the bootloader loads the i386 kernel at a non-aligned address and
+	  CONFIG_RELOCATABLE is not set, the kernel will ignore the run time
 	  load address and decompress itself to the address it has been
 	  compiled for and run from there. The address for which kernel is
 	  compiled already meets above alignment restrictions. Hence the
arch/x86/boot/compressed/head_64.S +0 -4
index d9dab940ff62..8a964a4d45c2 100644
@@ -143,7 +143,6 @@ SYM_FUNC_START(startup_32)
  * for safe in-place decompression.
  */
 
-#ifdef CONFIG_RELOCATABLE
 	movl	%ebp, %ebx
 	movl	BP_kernel_alignment(%esi), %eax
 	decl	%eax
@@ -152,7 +151,6 @@ SYM_FUNC_START(startup_32)
 	andl	%eax, %ebx
 	cmpl	$LOAD_PHYSICAL_ADDR, %ebx
 	jae	1f
-#endif
 	movl	$LOAD_PHYSICAL_ADDR, %ebx
 1:
 
@@ -312,7 +310,6 @@ SYM_CODE_START(startup_64)
 	 */
 
 	/* Start with the delta to where the kernel will run at. */
-#ifdef CONFIG_RELOCATABLE
 	leaq	startup_32(%rip) /* - $startup_32 */, %rbp
 	movl	BP_kernel_alignment(%rsi), %eax
 	decl	%eax
@@ -321,7 +318,6 @@ SYM_CODE_START(startup_64)
 	andq	%rax, %rbp
 	cmpq	$LOAD_PHYSICAL_ADDR, %rbp
 	jae	1f
-#endif
 	movq	$LOAD_PHYSICAL_ADDR, %rbp
 1:
 
arch/x86/boot/compressed/misc.c +3 -5
index 0f41ca0e52c0..d37569e7ee10 100644
@@ -314,12 +314,10 @@ static size_t parse_elf(void *output)
 			if ((phdr->p_align % 0x200000) != 0)
 				error("Alignment of LOAD segment isn't multiple of 2MB");
 #endif
-#ifdef CONFIG_RELOCATABLE
-			dest = output;
-			dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);
-#else
 			dest = (void *)(phdr->p_paddr);
-#endif
+			if (IS_ENABLED(CONFIG_X86_64) ||
+			    IS_ENABLED(CONFIG_RELOCATABLE))
+				dest += (unsigned long)output - LOAD_PHYSICAL_ADDR;
 			memmove(dest, output + phdr->p_offset, phdr->p_filesz);
 			break;
 		default: /* Ignore other PT_* */ break;
arch/x86/boot/header.S +2 -6
index 9bea5a1e2c52..b72e6055e103 100644
@@ -332,7 +332,7 @@ initrd_addr_max: .long 0x7fffffff
 kernel_alignment:  .long CONFIG_PHYSICAL_ALIGN	#physical addr alignment
 						#required for protected mode
 						#kernel
-#ifdef CONFIG_RELOCATABLE
+#if defined(CONFIG_RELOCATABLE) || defined(CONFIG_X86_64)
 relocatable_kernel:    .byte 1
 #else
 relocatable_kernel:    .byte 0
@@ -342,14 +342,10 @@ min_alignment:		.byte MIN_KERNEL_ALIGN_LG2	# minimum alignment
 xloadflags:
 #ifdef CONFIG_X86_64
 # define XLF0 XLF_KERNEL_64			/* 64-bit kernel */
-#else
-# define XLF0 0
-#endif
-
-#if defined(CONFIG_RELOCATABLE) && defined(CONFIG_X86_64)
    /* kernel/boot_param/ramdisk could be loaded above 4g */
 # define XLF1 XLF_CAN_BE_LOADED_ABOVE_4G
 #else
+# define XLF0 0
 # define XLF1 0
 #endif
 
Signature · 1 line
2.47.3
author maintainer
#18 lore raw

PATCH 5/19 x86/efistub: Simplify early remapping of kernel text

In reply to Ard Biesheuvel

Now that the kernel's .text, .rodata and .inittext are all covered by a single ELF segment, there is no need to remap .inittext separately. Instead, remap the entire region in a single call.

This remapping is needed because the EFI stub hands over to the core kernel while running in long mode, using the page tables provided by the firmware. Recent so-called 'MS secured core' (tm) PCs are more strict when it comes to separating writable from executable mappings, and so for compatibility with such systems, any code that may be callable during early boot (i.e., before the kernel switches to its own page tables) must be remapped executable explicitly.

---
 arch/x86/boot/compressed/Makefile       | 2 +-
 arch/x86/boot/compressed/misc.c         | 4 +---
 arch/x86/include/asm/boot.h             | 2 --
 arch/x86/kernel/vmlinux.lds.S           | 2 --
 drivers/firmware/efi/libstub/x86-stub.c | 4 +---
 5 files changed, 3 insertions(+), 11 deletions(-)
arch/x86/boot/compressed/Makefile +1 -1
index 68f9d7a1683b..bc071bdcd11e 100644
@@ -76,7 +76,7 @@ LDFLAGS_vmlinux += -T
 hostprogs	:= mkpiggy
 HOST_EXTRACFLAGS += -I$(srctree)/tools/include
 
-sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__start_rodata\|_sinittext\|__inittext_end\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p'
+sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__data_segment_start\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p'
 
 quiet_cmd_voffset = VOFFSET $@
       cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
arch/x86/boot/compressed/misc.c +1 -3
index d37569e7ee10..1ea419cf88fe 100644
@@ -329,9 +329,7 @@ static size_t parse_elf(void *output)
 	return ehdr.e_entry - LOAD_PHYSICAL_ADDR;
 }
 
-const unsigned long kernel_text_size = VO___start_rodata - VO__text;
-const unsigned long kernel_inittext_offset = VO__sinittext - VO__text;
-const unsigned long kernel_inittext_size = VO___inittext_end - VO__sinittext;
+const unsigned long kernel_text_size = VO___data_segment_start - VO__text;
 const unsigned long kernel_total_size = VO__end - VO__text;
 
 static u8 boot_heap[BOOT_HEAP_SIZE] __aligned(4);
arch/x86/include/asm/boot.h +0 -2
index f7b67cb73915..02b23aa78955 100644
@@ -82,8 +82,6 @@
 #ifndef __ASSEMBLER__
 extern unsigned int output_len;
 extern const unsigned long kernel_text_size;
-extern const unsigned long kernel_inittext_offset;
-extern const unsigned long kernel_inittext_size;
 extern const unsigned long kernel_total_size;
 
 unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr,
arch/x86/kernel/vmlinux.lds.S +0 -2
index 1dee2987c42b..6772fe9a9957 100644
@@ -196,8 +196,6 @@ SECTIONS
 	 */
 	.altinstr_aux : AT(ADDR(.altinstr_aux) - LOAD_OFFSET) {
 		*(.altinstr_aux)
-		. = ALIGN(PAGE_SIZE);
-		__inittext_end = .;
 	}
 
 	DATA_SEGMENT_START
drivers/firmware/efi/libstub/x86-stub.c +1 -3
index cef32e2c82d8..ffe30ef73fda 100644
@@ -890,9 +890,7 @@ static efi_status_t efi_decompress_kernel(unsigned long *kernel_entry,
 
 	*kernel_entry = addr + entry;
 
-	return efi_adjust_memory_range_protection(addr, kernel_text_size) ?:
-	       efi_adjust_memory_range_protection(addr + kernel_inittext_offset,
-						  kernel_inittext_size);
+	return efi_adjust_memory_range_protection(addr, kernel_text_size);
 }
 
 static void __noreturn enter_kernel(unsigned long kernel_addr,
Signature · 1 line
2.47.3
author maintainer
#19 lore raw

PATCH 6/19 alloc_tag: Use __ prefixed ELF section names

In reply to Ard Biesheuvel

The compiler will emit static relocations related to a section .foo into a separate section called .rela.foo, i.e., it just appends the section name to the string ".rela"

When section names start with . or __, this results in section names that are correctly matched by the various pattern rules in the various linker scripts across the tree.

Without any such leading delimiter, it may lead to spurious warnings such as

Quoted 1 line
Quoted 1 line

ld: warning: orphan section .relaalloc_tags' from init/main.o' being placed in section `.rela.dyn'

ld: warning: dot moved backwards before `.rela.dyn'
ld: .tmp_vmlinux1: section `.rela.dyn' can't be allocated in segment 1

Fix this by renaming the section to __alloc_tags. While at it, tweak the headers so that the definition appears only a single time.

---
 include/asm-generic/codetag.lds.h | 14 +++++++++-----
 include/linux/alloc_tag.h         | 11 ++++++-----
 lib/alloc_tag.c                   |  6 +++---
 3 files changed, 18 insertions(+), 13 deletions(-)
include/asm-generic/codetag.lds.h +9 -5
index a14f4bdafdda..d7ff181862da 100644
@@ -2,6 +2,10 @@
 #ifndef __ASM_GENERIC_CODETAG_LDS_H
 #define __ASM_GENERIC_CODETAG_LDS_H
 
+#include <linux/compiler_types.h>
+
+#define ALLOC_TAG_SECTION_NAME	__alloc_tags
+
 #ifdef CONFIG_MEM_ALLOC_PROFILING
 #define IF_MEM_ALLOC_PROFILING(...) __VA_ARGS__
 #else
@@ -10,15 +14,15 @@
 
 #define SECTION_WITH_BOUNDARIES(_name)	\
 	. = ALIGN(8);			\
-	__start_##_name = .;		\
+	__PASTE(__start_, _name) = .;	\
 	KEEP(*(_name))			\
-	__stop_##_name = .;
+	__PASTE(__stop_, _name) = .;
 
 #define CODETAG_SECTIONS()		\
-	IF_MEM_ALLOC_PROFILING(SECTION_WITH_BOUNDARIES(alloc_tags))
+	IF_MEM_ALLOC_PROFILING(SECTION_WITH_BOUNDARIES(ALLOC_TAG_SECTION_NAME))
 
 #define MOD_SEPARATE_CODETAG_SECTION(_name)	\
-	.codetag.##_name : {			\
+	.codetag._name : {			\
 		SECTION_WITH_BOUNDARIES(_name)	\
 	}
 
@@ -28,6 +32,6 @@
  * unload them individually once unused.
  */
 #define MOD_SEPARATE_CODETAG_SECTIONS()		\
-	IF_MEM_ALLOC_PROFILING(MOD_SEPARATE_CODETAG_SECTION(alloc_tags))
+	IF_MEM_ALLOC_PROFILING(MOD_SEPARATE_CODETAG_SECTION(ALLOC_TAG_SECTION_NAME))
 
 #endif /* __ASM_GENERIC_CODETAG_LDS_H */
include/linux/alloc_tag.h +6 -5
index d40ac39bfbe8..f39d85b05b8a 100644
@@ -15,6 +15,9 @@
 #include <linux/static_key.h>
 #include <linux/irqflags.h>
 
+/* for ALLOC_TAG_SECTION_NAME */
+#include <asm-generic/codetag.lds.h>
+
 struct alloc_tag_counters {
 	u64 bytes;
 	u64 calls;
@@ -74,8 +77,6 @@ static inline void set_codetag_empty(union codetag_ref *ref)
 
 #ifdef CONFIG_MEM_ALLOC_PROFILING
 
-#define ALLOC_TAG_SECTION_NAME	"alloc_tags"
-
 struct codetag_bytes {
 	struct codetag *ct;
 	s64 bytes;
@@ -98,7 +99,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag);
 
 #define DEFINE_ALLOC_TAG(_alloc_tag)						\
 	static struct alloc_tag _alloc_tag __used __aligned(8)			\
-	__section(ALLOC_TAG_SECTION_NAME) = {					\
+	__section(__stringify(ALLOC_TAG_SECTION_NAME)) = {			\
 		.ct = CODE_TAG_INIT,						\
 		.counters = &_shared_alloc_tag };
 
@@ -108,7 +109,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag);
 
 #define DEFINE_ALLOC_TAG(_alloc_tag)						\
 	static struct alloc_tag _alloc_tag __used __aligned(8)			\
-	__section(ALLOC_TAG_SECTION_NAME) = {					\
+	__section(__stringify(ALLOC_TAG_SECTION_NAME)) = {			\
 		.ct = CODE_TAG_INIT,						\
 		.counters = NULL };
 
@@ -117,7 +118,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag);
 #define DEFINE_ALLOC_TAG(_alloc_tag)						\
 	static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr);	\
 	static struct alloc_tag _alloc_tag __used __aligned(8)			\
-	__section(ALLOC_TAG_SECTION_NAME) = {					\
+	__section(__stringify(ALLOC_TAG_SECTION_NAME)) = {			\
 		.ct = CODE_TAG_INIT,						\
 		.counters = &_alloc_tag_cntr };
 
lib/alloc_tag.c +3 -3
index 27fee57a5c91..3eff7e912521 100644
@@ -15,8 +15,8 @@
 
 #define ALLOCINFO_FILE_NAME		"allocinfo"
 #define MODULE_ALLOC_TAG_VMAP_SIZE	(100000UL * sizeof(struct alloc_tag))
-#define SECTION_START(NAME)		(CODETAG_SECTION_START_PREFIX NAME)
-#define SECTION_STOP(NAME)		(CODETAG_SECTION_STOP_PREFIX NAME)
+#define SECTION_START(NAME)		(CODETAG_SECTION_START_PREFIX #NAME)
+#define SECTION_STOP(NAME)		(CODETAG_SECTION_STOP_PREFIX #NAME)
 
 #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT
 static bool mem_profiling_support = true;
@@ -810,7 +810,7 @@ static inline void sysctl_init(void) {}
 static int __init alloc_tag_init(void)
 {
 	const struct codetag_type_desc desc = {
-		.section		= ALLOC_TAG_SECTION_NAME,
+		.section		= __stringify(ALLOC_TAG_SECTION_NAME),
 		.tag_size		= sizeof(struct alloc_tag),
 #ifdef CONFIG_MODULES
 		.needs_section_mem	= needs_section_mem,
Signature · 1 line
2.47.3
author maintainer
#20 lore raw

PATCH 7/19 tools/objtool: Treat indirect ftrace calls as direct calls

In reply to Ard Biesheuvel

In some cases, the compiler may rely on indirect calls using GOT slots as memory operands to emit function calls. This leaves it up to the linker to relax the call to a direct call if possible, i.e., if the destination address is known at link time and in range, which may not be the case when building shared libraries for user space.

On x86, this may happen when building in PIC mode with ftrace enabled, and given that vmlinux is a fully linked binary, this relaxation is always possible, and therefore mandatory per the x86_64 psABI.

This means that the indirect calls to __fentry__ that are observeable in vmlinux.o will have been converted to direct calls in vmlinux, and can be treated as such by objtool.

---
 tools/objtool/check.c | 32 ++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
tools/objtool/check.c +30 -2
index 3f7999317f4d..765f818af839 100644
@@ -1660,11 +1660,39 @@ static int add_call_destinations(struct objtool_file *file)
 
 	for_each_insn(file, insn) {
 		struct symbol *func = insn_func(insn);
-		if (insn->type != INSN_CALL)
+		if (insn->type != INSN_CALL &&
+		    insn->type != INSN_CALL_DYNAMIC)
 			continue;
 
 		reloc = insn_reloc(file, insn);
-		if (!reloc) {
+		if (insn->type == INSN_CALL_DYNAMIC) {
+			if (!reloc)
+				continue;
+
+			/*
+			 * GCC 13 and older on x86 will always emit the call to
+			 * __fentry__ using a relaxable GOT-based symbol
+			 * reference when operating in PIC mode, i.e.,
+			 *
+			 *   call   *0x0(%rip)
+			 *             R_X86_64_GOTPCRELX  __fentry__-0x4
+			 *
+			 * where it is left up to the linker to relax this into
+			 *
+			 *   call   __fentry__
+			 *   nop
+			 *
+			 * if __fentry__ turns out to be DSO local, which is
+			 * always the case for vmlinux. Given that this
+			 * relaxation is mandatory per the x86_64 psABI, these
+			 * calls can simply be treated as direct calls.
+			 */
+			if (arch_ftrace_match(reloc->sym->name)) {
+				insn->type = INSN_CALL;
+				add_call_dest(file, insn, reloc->sym, false);
+			}
+
+		} else if (!reloc) {
 			dest_off = arch_jump_destination(insn);
 			dest = find_call_destination(insn->sec, dest_off);
 
Signature · 1 line
2.47.3
author maintainer
#21 lore raw

PATCH 8/19 x86: Use PIE codegen for the relocatable 64-bit kernel

In reply to Ard Biesheuvel

As an intermediate step towards enabling PIE linking for the x86_64 KASLR kernel, enable PIE codegen for all C and Rust objects that are linked into the kernel proper. Add a Kconfig option RELOCATABLE_PIE for this, depending on RELR support in the linker, as the relocation tables will blow up the kernel image otherwise.

This results in a code size increase of between 0.2% (clang) and 0.5% (gcc). Performance (hackbench) appears to be unaffected across several different uarchs.

---
 arch/x86/Kconfig                  |  4 ++++
 arch/x86/Makefile                 | 19 ++++++++++++++++++-
 arch/x86/boot/Makefile            |  1 +
 arch/x86/boot/compressed/Makefile |  2 +-
 arch/x86/entry/vdso/Makefile      |  1 +
 arch/x86/realmode/rm/Makefile     |  1 +
 include/asm-generic/vmlinux.lds.h |  1 +
 include/linux/hidden.h            |  2 ++
 8 files changed, 29 insertions(+), 2 deletions(-)
arch/x86/Kconfig +4 -0
index bf51e17d5813..b3a64cfe04cf 100644
@@ -2067,6 +2067,7 @@ config PHYSICAL_START
 config RELOCATABLE
 	bool "Build a relocatable kernel" if X86_32
 	default X86_32
+	select RELOCATABLE_PIE if TOOLS_SUPPORT_RELR
 	help
 	  This builds a kernel image that retains relocation information so it
 	  can be placed someplace besides the default PAGE_OFFSET + 1MB. This
@@ -2087,6 +2088,9 @@ config RELOCATABLE
 	  it has been loaded at and the compile time physical address
 	  (CONFIG_PHYSICAL_START) is used as the minimum location.
 
+config RELOCATABLE_PIE
+	bool
+
 config RANDOMIZE_BASE
 	bool "Randomize the address of the kernel image (KASLR)"
 	select RELOCATABLE
arch/x86/Makefile +18 -1
index 1d403a3612ea..b211d6c950aa 100644
@@ -89,6 +89,8 @@ ifdef CONFIG_CC_IS_GCC
 CC_FLAGS_FPU += -mhard-float
 endif
 
+rustflags-nojumptables := $(if $(call rustc-min-version,109300),-Cjump-tables=n,-Zno-jump-tables)
+
 ifeq ($(CONFIG_X86_KERNEL_IBT),y)
 #
 # Kernel IBT has S_CET.NOTRACK_EN=0, as such the compilers must not generate
@@ -100,7 +102,7 @@ ifeq ($(CONFIG_X86_KERNEL_IBT),y)
 #   https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104816
 #
 KBUILD_CFLAGS += $(call cc-option,-fcf-protection=branch -fno-jump-tables)
-KBUILD_RUSTFLAGS += -Zcf-protection=branch $(if $(call rustc-min-version,109300),-Cjump-tables=n,-Zno-jump-tables)
+KBUILD_RUSTFLAGS += -Zcf-protection=branch $(rustflags-nojumptables)
 else
 KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none)
 endif
@@ -178,6 +180,21 @@ endif
         KBUILD_RUSTFLAGS += -Ccode-model=kernel
 
         percpu_seg := gs
+
+        pie-ccflags-$(CONFIG_CC_IS_GCC) += $(call cc-option.-mdirect-extern-access)
+        pie-ccflags-$(CONFIG_CC_IS_CLANG) += -fdirect-access-external-data
+
+        # objtool gets confused by unannotated PIC flavor jump tables
+        pie-ccflags-y += $(call cc-option,-fannotate-jump-tables,-fno-jump-tables)
+
+        pie-cflags-$(CONFIG_RELOCATABLE_PIE) := $(pie-ccflags-y) -fpie -mcmodel=small \
+                                -include $(srctree)/include/linux/hidden.h
+        pie-rustflags-$(CONFIG_RELOCATABLE_PIE) := -Crelocation-model=pie \
+                                -Ccode-model=small -Zdirect-access-external-data=yes \
+                                $(rustflags-nojumptables)
+
+        KBUILD_CFLAGS_KERNEL    += $(pie-cflags-y)
+        KBUILD_RUSTFLAGS_KERNEL += $(pie-rustflags-y)
 endif
 
 ifeq ($(CONFIG_STACKPROTECTOR),y)
arch/x86/boot/Makefile +1 -0
index 3f9fb3698d66..491b3b2a9a02 100644
@@ -55,6 +55,7 @@ KBUILD_CFLAGS	:= $(REALMODE_CFLAGS) -D_SETUP
 KBUILD_AFLAGS	:= $(KBUILD_CFLAGS) -D__ASSEMBLY__
 KBUILD_CFLAGS	+= -fno-asynchronous-unwind-tables
 KBUILD_CFLAGS	+= $(CONFIG_CC_IMPLICIT_FALLTHROUGH)
+KBUILD_CFLAGS_KERNEL :=
 
 $(obj)/bzImage: asflags-y  := $(SVGA_MODE)
 
arch/x86/boot/compressed/Makefile +1 -1
index bc071bdcd11e..96099b5d1064 100644
@@ -76,7 +76,7 @@ LDFLAGS_vmlinux += -T
 hostprogs	:= mkpiggy
 HOST_EXTRACFLAGS += -I$(srctree)/tools/include
 
-sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__data_segment_start\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p'
+sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDdGRSTtVW] \(_text\|__data_segment_start\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p'
 
 quiet_cmd_voffset = VOFFSET $@
       cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
arch/x86/entry/vdso/Makefile +1 -0
index f247f5f5cb44..bf4221a0fc08 100644
@@ -143,6 +143,7 @@ endif
 endif
 
 $(obj)/vdso32.so.dbg: KBUILD_CFLAGS = $(KBUILD_CFLAGS_32)
+$(obj)/vdso32.so.dbg: KBUILD_CFLAGS_KERNEL :=
 
 $(obj)/vdso32.so.dbg: $(obj)/vdso32/vdso32.lds $(vobjs32) FORCE
 	$(call if_changed,vdso_and_check)
arch/x86/realmode/rm/Makefile +1 -0
index a0fb39abc5c8..70bf0a26da91 100644
@@ -67,3 +67,4 @@ KBUILD_CFLAGS	:= $(REALMODE_CFLAGS) -D_SETUP -D_WAKEUP \
 		   -I$(srctree)/arch/x86/boot
 KBUILD_AFLAGS	:= $(KBUILD_CFLAGS) -D__ASSEMBLY__
 KBUILD_CFLAGS	+= -fno-asynchronous-unwind-tables
+KBUILD_CFLAGS_KERNEL :=
include/asm-generic/vmlinux.lds.h +1 -0
index 8ca130af301f..1782b6b87b2d 100644
@@ -373,6 +373,7 @@
 	*(DATA_MAIN)							\
 	*(.data..decrypted)						\
 	*(.ref.data)							\
+	*(.data.rel*)							\
 	*(.data..shared_aligned) /* percpu related */			\
 	*(.data..unlikely)						\
 	__start_once = .;						\
include/linux/hidden.h +2 -0
index 49a17b6b5962..2ad764c0ca18 100644
@@ -16,4 +16,6 @@
  * giving them 'hidden' visibility.
  */
 
+#ifndef __BINDGEN__
 #pragma GCC visibility push(hidden)
+#endif
Signature · 1 line
2.47.3
author maintainer
#24 lore raw

PATCH 9/19 x86/pm-trace: Use RIP-relative accesses for .tracedata

In reply to Ard Biesheuvel

Use RIP-relative accesses and 32-bit offsets for .tracedata, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/include/asm/pm-trace.h | 4 ++--
 drivers/base/power/trace.c      | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/pm-trace.h +2 -2
index bfa32aa428e5..123faf978473 100644
@@ -8,10 +8,10 @@
 do {								\
 	if (pm_trace_enabled) {					\
 		const void *tracedata;				\
-		asm volatile(_ASM_MOV " $1f,%0\n"		\
+		asm volatile("lea " _ASM_RIP(1f) ", %0\n"	\
 			     ".section .tracedata,\"a\"\n"	\
 			     "1:\t.word %c1\n\t"		\
-			     _ASM_PTR " %c2\n"			\
+			     ".long %c2 - .\n"			\
 			     ".previous"			\
 			     :"=r" (tracedata)			\
 			     : "i" (__LINE__), "i" (__FILE__));	\
drivers/base/power/trace.c +3 -3
index d8da7195bb00..111be5825529 100644
@@ -167,7 +167,7 @@ EXPORT_SYMBOL(set_trace_device);
 void generate_pm_trace(const void *tracedata, unsigned int user)
 {
 	unsigned short lineno = *(unsigned short *)tracedata;
-	const char *file = *(const char **)(tracedata + 2);
+	const char *file = offset_to_ptr((int *)(tracedata + 2));
 	unsigned int user_hash_value, file_hash_value;
 
 	if (!x86_platform.legacy.rtc)
@@ -187,9 +187,9 @@ static int show_file_hash(unsigned int value)
 
 	match = 0;
 	for (tracedata = __tracedata_start ; tracedata < __tracedata_end ;
-			tracedata += 2 + sizeof(unsigned long)) {
+			tracedata += 2 + sizeof(int)) {
 		unsigned short lineno = *(unsigned short *)tracedata;
-		const char *file = *(const char **)(tracedata + 2);
+		const char *file = offset_to_ptr((int *)(tracedata + 2));
 		unsigned int hash = hash_string(lineno, file, FILEHASH);
 		if (hash != value)
 			continue;
Signature · 1 line
2.47.3
author maintainer
#25 lore raw

PATCH 10/19 x86/kvm: Use RIP-relative addressing

In reply to Ard Biesheuvel

Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/kernel/kvm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/kvm.c +3 -2
index df78ddee0abb..1a0335f328e1 100644
@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);
  * restoring to/from the stack.
  */
 #define PV_VCPU_PREEMPTED_ASM						     \
- "movq   __per_cpu_offset(,%rdi,8), %rax\n\t"				     \
- "cmpb   $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \
+ "0:leaq 0b(%rip), %rax\n\t"						     \
+ "addq   __per_cpu_offset - 0b(%rax,%rdi,8), %rax\n\t"			     \
+ "cmpb   $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time-0b(%rax)\n\t" \
  "setne  %al\n\t"
 
 DEFINE_ASM_FUNC(__raw_callee_save___kvm_vcpu_is_preempted,
Signature · 1 line
2.47.3
author maintainer
#30 lore raw

PATCH 11/19 x86/rethook: Use RIP-relative reference for fake return address

In reply to Ard Biesheuvel

Pushing an immediate absolute address to the stack is not permitted when linking x86_64 code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.

So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.

---
 arch/x86/kernel/rethook.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/rethook.c +5 -1
index 85e2f2d16a90..50812ac718b0 100644
@@ -11,6 +11,10 @@
 
 __visible void arch_rethook_trampoline_callback(struct pt_regs *regs);
 
+#ifdef CONFIG_X86_64
+static __used void * const __arch_rethook_trampoline = &arch_rethook_trampoline;
+#endif
+
 #ifndef ANNOTATE_NOENDBR
 #define ANNOTATE_NOENDBR
 #endif
@@ -27,7 +31,7 @@ asm(
 #ifdef CONFIG_X86_64
 	ANNOTATE_NOENDBR "\n"	/* This is only jumped from ret instruction */
 	/* Push a fake return address to tell the unwinder it's a rethook. */
-	"	pushq $arch_rethook_trampoline\n"
+	"	pushq __arch_rethook_trampoline(%rip)\n"
 	UNWIND_HINT_FUNC
 	"       pushq $" __stringify(__KERNEL_DS) "\n"
 	/* Save the 'sp - 16', this will be fixed later. */
Signature · 1 line
2.47.3
author maintainer
#34 lore raw

PATCH 12/19 x86/sync_core: Use RIP-relative addressing

In reply to Ard Biesheuvel

Use RIP-relative accesses for sync_core(). This removes a 32-bit absolute reference that requires fixing up at runtime when KASLR is enabled. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/include/asm/sync_core.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/sync_core.h +2 -1
index 96bda43538ee..547fdc690ecc 100644
@@ -31,7 +31,8 @@ static __always_inline void iret_to_self(void)
 		"pushfq\n\t"
 		"mov %%cs, %0\n\t"
 		"pushq %q0\n\t"
-		"pushq $1f\n\t"
+		"leaq 1f(%%rip), %q0\n\t"
+		"pushq %q0\n\t"
 		"iretq\n\t"
 		"1:"
 		: "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory");
Signature · 1 line
2.47.3
author maintainer
#35 lore raw

PATCH 13/19 x86/entry_64: Use RIP-relative addressing

In reply to Ard Biesheuvel

Replace a couple of instances in the x86_64 entry code where the absolute address of a symbol is taken in a manner that is not supported when linking in PIE mode, and use RIP-relative references instead, which don't require boot-time fixups at all.

---
 arch/x86/entry/calling.h  |  9 +++++----
 arch/x86/entry/entry_64.S | 14 +++++++-------
 2 files changed, 12 insertions(+), 11 deletions(-)
arch/x86/entry/calling.h +5 -4
index 77e2d920a640..a37b402432a3 100644
@@ -376,8 +376,8 @@ For 32-bit we have the following conventions - kernel is built with
 .endm
 
 .macro SAVE_AND_SET_GSBASE scratch_reg:req save_reg:req
+	GET_PERCPU_BASE \scratch_reg \save_reg
 	rdgsbase \save_reg
-	GET_PERCPU_BASE \scratch_reg
 	wrgsbase \scratch_reg
 .endm
 
@@ -413,15 +413,16 @@ For 32-bit we have the following conventions - kernel is built with
  * Thus the kernel would consume a guest's TSC_AUX if an NMI arrives
  * while running KVM's run loop.
  */
-.macro GET_PERCPU_BASE reg:req
+.macro GET_PERCPU_BASE reg:req scratch:req
 	LOAD_CPU_AND_NODE_SEG_LIMIT \reg
 	andq	$VDSO_CPUNODE_MASK, \reg
-	movq	__per_cpu_offset(, \reg, 8), \reg
+	leaq	__per_cpu_offset(%rip), \scratch
+	movq	(\scratch, \reg, 8), \reg
 .endm
 
 #else
 
-.macro GET_PERCPU_BASE reg:req
+.macro GET_PERCPU_BASE reg:req scratch:req
 	movq	pcpu_unit_offsets(%rip), \reg
 .endm
 
arch/x86/entry/entry_64.S +7 -7
index f9983a1907bf..77584f5ebb4b 100644
@@ -1040,7 +1040,8 @@ SYM_CODE_START(error_entry)
 	movl	%ecx, %eax			/* zero extend */
 	cmpq	%rax, RIP+8(%rsp)
 	je	.Lbstep_iret
-	cmpq	$.Lgs_change, RIP+8(%rsp)
+	leaq	.Lgs_change(%rip), %rcx
+	cmpq	%rcx, RIP+8(%rsp)
 	jne	.Lerror_entry_done_lfence
 
 	/*
@@ -1252,10 +1253,10 @@ SYM_CODE_START(asm_exc_nmi)
 	 * the outer NMI.
 	 */
 
-	movq	$repeat_nmi, %rdx
+	leaq	repeat_nmi(%rip), %rdx
 	cmpq	8(%rsp), %rdx
 	ja	1f
-	movq	$end_repeat_nmi, %rdx
+	leaq	end_repeat_nmi(%rip), %rdx
 	cmpq	8(%rsp), %rdx
 	ja	nested_nmi_out
 1:
@@ -1309,7 +1310,8 @@ nested_nmi:
 	pushq	%rdx
 	pushfq
 	pushq	$__KERNEL_CS
-	pushq	$repeat_nmi
+	leaq	repeat_nmi(%rip), %rdx
+	pushq	%rdx
 
 	/* Put stack back */
 	addq	$(6*8), %rsp
@@ -1348,10 +1350,8 @@ first_nmi:
 	addq	$8, (%rsp)	/* Fix up RSP */
 	pushfq			/* RFLAGS */
 	pushq	$__KERNEL_CS	/* CS */
-	pushq	$1f		/* RIP */
-	iretq			/* continues at repeat_nmi below */
+	call	native_irq_return_iret
 	UNWIND_HINT_IRET_REGS
-1:
 #endif
 
 repeat_nmi:
Signature · 1 line
2.47.3
author maintainer
#36 lore raw

PATCH 14/19 x86/hibernate: Prefer RIP-relative accesses

In reply to Ard Biesheuvel

Replace some absolute symbol references with RIP-relative ones, to avoid fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/power/hibernate_asm_64.S | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/power/hibernate_asm_64.S +2 -2
index c73be0a02a6c..173df717275a 100644
@@ -40,7 +40,7 @@ SYM_FUNC_START(restore_registers)
 	movq	%rax, %cr4;  # turn PGE back on
 
 	/* We don't restore %rax, it must be 0 anyway */
-	movq	$saved_context, %rax
+	leaq	saved_context(%rip), %rax
 	movq	pt_regs_sp(%rax), %rsp
 	movq	pt_regs_bp(%rax), %rbp
 	movq	pt_regs_si(%rax), %rsi
@@ -71,7 +71,7 @@ SYM_FUNC_START(restore_registers)
 SYM_FUNC_END(restore_registers)
 
 SYM_FUNC_START(swsusp_arch_suspend)
-	movq	$saved_context, %rax
+	leaq	saved_context(%rip), %rax
 	movq	%rsp, pt_regs_sp(%rax)
 	movq	%rbp, pt_regs_bp(%rax)
 	movq	%rsi, pt_regs_si(%rax)
Signature · 1 line
2.47.3
author maintainer
#37 lore raw

PATCH 15/19 x64/acpi: Use PIC-compatible references in wakeup_64.S

In reply to Ard Biesheuvel

Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.

Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.

---
 arch/x86/kernel/acpi/wakeup_64.S | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +4 -7
index 04f561f75e99..15233a4e1c95 100644
@@ -14,7 +14,7 @@
 
 .code64
 	/*
-	 * Hooray, we are in Long 64-bit mode (but still running in low memory)
+	 * Hooray, we are in Long 64-bit mode
 	 */
 SYM_FUNC_START(wakeup_long64)
 	ANNOTATE_NOENDBR
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64)
 	movq	saved_rsi(%rip), %rsi
 	movq	saved_rbp(%rip), %rbp
 
-	movq	saved_rip(%rip), %rax
+	leaq	.Lresume_point(%rip), %rax
 	ANNOTATE_RETPOLINE_SAFE
 	jmp	*%rax
 SYM_FUNC_END(wakeup_long64)
@@ -52,7 +52,7 @@ SYM_FUNC_START(do_suspend_lowlevel)
 	xorl	%eax, %eax
 	call	save_processor_state
 
-	movq	$saved_context, %rax
+	leaq	saved_context(%rip), %rax
 	movq	%rsp, pt_regs_sp(%rax)
 	movq	%rbp, pt_regs_bp(%rax)
 	movq	%rsi, pt_regs_si(%rax)
@@ -71,8 +71,6 @@ SYM_FUNC_START(do_suspend_lowlevel)
 	pushfq
 	popq	pt_regs_flags(%rax)
 
-	movq	$.Lresume_point, saved_rip(%rip)
-
 	movq	%rsp, saved_rsp(%rip)
 	movq	%rbp, saved_rbp(%rip)
 	movq	%rbx, saved_rbx(%rip)
@@ -90,7 +88,7 @@ SYM_FUNC_START(do_suspend_lowlevel)
 .Lresume_point:
 	ANNOTATE_NOENDBR
 	/* We don't restore %rax, it must be 0 anyway */
-	movq	$saved_context, %rax
+	leaq	saved_context(%rip), %rax
 	movq	saved_context_cr4(%rax), %rbx
 	movq	%rbx, %cr4
 	movq	saved_context_cr3(%rax), %rbx
@@ -139,7 +137,6 @@ saved_rsi:		.quad	0
 saved_rdi:		.quad	0
 saved_rbx:		.quad	0
 
-saved_rip:		.quad	0
 saved_rsp:		.quad	0
 
 SYM_DATA(saved_magic,	.quad	0)
Signature · 1 line
2.47.3
author maintainer
#44 lore raw

PATCH 16/19 x86/kexec: Use 64-bit wide absolute reference from relocated code

In reply to Ard Biesheuvel

The virtual_mapped() kexec routine runs from a different virtual address than it was linked at, and so it needs to use an absolute reference to load the address of 'saved_context'. Change this reference to a 64-bit wide one, to make the code compatible with linking in PIE mode.

---
 arch/x86/kernel/relocate_kernel_64.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/relocate_kernel_64.S +1 -1
index 4ffba68dc57b..3fc1a3002e32 100644
@@ -311,7 +311,7 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped)
 
 #ifdef CONFIG_KEXEC_JUMP
 	/* Saved in save_processor_state. */
-	movq    $saved_context, %rax
+	movabsq $saved_context, %rax
 	lgdt    saved_context_gdt_desc(%rax)
 #endif
 
Signature · 1 line
2.47.3
author maintainer
#45 lore raw

PATCH 17/19 x86/head64: Avoid absolute references in startup asm

In reply to Ard Biesheuvel

Replace a couple of occurrences of absolute references with RIP-relative ones. This removes the need for boot-time fixups. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/kernel/head_64.S | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
arch/x86/kernel/head_64.S +10 -7
index 21816b48537c..2c666c8c4519 100644
@@ -179,8 +179,9 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
 	xorl	%r15d, %r15d
 
 	/* Derive the runtime physical address of init_top_pgt[] */
-	movq	phys_base(%rip), %rax
-	addq	$(init_top_pgt - __START_KERNEL_map), %rax
+	leaq	init_top_pgt(%rip), %rax
+	subq	$__START_KERNEL_map, %rax
+	addq	phys_base(%rip), %rax
 
 	/*
 	 * Retrieve the modifier (SME encryption mask if SME is active) to be
@@ -232,6 +233,9 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 	btsl	$X86_CR4_PGE_BIT, %ecx
 	movq	%rcx, %cr4
 
+	/* Use .text as an anchor to emit PC-relative symbol references */
+	leaq	.text(%rip), %rbx
+
 #ifdef CONFIG_SMP
 	/*
 	 * For parallel boot, the APIC ID is read from the APIC, and then
@@ -288,10 +292,9 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 .Llookup_AP:
 	/* EAX contains the APIC ID of the current CPU */
 	xorl	%ecx, %ecx
-	leaq	cpuid_to_apicid(%rip), %rbx
 
 .Lfind_cpunr:
-	cmpl	(%rbx,%rcx,4), %eax
+	cmpl	cpuid_to_apicid - .text(%rbx,%rcx,4), %eax
 	jz	.Lsetup_cpu
 	inc	%ecx
 #ifdef CONFIG_FORCE_NR_CPUS
@@ -311,7 +314,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 
 .Lsetup_cpu:
 	/* Get the per cpu offset for the given CPU# which is in ECX */
-	movq	__per_cpu_offset(,%rcx,8), %rdx
+	movq	__per_cpu_offset - .text(%rbx,%rcx,8), %rdx
 #else
 	xorl	%edx, %edx /* zero-extended to clear all of RDX */
 #endif /* CONFIG_SMP */
@@ -322,7 +325,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 	 *
 	 * RDX contains the per-cpu offset
 	 */
-	movq	current_task(%rdx), %rax
+	movq	current_task - .text(%rbx,%rdx), %rax
 	movq	TASK_threadsp(%rax), %rsp
 
 	/*
@@ -343,7 +346,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 	 */
 	subq	$16, %rsp
 	movw	$(GDT_SIZE-1), (%rsp)
-	leaq	gdt_page(%rdx), %rax
+	leaq	gdt_page - .text(%rbx,%rdx), %rax
 	movq	%rax, 2(%rsp)
 	lgdt	(%rsp)
 	addq	$16, %rsp
Signature · 1 line
2.47.3
author maintainer
#46 lore raw

PATCH 18/19 x86/boot: Implement support for RELA/RELR/REL runtime relocations

In reply to Ard Biesheuvel

Given that the decompressor already incorporates an ELF loader that parses the program headers of the decompressed ELF image, support for dealing with the PT_DYNAMIC program header can be added quite easily, which describes the location of the RELA and RELR relocation tables in the image.

This is a more efficient, and more idiomatic format, which allows the handling of boot-time randomization (KASLR) in a generic manner, rather than based on a bespoke x86-specific relocation format. This is a prerequisite for enabling further hardening measures that are implemented in the ELF domain, i.e., fgkaslr.

---
 arch/x86/boot/compressed/misc.c | 73 +++++++++++++++++++-
 include/uapi/linux/elf.h        |  3 +
 2 files changed, 74 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/misc.c +71 -2
index 1ea419cf88fe..bc5677e697ca 100644
@@ -278,7 +278,68 @@ static inline void handle_relocations(void *output, unsigned long output_len,
 { }
 #endif
 
-static size_t parse_elf(void *output)
+#define ELF(type) __PASTE(__PASTE(Elf, __LONG_WIDTH__), __PASTE(_, type))
+
+static void handle_dynamic(const ELF(Dyn) *dyn, unsigned long p2v_offset,
+			   unsigned long va_shift)
+{
+	const ELF(Rela) *rela = NULL;
+	const ELF(Rel) *rel = NULL;
+	unsigned long *relr = NULL;
+	unsigned long *place;
+	int relasize = 0;
+	int relrsize = 0;
+	int relsize = 0;
+
+	for (auto d = dyn; d->d_tag != DT_NULL; d++) {
+		switch (d->d_tag) {
+		case DT_RELA:
+			rela = (void *)(d->d_un.d_ptr + p2v_offset);
+			break;
+		case DT_RELASZ:
+			relasize = d->d_un.d_val;
+			break;
+		case DT_RELR:
+			relr = (void *)(d->d_un.d_ptr + p2v_offset);
+			break;
+		case DT_RELRSZ:
+			relrsize = d->d_un.d_val;
+			break;
+		case DT_REL:
+			rel = (void *)(d->d_un.d_ptr + p2v_offset);
+			break;
+		case DT_RELSZ:
+			relsize = d->d_un.d_val;
+			break;
+		}
+	}
+
+	for (int i = 0; i < relasize / sizeof(*rela); i++) {
+		place = (unsigned long *)(rela[i].r_offset + p2v_offset);
+		*place += va_shift;
+	}
+
+	for (int i = 0; i < relrsize / sizeof(*relr); i++) {
+		if ((relr[i] & 1) == 0) {
+			place = (unsigned long *)(relr[i] + p2v_offset);
+			*place++ += va_shift;
+			continue;
+		}
+
+		for (unsigned long *p = place, r = relr[i] >> 1; r; p++, r >>= 1)
+			if (r & 1)
+				*p += va_shift;
+		place += 8 * sizeof(*relr) - 1;
+	}
+
+	for (int i = 0; i < relsize / sizeof(*rel); i++) {
+		place = (unsigned long *)(rel[i].r_offset + p2v_offset);
+		*place += va_shift;
+	}
+
+}
+
+static size_t parse_elf(void *output, u64 va_shift)
 {
 #ifdef CONFIG_X86_64
 	Elf64_Ehdr ehdr;
@@ -320,6 +381,12 @@ static size_t parse_elf(void *output)
 				dest += (unsigned long)output - LOAD_PHYSICAL_ADDR;
 			memmove(dest, output + phdr->p_offset, phdr->p_filesz);
 			break;
+		case PT_DYNAMIC:
+			if (!va_shift)
+				break;
+			dest = (void *)(output + phdr->p_paddr - LOAD_PHYSICAL_ADDR);
+			handle_dynamic(dest, (unsigned long)dest - phdr->p_vaddr, va_shift);
+			break;
 		default: /* Ignore other PT_* */ break;
 		}
 	}
@@ -351,7 +418,9 @@ unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr,
 			 NULL, error) < 0)
 		return ULONG_MAX;
 
-	entry = parse_elf(outbuf);
+	if (IS_ENABLED(CONFIG_X86_32))
+		virt_addr = (unsigned long)outbuf;
+	entry = parse_elf(outbuf, virt_addr - LOAD_PHYSICAL_ADDR);
 	handle_relocations(outbuf, output_len, virt_addr);
 
 	return entry;
include/uapi/linux/elf.h +3 -0
index 819ded2d39de..868cd67f0ea7 100644
@@ -103,6 +103,9 @@ typedef __u16	Elf64_Versym;
 #define DT_TEXTREL	22
 #define DT_JMPREL	23
 #define DT_ENCODING	32
+#define DT_RELRSZ	35
+#define DT_RELR		36
+#define DT_RELRENT	37
 #define OLD_DT_LOOS	0x60000000
 #define DT_LOOS		0x6000000d
 #define DT_HIOS		0x6ffff000
Signature · 1 line
2.47.3
author maintainer
#47 lore raw

PATCH 19/19 x86/kernel: Switch to PIE linking for the relocatable kernel

In reply to Ard Biesheuvel

If the toolchain supports RELR relocation packing, build the virtually relocatable kernels as Position Independent (PIE) Executables. This results in more efficient relocation processing for the virtual displacement of the kernel applied at boot, using RELR relocations that take up only a fraction of the space occupied by ordinary RELA relocations.

More importantly, it instructs the linker to generate a binary that is really meant to be relocated at boot, using data structures that are intended for this purpose. Doing so is important for a couple of reasons:

- Relying on --emit-relocs is problematic, because it produces the static relocations that are consumed by the linker as input, and these are not meant for describing a runtime relocatable image. For example, the linker may apply relaxations that result in the code and the static relocation going out of sync (and ld.bfd and ld.lld already handle this in a different way).

- The 'relocs' tool relies on manually kept allow/deny lists of symbol names. These are needed because ELF absolute/relative symbol designations are often inaccurate.

- x86 deviates from other architectures in the kernel when it comes to its implementation of boot-time relocation, making it difficult to implement further enhancements (e.g., fgkaslr, EFI zboot) in a portable manner.

Note that this means that all codegen on x86_64 should be position independent, to be compatible with PIE linking, but only if KASLR is enabled. On i386, no changes to the codegen are needed, as the ordinary position dependent relocation model is supported by the linker when operating in PIE mode.

---
 arch/x86/Kconfig              |  3 ++-
 arch/x86/Makefile             |  5 +++++
 arch/x86/kernel/vmlinux.lds.S | 18 ++++++++++++++++++
 3 files changed, 25 insertions(+), 1 deletion(-)
arch/x86/Kconfig +2 -1
index b3a64cfe04cf..2aa50aa8dc68 100644
@@ -103,6 +103,7 @@ config X86
 	select ARCH_HAS_NONLEAF_PMD_YOUNG	if PGTABLE_LEVELS > 2
 	select ARCH_HAS_UACCESS_FLUSHCACHE	if X86_64
 	select ARCH_HAS_COPY_MC			if X86_64
+	select ARCH_HAS_RELR
 	select ARCH_HAS_SET_MEMORY
 	select ARCH_HAS_SET_DIRECT_MAP
 	select ARCH_HAS_STRICT_KERNEL_RWX
@@ -2129,7 +2130,7 @@ config RANDOMIZE_BASE
 # Relocation on x86 needs some additional build support
 config X86_NEED_RELOCS
 	def_bool y
-	depends on RELOCATABLE
+	depends on RELOCATABLE && !TOOLS_SUPPORT_RELR
 	select ARCH_VMLINUX_NEEDS_RELOCS
 
 config PHYSICAL_ALIGN
arch/x86/Makefile +5 -0
index b211d6c950aa..7eac705c4ff4 100644
@@ -258,6 +258,11 @@ endif
 
 KBUILD_LDFLAGS += -m elf_$(UTS_MACHINE)
 
+ldflags-pie-$(CONFIG_LD_IS_LLD)		:= --apply-dynamic-relocs
+ldflags-pie-$(CONFIG_LD_IS_BFD)		:= -z call-nop=suffix-nop
+ldflags-$(CONFIG_RELOCATABLE_PIE)	:= --pie -z notext $(ldflags-pie-y)
+LDFLAGS_vmlinux				+= $(ldflags-y)
+
 #
 # The 64-bit kernel must be aligned to 2MB.  Pass -z max-page-size=0x200000 to
 # the linker to force 2MB page size regardless of the default page size used
arch/x86/kernel/vmlinux.lds.S +18 -0
index 6772fe9a9957..cfaf6ab80684 100644
@@ -127,6 +127,9 @@ PHDRS {
 	text PT_LOAD FLAGS(5);          /* R_E */
 	data PT_LOAD FLAGS(6);          /* RW_ */
 	note PT_NOTE FLAGS(0);          /* ___ */
+#ifdef CONFIG_RELOCATABLE_PIE
+	dynamic PT_DYNAMIC;
+#endif
 }
 
 SECTIONS
@@ -201,6 +204,21 @@ SECTIONS
 	DATA_SEGMENT_START
 	INIT_DATA_SECTION(16) :data
 
+#ifdef CONFIG_RELOCATABLE_PIE
+	/DISCARD/ : {
+		*(.interp .dynbss .eh_frame .sframe .relr.auth.dyn)
+	}
+
+	.dynamic	: { *(.dynamic) } :dynamic :data
+	.dynstr		: { *(.dynstr) } :data
+	.dynsym		: { *(.dynsym) }
+	.gnu.hash	: { *(.gnu.hash) }
+	.hash		: { *(.hash) }
+	.init.rela	: { *(.rela.*) *(.rela_*) }
+	.init.rel	: { *(.rel.*) *(.rel_*) }
+	.init.relr	: { *(.relr.*) }
+#endif
+
 	.x86_cpu_dev.init : AT(ADDR(.x86_cpu_dev.init) - LOAD_OFFSET) {
 		__x86_cpu_dev_start = .;
 		*(.x86_cpu_dev.init)
Signature · 1 line
2.47.3
#31 lore raw

Re: [RFC/RFT PATCH 11/19] x86/rethook: Use RIP-relative reference for fake return address

In reply to Ard Biesheuvel
Quoted 7 lines · On Thu, 8 Jan 2026 09:25:38 +0000 Ard Biesheuvel <[email protected]> wrote:

Pushing an immediate absolute address to the stack is not permitted when linking x86_64 code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.

So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.

The comment implies the address is 'fake'.
Does that mean it could just be a constant?
Clearly the unwinder would need the same change.

David

Quoted 30 lines
---
 arch/x86/kernel/rethook.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/rethook.c +5 -1
index 85e2f2d16a90..50812ac718b0 100644
@@ -11,6 +11,10 @@
 
 __visible void arch_rethook_trampoline_callback(struct pt_regs *regs);
 
+#ifdef CONFIG_X86_64
+static __used void * const __arch_rethook_trampoline = &arch_rethook_trampoline;
+#endif
+
 #ifndef ANNOTATE_NOENDBR
 #define ANNOTATE_NOENDBR
 #endif
@@ -27,7 +31,7 @@ asm(
 #ifdef CONFIG_X86_64
 	ANNOTATE_NOENDBR "\n"	/* This is only jumped from ret instruction */
 	/* Push a fake return address to tell the unwinder it's a rethook. */
-	"	pushq $arch_rethook_trampoline\n"
+	"	pushq __arch_rethook_trampoline(%rip)\n"
 	UNWIND_HINT_FUNC
 	"       pushq $" __stringify(__KERNEL_DS) "\n"
 	/* Save the 'sp - 16', this will be fixed later. */
author maintainer
#32 lore raw

Re: [RFC/RFT PATCH 11/19] x86/rethook: Use RIP-relative reference for fake return address

Quoted 9 lines · David Laight
Quoted 5 lines · On Thu, 8 Jan 2026 09:25:38 +0000 Ard Biesheuvel <[email protected]> wrote:

Pushing an immediate absolute address to the stack is not permitted when linking x86_64 code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.

Quoted 5 lines
Quoted 2 lines

So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.

The comment implies the address is 'fake'.
Does that mean it could just be a constant?

It could be, but it isn't, across all architectures.

Quoted 2 lines

Clearly the unwinder would need the same change.

Why? The value being pushed is the same.

author maintainer
#33 lore raw

Re: [RFC/RFT PATCH 11/19] x86/rethook: Use RIP-relative reference for fake return address

Quoted 17 lines · Ard Biesheuvel
Quoted 9 lines · David Laight
Quoted 5 lines · On Thu, 8 Jan 2026 09:25:38 +0000 Ard Biesheuvel <[email protected]> wrote:

Pushing an immediate absolute address to the stack is not permitted when linking x86_64 code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.

Quoted 5 lines
Quoted 2 lines

So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.

The comment implies the address is 'fake'.
Does that mean it could just be a constant?

Quoted 6 lines

It could be, but it isn't, across all architectures.

Quoted 2 lines

Clearly the unwinder would need the same change.

Why? The value being pushed is the same.

Never mind - I guess you meant 'the unwinder would need to use the constant too'

#48 lore raw
In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>
Date: Thu, 8 Jan 2026 09:25:27 +0000

Quoted 29 lines

This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].

The main sticking point is the fact that PIE linking on x86_64 requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.

This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)

So now that we have some actual numbers, I would like to try and revisit this discussion, and get a conclusion on whether this is really a non-starter. Note that only the KASLR kernel would rely on this, and disabling CONFIG_RANDOMIZE_BASE will revert to the current situation (provided that patch #4 is applied)

Some minor asm tweaks are needed too (patches #9 - #17), but those all seem uncontroversial to me.

The first 5 patches are general cleanup, and could be taken into consideration independently of the discussion around PIC codegen.

Quoted 11 lines

[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).

In case anybody is interested...
The latest (to my knowledge) experiments with FG-KALSR was my side project reviving Kristen's old series (and then rewriting it completely): [0]

I haven't worked on it since then, as I work in an XDP/netmem/whatever team, i.e. networking, not x86, and free time for side projects shrunk severely since 2022.

Maybe someone would pick it up again some day, just like I picked up Kristen's series back then...

[0] https://github.com/alobakin/linux/commits/fgkaslr

Thanks,
Olek

#49 lore raw
In reply to Ard Biesheuvel
Quoted 4 lines · Ard Biesheuvel

This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].

Kristin Accardi had fg-kasrl running without that, didn't she?

From your footnotes, it looks like what you are *really* asking for is to pessimize x86 code to benefit other architectures. That isn't inherently wrong, but stating it as you have above is dishonest.

Quoted 11 lines

The main sticking point is the fact that PIE linking on x86_64 requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.

This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)

Could you show some examples of how the code changes?

-hpa

Quoted 13 lines

[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).

Quoted 6 lines

[2] EFI zboot is already used by various architectures that have no decompressor stage at all (arm64, RISC-V, LoongArch), and this format can be combined with an ELF payload too. EFI zboot accommodates non-EFI boot chains by describing the size, offset, payload type and compression type in its header, so that it can be extracted and booted by other means.

The bzImage format already have that for all practical purposes. We *really* don't want to introduce a new binary format for the x86 kernel. A bunch of such attempts have been done in the past, and it is nothing but a mess that breaks things, because now you are encouraging different bootloaders to support a non-overlapping set of binary formats.

STRONG NAK on that one.

-hpa

#38 lore raw

Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S

In reply to Ard Biesheuvel
Quoted 27 lines · Ard Biesheuvel

Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.

Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.

---
 arch/x86/kernel/acpi/wakeup_64.S | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +1 -1
index 04f561f75e99..15233a4e1c95 100644
@@ -14,7 +14,7 @@
 
 .code64
        /*
-        * Hooray, we are in Long 64-bit mode (but still running in low memory)
+        * Hooray, we are in Long 64-bit mode
         */
 SYM_FUNC_START(wakeup_long64)
        ANNOTATE_NOENDBR

arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46

Quoted 7 lines
arch/x86/kernel/acpi/wakeup_64.S +1 -1
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64)
        movq    saved_rsi(%rip), %rsi
        movq    saved_rbp(%rip), %rbp
 
-       movq    saved_rip(%rip), %rax
+       leaq    .Lresume_point(%rip), %rax
        ANNOTATE_RETPOLINE_SAFE
        jmp     *%rax

If this is already running on the virtual mapping, this can simply be changed to a direct jump.

Brian Gerst

author maintainer
#39 lore raw

Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S

Quoted 32 lines · Brian Gerst
Quoted 30 lines · Ard Biesheuvel

Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.

Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.

---
 arch/x86/kernel/acpi/wakeup_64.S | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +1 -1
index 04f561f75e99..15233a4e1c95 100644
@@ -14,7 +14,7 @@
 
 .code64
        /*
-        * Hooray, we are in Long 64-bit mode (but still running in low memory)
+        * Hooray, we are in Long 64-bit mode
         */
 SYM_FUNC_START(wakeup_long64)
        ANNOTATE_NOENDBR
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64)
        movq    saved_rsi(%rip), %rsi
        movq    saved_rbp(%rip), %rbp
 
Quoted 8 lines

arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46

Quoted 4 lines
arch/x86/kernel/acpi/wakeup_64.S +1 -1
-       movq    saved_rip(%rip), %rax
+       leaq    .Lresume_point(%rip), %rax
        ANNOTATE_RETPOLINE_SAFE
        jmp     *%rax

If this is already running on the virtual mapping, this can simply be changed to a direct jump.

Indeed, but I couldn't figure out how to do so without making objtool unhappy.

author maintainer
#50 lore raw
In reply to H. Peter Anvin
Quoted 9 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel

This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].

Kristin Accardi had fg-kasrl running without that, didn't she?

Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.

One could argue that this example is just a Clang bug, but it is very difficult to make that case with the toolchain developers, given that --emit-relocs (which is what tells the linker to emit the relocations that it received as input) has no specification, and some linker relaxations are not representable as static relocations to begin with (but to be fair, that currently mostly affects other architectures, but there is no reason this could never happen on x86)

Doing fgkaslr properly (IMHO) means supporting things like live patch and debug seamlessly, and in a portable manner. Toolchain support is critical, and securing that for a one-off x86 implementation rather than one that can be used across architectures and other bare-metal projects is going to be difficult.

Quoted 4 lines

From your footnotes, it looks like what you are *really* asking for is to pessimize x86 code to benefit other architectures. That isn't inherently wrong, but stating it as you have above is dishonest.

I was hoping to save the ad-hominems for later in the thread, when things *really* heat up.

The point is not to benefit other architectures. The point is to implement something once, and deploy it on all architectures in the same way. ELF is the greatest common denominator across the entire ecosystem, and so using idiomatic ELF to describe how to load the image and how to move it around in the virtual address space is on obvious choice.

Quoted 6 lines
Quoted 6 lines

The main sticking point is the fact that PIE linking on x86_64 requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.

Quoted 8 lines
Quoted 5 lines

This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)

Could you show some examples of how the code changes?

Taking the address of a symbol (same code size)

0: 48 c7 c0 00 00 00 00 mov $0x0,%rax
3: R_X86_64_32S sym

7: 48 8d 05 00 00 00 00 lea 0x0(%rip),%rax # 0xe
a: R_X86_64_PC32

Loading a global variable from memory (one byte shorter in PIC)

e: 48 8b 04 25 00 00 00 mov 0x0,%rax
15: 00
12: R_X86_64_32S sym

16: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 0x1d
19: R_X86_64_PC32 sym-0x4

Indexing a global array (3 bytes longer in PIC, needs an additional GPR if source and destination are the same)

1d: 48 8b 04 c5 00 00 00 mov 0x0(,%rax,8),%rax
24: 00
21: R_X86_64_32S array

25: 48 8d 15 00 00 00 00 lea 0x0(%rip),%rdx # 0x2c
28: R_X86_64_PC32 array-0x4
2c: 48 8b 04 c2 mov (%rdx,%rax,8),%rax

Pushing the address of a symbol to the stack ((3 bytes longer in PIC, needs an additional GPR)

30: 68 00 00 00 00 push $0x0
31: R_X86_64_32S sym

35: 48 8d 05 00 00 00 00 lea 0x0(%rip),%rax # 0x3c
38: R_X86_64_PC32 sym-0x4
3c: 50 push %rax

Jump tables look completely different, but the table itself is only half the size. Even for non-PIC, jump tables are problematic for objtool, and so these need to be annotated by the compiler. I have some unfinished Clang patches that implement this, which I hope to get back to soon.

The asm patches in the series should give a good impression of how the code changes.

Quoted 20 lines
Quoted 13 lines

[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).

Quoted 6 lines

[2] EFI zboot is already used by various architectures that have no decompressor stage at all (arm64, RISC-V, LoongArch), and this format can be combined with an ELF payload too. EFI zboot accommodates non-EFI boot chains by describing the size, offset, payload type and compression type in its header, so that it can be extracted and booted by other means.

Quoted 8 lines

The bzImage format already have that for all practical purposes. We *really* don't want to introduce a new binary format for the x86 kernel. A bunch of such attempts have been done in the past, and it is nothing but a mess that breaks things, because now you are encouraging different bootloaders to support a non-overlapping set of binary formats.

STRONG NAK on that one.

I think it should be feasible to implement a hybrid bzImage/EFI zboot format. There is already prior art in loaders that decompress the ELF payload directly (Xen).

Given that a x86_64 bootloader running in long mode needs to do very little beyond loading the ELF at some arbitrary 2M aligned offset and calling the entrypoint with a struct bootparams in %RDI, most of the logic in the decompressor is really only needed when booting in 32-bit mode.

So I think there is value in having a generic boot format that can be consumed by EFI directly, or by a generic ELF vmlinux loader (library) that understands the EFI zboot format and knows how to extract the ELF payload. I'd strongly prefer only a single idiom for describing the relocations in the image.

On other architectures (i.e., without decompressor), EFI zboot would be a prerequisite for fgkaslr, but it is up to the platform to decide whether to boot via EFI or load the ELF and apply the relocations. On x86_64, the same tooling would work seamlessly, but the decompressor could apply the relocations itself as well.

#40 lore raw

Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S

In reply to Ard Biesheuvel
Quoted 43 lines · Ard Biesheuvel
Quoted 32 lines · Brian Gerst
Quoted 30 lines · Ard Biesheuvel

Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.

Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.

---
 arch/x86/kernel/acpi/wakeup_64.S | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +1 -1
index 04f561f75e99..15233a4e1c95 100644
@@ -14,7 +14,7 @@
 
 .code64
        /*
-        * Hooray, we are in Long 64-bit mode (but still running in low memory)
+        * Hooray, we are in Long 64-bit mode
         */
 SYM_FUNC_START(wakeup_long64)
        ANNOTATE_NOENDBR
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64)
        movq    saved_rsi(%rip), %rsi
        movq    saved_rbp(%rip), %rbp
 
Quoted 8 lines

arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46

Quoted 4 lines
arch/x86/kernel/acpi/wakeup_64.S +1 -1
-       movq    saved_rip(%rip), %rax
+       leaq    .Lresume_point(%rip), %rax
        ANNOTATE_RETPOLINE_SAFE
        jmp     *%rax

If this is already running on the virtual mapping, this can simply be changed to a direct jump.

Quoted 1 line

Indeed, but I couldn't figure out how to do so without making objtool unhappy.

I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?

author maintainer
#41 lore raw

Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S

Quoted 45 lines · Brian Gerst
Quoted 43 lines · Ard Biesheuvel
Quoted 40 lines · Brian Gerst
Quoted 27 lines · Ard Biesheuvel

Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.

Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.

---
 arch/x86/kernel/acpi/wakeup_64.S | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +1 -1
index 04f561f75e99..15233a4e1c95 100644
@@ -14,7 +14,7 @@
 
 .code64
        /*
-        * Hooray, we are in Long 64-bit mode (but still running in low memory)
+        * Hooray, we are in Long 64-bit mode
         */
 SYM_FUNC_START(wakeup_long64)
        ANNOTATE_NOENDBR

arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46

Quoted 7 lines
arch/x86/kernel/acpi/wakeup_64.S +1 -1
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64)
        movq    saved_rsi(%rip), %rsi
        movq    saved_rbp(%rip), %rbp
 
-       movq    saved_rip(%rip), %rax
+       leaq    .Lresume_point(%rip), %rax
        ANNOTATE_RETPOLINE_SAFE
        jmp     *%rax

If this is already running on the virtual mapping, this can simply be changed to a direct jump.

Quoted 4 lines
Quoted 1 line

Indeed, but I couldn't figure out how to do so without making objtool unhappy.

I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?

arch/x86/kernel/acpi/wakeup_64.o: warning: objtool: wakeup_long64() falls through to next function do_suspend_lowlevel()

author maintainer
#42 lore raw

Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S

Quoted 47 lines · Ard Biesheuvel
Quoted 45 lines · Brian Gerst
Quoted 43 lines · Ard Biesheuvel
Quoted 40 lines · Brian Gerst
Quoted 27 lines · Ard Biesheuvel

Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.

Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.

---
 arch/x86/kernel/acpi/wakeup_64.S | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +1 -1
index 04f561f75e99..15233a4e1c95 100644
@@ -14,7 +14,7 @@
 
 .code64
        /*
-        * Hooray, we are in Long 64-bit mode (but still running in low memory)
+        * Hooray, we are in Long 64-bit mode
         */
 SYM_FUNC_START(wakeup_long64)
        ANNOTATE_NOENDBR

arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46

Quoted 7 lines
arch/x86/kernel/acpi/wakeup_64.S +1 -1
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64)
        movq    saved_rsi(%rip), %rsi
        movq    saved_rbp(%rip), %rbp
 
-       movq    saved_rip(%rip), %rax
+       leaq    .Lresume_point(%rip), %rax
        ANNOTATE_RETPOLINE_SAFE
        jmp     *%rax

If this is already running on the virtual mapping, this can simply be changed to a direct jump.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Indeed, but I couldn't figure out how to do so without making objtool unhappy.

I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?

arch/x86/kernel/acpi/wakeup_64.o: warning: objtool: wakeup_long64() falls through to next function do_suspend_lowlevel()

Note that this is x86_64_defconfig with CONFIG_X86_KERNEL_IBT disabled.

#43 lore raw

Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S

Quoted 51 lines · Ard Biesheuvel
Quoted 49 lines · Ard Biesheuvel
Quoted 47 lines · Brian Gerst
Quoted 44 lines · Ard Biesheuvel
Quoted 32 lines · Brian Gerst
Quoted 30 lines · Ard Biesheuvel

Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.

Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.

---
 arch/x86/kernel/acpi/wakeup_64.S | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +1 -1
index 04f561f75e99..15233a4e1c95 100644
@@ -14,7 +14,7 @@
 
 .code64
        /*
-        * Hooray, we are in Long 64-bit mode (but still running in low memory)
+        * Hooray, we are in Long 64-bit mode
         */
 SYM_FUNC_START(wakeup_long64)
        ANNOTATE_NOENDBR
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64)
        movq    saved_rsi(%rip), %rsi
        movq    saved_rbp(%rip), %rbp
 
Quoted 8 lines

arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46

Quoted 4 lines
arch/x86/kernel/acpi/wakeup_64.S +1 -1
-       movq    saved_rip(%rip), %rax
+       leaq    .Lresume_point(%rip), %rax
        ANNOTATE_RETPOLINE_SAFE
        jmp     *%rax

If this is already running on the virtual mapping, this can simply be changed to a direct jump.

Indeed, but I couldn't figure out how to do so without making objtool unhappy.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?

arch/x86/kernel/acpi/wakeup_64.o: warning: objtool: wakeup_long64() falls through to next function do_suspend_lowlevel()

Note that this is x86_64_defconfig with CONFIG_X86_KERNEL_IBT disabled.

I do see the error now. I had missed it when building the .o file and thought it would show when linking vmlinux.

I was able to make objtool happy by embedding wakeup_long64() into do_suspend_lowlevel() and using SYM_INNER_LABEL_ALIGN().

#22 lore raw

Re: [RFC/RFT PATCH 08/19] x86: Use PIE codegen for the relocatable 64-bit kernel

In reply to Ard Biesheuvel

arch/x86/Makefile endif · line 184

Quoted 12 lines · Ard Biesheuvel

As an intermediate step towards enabling PIE linking for the x86_64 KASLR kernel, enable PIE codegen for all C and Rust objects that are linked into the kernel proper. Add a Kconfig option RELOCATABLE_PIE for this, depending on RELR support in the linker, as the relocation tables will blow up the kernel image otherwise.

KBUILD_CFLAGS += $(call cc-option,-fcf-protection=branch -fno-jump-tables)
...

arch/x86/Makefile +2 -0
+
+        pie-ccflags-$(CONFIG_CC_IS_GCC) += $(call cc-option.-mdirect-extern-access)

I think that dot there next to -mdirect-extern-accesss should have been a comma.

author maintainer
#23 lore raw

Re: [RFC/RFT PATCH 08/19] x86: Use PIE codegen for the relocatable 64-bit kernel

Quoted 11 lines · Jan Engelhardt
Quoted 6 lines · Ard Biesheuvel

As an intermediate step towards enabling PIE linking for the x86_64 KASLR kernel, enable PIE codegen for all C and Rust objects that are linked into the kernel proper. Add a Kconfig option RELOCATABLE_PIE for this, depending on RELR support in the linker, as the relocation tables will blow up the kernel image otherwise.

Quoted 8 lines

arch/x86/Makefile endif · line 184

Quoted 4 lines

KBUILD_CFLAGS += $(call cc-option,-fcf-protection=branch -fno-jump-tables)
...

arch/x86/Makefile +2 -0
+
+        pie-ccflags-$(CONFIG_CC_IS_GCC) += $(call cc-option.-mdirect-extern-access)

I think that dot there next to -mdirect-extern-accesss should have been a comma.

Indeed, thanks for spotting that!

#15 lore raw

Re: [RFC/RFT PATCH 04/19] x86: Make the 64-bit bzImage always physically relocatable

In reply to Ard Biesheuvel
Quoted 11 lines · Ard Biesheuvel

On x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.

On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.

Quoted 7 lines

Clarify this in Kconfig and in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whether CONFIG_RELOCATABLE is set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite for EFI_STUB and RANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.

I don't see any reason to support non-relocatable kernels anymore. In fact, in a patchset I am working on I have already removed it.

-hpa

#16 lore raw

Re: [RFC/RFT PATCH 04/19] x86: Make the 64-bit bzImage always physically relocatable

Quoted 20 lines · On Sun, 11 Jan 2026 20:01:02 -0800 "H. Peter Anvin" <[email protected]> wrote:
Quoted 11 lines · Ard Biesheuvel

On x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.

On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.

Quoted 7 lines

Clarify this in Kconfig and in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whether CONFIG_RELOCATABLE is set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite for EFI_STUB and RANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.

Quoted 2 lines

I don't see any reason to support non-relocatable kernels anymore. In fact, in a patchset I am working on I have already removed it.

For just 64bit, or 32bit as well?
The 'bloat' for 32bit will be higher due to the lack of pc-relative addressing.

David

Quoted 4 lines

-hpa

#17 lore raw

Re: [RFC/RFT PATCH 04/19] x86: Make the 64-bit bzImage always physically relocatable

Quoted 23 lines · David Laight
Quoted 20 lines · On Sun, 11 Jan 2026 20:01:02 -0800 "H. Peter Anvin" <[email protected]> wrote:
Quoted 18 lines · Ard Biesheuvel

On x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.

On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.

Clarify this in Kconfig and in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whether CONFIG_RELOCATABLE is set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite for EFI_STUB and RANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.

Quoted 14 lines
Quoted 2 lines

I don't see any reason to support non-relocatable kernels anymore. In fact, in a patchset I am working on I have already removed it.

For just 64bit, or 32bit as well?
The 'bloat' for 32bit will be higher due to the lack of pc-relative addressing.

David

Quoted 4 lines

-hpa

Either. The bloat is strictly boot time.

maintainer
#51 lore raw
In reply to Ard Biesheuvel
Quoted 9 lines · Ard Biesheuvel
Quoted 8 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel

This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].

Kristin Accardi had fg-kasrl running without that, didn't she?

I understand "such as fg-kaslr" to have been just a terse way of saying "such as a complete multi-architectural fg-kaslr"

Quoted 11 lines

Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.

Another historical bit of context is that one of the main reasons Kristen's fg-kaslr got stuck was the linker support needed for (the 65k worth of) section pass-through. That never got resolved, and the solutions either required huge linker files (that tickled performance flaws in the linkers) that resulted in 10 minute linking times, or to disable all the orphan section handling, which was a regression in our sanity checking and bug-finding.

So, getting a well-behaved fg-kaslr still needs toolchain support, and getting there is going to need further design work. As far as PIE, this just makes the fg-kaslr toolchain work easier (fewer special cases), along with all the other benefits of moving to PIE.

-Kees

Signature · 1 line
Kees Cook
#26 lore raw

Re: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing

In reply to Ard Biesheuvel
Quoted 15 lines · Ard Biesheuvel

Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/kernel/kvm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/kvm.c +0 -0
index df78ddee0abb..1a0335f328e1 100644

arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810

Quoted 6 lines
arch/x86/kernel/kvm.c +1 -2
@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);
  * restoring to/from the stack.
  */
 #define PV_VCPU_PREEMPTED_ASM						     \
- "movq   __per_cpu_offset(,%rdi,8), %rax\n\t"				     \
- "cmpb   $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \
+ "0:leaq 0b(%rip), %rax\n\t"						     \

Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.

It might also be worth calling out in the changelog that this function is called across CPUs, e.g. from kvm_smp_send_call_func_ipi(), and thus can't use gs:
or any other "normal" method for accessing per-CPU data.

Quoted 8 lines
arch/x86/kernel/kvm.c +2 -0
+ "addq   __per_cpu_offset - 0b(%rax,%rdi,8), %rax\n\t"			     \
+ "cmpb   $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time-0b(%rax)\n\t" \
  "setne  %al\n\t"
 
 DEFINE_ASM_FUNC(__raw_callee_save___kvm_vcpu_is_preempted,
Signature · 1 line
2.47.3
#27 lore raw

Re: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing

Quoted 11 lines · On Tue, 20 Jan 2026 09:04:26 -0800 Sean Christopherson <[email protected]>…
Quoted 10 lines · Ard Biesheuvel

Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/kernel/kvm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Quoted 15 lines

arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810

Quoted 11 lines
arch/x86/kernel/kvm.c +1 -2
index df78ddee0abb..1a0335f328e1 100644
@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);
  * restoring to/from the stack.
  */
 #define PV_VCPU_PREEMPTED_ASM						     \
- "movq   __per_cpu_offset(,%rdi,8), %rax\n\t"				     \
- "cmpb   $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \
+ "0:leaq 0b(%rip), %rax\n\t"						     \  

Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.

I remember taking a while to grok that as well.

Can't you just use . as in:

	leaq	.(%rip), %rax

shame the assembler doesn't understand:

	movq	%rip, %rax

(maybe it does...)

David

#52 lore raw
In reply to Kees Cook
Quoted 34 lines · Kees Cook
Quoted 9 lines · Ard Biesheuvel
Quoted 8 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel

This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].

Kristin Accardi had fg-kasrl running without that, didn't she?

I understand "such as fg-kaslr" to have been just a terse way of saying "such as a complete multi-architectural fg-kaslr"

Quoted 11 lines

Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.

Another historical bit of context is that one of the main reasons Kristen's fg-kaslr got stuck was the linker support needed for (the 65k worth of) section pass-through. That never got resolved, and the solutions either required huge linker files (that tickled performance flaws in the linkers) that resulted in 10 minute linking times, or to disable all the orphan section handling, which was a regression in our sanity checking and bug-finding.

Quoted 5 lines

So, getting a well-behaved fg-kaslr still needs toolchain support, and getting there is going to need further design work. As far as PIE, this just makes the fg-kaslr toolchain work easier (fewer special cases), along with all the other benefits of moving to PIE.

As I *explicitly* stated earlier, there isn't anything inherently wrong with putting a small onus on x86 in order to make the general Linux code better -- but please, be honest about it *so we know what the actual tradeoffs are*.

For x86, we really do want to maintain the kernel memory model, which allows us to directly reference symbols in complex address expressions and to directly jump across modules. This means the "PIE" will need to be different from the way PIE works in user space, which is in part designed to avoid needing to dirty readonly pages, which would inhibit sharing -- which is explicitly NOT a concern for the kernel.

So that is one thing that the toolchain needs to be able to do.

I fully expect that we will continue to need to have some kinds of overrides for specific symbols, too, because there aren't any really sane ways to express them to the toolchain; this especially applies to linker-script and some assembly symbols. For example, the real-mode code (which uses the reloc tool as well) has to support segment and segbase-relative relocations, which are something that ELF simply has no concept of.

I have a lot more of an issue with trying to change the x86 boot protocol, simply because the way booting works in x86 has been incredibly successful; yes, the bzImage file format is ugly as ****, but that is a direct result of 34 years of continuous backwards compatibility. One of the reasons we have been able to do that is that we have *explicitly* rejected other boot models, such as Grub's self-declared Multiboot "standard" (which they have had to revise multiple times by now) and the early Xen boot model of booting vmlinux directly. We have added *many* capabilities to bzImage as needed, and it has turned out to be quite flexible in the end.

That, in turn, has been possible exactly *because* the Linux kernel provides a "prekernel". I don't even really like calling it the "decompressor" anymore; it really has developed far beyond that.

Every time you introduce a new boot model you take a serious risk that a boot loader author will say "hey, I'll just support this new model", and you *also* take the serious risk that the boot model isn't adequate. When the Grub authors says "we started using the 'modern' Linux entry point" -- meaning they would bypass the entry stub and call the kernel entry point directly -- they *reduced* the overall functionality, because history has shown that:

a. The kernel image is far easier for the end user to update than the potentially many bootloaders, because the bootloader depends on the deployment model (consider network booting, for example, or when the bootloader is in firmware.)

b. The kernel image provides ONE CENTRAL PLACE to add features, fix bugs, and develop workarounds for strange systems.

Thus, I will do anything I can to continue to veto changes to the x86 boot model, unless they come with a VERY VERY good motivation.

We originally made the mistake with EFI to leave too much to the bootloader; because of the downsides above we really needed to backpedal and take over control much earlier in the flow, just as with BIOS -- apparently to the Grub developers' utterly inexplicable chagrin.

-hpa

author maintainer
#28 lore raw

Re: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing

In reply to David Laight
Quoted 31 lines · David Laight
Quoted 11 lines · On Tue, 20 Jan 2026 09:04:26 -0800 Sean Christopherson <[email protected]>…
Quoted 10 lines · Ard Biesheuvel

Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/kernel/kvm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Quoted 15 lines

arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810

Quoted 11 lines
arch/x86/kernel/kvm.c +1 -2
index df78ddee0abb..1a0335f328e1 100644
@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);
  * restoring to/from the stack.
  */
 #define PV_VCPU_PREEMPTED_ASM                                                   \
- "movq   __per_cpu_offset(,%rdi,8), %rax\n\t"                                   \
- "cmpb   $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \
+ "0:leaq 0b(%rip), %rax\n\t"                                                    \

Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.

Quoted 5 lines

I remember taking a while to grok that as well.

Can't you just use . as in:

        leaq    .(%rip), %rax

How would the other two instructions referring to '0b' in their immediates refer to '.' in that case?

#29 lore raw

Re: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing

Quoted 32 lines · On Tue, 20 Jan 2026 21:54:30 +0100 Ard Biesheuvel <[email protected]> wrote:
Quoted 31 lines · David Laight
Quoted 26 lines · On Tue, 20 Jan 2026 09:04:26 -0800 Sean Christopherson <[email protected]>…
Quoted 15 lines · Ard Biesheuvel

Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.

---
 arch/x86/kernel/kvm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/kvm.c +0 -0
index df78ddee0abb..1a0335f328e1 100644

arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810

Quoted 6 lines
arch/x86/kernel/kvm.c +1 -2
@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);
  * restoring to/from the stack.
  */
 #define PV_VCPU_PREEMPTED_ASM                                                   \
- "movq   __per_cpu_offset(,%rdi,8), %rax\n\t"                                   \
- "cmpb   $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \
+ "0:leaq 0b(%rip), %rax\n\t"                                                    \  

Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.

Quoted 8 lines
Quoted 5 lines

I remember taking a while to grok that as well.

Can't you just use . as in:

        leaq    .(%rip), %rax

How would the other two instructions referring to '0b' in their immediates refer to '.' in that case?

I'd forgotten about those, not in the quoted bit of patch :-(

Traditionally (going back to MACRO-11) numeric labels would start at 10 and go up in 10s, TECO had a nice macro to renumber them. (Showing my age again)

David

author maintainer
#53 lore raw
In reply to H. Peter Anvin
Quoted 42 lines · H. Peter Anvin
Quoted 34 lines · Kees Cook
Quoted 9 lines · Ard Biesheuvel
Quoted 8 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel

This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].

Kristin Accardi had fg-kasrl running without that, didn't she?

I understand "such as fg-kaslr" to have been just a terse way of saying "such as a complete multi-architectural fg-kaslr"

Quoted 11 lines

Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.

Another historical bit of context is that one of the main reasons Kristen's fg-kaslr got stuck was the linker support needed for (the 65k worth of) section pass-through. That never got resolved, and the solutions either required huge linker files (that tickled performance flaws in the linkers) that resulted in 10 minute linking times, or to disable all the orphan section handling, which was a regression in our sanity checking and bug-finding.

Quoted 5 lines

So, getting a well-behaved fg-kaslr still needs toolchain support, and getting there is going to need further design work. As far as PIE, this just makes the fg-kaslr toolchain work easier (fewer special cases), along with all the other benefits of moving to PIE.

Quoted 4 lines

As I *explicitly* stated earlier, there isn't anything inherently wrong with putting a small onus on x86 in order to make the general Linux code better -- but please, be honest about it *so we know what the actual tradeoffs are*.

It is not just about the general Linux code. The x86 fgkaslr implementation was never merged because the toolchain side needs changes. And convincing the toolchain maintainers to take our changes is difficult if we keep using relocation tables that are not fit for purpose to perform runtime fixups on code that was built using the 'kernel' code model, which is explicitly position dependent.

Quoted 3 lines

For x86, we really do want to maintain the kernel memory model, which allows us to directly reference symbols in complex address expressions and to directly jump across modules.

AIUI, those complex address expressions are mostly indexed loads from global arrays, which do get slightly less efficient, but not in a way that was noticeable in any benchmarking I did (or LKP for that matter, which generally sniffs out any performance regressions). This includes jump tables, but as I already explained, RIP-relative jump tables have an upside too, given that the table itself is only half the size.

The ability to directly jump across modules is not affected at all by these changes.

Quoted 5 lines

This means the "PIE" will need to be different
from the way PIE works in user space, which is in part designed to avoid needing to dirty readonly pages, which would inhibit sharing -- which is explicitly NOT a concern for the kernel.

This is already implemented in this series: no GOT entries are permitted, and text relocations are allowed.

Quoted 2 lines

So that is one thing that the toolchain needs to be able to do.

It already can, and this series makes use of it.

Note that the size of the relocation table taken from an allmodconfig bzImage drops from 7.3 M to 2.4 M (defconfig goes from 800k to 45k), so there is a minor intrinsic benefit to these changes as well. But it is mostly about moving away from bespoke tooling and formats that are becoming more of a maintenance burden as the number of supported toolchains and languages increases.

Quoted 7 lines

I fully expect that we will continue to need to have some kinds of overrides for specific symbols, too, because there aren't any really sane ways to express them to the toolchain; this especially applies to linker-script and some assembly symbols. For example, the real-mode code (which uses the reloc tool as well) has to support segment and segbase-relative relocations, which are something that ELF simply has no concept of.

The real mode trampoline is not affected at all by these changes, given that it is built as a separate executable. Using a bespoke relocation format there is fine, because it is internal ABI.

Quoted 10 lines

I have a lot more of an issue with trying to change the x86 boot protocol, simply because the way booting works in x86 has been incredibly successful; yes, the bzImage file format is ugly as ****, but that is a direct result of 34 years of continuous backwards compatibility. One of the reasons we have been able to do that is that we have *explicitly* rejected other boot models, such as Grub's self-declared Multiboot "standard" (which they have had to revise multiple times by now) and the early Xen boot model of booting vmlinux directly. We have added *many* capabilities to bzImage as needed, and it has turned out to be quite flexible in the end.

Quoted 4 lines

That, in turn, has been possible exactly *because* the Linux kernel provides a "prekernel". I don't even really like calling it the "decompressor" anymore; it really has developed far beyond that.

The decompressor is needed when booting the 64-bit kernel from a boot loader that calls it in 32-bit mode.

When entering in long mode, with all memory mapped 1:1 (or at least, the kernel image itself, and all assets in memory that the bootloader exposes to the kernel), the decompressor does nothing useful, and all the problems it solves (by doing demand paging etc) only exist because it created them in the first place.

SEV-SNP confidential compute made an even bigger mess of this, because it can trigger #VC exceptions too, which also need to be handled.

Note that the EFI stub does not bother with the decompressor anymore, and unpacks and boots vmlinux directly. This was needed because the decompressor fundamentally relies on memory that is both writable and executable (as it moves its own executable image around in memory), which is difficult to reconcile with recent PC firmware implementations that are pedantic about mapping memory RWX.

But actually, I am not proposing to get rid of bzImage. I am proposing to make it more transparent so generic bootloader components can be constructed that consume the ELF directly.

maintainer
#2 lore raw

Re: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section

In reply to Ard Biesheuvel
Quoted 32 lines · Ard Biesheuvel

Currently, idt_table is allocated as page-aligned .bss, and remapped read-only after init. This breaks a 2 MiB large page into 4k page mappings, which defeats some of the effort done at boot to map the kernel image using large pages, for improved TLB efficiency.

Mark this allocation as __ro_after_init instead, so it will be made read-only automatically after boot, without breaking up large page mappings.

This also fixes a latent bug on i386, where the size of idt_table is less than a page, and so remapping it read-only could potentially affect other read-write variables too, if those are not page-aligned as well.

---
 arch/x86/kernel/idt.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
arch/x86/kernel/idt.c +1 -1
index f445bec516a0..d6da25d7964f 100644
@@ -170,7 +170,7 @@ static const __initconst struct idt_data apic_idts[] = {
 };
 
 /* Must be page-aligned because the real IDT is used in the cpu entry area */
-static gate_desc idt_table[IDT_ENTRIES] __page_aligned_bss;
+static gate_desc idt_table[IDT_ENTRIES] __aligned(PAGE_SIZE) __ro_after_init;
 
 static struct desc_ptr idt_descr __ro_after_init = {
 	.size		= IDT_TABLE_SIZE - 1,

arch/x86/kernel/idt.c void __init idt_setup_apic_and_irq_gates(void) · line 312

Quoted 8 lines
arch/x86/kernel/idt.c +0 -3
@@ -308,9 +308,6 @@ void __init idt_setup_apic_and_irq_gates(void)
 	idt_map_in_cea();
 	load_idt(&idt_descr);
 
-	/* Make the IDT table read only */
-	set_memory_ro((unsigned long)&idt_table, 1);
-
 	idt_setup_done = true;
 }

Good idea, except my guest shows me something else:

before:

[    0.186281] IDT table: 0xffffffff89c7f000
0xffffffff89c00000-0xffffffff89c7f000         508K     RW                 GLB NX pte
0xffffffff89c7f000-0xffffffff89c80000           4K     ro                 GLB NX pte
0xffffffff89c80000-0xffffffff89e00000        1536K     RW                 GLB NX pte
0xffffffff89e00000-0xffffffff8be00000          32M     RW         PSE     GLB NX pmd

This is clearly a single, 4K RO pageframe right in the middle of a splintered 2M page.

after:

[    0.180635] IDT table: 0xffffffff822cf000
0xffffffff81e00000-0xffffffff82200000           4M     ro         PSE     GLB NX pmd
0xffffffff82200000-0xffffffff8236f000        1468K     ro                 GLB NX pte
0xffffffff8236f000-0xffffffff82400000         580K     RW                 GLB NX pte
0xffffffff82400000-0xffffffff89800000         116M     RW         PSE     GLB NX pmd

but after applying your patch it looks like it still broke the 2M mapping as the remaining piece is RW.

If I do this:

static gate_desc idt_table[IDT_ENTRIES] __aligned(PMD_SIZE) __ro_after_init;

it still doesn't help:

[    0.197808] IDT table: 0xffffffff82800000
0xffffffff81e00000-0xffffffff82800000          10M     ro         PSE     GLB NX pmd
0xffffffff82800000-0xffffffff828a0000         640K     ro                 GLB NX pte
0xffffffff828a0000-0xffffffff82a00000        1408K     RW                 GLB NX pte
0xffffffff82a00000-0xffffffff89e00000         116M     RW         PSE     GLB NX pmd

because that trailing piece of the 2M page is still RW.

And who knows what else am I breaking when doing this:

[    2.368601] ------------[ cut here ]------------
[    2.389816] [CRTC:35:crtc-0] vblank wait timed out
[    2.396676] WARNING: drivers/gpu/drm/drm_atomic_helper.c:1920 at drm_atomic_helper_wait_for_vblanks.part.0+0x1ba/0x1e0, CPU#1: kworker/1:0/57
[    2.406715] Modules linked in:
[    2.408462] CPU: 1 UID: 0 PID: 57 Comm: kworker/1:0 Not tainted 6.19.0-rc6+ #4 PREEMPT(full)

...

I don't know, sacrificing a 2M page just for the idt_table and so that it doesn't get splintered, not sure it is worth it.

Hmmm.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#3 lore raw

Re: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section

Quoted 61 lines · Borislav Petkov
Quoted 32 lines · Ard Biesheuvel

Currently, idt_table is allocated as page-aligned .bss, and remapped read-only after init. This breaks a 2 MiB large page into 4k page mappings, which defeats some of the effort done at boot to map the kernel image using large pages, for improved TLB efficiency.

Mark this allocation as __ro_after_init instead, so it will be made read-only automatically after boot, without breaking up large page mappings.

This also fixes a latent bug on i386, where the size of idt_table is less than a page, and so remapping it read-only could potentially affect other read-write variables too, if those are not page-aligned as well.

---
 arch/x86/kernel/idt.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
arch/x86/kernel/idt.c +1 -1
index f445bec516a0..d6da25d7964f 100644
@@ -170,7 +170,7 @@ static const __initconst struct idt_data apic_idts[] = {
 };
 
 /* Must be page-aligned because the real IDT is used in the cpu entry area */
-static gate_desc idt_table[IDT_ENTRIES] __page_aligned_bss;
+static gate_desc idt_table[IDT_ENTRIES] __aligned(PAGE_SIZE) __ro_after_init;
 
 static struct desc_ptr idt_descr __ro_after_init = {
      .size           = IDT_TABLE_SIZE - 1,

arch/x86/kernel/idt.c void __init idt_setup_apic_and_irq_gates(void) · line 312

Quoted 8 lines
arch/x86/kernel/idt.c +0 -3
@@ -308,9 +308,6 @@ void __init idt_setup_apic_and_irq_gates(void)
      idt_map_in_cea();
      load_idt(&idt_descr);
 
-     /* Make the IDT table read only */
-     set_memory_ro((unsigned long)&idt_table, 1);
-
      idt_setup_done = true;
 }

Good idea, except my guest shows me something else:

before:

[    0.186281] IDT table: 0xffffffff89c7f000
0xffffffff89c00000-0xffffffff89c7f000         508K     RW                 GLB NX pte
0xffffffff89c7f000-0xffffffff89c80000           4K     ro                 GLB NX pte
0xffffffff89c80000-0xffffffff89e00000        1536K     RW                 GLB NX pte
0xffffffff89e00000-0xffffffff8be00000          32M     RW         PSE     GLB NX pmd

This is clearly a single, 4K RO pageframe right in the middle of a splintered 2M page.

after:

[    0.180635] IDT table: 0xffffffff822cf000
Quoted 8 lines
0xffffffff81e00000-0xffffffff82200000           4M     ro         PSE     GLB NX pmd
0xffffffff82200000-0xffffffff8236f000        1468K     ro                 GLB NX pte
0xffffffff8236f000-0xffffffff82400000         580K     RW                 GLB NX pte
0xffffffff82400000-0xffffffff89800000         116M     RW         PSE     GLB NX pmd

but after applying your patch it looks like it still broke the 2M mapping as the remaining piece is RW.

That is because the init region is between .data and .bss, which are not 2M aligned, and so freeing/remapping the init region breaks the hugepage mapping of the region between them. This is addressed in patch #3.

maintainer
#4 lore raw

Re: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section

Quoted 4 lines · Ard Biesheuvel

That is because the init region is between .data and .bss, which are not 2M aligned, and so freeing/remapping the init region breaks the hugepage mapping of the region between them. This is addressed in patch #3.

Ok, I'll continue looking but this commit message should not mislead in stating that it prevents the "breaking up large page mappings." I'll fix it up if I end up picking up this or you fix it up in your next revision please.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#5 lore raw

Re: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section

Quoted 9 lines · Borislav Petkov
Quoted 4 lines · Ard Biesheuvel

That is because the init region is between .data and .bss, which are not 2M aligned, and so freeing/remapping the init region breaks the hugepage mapping of the region between them. This is addressed in patch #3.

Ok, I'll continue looking but this commit message should not mislead in stating that it prevents the "breaking up large page mappings."

It does. It just doesn't prevent if from happening for other reasons.

Quoted 3 lines

I'll fix it up
if I end up picking up this or you fix it up in your next revision please.

Ack.

maintainer
#6 lore raw

Re: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section

Quoted 1 line · Ard Biesheuvel

It does. It just doesn't prevent if from happening for other reasons.

So this reads to me like "it does but it doesn't". Huh?!?

C'mon Ard, let's be more precise pls.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#7 lore raw

Re: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section

Quoted 8 lines · Borislav Petkov
Quoted 1 line · Ard Biesheuvel

It does. It just doesn't prevent if from happening for other reasons.

So this reads to me like "it does but it doesn't". Huh?!?

C'mon Ard, let's be more precise pls.

Fair enough. What I meant to say is that if your BSS placed on a 2M boundary or is sufficiently larger than that, some of it will be mapped using PMDs, and this change will prevent those from being broken up into page mappings. If your BSS is too small to be covered by huge page mappings in the first place, this change obviously does nothing (except fixing a bug on i386)

maintainer
#8 lore raw

Re: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section

Quoted 6 lines · Ard Biesheuvel

Fair enough. What I meant to say is that if your BSS placed on a 2M boundary or is sufficiently larger than that, some of it will be mapped using PMDs, and this change will prevent those from being broken up into page mappings. If your BSS is too small to be covered by huge page mappings in the first place, this change obviously does nothing (except fixing a bug on i386)

That makes more sense. :)

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
merge bot Applied
#10 lore raw

[tip: x86/sev] x86/sev: Don't emit BSS_DECRYPTED section unless it is in use

In reply to Ard Biesheuvel
The following commit has been merged into the x86/sev branch of tip:

The following commit has been merged into the x86/sev branch of tip:

Commit-ID:     8c89d3ad3095808ac130c535ad7ed3d1344d5986
Gitweb:        https://git.kernel.org/tip/8c89d3ad3095808ac130c535ad7ed3d1344d5986
Author:        Ard Biesheuvel <[email protected]>
AuthorDate:    Thu, 08 Jan 2026 09:25:29 
Committer:     Borislav Petkov (AMD) <[email protected]>
CommitterDate: Sat, 31 Jan 2026 14:42:53 +01:00

x86/sev: Don't emit BSS_DECRYPTED section unless it is in use

The BSS_DECRYPTED section that gets emitted into .bss will be empty if CONFIG_AMD_MEM_ENCRYPT is not defined. However, due to the fact that it is injected into .bss rather than emitted as a separate section, the 2 MiB alignment that it specifies is still taken into account unconditionally, pushing .bss out to the next 2 MiB boundary, leaving a gap that is never freed.

So only emit a non-empty BSS_DECRYPTED section if it is going to be used. In that case, it would still be nice to free the padding, but that is left for later.

---
 arch/x86/kernel/vmlinux.lds.S | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
arch/x86/kernel/vmlinux.lds.S +12 -9
index d7af4a6..3a24a3f 100644
@@ -67,7 +67,18 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 
 #define ALIGN_ENTRY_TEXT_BEGIN	. = ALIGN(PMD_SIZE);
 #define ALIGN_ENTRY_TEXT_END	. = ALIGN(PMD_SIZE);
+#else
+
+#define X86_ALIGN_RODATA_BEGIN
+#define X86_ALIGN_RODATA_END					\
+		. = ALIGN(PAGE_SIZE);				\
+		__end_rodata_aligned = .;
 
+#define ALIGN_ENTRY_TEXT_BEGIN
+#define ALIGN_ENTRY_TEXT_END
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
 /*
  * This section contains data which will be mapped as decrypted. Memory
  * encryption operates on a page basis. Make this section PMD-aligned
@@ -88,17 +99,9 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 	__pi___end_bss_decrypted = .;				\
 
 #else
-
-#define X86_ALIGN_RODATA_BEGIN
-#define X86_ALIGN_RODATA_END					\
-		. = ALIGN(PAGE_SIZE);				\
-		__end_rodata_aligned = .;
-
-#define ALIGN_ENTRY_TEXT_BEGIN
-#define ALIGN_ENTRY_TEXT_END
 #define BSS_DECRYPTED
-
 #endif
+
 #if defined(CONFIG_X86_64) && defined(CONFIG_KEXEC_CORE)
 #define KEXEC_RELOCATE_KERNEL					\
 	. = ALIGN(0x100);					\
maintainer
#12 lore raw

Re: [RFC/RFT PATCH 03/19] x86: Combine .data with .bss in kernel mapping

In reply to Ard Biesheuvel
Quoted 76 lines · Ard Biesheuvel

The primary mapping of the kernel image is made using huge pages where possible, mostly to minimize TLB pressure (Only the entry text section requires alignment to 2 MiB). This involves some rounding and padding of the .text and .rodata sections, resulting in gaps. These gaps are smaller than a huge page, and are remapped using different permissions, resulting in fragmentation of the huge page mappings at the edges of those regions.

Similarly, there is a gap between .data and .bss, where the init text and data regions reside. This means that the end of the .data region and the start of the .bss region are not covered by huge page mappings either, even though both regions use the same permissions (RW+NX).

Improve the situation, by placing .data and .bss adjacently in the linker map, and putting the init text and data regions after .rodata, taking the place of the rodata/data gap. This results in one fewer gap, and a more efficient mapping of the .data and .bss regions.

To preserve the x86_64 ELF layout with PT_LOAD regions aligned to 2 MiB, start the second ELF segment at .init.data and align it to 2 MiB. The resulting padding will be covered by the init region and will be freed along with it after boot.

defconfig + Clang 19:

Before:

  0xffffffff81000000-0xffffffff82200000    18M  ro  PSE  GLB x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro       GLB x  pte
  0xffffffff8231c000-0xffffffff82400000   912K  RW       GLB NX pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE  GLB NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro       GLB NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K  RW       GLB NX pte
  0xffffffff82c00000-0xffffffff83400000     8M  RW  PSE  GLB NX pmd
  0xffffffff83400000-0xffffffff83800000     4M  RW       GLB NX pte

After:

  0xffffffff81000000-0xffffffff82200000    18M  ro  PSE  GLB x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro       GLB x  pte
  0xffffffff8231c000-0xffffffff82400000   912K  RW       GLB NX pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE  GLB NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro       GLB NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K  RW       GLB NX pte
  0xffffffff82c00000-0xffffffff82e00000     2M  RW  PSE  GLB NX pmd
  0xffffffff82e00000-0xffffffff83000000     2M  RW       GLB NX pte
  0xffffffff83000000-0xffffffff83800000     8M  RW  PSE  GLB NX pmd

With the gaps removed/unmapped (pti=on)

Before:

  0xffffffff81000000-0xffffffff81200000     2M  ro  PSE  GLB x  pmd
  0xffffffff81200000-0xffffffff82200000    16M  ro  PSE      x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro           x  pte
  0xffffffff8231c000-0xffffffff82400000   912K                  pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE      NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro           NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K                  pte
  0xffffffff82c00000-0xffffffff83400000     8M  RW  PSE      NX pmd
  0xffffffff83400000-0xffffffff8342a000   168K  RW           NX pte

0xffffffff8342a000-0xffffffff836f3000 2852K pte
0xffffffff836f3000-0xffffffff83800000 1076K RW NX pte

After:

  0xffffffff81000000-0xffffffff81200000     2M  ro  PSE  GLB x  pmd
  0xffffffff81200000-0xffffffff82200000    16M  ro  PSE      x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro           x  pte
  0xffffffff8231c000-0xffffffff82400000   912K                  pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE      NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro           NX pte
  0xffffffff82b40000-0xffffffff82e3d000  3060K                  pte
  0xffffffff82e3d000-0xffffffff83000000  1804K  RW           NX pte
  0xffffffff83000000-0xffffffff83800000     8M  RW  PSE      NX pmd
Quoted 6 lines
---
 arch/x86/kernel/vmlinux.lds.S | 91 +++++++++++---------
 arch/x86/mm/init_64.c         |  5 +-
 arch/x86/mm/pat/set_memory.c  |  2 +-
 3 files changed, 52 insertions(+), 46 deletions(-)

I guess we could do this - I don't see why not... we'll have to take it for a longer spin tho.

arch/x86/kernel/vmlinux.lds.S const_cpu_current_top_of_stack = cpu_current_top_of_stack; · line 65

Quoted 10 lines
arch/x86/kernel/vmlinux.lds.S +0 -2
index 3a24a3fc55f5..1dee2987c42b 100644
@@ -61,12 +61,15 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 #define X86_ALIGN_RODATA_BEGIN	. = ALIGN(HPAGE_SIZE);
 
 #define X86_ALIGN_RODATA_END					\
-		. = ALIGN(HPAGE_SIZE);				\
-		__end_rodata_hpage_align = .;			\
$ git grep __end_rodata_hpage_align
arch/x86/include/asm/sections.h:13:extern char __end_rodata_hpage_align[];
arch/x86/tools/relocs.c:93:     "__end_rodata_hpage_align|"

I guess you wanna remove those too and say that that marker is unused. Better yet do that in a pre-patch.

Quoted 13 lines
arch/x86/kernel/vmlinux.lds.S +6 -1
-		__end_rodata_aligned = .;
+		. = ALIGN(PAGE_SIZE);				\
+		__end_rodata_aligned = ALIGN(HPAGE_SIZE);
 
 #define ALIGN_ENTRY_TEXT_BEGIN	. = ALIGN(PMD_SIZE);
 #define ALIGN_ENTRY_TEXT_END	. = ALIGN(PMD_SIZE);
+
+#define DATA_SEGMENT_START					\
+	. = ALIGN(HPAGE_SIZE);					\
+	__data_segment_start = .;
 #else
 
 #define X86_ALIGN_RODATA_BEGIN

...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#13 lore raw

Re: [RFC/RFT PATCH 03/19] x86: Combine .data with .bss in kernel mapping

Quoted 102 lines · Borislav Petkov
Quoted 76 lines · Ard Biesheuvel

The primary mapping of the kernel image is made using huge pages where possible, mostly to minimize TLB pressure (Only the entry text section requires alignment to 2 MiB). This involves some rounding and padding of the .text and .rodata sections, resulting in gaps. These gaps are smaller than a huge page, and are remapped using different permissions, resulting in fragmentation of the huge page mappings at the edges of those regions.

Similarly, there is a gap between .data and .bss, where the init text and data regions reside. This means that the end of the .data region and the start of the .bss region are not covered by huge page mappings either, even though both regions use the same permissions (RW+NX).

Improve the situation, by placing .data and .bss adjacently in the linker map, and putting the init text and data regions after .rodata, taking the place of the rodata/data gap. This results in one fewer gap, and a more efficient mapping of the .data and .bss regions.

To preserve the x86_64 ELF layout with PT_LOAD regions aligned to 2 MiB, start the second ELF segment at .init.data and align it to 2 MiB. The resulting padding will be covered by the init region and will be freed along with it after boot.

defconfig + Clang 19:

Before:

  0xffffffff81000000-0xffffffff82200000    18M  ro  PSE  GLB x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro       GLB x  pte
  0xffffffff8231c000-0xffffffff82400000   912K  RW       GLB NX pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE  GLB NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro       GLB NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K  RW       GLB NX pte
  0xffffffff82c00000-0xffffffff83400000     8M  RW  PSE  GLB NX pmd
  0xffffffff83400000-0xffffffff83800000     4M  RW       GLB NX pte

After:

  0xffffffff81000000-0xffffffff82200000    18M  ro  PSE  GLB x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro       GLB x  pte
  0xffffffff8231c000-0xffffffff82400000   912K  RW       GLB NX pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE  GLB NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro       GLB NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K  RW       GLB NX pte
  0xffffffff82c00000-0xffffffff82e00000     2M  RW  PSE  GLB NX pmd
  0xffffffff82e00000-0xffffffff83000000     2M  RW       GLB NX pte
  0xffffffff83000000-0xffffffff83800000     8M  RW  PSE  GLB NX pmd

With the gaps removed/unmapped (pti=on)

Before:

  0xffffffff81000000-0xffffffff81200000     2M  ro  PSE  GLB x  pmd
  0xffffffff81200000-0xffffffff82200000    16M  ro  PSE      x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro           x  pte
  0xffffffff8231c000-0xffffffff82400000   912K                  pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE      NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro           NX pte
  0xffffffff82b40000-0xffffffff82c00000   768K                  pte
  0xffffffff82c00000-0xffffffff83400000     8M  RW  PSE      NX pmd
  0xffffffff83400000-0xffffffff8342a000   168K  RW           NX pte

0xffffffff8342a000-0xffffffff836f3000 2852K pte
0xffffffff836f3000-0xffffffff83800000 1076K RW NX pte

After:

  0xffffffff81000000-0xffffffff81200000     2M  ro  PSE  GLB x  pmd
  0xffffffff81200000-0xffffffff82200000    16M  ro  PSE      x  pmd
  0xffffffff82200000-0xffffffff8231c000  1136K  ro           x  pte
  0xffffffff8231c000-0xffffffff82400000   912K                  pte
  0xffffffff82400000-0xffffffff82a00000     6M  ro  PSE      NX pmd
  0xffffffff82a00000-0xffffffff82b40000  1280K  ro           NX pte
  0xffffffff82b40000-0xffffffff82e3d000  3060K                  pte
  0xffffffff82e3d000-0xffffffff83000000  1804K  RW           NX pte
  0xffffffff83000000-0xffffffff83800000     8M  RW  PSE      NX pmd
Quoted 6 lines
---
 arch/x86/kernel/vmlinux.lds.S | 91 +++++++++++---------
 arch/x86/mm/init_64.c         |  5 +-
 arch/x86/mm/pat/set_memory.c  |  2 +-
 3 files changed, 52 insertions(+), 46 deletions(-)

I guess we could do this - I don't see why not... we'll have to take it for a longer spin tho.

arch/x86/kernel/vmlinux.lds.S const_cpu_current_top_of_stack = cpu_current_top_of_stack; · line 65

Quoted 10 lines
arch/x86/kernel/vmlinux.lds.S +0 -2
index 3a24a3fc55f5..1dee2987c42b 100644
@@ -61,12 +61,15 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack;
 #define X86_ALIGN_RODATA_BEGIN	. = ALIGN(HPAGE_SIZE);
 
 #define X86_ALIGN_RODATA_END					\
-		. = ALIGN(HPAGE_SIZE);				\
-		__end_rodata_hpage_align = .;			\
$ git grep __end_rodata_hpage_align
arch/x86/include/asm/sections.h:13:extern char __end_rodata_hpage_align[];
arch/x86/tools/relocs.c:93:     "__end_rodata_hpage_align|"
Quoted 3 lines

I guess you wanna remove those too and say that that marker is unused. Better yet do that in a pre-patch.

Indeed. When __end_rodata_hpage_align exists, it is always equal to __end_rodata_aligned, so it can just be dropped entirely.