Link the relocatable x86 kernel as PIE
Subsystems (19)
Ard Biesheuvel
maintainercover Link the relocatable x86 kernel as PIE
This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of x86_64 vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].
The main sticking point is the fact that PIE linking on x86_64 requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.
This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)
So now that we have some actual numbers, I would like to try and revisit this discussion, and get a conclusion on whether this is really a non-starter. Note that only the KASLR kernel would rely on this, and disabling CONFIG_RANDOMIZE_BASE will revert to the current situation (provided that patch #4 is applied)
Some minor asm tweaks are needed too (patches #9 - #17), but those all seem uncontroversial to me.
The first 5 patches are general cleanup, and could be taken into consideration independently of the discussion around PIC codegen.
[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).
[2] EFI zboot is already used by various architectures that have no decompressor stage at all (arm64, RISC-V, LoongArch), and this format can be combined with an ELF payload too. EFI zboot accommodates non-EFI boot chains by describing the size, offset, payload type and compression type in its header, so that it can be extracted and booted by other means.
Ard Biesheuvel (19):
x86/idt: Move idt_table to __ro_after_init section x86/sev: Don't emit BSS_DECRYPT section unless it is in use
x86: Combine .data with .bss in kernel mapping
x86: Make the 64-bit bzImage always physically relocatable x86/efistub: Simplify early remapping of kernel text
alloc_tag: Use __ prefixed ELF section names tools/objtool: Treat indirect ftrace calls as direct calls
x86: Use PIE codegen for the relocatable 64-bit kernel x86/pm-trace: Use RIP-relative accesses for .tracedata x86/kvm: Use RIP-relative addressing
x86/rethook: Use RIP-relative reference for fake return address x86/sync_core: Use RIP-relative addressing x86/entry_64: Use RIP-relative addressing x86/hibernate: Prefer RIP-relative accesses
x64/acpi: Use PIC-compatible references in wakeup_64.S x86/kexec: Use 64-bit wide absolute reference from relocated code x86/head64: Avoid absolute references in startup asm x86/boot: Implement support for RELA/RELR/REL runtime relocations x86/kernel: Switch to PIE linking for the relocatable kernel
arch/x86/Kconfig | 45 ++++--- arch/x86/Makefile | 24 +++- arch/x86/boot/Makefile | 1 + arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/head_64.S | 4 - arch/x86/boot/compressed/misc.c | 85 +++++++++++-- arch/x86/boot/header.S | 8 +- arch/x86/entry/calling.h | 9 +- arch/x86/entry/entry_64.S | 14 +-- arch/x86/entry/vdso/Makefile | 1 + arch/x86/include/asm/boot.h | 2 - arch/x86/include/asm/pm-trace.h | 4 +- arch/x86/include/asm/sync_core.h | 3 +- arch/x86/kernel/acpi/wakeup_64.S | 11 +- arch/x86/kernel/head_64.S | 17 +-- arch/x86/kernel/idt.c | 5 +- arch/x86/kernel/kvm.c | 5 +- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kernel/rethook.c | 6 +- arch/x86/kernel/vmlinux.lds.S | 132 ++++++++++++-------- arch/x86/mm/init_64.c | 5 +- arch/x86/mm/pat/set_memory.c | 2 +- arch/x86/power/hibernate_asm_64.S | 4 +- arch/x86/realmode/rm/Makefile | 1 + drivers/base/power/trace.c | 6 +- drivers/firmware/efi/libstub/x86-stub.c | 4 +- include/asm-generic/codetag.lds.h | 14 ++- include/asm-generic/vmlinux.lds.h | 1 + include/linux/alloc_tag.h | 11 +- include/linux/hidden.h | 2 + include/uapi/linux/elf.h | 3 + lib/alloc_tag.c | 6 +- tools/objtool/check.c | 32 ++++- 33 files changed, 315 insertions(+), 156 deletions(-)
base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 1/19 x86/idt: Move idt_table to __ro_after_init section
Currently, idt_table is allocated as page-aligned .bss, and remapped read-only after init. This breaks a 2 MiB large page into 4k page mappings, which defeats some of the effort done at boot to map the kernel image using large pages, for improved TLB efficiency.
Mark this allocation as __ro_after_init instead, so it will be made read-only automatically after boot, without breaking up large page mappings.
This also fixes a latent bug on i386, where the size of idt_table is less than a page, and so remapping it read-only could potentially affect other read-write variables too, if those are not page-aligned as well.
--- arch/x86/kernel/idt.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
arch/x86/kernel/idt.c +1 -4
@@ -170,7 +170,7 @@ static const __initconst struct idt_data apic_idts[] = { };/* Must be page-aligned because the real IDT is used in the cpu entry area */ -static gate_desc idt_table[IDT_ENTRIES] __page_aligned_bss; +static gate_desc idt_table[IDT_ENTRIES ]__aligned(PAGE_SIZE) __ro_after_init ;static struct desc_ptr idt_descr__ro_after_init = { .size =IDT_TABLE_SIZE -1 ,
@@ -308,9 +308,6 @@ void __init idt_setup_apic_and_irq_gates(void)idt_map_in_cea ();load_idt (&idt_descr); -/* Make the IDT table read only */ -set_memory_ro ((unsigned long )&idt_table,1 ); - idt_setup_done = true; }
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 2/19 x86/sev: Don't emit BSS_DECRYPT section unless it is in use
In reply to Ard BiesheuvelThe BSS_DECRYPT section that gets emitted into .bss will be empty if CONFIG_AMD_MEM_ENCRYPT is not defined. However, due to the fact that it is injected into .bss rather than emitted as a separate section, the 2 MiB alignment that it specifies is still taken into account unconditionally, pushing .bss out to the next 2 MiB boundary, leaving a gap that is never freed.
So only emit a non-empty BSS_DECRYPT section if it is going to be used. In that case, it would still be nice to free the padding, but that is left for later.
--- arch/x86/kernel/vmlinux.lds.S | 21 +++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
arch/x86/kernel/vmlinux.lds.S +12 -9
@@ -67,7 +67,18 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define ALIGN_ENTRY_TEXT_BEGIN . = ALIGN(PMD_SIZE); #define ALIGN_ENTRY_TEXT_END . = ALIGN(PMD_SIZE); +#else + +#define X86_ALIGN_RODATA_BEGIN +#define X86_ALIGN_RODATA_END \ + . = ALIGN(PAGE_SIZE); \ + __end_rodata_aligned = .; +#define ALIGN_ENTRY_TEXT_BEGIN +#define ALIGN_ENTRY_TEXT_END +#endif + +#ifdef CONFIG_AMD_MEM_ENCRYPT /* * This section contains data which will be mapped as decrypted. Memory * encryption operates on a page basis. Make this section PMD-aligned
@@ -88,17 +99,9 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; __pi___end_bss_decrypted = .; \ #else - -#define X86_ALIGN_RODATA_BEGIN -#define X86_ALIGN_RODATA_END \ - . = ALIGN(PAGE_SIZE); \ - __end_rodata_aligned = .; - -#define ALIGN_ENTRY_TEXT_BEGIN -#define ALIGN_ENTRY_TEXT_END #define BSS_DECRYPTED - #endif + #if defined(CONFIG_X86_64) && defined(CONFIG_KEXEC_CORE) #define KEXEC_RELOCATE_KERNEL \ . = ALIGN(0x100); \
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 3/19 x86: Combine .data with .bss in kernel mapping
In reply to Ard BiesheuvelThe primary mapping of the kernel image is made using huge pages where possible, mostly to minimize TLB pressure (Only the entry text section requires alignment to 2 MiB). This involves some rounding and padding of the .text and .rodata sections, resulting in gaps. These gaps are smaller than a huge page, and are remapped using different permissions, resulting in fragmentation of the huge page mappings at the edges of those regions.
Similarly, there is a gap between .data and .bss, where the init text and data regions reside. This means that the end of the .data region and the start of the .bss region are not covered by huge page mappings either, even though both regions use the same permissions (RW+NX).
Improve the situation, by placing .data and .bss adjacently in the linker map, and putting the init text and data regions after .rodata, taking the place of the rodata/data gap. This results in one fewer gap, and a more efficient mapping of the .data and .bss regions.
To preserve the x86_64 ELF layout with PT_LOAD regions aligned to 2 MiB, start the second ELF segment at .init.data and align it to 2 MiB. The resulting padding will be covered by the init region and will be freed along with it after boot.
defconfig + Clang 19:
Before:
0xffffffff81000000-0xffffffff82200000 18M ro PSE GLB x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro GLB x pte 0xffffffff8231c000-0xffffffff82400000 912K RW GLB NX pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE GLB NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro GLB NX pte 0xffffffff82b40000-0xffffffff82c00000 768K RW GLB NX pte 0xffffffff82c00000-0xffffffff83400000 8M RW PSE GLB NX pmd 0xffffffff83400000-0xffffffff83800000 4M RW GLB NX pte
After:
0xffffffff81000000-0xffffffff82200000 18M ro PSE GLB x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro GLB x pte 0xffffffff8231c000-0xffffffff82400000 912K RW GLB NX pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE GLB NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro GLB NX pte 0xffffffff82b40000-0xffffffff82c00000 768K RW GLB NX pte 0xffffffff82c00000-0xffffffff82e00000 2M RW PSE GLB NX pmd 0xffffffff82e00000-0xffffffff83000000 2M RW GLB NX pte 0xffffffff83000000-0xffffffff83800000 8M RW PSE GLB NX pmd
With the gaps removed/unmapped (pti=on)
Before:
0xffffffff81000000-0xffffffff81200000 2M ro PSE GLB x pmd 0xffffffff81200000-0xffffffff82200000 16M ro PSE x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro x pte 0xffffffff8231c000-0xffffffff82400000 912K pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro NX pte 0xffffffff82b40000-0xffffffff82c00000 768K pte 0xffffffff82c00000-0xffffffff83400000 8M RW PSE NX pmd 0xffffffff83400000-0xffffffff8342a000 168K RW NX pte
0xffffffff8342a000-0xffffffff836f3000 2852K pte
0xffffffff836f3000-0xffffffff83800000 1076K RW NX pte
After:
0xffffffff81000000-0xffffffff81200000 2M ro PSE GLB x pmd 0xffffffff81200000-0xffffffff82200000 16M ro PSE x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro x pte 0xffffffff8231c000-0xffffffff82400000 912K pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro NX pte 0xffffffff82b40000-0xffffffff82e3d000 3060K pte 0xffffffff82e3d000-0xffffffff83000000 1804K RW NX pte 0xffffffff83000000-0xffffffff83800000 8M RW PSE NX pmd
--- arch/x86/kernel/vmlinux.lds.S | 91 +++++++++++--------- arch/x86/mm/init_64.c | 5 +- arch/x86/mm/pat/set_memory.c | 2 +- 3 files changed, 52 insertions(+), 46 deletions(-)
arch/x86/kernel/vmlinux.lds.S +50 -41
@@ -61,12 +61,15 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define X86_ALIGN_RODATA_BEGIN . = ALIGN(HPAGE_SIZE); #define X86_ALIGN_RODATA_END \ - . = ALIGN(HPAGE_SIZE); \ - __end_rodata_hpage_align = .; \ - __end_rodata_aligned = .; + . = ALIGN(PAGE_SIZE); \ + __end_rodata_aligned = ALIGN(HPAGE_SIZE); #define ALIGN_ENTRY_TEXT_BEGIN . = ALIGN(PMD_SIZE); #define ALIGN_ENTRY_TEXT_END . = ALIGN(PMD_SIZE); + +#define DATA_SEGMENT_START \ + . = ALIGN(HPAGE_SIZE); \ + __data_segment_start = .; #else #define X86_ALIGN_RODATA_BEGIN
@@ -76,9 +79,14 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define ALIGN_ENTRY_TEXT_BEGIN #define ALIGN_ENTRY_TEXT_END + +#define DATA_SEGMENT_START \ + . = ALIGN(PAGE_SIZE); \ + __data_segment_start = .; #endif #ifdef CONFIG_AMD_MEM_ENCRYPT + /* * This section contains data which will be mapped as decrypted. Memory * encryption operates on a page basis. Make this section PMD-aligned
@@ -171,43 +179,6 @@ SECTIONS RO_DATA(PAGE_SIZE) X86_ALIGN_RODATA_END - /* Data */ - .data : AT(ADDR(.data) - LOAD_OFFSET) { - /* Start of data section */ - _sdata = .; - - /* init_task */ - INIT_TASK_DATA(THREAD_SIZE) - - /* equivalent to task_pt_regs(&init_task) */ - __top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE; - -#ifdef CONFIG_X86_32 - /* 32 bit has nosave before _edata */ - NOSAVE_DATA -#endif - - PAGE_ALIGNED_DATA(PAGE_SIZE) - - CACHE_HOT_DATA(L1_CACHE_BYTES) - - CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES) - - DATA_DATA - CONSTRUCTORS - KEXEC_RELOCATE_KERNEL - - /* rarely changed data like cpu maps */ - READ_MOSTLY_DATA(INTERNODE_CACHE_BYTES) - - /* End of data section */ - _edata = .; - } :data - - BUG_TABLE - - ORC_UNWIND_TABLE - /* Init code and data - will be freed after init */ . = ALIGN(PAGE_SIZE); .init.begin : AT(ADDR(.init.begin) - LOAD_OFFSET) {
@@ -229,7 +200,8 @@ SECTIONS __inittext_end = .; } - INIT_DATA_SECTION(16) + DATA_SEGMENT_START + INIT_DATA_SECTION(16) :data .x86_cpu_dev.init : AT(ADDR(.x86_cpu_dev.init) - LOAD_OFFSET) { __x86_cpu_dev_start = .;
@@ -358,6 +330,43 @@ SECTIONS __smp_locks_end = .; } + /* Data */ + .data : AT(ADDR(.data) - LOAD_OFFSET) { + /* Start of data section */ + _sdata = .; + + /* init_task */ + INIT_TASK_DATA(THREAD_SIZE) + + /* equivalent to task_pt_regs(&init_task) */ + __top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE; + +#ifdef CONFIG_X86_32 + /* 32 bit has nosave before _edata */ + NOSAVE_DATA +#endif + + PAGE_ALIGNED_DATA(PAGE_SIZE) + + CACHE_HOT_DATA(L1_CACHE_BYTES) + + CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES) + + DATA_DATA + CONSTRUCTORS + KEXEC_RELOCATE_KERNEL + + /* rarely changed data like cpu maps */ + READ_MOSTLY_DATA(INTERNODE_CACHE_BYTES) + + /* End of data section */ + _edata = .; + } + + BUG_TABLE + + ORC_UNWIND_TABLE + #ifdef CONFIG_X86_64 .data_nosave : AT(ADDR(.data_nosave) - LOAD_OFFSET) { NOSAVE_DATA
arch/x86/mm/init_64.c +1 -4
@@ -1397,9 +1397,8 @@ void mark_rodata_ro(void) {unsigned long start =PFN_ALIGN (_text);unsigned long rodata_start =PFN_ALIGN (__start_rodata); -unsigned long end = (unsigned long )__end_rodata_hpage_align; +unsigned long end = (unsigned long )__end_rodata;unsigned long text_end =PFN_ALIGN (_etext); -unsigned long rodata_end =PFN_ALIGN (__end_rodata);unsigned long all_end; printk(KERN_INFO "Write protecting the kernel read-only data: %luk\n",
@@ -1435,8 +1434,6 @@ void mark_rodata_ro(void)free_kernel_image_pages ("unused kernel image (text/rodata gap)" , (void *)text_end, (void *)rodata_start); -free_kernel_image_pages ("unused kernel image (rodata/data gap)" , - (void *)rodata_end, (void *)_sdata); }/*
arch/x86/mm/pat/set_memory.c +1 -1
@@ -554,7 +554,7 @@ static pgprotval_t protect_kernel_text(unsigned long start, unsigned long end)static pgprotval_t protect_kernel_text_ro (unsigned long start,unsigned long end) { -unsigned long t_end = (unsigned long )__end_rodata_hpage_align -1 ; +unsigned long t_end = (unsigned long )__end_rodata -1 ;unsigned long t_start = (unsigned long )_text;unsigned int level;
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 4/19 x86: Make the 64-bit bzImage always physically relocatable
In reply to Ard BiesheuvelOn x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.
On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.
Clarify this in Kconfig and in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whether CONFIG_RELOCATABLE is set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite for EFI_STUB and RANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.
--- arch/x86/Kconfig | 40 ++++++++++++-------- arch/x86/boot/compressed/head_64.S | 4 -- arch/x86/boot/compressed/misc.c | 8 ++-- arch/x86/boot/header.S | 8 +--- 4 files changed, 29 insertions(+), 31 deletions(-)
arch/x86/Kconfig +24 -16
@@ -1931,7 +1931,7 @@ config EFI config EFI_STUB bool "EFI stub support" depends on EFI - select RELOCATABLE + select RELOCATABLE if X86_32 help This kernel feature allows a bzImage to be loaded directly by EFI firmware without the use of a bootloader.
@@ -2028,8 +2028,9 @@ config PHYSICAL_START help This gives the physical address where the kernel is loaded. - If the kernel is not relocatable (CONFIG_RELOCATABLE=n) then bzImage - will decompress itself to above physical address and run from there. + If the kernel is not relocatable (CONFIG_RELOCATABLE=n) and built for + i386, then the bzImage will decompress itself to the above physical + address and run from there. Otherwise, bzImage will run from the address where it has been loaded by the boot loader. The only exception is if it is loaded below the above physical address, in which case it will relocate itself there.
@@ -2064,16 +2065,22 @@ config PHYSICAL_START Don't change this unless you know what you are doing. config RELOCATABLE - bool "Build a relocatable kernel" - default y + bool "Build a relocatable kernel" if X86_32 + default X86_32 help - This builds a kernel image that retains relocation information - so it can be loaded someplace besides the default 1MB. + This builds a kernel image that retains relocation information so it + can be placed someplace besides the default PAGE_OFFSET + 1MB. This + is a prerequisite for KASLR. The relocations tend to make the kernel binary about 10% larger, but are discarded at runtime. - One use is for the kexec on panic case where the recovery kernel - must live at a different physical address than the primary + On i386, where the virtual and physical load offset of the kernel + must be equal, this also allows the kernel image to be placed at a + physical load address that differs from the compile time default. On + x86_64, this is always permitted. + + One use is for the kexec on panic case on i386, where the recovery + kernel must live at a different physical address than the primary kernel. Note: If CONFIG_RELOCATABLE=y, then the kernel runs from the address
@@ -2082,7 +2089,7 @@ config RELOCATABLE config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)" - depends on RELOCATABLE + select RELOCATABLE default y help In support of Kernel Address Space Layout Randomization (KASLR),
@@ -2118,7 +2125,7 @@ config RANDOMIZE_BASE # Relocation on x86 needs some additional build support config X86_NEED_RELOCS def_bool y - depends on RANDOMIZE_BASE || (X86_32 && RELOCATABLE) + depends on RELOCATABLE select ARCH_VMLINUX_NEEDS_RELOCS config PHYSICAL_ALIGN
@@ -2131,12 +2138,13 @@ config PHYSICAL_ALIGN where kernel is loaded and run from. Kernel is compiled for an address which meets above alignment restriction. - If bootloader loads the kernel at a non-aligned address and - CONFIG_RELOCATABLE is set, kernel will move itself to nearest - address aligned to above value and run from there. + If the bootloader loads the kernel at a non-aligned address and it + is built for x86_64 or CONFIG_RELOCATABLE is set, the kernel will + move itself to the nearest address aligned to above value and run + from there. - If bootloader loads the kernel at a non-aligned address and - CONFIG_RELOCATABLE is not set, kernel will ignore the run time + If the bootloader loads the i386 kernel at a non-aligned address and + CONFIG_RELOCATABLE is not set, the kernel will ignore the run time load address and decompress itself to the address it has been compiled for and run from there. The address for which kernel is compiled already meets above alignment restrictions. Hence the
arch/x86/boot/compressed/head_64.S +0 -4
@@ -143,7 +143,6 @@ SYM_FUNC_START(startup_32) * for safe in-place decompression. */ -#ifdef CONFIG_RELOCATABLE movl %ebp, %ebx movl BP_kernel_alignment(%esi), %eax decl %eax
@@ -152,7 +151,6 @@ SYM_FUNC_START(startup_32) andl %eax, %ebx cmpl $LOAD_PHYSICAL_ADDR, %ebx jae 1f -#endif movl $LOAD_PHYSICAL_ADDR, %ebx 1:
@@ -312,7 +310,6 @@ SYM_CODE_START(startup_64) */ /* Start with the delta to where the kernel will run at. */ -#ifdef CONFIG_RELOCATABLE leaq startup_32(%rip) /* - $startup_32 */, %rbp movl BP_kernel_alignment(%rsi), %eax decl %eax
@@ -321,7 +318,6 @@ SYM_CODE_START(startup_64) andq %rax, %rbp cmpq $LOAD_PHYSICAL_ADDR, %rbp jae 1f -#endif movq $LOAD_PHYSICAL_ADDR, %rbp 1:
arch/x86/boot/compressed/misc.c +3 -5
@@ -314,12 +314,10 @@ static size_t parse_elf(void *output)if ((phdr->p_align %0x200000 ) !=0 )error ("Alignment of LOAD segment isn't multiple of 2MB" );#endif -#ifdef CONFIG_RELOCATABLE - dest = output; - dest += (phdr->p_paddr -LOAD_PHYSICAL_ADDR ); -#else dest = (void *)(phdr->p_paddr); -#endif +if (IS_ENABLED (CONFIG_X86_64 ) || +IS_ENABLED (CONFIG_RELOCATABLE )) + dest += (unsigned long )output -LOAD_PHYSICAL_ADDR ;memmove (dest, output + phdr->p_offset, phdr->p_filesz);break ;default :/* Ignore other PT_* */ break ;
arch/x86/boot/header.S +2 -6
@@ -332,7 +332,7 @@ initrd_addr_max: .long 0x7fffffff kernel_alignment: .long CONFIG_PHYSICAL_ALIGN #physical addr alignment #required for protected mode #kernel -#ifdef CONFIG_RELOCATABLE +#if defined(CONFIG_RELOCATABLE) || defined(CONFIG_X86_64) relocatable_kernel: .byte 1 #else relocatable_kernel: .byte 0
@@ -342,14 +342,10 @@ min_alignment: .byte MIN_KERNEL_ALIGN_LG2 # minimum alignment xloadflags: #ifdef CONFIG_X86_64 # define XLF0 XLF_KERNEL_64 /* 64-bit kernel */ -#else -# define XLF0 0 -#endif - -#if defined(CONFIG_RELOCATABLE) && defined(CONFIG_X86_64) /* kernel/boot_param/ramdisk could be loaded above 4g */ # define XLF1 XLF_CAN_BE_LOADED_ABOVE_4G #else +# define XLF0 0 # define XLF1 0 #endif
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 5/19 x86/efistub: Simplify early remapping of kernel text
In reply to Ard BiesheuvelNow that the kernel's .text, .rodata and .inittext are all covered by a single ELF segment, there is no need to remap .inittext separately. Instead, remap the entire region in a single call.
This remapping is needed because the EFI stub hands over to the core kernel while running in long mode, using the page tables provided by the firmware. Recent so-called 'MS secured core' (tm) PCs are more strict when it comes to separating writable from executable mappings, and so for compatibility with such systems, any code that may be callable during early boot (i.e., before the kernel switches to its own page tables) must be remapped executable explicitly.
--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/misc.c | 4 +--- arch/x86/include/asm/boot.h | 2 -- arch/x86/kernel/vmlinux.lds.S | 2 -- drivers/firmware/efi/libstub/x86-stub.c | 4 +--- 5 files changed, 3 insertions(+), 11 deletions(-)
arch/x86/boot/compressed/Makefile +1 -1
@@ -76,7 +76,7 @@ LDFLAGS_vmlinux += -T hostprogs := mkpiggy HOST_EXTRACFLAGS += -I$(srctree)/tools/include -sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__start_rodata\|_sinittext\|__inittext_end\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' +sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__data_segment_start\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' quiet_cmd_voffset = VOFFSET $@ cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
arch/x86/boot/compressed/misc.c +1 -3
@@ -329,9 +329,7 @@ static size_t parse_elf(void *output)return ehdr.e_entry -LOAD_PHYSICAL_ADDR ; } -const unsigned long kernel_text_size = VO___start_rodata - VO__text; -const unsigned long kernel_inittext_offset = VO__sinittext - VO__text; -const unsigned long kernel_inittext_size = VO___inittext_end - VO__sinittext; +const unsigned long kernel_text_size = VO___data_segment_start - VO__text;const unsigned long kernel_total_size = VO__end - VO__text;static u8 boot_heap[BOOT_HEAP_SIZE ]__aligned(4) ;
arch/x86/include/asm/boot.h +0 -2
@@ -82,8 +82,6 @@#ifndef __ASSEMBLER__extern unsigned int output_len;extern const unsigned long kernel_text_size; -extern const unsigned long kernel_inittext_offset; -extern const unsigned long kernel_inittext_size;extern const unsigned long kernel_total_size; unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr,
arch/x86/kernel/vmlinux.lds.S +0 -2
@@ -196,8 +196,6 @@ SECTIONS */ .altinstr_aux : AT(ADDR(.altinstr_aux) - LOAD_OFFSET) { *(.altinstr_aux) - . = ALIGN(PAGE_SIZE); - __inittext_end = .; } DATA_SEGMENT_START
drivers/firmware/efi/libstub/x86-stub.c +1 -3
@@ -890,9 +890,7 @@ static efi_status_t efi_decompress_kernel(unsigned long *kernel_entry, *kernel_entry = addr + entry; -return efi_adjust_memory_range_protection (addr, kernel_text_size) ?: -efi_adjust_memory_range_protection (addr + kernel_inittext_offset, - kernel_inittext_size); +return efi_adjust_memory_range_protection (addr, kernel_text_size); } static void __noreturn enter_kernel(unsigned long kernel_addr,
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 6/19 alloc_tag: Use __ prefixed ELF section names
In reply to Ard BiesheuvelThe compiler will emit static relocations related to a section .foo into a separate section called .rela.foo, i.e., it just appends the section name to the string ".rela"
When section names start with . or __, this results in section names that are correctly matched by the various pattern rules in the various linker scripts across the tree.
Without any such leading delimiter, it may lead to spurious warnings such as
Quoted 1 line
Quoted 1 line
ld: warning: orphan section
.relaalloc_tags' frominit/main.o' being placed in section `.rela.dyn'
ld: warning: dot moved backwards before `.rela.dyn'
ld: .tmp_vmlinux1: section `.rela.dyn' can't be allocated in segment 1
Fix this by renaming the section to __alloc_tags. While at it, tweak the headers so that the definition appears only a single time.
--- include/asm-generic/codetag.lds.h | 14 +++++++++----- include/linux/alloc_tag.h | 11 ++++++----- lib/alloc_tag.c | 6 +++--- 3 files changed, 18 insertions(+), 13 deletions(-)
include/asm-generic/codetag.lds.h +9 -5
@@ -2,6 +2,10 @@#ifndef __ASM_GENERIC_CODETAG_LDS_H#define __ASM_GENERIC_CODETAG_LDS_H +#include <linux/compiler_types.h> + +#define ALLOC_TAG_SECTION_NAME __alloc_tags +#ifdef CONFIG_MEM_ALLOC_PROFILING #define IF_MEM_ALLOC_PROFILING (...) __VA_ARGS__#else
@@ -10,15 +14,15 @@#define SECTION_WITH_BOUNDARIES (_name) \ . = ALIGN(8); \ - __start_##_name = .; \ + __PASTE(__start_, _name) = .; \ KEEP(*(_name)) \ - __stop_##_name = .; + __PASTE(__stop_, _name) = .;#define CODETAG_SECTIONS () \ - IF_MEM_ALLOC_PROFILING(SECTION_WITH_BOUNDARIES(alloc_tags)) + IF_MEM_ALLOC_PROFILING(SECTION_WITH_BOUNDARIES(ALLOC_TAG_SECTION_NAME))#define MOD_SEPARATE_CODETAG_SECTION (_name) \ - .codetag.##_name : { \ + .codetag._name : { \ SECTION_WITH_BOUNDARIES(_name) \ }
@@ -28,6 +32,6 @@* unload them individually once unused. */ #define MOD_SEPARATE_CODETAG_SECTIONS () \ - IF_MEM_ALLOC_PROFILING(MOD_SEPARATE_CODETAG_SECTION(alloc_tags)) + IF_MEM_ALLOC_PROFILING(MOD_SEPARATE_CODETAG_SECTION(ALLOC_TAG_SECTION_NAME))#endif /* __ASM_GENERIC_CODETAG_LDS_H */
include/linux/alloc_tag.h +6 -5
@@ -15,6 +15,9 @@#include <linux/static_key.h> #include <linux/irqflags.h> +/* for ALLOC_TAG_SECTION_NAME */ +#include <asm-generic/codetag.lds.h> +struct alloc_tag_counters {u64 bytes;u64 calls;
@@ -74,8 +77,6 @@ static inline void set_codetag_empty(union codetag_ref *ref) #ifdef CONFIG_MEM_ALLOC_PROFILING -#define ALLOC_TAG_SECTION_NAME "alloc_tags" - struct codetag_bytes { struct codetag *ct; s64 bytes;
@@ -98,7 +99,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag);#define DEFINE_ALLOC_TAG (_alloc_tag) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section(ALLOC_TAG_SECTION_NAME) = { \ + __section(__stringify(ALLOC_TAG_SECTION_NAME)) = { \ .ct = CODE_TAG_INIT, \ .counters = &_shared_alloc_tag };
@@ -108,7 +109,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag);#define DEFINE_ALLOC_TAG (_alloc_tag) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section(ALLOC_TAG_SECTION_NAME) = { \ + __section(__stringify(ALLOC_TAG_SECTION_NAME)) = { \ .ct = CODE_TAG_INIT, \ .counters = NULL };
@@ -117,7 +118,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag);#define DEFINE_ALLOC_TAG (_alloc_tag) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section(ALLOC_TAG_SECTION_NAME) = { \ + __section(__stringify(ALLOC_TAG_SECTION_NAME)) = { \ .ct = CODE_TAG_INIT, \ .counters = &_alloc_tag_cntr };
lib/alloc_tag.c +3 -3
@@ -15,8 +15,8 @@#define ALLOCINFO_FILE_NAME "allocinfo"#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) -#define SECTION_START (NAME ) (CODETAG_SECTION_START_PREFIX NAME) -#define SECTION_STOP (NAME ) (CODETAG_SECTION_STOP_PREFIX NAME) +#define SECTION_START (NAME ) (CODETAG_SECTION_START_PREFIX #NAME) +#define SECTION_STOP (NAME ) (CODETAG_SECTION_STOP_PREFIX #NAME)#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT static bool mem_profiling_support = true;
@@ -810,7 +810,7 @@ static inline void sysctl_init(void) {}static int __init alloc_tag_init (void ) {const struct codetag_type_desc desc = { - .section =ALLOC_TAG_SECTION_NAME , + .section =__stringify (ALLOC_TAG_SECTION_NAME ), .tag_size =sizeof (struct alloc_tag ), #ifdef CONFIG_MODULES .needs_section_mem = needs_section_mem,
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 7/19 tools/objtool: Treat indirect ftrace calls as direct calls
In reply to Ard BiesheuvelIn some cases, the compiler may rely on indirect calls using GOT slots as memory operands to emit function calls. This leaves it up to the linker to relax the call to a direct call if possible, i.e., if the destination address is known at link time and in range, which may not be the case when building shared libraries for user space.
On x86, this may happen when building in PIC mode with ftrace enabled, and given that vmlinux is a fully linked binary, this relaxation is always possible, and therefore mandatory per the x86_64 psABI.
This means that the indirect calls to __fentry__ that are observeable in vmlinux.o will have been converted to direct calls in vmlinux, and can be treated as such by objtool.
--- tools/objtool/check.c | 32 ++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
tools/objtool/check.c +30 -2
@@ -1660,11 +1660,39 @@ static int add_call_destinations(struct objtool_file *file)for_each_insn (file, insn) {struct symbol *func =insn_func (insn); -if (insn->type !=INSN_CALL ) +if (insn->type !=INSN_CALL && + insn->type !=INSN_CALL_DYNAMIC )continue ; reloc =insn_reloc (file, insn); -if (!reloc) { +if (insn->type ==INSN_CALL_DYNAMIC ) { +if (!reloc) +continue ; + +/* +* GCC 13 and older on x86 will always emit the call to +* __fentry__ using a relaxable GOT-based symbol +* reference when operating in PIC mode, i.e., +* +* call *0x0(%rip) +* R_X86_64_GOTPCRELX __fentry__-0x4 +* +* where it is left up to the linker to relax this into +* +* call __fentry__ +* nop +* +* if __fentry__ turns out to be DSO local, which is +* always the case for vmlinux. Given that this +* relaxation is mandatory per the x86_64 psABI, these +* calls can simply be treated as direct calls. +*/ +if (arch_ftrace_match (reloc->sym->name)) { + insn->type =INSN_CALL ; +add_call_dest (file, insn, reloc->sym, false); + } + + }else if (!reloc) { dest_off =arch_jump_destination (insn); dest =find_call_destination (insn->sec, dest_off);
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 8/19 x86: Use PIE codegen for the relocatable 64-bit kernel
In reply to Ard BiesheuvelAs an intermediate step towards enabling PIE linking for the x86_64 KASLR kernel, enable PIE codegen for all C and Rust objects that are linked into the kernel proper. Add a Kconfig option RELOCATABLE_PIE for this, depending on RELR support in the linker, as the relocation tables will blow up the kernel image otherwise.
This results in a code size increase of between 0.2% (clang) and 0.5% (gcc). Performance (hackbench) appears to be unaffected across several different uarchs.
--- arch/x86/Kconfig | 4 ++++ arch/x86/Makefile | 19 ++++++++++++++++++- arch/x86/boot/Makefile | 1 + arch/x86/boot/compressed/Makefile | 2 +- arch/x86/entry/vdso/Makefile | 1 + arch/x86/realmode/rm/Makefile | 1 + include/asm-generic/vmlinux.lds.h | 1 + include/linux/hidden.h | 2 ++ 8 files changed, 29 insertions(+), 2 deletions(-)
arch/x86/Kconfig +4 -0
@@ -2067,6 +2067,7 @@ config PHYSICAL_START config RELOCATABLE bool "Build a relocatable kernel" if X86_32 default X86_32 + select RELOCATABLE_PIE if TOOLS_SUPPORT_RELR help This builds a kernel image that retains relocation information so it can be placed someplace besides the default PAGE_OFFSET + 1MB. This
@@ -2087,6 +2088,9 @@ config RELOCATABLE it has been loaded at and the compile time physical address (CONFIG_PHYSICAL_START) is used as the minimum location. +config RELOCATABLE_PIE + bool + config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)" select RELOCATABLE
arch/x86/Makefile +18 -1
@@ -89,6 +89,8 @@ ifdef CONFIG_CC_IS_GCC CC_FLAGS_FPU += -mhard-float endif +rustflags-nojumptables := $(if $(call rustc-min-version,109300),-Cjump-tables=n,-Zno-jump-tables) + ifeq ($(CONFIG_X86_KERNEL_IBT),y) # # Kernel IBT has S_CET.NOTRACK_EN=0, as such the compilers must not generate
@@ -100,7 +102,7 @@ ifeq ($(CONFIG_X86_KERNEL_IBT),y) # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104816 # KBUILD_CFLAGS += $(call cc-option,-fcf-protection=branch -fno-jump-tables) -KBUILD_RUSTFLAGS += -Zcf-protection=branch $(if $(call rustc-min-version,109300),-Cjump-tables=n,-Zno-jump-tables) +KBUILD_RUSTFLAGS += -Zcf-protection=branch $(rustflags-nojumptables) else KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none) endif
@@ -178,6 +180,21 @@ endif KBUILD_RUSTFLAGS += -Ccode-model=kernel percpu_seg := gs + + pie-ccflags-$(CONFIG_CC_IS_GCC) += $(call cc-option.-mdirect-extern-access) + pie-ccflags-$(CONFIG_CC_IS_CLANG) += -fdirect-access-external-data + + # objtool gets confused by unannotated PIC flavor jump tables + pie-ccflags-y += $(call cc-option,-fannotate-jump-tables,-fno-jump-tables) + + pie-cflags-$(CONFIG_RELOCATABLE_PIE) := $(pie-ccflags-y) -fpie -mcmodel=small \ + -include $(srctree)/include/linux/hidden.h + pie-rustflags-$(CONFIG_RELOCATABLE_PIE) := -Crelocation-model=pie \ + -Ccode-model=small -Zdirect-access-external-data=yes \ + $(rustflags-nojumptables) + + KBUILD_CFLAGS_KERNEL += $(pie-cflags-y) + KBUILD_RUSTFLAGS_KERNEL += $(pie-rustflags-y) endif ifeq ($(CONFIG_STACKPROTECTOR),y)
arch/x86/boot/Makefile +1 -0
@@ -55,6 +55,7 @@ KBUILD_CFLAGS := $(REALMODE_CFLAGS) -D_SETUP KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables KBUILD_CFLAGS += $(CONFIG_CC_IMPLICIT_FALLTHROUGH) +KBUILD_CFLAGS_KERNEL := $(obj)/bzImage: asflags-y := $(SVGA_MODE)
arch/x86/boot/compressed/Makefile +1 -1
@@ -76,7 +76,7 @@ LDFLAGS_vmlinux += -T hostprogs := mkpiggy HOST_EXTRACFLAGS += -I$(srctree)/tools/include -sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__data_segment_start\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' +sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDdGRSTtVW] \(_text\|__data_segment_start\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' quiet_cmd_voffset = VOFFSET $@ cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
arch/x86/entry/vdso/Makefile +1 -0
@@ -143,6 +143,7 @@ endif endif $(obj)/vdso32.so.dbg: KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) +$(obj)/vdso32.so.dbg: KBUILD_CFLAGS_KERNEL := $(obj)/vdso32.so.dbg: $(obj)/vdso32/vdso32.lds $(vobjs32) FORCE $(call if_changed,vdso_and_check)
arch/x86/realmode/rm/Makefile +1 -0
@@ -67,3 +67,4 @@ KBUILD_CFLAGS := $(REALMODE_CFLAGS) -D_SETUP -D_WAKEUP \ -I$(srctree)/arch/x86/boot KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables +KBUILD_CFLAGS_KERNEL :=
include/asm-generic/vmlinux.lds.h +1 -0
@@ -373,6 +373,7 @@ *(DATA_MAIN) \ *(.data..decrypted) \ *(.ref.data) \ + *(.data.rel*) \ *(.data..shared_aligned) /* percpu related */ \ *(.data..unlikely) \ __start_once = .; \
include/linux/hidden.h +2 -0
@@ -16,4 +16,6 @@* giving them 'hidden' visibility. */ +#ifndef __BINDGEN__#pragma GCC visibility push(hidden) +#endif
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 9/19 x86/pm-trace: Use RIP-relative accesses for .tracedata
In reply to Ard BiesheuvelUse RIP-relative accesses and 32-bit offsets for .tracedata, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
--- arch/x86/include/asm/pm-trace.h | 4 ++-- drivers/base/power/trace.c | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/pm-trace.h +2 -2
@@ -8,10 +8,10 @@ do { \ if (pm_trace_enabled) { \ const void *tracedata; \ - asm volatile(_ASM_MOV " $1f,%0\n" \ + asm volatile("lea " _ASM_RIP(1f) ", %0\n" \ ".section .tracedata,\"a\"\n" \ "1:\t.word %c1\n\t" \ - _ASM_PTR " %c2\n" \ + ".long %c2 - .\n" \ ".previous" \ :"=r" (tracedata) \ : "i" (__LINE__), "i" (__FILE__)); \
drivers/base/power/trace.c +3 -3
@@ -167,7 +167,7 @@ EXPORT_SYMBOL(set_trace_device);void generate_pm_trace (const void *tracedata,unsigned int user) {unsigned short lineno = *(unsigned short *)tracedata; -const char *file = *(const char **)(tracedata +2 ); +const char *file =offset_to_ptr ((int *)(tracedata +2 ));unsigned int user_hash_value, file_hash_value;if (!x86_platform.legacy.rtc)
@@ -187,9 +187,9 @@ static int show_file_hash(unsigned int value) match =0 ;for (tracedata = __tracedata_start ; tracedata < __tracedata_end ; - tracedata +=2 +sizeof (unsigned long )) { + tracedata +=2 +sizeof (int )) {unsigned short lineno = *(unsigned short *)tracedata; -const char *file = *(const char **)(tracedata +2 ); +const char *file =offset_to_ptr ((int *)(tracedata +2 ));unsigned int hash =hash_string (lineno, file,FILEHASH );if (hash != value)continue ;
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 10/19 x86/kvm: Use RIP-relative addressing
In reply to Ard BiesheuvelReplace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
--- arch/x86/kernel/kvm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/kvm.c +3 -2
@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);* restoring to/from the stack. */ #define PV_VCPU_PREEMPTED_ASM \ - "movq __per_cpu_offset(,%rdi,8), %rax\n\t" \ - "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \ + "0:leaq 0b(%rip), %rax\n\t" \ + "addq __per_cpu_offset - 0b(%rax,%rdi,8), %rax\n\t" \ + "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time-0b(%rax)\n\t" \ "setne %al\n\t" DEFINE_ASM_FUNC(__raw_callee_save___kvm_vcpu_is_preempted,
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 11/19 x86/rethook: Use RIP-relative reference for fake return address
In reply to Ard BiesheuvelPushing an immediate absolute address to the stack is not permitted when linking x86_64 code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.
So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.
--- arch/x86/kernel/rethook.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/rethook.c +5 -1
@@ -11,6 +11,10 @@ __visible void arch_rethook_trampoline_callback(struct pt_regs *regs); +#ifdef CONFIG_X86_64 +static __used void *const __arch_rethook_trampoline = &arch_rethook_trampoline; +#endif +#ifndef ANNOTATE_NOENDBR #define ANNOTATE_NOENDBR #endif
@@ -27,7 +31,7 @@ asm(#ifdef CONFIG_X86_64 ANNOTATE_NOENDBR "\n" /* This is only jumped from ret instruction */ /* Push a fake return address to tell the unwinder it's a rethook. */ -" pushq $arch_rethook_trampoline\n" +" pushq __arch_rethook_trampoline(%rip)\n" UNWIND_HINT_FUNC " pushq $" __stringify(__KERNEL_DS) "\n"/* Save the 'sp - 16', this will be fixed later. */
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 12/19 x86/sync_core: Use RIP-relative addressing
In reply to Ard BiesheuvelUse RIP-relative accesses for sync_core(). This removes a 32-bit absolute reference that requires fixing up at runtime when KASLR is enabled. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
--- arch/x86/include/asm/sync_core.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/sync_core.h +2 -1
@@ -31,7 +31,8 @@ static __always_inline void iret_to_self(void)"pushfq\n\t" "mov %%cs, %0\n\t" "pushq %q0\n\t" -"pushq $1f\n\t" +"leaq 1f(%%rip), %q0\n\t" +"pushq %q0\n\t" "iretq\n\t" "1:" : "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory");
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 13/19 x86/entry_64: Use RIP-relative addressing
In reply to Ard BiesheuvelReplace a couple of instances in the x86_64 entry code where the absolute address of a symbol is taken in a manner that is not supported when linking in PIE mode, and use RIP-relative references instead, which don't require boot-time fixups at all.
--- arch/x86/entry/calling.h | 9 +++++---- arch/x86/entry/entry_64.S | 14 +++++++------- 2 files changed, 12 insertions(+), 11 deletions(-)
arch/x86/entry/calling.h +5 -4
@@ -376,8 +376,8 @@ For 32-bit we have the following conventions - kernel is built with .endm .macro SAVE_AND_SET_GSBASE scratch_reg:req save_reg:req + GET_PERCPU_BASE \scratch_reg \save_reg rdgsbase \save_reg - GET_PERCPU_BASE \scratch_reg wrgsbase \scratch_reg .endm
@@ -413,15 +413,16 @@ For 32-bit we have the following conventions - kernel is built with* Thus the kernel would consume a guest's TSC_AUX if an NMI arrives * while running KVM's run loop. */ -.macro GET_PERCPU_BASE reg:req +.macro GET_PERCPU_BASE reg:req scratch:req LOAD_CPU_AND_NODE_SEG_LIMIT \reg andq $VDSO_CPUNODE_MASK, \reg - movq __per_cpu_offset(, \reg, 8), \reg + leaq __per_cpu_offset(%rip), \scratch + movq (\scratch, \reg, 8), \reg .endm #else -.macro GET_PERCPU_BASE reg:req +.macro GET_PERCPU_BASE reg:req scratch:req movq pcpu_unit_offsets(%rip), \reg .endm
arch/x86/entry/entry_64.S +7 -7
@@ -1040,7 +1040,8 @@ SYM_CODE_START(error_entry) movl %ecx, %eax /* zero extend */ cmpq %rax, RIP+8(%rsp) je .Lbstep_iret - cmpq $.Lgs_change, RIP+8(%rsp) + leaq .Lgs_change(%rip), %rcx + cmpq %rcx, RIP+8(%rsp) jne .Lerror_entry_done_lfence /*
@@ -1252,10 +1253,10 @@ SYM_CODE_START(asm_exc_nmi) * the outer NMI. */ - movq $repeat_nmi, %rdx + leaq repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja 1f - movq $end_repeat_nmi, %rdx + leaq end_repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja nested_nmi_out 1:
@@ -1309,7 +1310,8 @@ nested_nmi: pushq %rdx pushfq pushq $__KERNEL_CS - pushq $repeat_nmi + leaq repeat_nmi(%rip), %rdx + pushq %rdx /* Put stack back */ addq $(6*8), %rsp
@@ -1348,10 +1350,8 @@ first_nmi: addq $8, (%rsp) /* Fix up RSP */ pushfq /* RFLAGS */ pushq $__KERNEL_CS /* CS */ - pushq $1f /* RIP */ - iretq /* continues at repeat_nmi below */ + call native_irq_return_iret UNWIND_HINT_IRET_REGS -1: #endif repeat_nmi:
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 14/19 x86/hibernate: Prefer RIP-relative accesses
In reply to Ard BiesheuvelReplace some absolute symbol references with RIP-relative ones, to avoid fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
--- arch/x86/power/hibernate_asm_64.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/power/hibernate_asm_64.S +2 -2
@@ -40,7 +40,7 @@ SYM_FUNC_START(restore_registers) movq %rax, %cr4; # turn PGE back on /* We don't restore %rax, it must be 0 anyway */ - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq pt_regs_sp(%rax), %rsp movq pt_regs_bp(%rax), %rbp movq pt_regs_si(%rax), %rsi
@@ -71,7 +71,7 @@ SYM_FUNC_START(restore_registers) SYM_FUNC_END(restore_registers) SYM_FUNC_START(swsusp_arch_suspend) - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax) movq %rsi, pt_regs_si(%rax)
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 15/19 x64/acpi: Use PIC-compatible references in wakeup_64.S
In reply to Ard BiesheuvelUse ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.
Note that wakeup_long64() runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.
--- arch/x86/kernel/acpi/wakeup_64.S | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +4 -7
@@ -14,7 +14,7 @@ .code64 /* - * Hooray, we are in Long 64-bit mode (but still running in low memory) + * Hooray, we are in Long 64-bit mode */ SYM_FUNC_START(wakeup_long64) ANNOTATE_NOENDBR
@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64) movq saved_rsi(%rip), %rsi movq saved_rbp(%rip), %rbp - movq saved_rip(%rip), %rax + leaq .Lresume_point(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%rax SYM_FUNC_END(wakeup_long64)
@@ -52,7 +52,7 @@ SYM_FUNC_START(do_suspend_lowlevel) xorl %eax, %eax call save_processor_state - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax) movq %rsi, pt_regs_si(%rax)
@@ -71,8 +71,6 @@ SYM_FUNC_START(do_suspend_lowlevel) pushfq popq pt_regs_flags(%rax) - movq $.Lresume_point, saved_rip(%rip) - movq %rsp, saved_rsp(%rip) movq %rbp, saved_rbp(%rip) movq %rbx, saved_rbx(%rip)
@@ -90,7 +88,7 @@ SYM_FUNC_START(do_suspend_lowlevel) .Lresume_point: ANNOTATE_NOENDBR /* We don't restore %rax, it must be 0 anyway */ - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq saved_context_cr4(%rax), %rbx movq %rbx, %cr4 movq saved_context_cr3(%rax), %rbx
@@ -139,7 +137,6 @@ saved_rsi: .quad 0 saved_rdi: .quad 0 saved_rbx: .quad 0 -saved_rip: .quad 0 saved_rsp: .quad 0 SYM_DATA(saved_magic, .quad 0)
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 16/19 x86/kexec: Use 64-bit wide absolute reference from relocated code
In reply to Ard BiesheuvelThe virtual_mapped() kexec routine runs from a different virtual address than it was linked at, and so it needs to use an absolute reference to load the address of 'saved_context'. Change this reference to a 64-bit wide one, to make the code compatible with linking in PIE mode.
--- arch/x86/kernel/relocate_kernel_64.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/relocate_kernel_64.S +1 -1
@@ -311,7 +311,7 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped) #ifdef CONFIG_KEXEC_JUMP /* Saved in save_processor_state. */ - movq $saved_context, %rax + movabsq $saved_context, %rax lgdt saved_context_gdt_desc(%rax) #endif
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 17/19 x86/head64: Avoid absolute references in startup asm
In reply to Ard BiesheuvelReplace a couple of occurrences of absolute references with RIP-relative ones. This removes the need for boot-time fixups. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
--- arch/x86/kernel/head_64.S | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
arch/x86/kernel/head_64.S +10 -7
@@ -179,8 +179,9 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) xorl %r15d, %r15d /* Derive the runtime physical address of init_top_pgt[] */ - movq phys_base(%rip), %rax - addq $(init_top_pgt - __START_KERNEL_map), %rax + leaq init_top_pgt(%rip), %rax + subq $__START_KERNEL_map, %rax + addq phys_base(%rip), %rax /* * Retrieve the modifier (SME encryption mask if SME is active) to be
@@ -232,6 +233,9 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 + /* Use .text as an anchor to emit PC-relative symbol references */ + leaq .text(%rip), %rbx + #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then
@@ -288,10 +292,9 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) .Llookup_AP: /* EAX contains the APIC ID of the current CPU */ xorl %ecx, %ecx - leaq cpuid_to_apicid(%rip), %rbx .Lfind_cpunr: - cmpl (%rbx,%rcx,4), %eax + cmpl cpuid_to_apicid - .text(%rbx,%rcx,4), %eax jz .Lsetup_cpu inc %ecx #ifdef CONFIG_FORCE_NR_CPUS
@@ -311,7 +314,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) .Lsetup_cpu: /* Get the per cpu offset for the given CPU# which is in ECX */ - movq __per_cpu_offset(,%rcx,8), %rdx + movq __per_cpu_offset - .text(%rbx,%rcx,8), %rdx #else xorl %edx, %edx /* zero-extended to clear all of RDX */ #endif /* CONFIG_SMP */
@@ -322,7 +325,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) * * RDX contains the per-cpu offset */ - movq current_task(%rdx), %rax + movq current_task - .text(%rbx,%rdx), %rax movq TASK_threadsp(%rax), %rsp /*
@@ -343,7 +346,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) */ subq $16, %rsp movw $(GDT_SIZE-1), (%rsp) - leaq gdt_page(%rdx), %rax + leaq gdt_page - .text(%rbx,%rdx), %rax movq %rax, 2(%rsp) lgdt (%rsp) addq $16, %rsp
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 18/19 x86/boot: Implement support for RELA/RELR/REL runtime relocations
In reply to Ard BiesheuvelGiven that the decompressor already incorporates an ELF loader that parses the program headers of the decompressed ELF image, support for dealing with the PT_DYNAMIC program header can be added quite easily, which describes the location of the RELA and RELR relocation tables in the image.
This is a more efficient, and more idiomatic format, which allows the handling of boot-time randomization (KASLR) in a generic manner, rather than based on a bespoke x86-specific relocation format. This is a prerequisite for enabling further hardening measures that are implemented in the ELF domain, i.e., fgkaslr.
--- arch/x86/boot/compressed/misc.c | 73 +++++++++++++++++++- include/uapi/linux/elf.h | 3 + 2 files changed, 74 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/misc.c +71 -2
@@ -278,7 +278,68 @@ static inline void handle_relocations(void *output, unsigned long output_len, { }#endif -static size_t parse_elf (void *output) +#define ELF (type) __PASTE(__PASTE(Elf, __LONG_WIDTH__), __PASTE(_, type)) + +static void handle_dynamic (const ELF (Dyn) *dyn,unsigned long p2v_offset, +unsigned long va_shift) +{ +const ELF (Rela) *rela =NULL ; +const ELF (Rel) *rel =NULL ; +unsigned long *relr =NULL ; +unsigned long *place; +int relasize =0 ; +int relrsize =0 ; +int relsize =0 ; + + for (auto d = dyn; d->d_tag != DT_NULL; d++) { +switch (d->d_tag) { +case DT_RELA : + rela = (void *)(d->d_un.d_ptr + p2v_offset); +break ; +case DT_RELASZ : + relasize = d->d_un.d_val; +break ; +case DT_RELR : + relr = (void *)(d->d_un.d_ptr + p2v_offset); +break ; +case DT_RELRSZ : + relrsize = d->d_un.d_val; +break ; +case DT_REL : + rel = (void *)(d->d_un.d_ptr + p2v_offset); +break ; +case DT_RELSZ : + relsize = d->d_un.d_val; +break ; + } + } + +for (int i =0 ; i < relasize /sizeof (*rela); i++) { + place = (unsigned long *)(rela[i].r_offset + p2v_offset); + *place += va_shift; + } + +for (int i =0 ; i < relrsize /sizeof (*relr); i++) { +if ((relr[i] &1 ) ==0 ) { + place = (unsigned long *)(relr[i] + p2v_offset); + *place++ += va_shift; +continue ; + } + +for (unsigned long *p = place, r = relr[i] >>1 ; r; p++, r >>=1 ) +if (r &1 ) + *p += va_shift; + place +=8 *sizeof (*relr) -1 ; + } + +for (int i =0 ; i < relsize /sizeof (*rel); i++) { + place = (unsigned long *)(rel[i].r_offset + p2v_offset); + *place += va_shift; + } + +} + +static size_t parse_elf (void *output,u64 va_shift) {#ifdef CONFIG_X86_64 Elf64_Ehdr ehdr;
@@ -320,6 +381,12 @@ static size_t parse_elf(void *output) dest += (unsigned long )output -LOAD_PHYSICAL_ADDR ;memmove (dest, output + phdr->p_offset, phdr->p_filesz);break ; +case PT_DYNAMIC : +if (!va_shift) +break ; + dest = (void *)(output + phdr->p_paddr -LOAD_PHYSICAL_ADDR ); +handle_dynamic (dest, (unsigned long )dest - phdr->p_vaddr, va_shift); +break ;default :/* Ignore other PT_* */ break ; } }
@@ -351,7 +418,9 @@ unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr, NULL, error) < 0)return ULONG_MAX ; - entry =parse_elf (outbuf); +if (IS_ENABLED (CONFIG_X86_32 )) + virt_addr = (unsigned long )outbuf; + entry =parse_elf (outbuf, virt_addr -LOAD_PHYSICAL_ADDR );handle_relocations (outbuf, output_len, virt_addr);return entry;
include/uapi/linux/elf.h +3 -0
@@ -103,6 +103,9 @@ typedef __u16 Elf64_Versym;#define DT_TEXTREL 22#define DT_JMPREL 23#define DT_ENCODING 32 +#define DT_RELRSZ 35 +#define DT_RELR 36 +#define DT_RELRENT 37#define OLD_DT_LOOS 0x60000000#define DT_LOOS 0x6000000d#define DT_HIOS 0x6ffff000
Signature · 1 line
2.47.3
Ard Biesheuvel
maintainerPATCH 19/19 x86/kernel: Switch to PIE linking for the relocatable kernel
In reply to Ard BiesheuvelIf the toolchain supports RELR relocation packing, build the virtually relocatable kernels as Position Independent (PIE) Executables. This results in more efficient relocation processing for the virtual displacement of the kernel applied at boot, using RELR relocations that take up only a fraction of the space occupied by ordinary RELA relocations.
More importantly, it instructs the linker to generate a binary that is really meant to be relocated at boot, using data structures that are intended for this purpose. Doing so is important for a couple of reasons:
- Relying on --emit-relocs is problematic, because it produces the static relocations that are consumed by the linker as input, and these are not meant for describing a runtime relocatable image. For example, the linker may apply relaxations that result in the code and the static relocation going out of sync (and ld.bfd and ld.lld already handle this in a different way).
- The 'relocs' tool relies on manually kept allow/deny lists of symbol names. These are needed because ELF absolute/relative symbol designations are often inaccurate.
- x86 deviates from other architectures in the kernel when it comes to its implementation of boot-time relocation, making it difficult to implement further enhancements (e.g., fgkaslr, EFI zboot) in a portable manner.
Note that this means that all codegen on x86_64 should be position independent, to be compatible with PIE linking, but only if KASLR is enabled. On i386, no changes to the codegen are needed, as the ordinary position dependent relocation model is supported by the linker when operating in PIE mode.
--- arch/x86/Kconfig | 3 ++- arch/x86/Makefile | 5 +++++ arch/x86/kernel/vmlinux.lds.S | 18 ++++++++++++++++++ 3 files changed, 25 insertions(+), 1 deletion(-)
arch/x86/Kconfig +2 -1
@@ -103,6 +103,7 @@ config X86 select ARCH_HAS_NONLEAF_PMD_YOUNG if PGTABLE_LEVELS > 2 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 + select ARCH_HAS_RELR select ARCH_HAS_SET_MEMORY select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_STRICT_KERNEL_RWX
@@ -2129,7 +2130,7 @@ config RANDOMIZE_BASE # Relocation on x86 needs some additional build support config X86_NEED_RELOCS def_bool y - depends on RELOCATABLE + depends on RELOCATABLE && !TOOLS_SUPPORT_RELR select ARCH_VMLINUX_NEEDS_RELOCS config PHYSICAL_ALIGN
arch/x86/Makefile +5 -0
@@ -258,6 +258,11 @@ endif KBUILD_LDFLAGS += -m elf_$(UTS_MACHINE) +ldflags-pie-$(CONFIG_LD_IS_LLD) := --apply-dynamic-relocs +ldflags-pie-$(CONFIG_LD_IS_BFD) := -z call-nop=suffix-nop +ldflags-$(CONFIG_RELOCATABLE_PIE) := --pie -z notext $(ldflags-pie-y) +LDFLAGS_vmlinux += $(ldflags-y) + # # The 64-bit kernel must be aligned to 2MB. Pass -z max-page-size=0x200000 to # the linker to force 2MB page size regardless of the default page size used
arch/x86/kernel/vmlinux.lds.S +18 -0
@@ -127,6 +127,9 @@ PHDRS { text PT_LOAD FLAGS(5); /* R_E */ data PT_LOAD FLAGS(6); /* RW_ */ note PT_NOTE FLAGS(0); /* ___ */ +#ifdef CONFIG_RELOCATABLE_PIE + dynamic PT_DYNAMIC; +#endif } SECTIONS
@@ -201,6 +204,21 @@ SECTIONS DATA_SEGMENT_START INIT_DATA_SECTION(16) :data +#ifdef CONFIG_RELOCATABLE_PIE + /DISCARD/ : { + *(.interp .dynbss .eh_frame .sframe .relr.auth.dyn) + } + + .dynamic : { *(.dynamic) } :dynamic :data + .dynstr : { *(.dynstr) } :data + .dynsym : { *(.dynsym) } + .gnu.hash : { *(.gnu.hash) } + .hash : { *(.hash) } + .init.rela : { *(.rela.*) *(.rela_*) } + .init.rel : { *(.rel.*) *(.rel_*) } + .init.relr : { *(.relr.*) } +#endif + .x86_cpu_dev.init : AT(ADDR(.x86_cpu_dev.init) - LOAD_OFFSET) { __x86_cpu_dev_start = .; *(.x86_cpu_dev.init)
Signature · 1 line
2.47.3
Re: [RFC/RFT PATCH 11/19] x86/rethook: Use RIP-relative reference for fake return address
In reply to Ard BiesheuvelQuoted 7 lines · On Thu, 8 Jan 2026 09:25:38 +0000 Ard Biesheuvel <[email protected]> wrote:
Pushing an immediate absolute address to the stack is not permitted when linking
x86_64code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.
The comment implies the address is 'fake'.
Does that mean it could just be a constant?
Clearly the unwinder would need the same change.
David
Quoted 30 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/rethook.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)arch/x86/kernel/rethook.c +5
-1@@ -11,6 +11,10 @@ __visible void arch_rethook_trampoline_callback(struct pt_regs *regs); +#ifdef CONFIG_X86_64 +static __used void *const __arch_rethook_trampoline = &arch_rethook_trampoline; +#endif +#ifndef ANNOTATE_NOENDBR #define ANNOTATE_NOENDBR #endif @@ -27,7 +31,7 @@ asm(#ifdef CONFIG_X86_64 ANNOTATE_NOENDBR "\n" /* This is only jumped from ret instruction */ /* Push a fake return address to tell the unwinder it's a rethook. */ -" pushq $arch_rethook_trampoline\n" +" pushq __arch_rethook_trampoline(%rip)\n" UNWIND_HINT_FUNC " pushq $" __stringify(__KERNEL_DS) "\n"/* Save the 'sp - 16', this will be fixed later. */
Ard Biesheuvel
maintainerRe: [RFC/RFT PATCH 11/19] x86/rethook: Use RIP-relative reference for fake return address
Quoted 9 lines · David Laight
Quoted 5 lines · On Thu, 8 Jan 2026 09:25:38 +0000 Ard Biesheuvel <[email protected]> wrote:
Pushing an immediate absolute address to the stack is not permitted when linking
x86_64code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.
Quoted 5 lines
Quoted 2 lines
So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.
The comment implies the address is 'fake'.
Does that mean it could just be a constant?
It could be, but it isn't, across all architectures.
Quoted 2 lines
Clearly the unwinder would need the same change.
Why? The value being pushed is the same.
Ard Biesheuvel
maintainerRe: [RFC/RFT PATCH 11/19] x86/rethook: Use RIP-relative reference for fake return address
Quoted 17 lines · Ard Biesheuvel
Quoted 9 lines · David Laight
Quoted 5 lines · On Thu, 8 Jan 2026 09:25:38 +0000 Ard Biesheuvel <[email protected]> wrote:
Pushing an immediate absolute address to the stack is not permitted when linking
x86_64code in PIE mode. Usually, the address can be taken using a RIP-relative LEA instruction, but this is not possible here as there are no available registers.Quoted 5 lines
Quoted 2 lines
So instead, take the address into a static global, and push it onto the stack using a RIP-relative memory operand.
The comment implies the address is 'fake'.
Does that mean it could just be a constant?
Quoted 6 lines
It could be, but it isn't, across all architectures.
Quoted 2 lines
Clearly the unwinder would need the same change.
Why? The value being pushed is the same.
Never mind - I guess you meant 'the unwinder would need to use the constant too'
From: Ard Biesheuvel <[email protected]>
Date: Thu, 8 Jan 2026 09:25:27 +0000
Quoted 29 lines
This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of
x86_64vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].The main sticking point is the fact that PIE linking on
x86_64requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)
So now that we have some actual numbers, I would like to try and revisit this discussion, and get a conclusion on whether this is really a non-starter. Note that only the KASLR kernel would rely on this, and disabling
CONFIG_RANDOMIZE_BASEwill revert to the current situation (provided that patch #4 is applied)Some minor asm tweaks are needed too (patches #9 - #17), but those all seem uncontroversial to me.
The first 5 patches are general cleanup, and could be taken into consideration independently of the discussion around PIC codegen.
Quoted 11 lines
[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).
In case anybody is interested...
The latest (to my knowledge) experiments with FG-KALSR was my side project reviving Kristen's old series (and then rewriting it completely): [0]
I haven't worked on it since then, as I work in an XDP/netmem/whatever team, i.e. networking, not x86, and free time for side projects shrunk severely since 2022.
Maybe someone would pick it up again some day, just like I picked up Kristen's series back then...
[0] https://github.com/alobakin/linux/commits/fgkaslr
Thanks,
Olek
H. Peter Anvin
reviewerQuoted 4 lines · Ard Biesheuvel
This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of
x86_64vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].
Kristin Accardi had fg-kasrl running without that, didn't she?
From your footnotes, it looks like what you are *really* asking for is to pessimize x86 code to benefit other architectures. That isn't inherently wrong, but stating it as you have above is dishonest.
Quoted 11 lines
The main sticking point is the fact that PIE linking on
x86_64requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)
Could you show some examples of how the code changes?
-hpa
Quoted 13 lines
[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).
Quoted 6 lines
[2] EFI zboot is already used by various architectures that have no decompressor stage at all (arm64, RISC-V, LoongArch), and this format can be combined with an ELF payload too. EFI zboot accommodates non-EFI boot chains by describing the size, offset, payload type and compression type in its header, so that it can be extracted and booted by other means.
The bzImage format already have that for all practical purposes. We *really* don't want to introduce a new binary format for the x86 kernel. A bunch of such attempts have been done in the past, and it is nothing but a mess that breaks things, because now you are encouraging different bootloaders to support a non-overlapping set of binary formats.
STRONG NAK on that one.
-hpa
Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S
In reply to Ard BiesheuvelQuoted 27 lines · Ard Biesheuvel
Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.
Note that
wakeup_long64()runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/acpi/wakeup_64.S | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -14,7 +14,7 @@ .code64 /* - * Hooray, we are in Long 64-bit mode (but still running in low memory) + * Hooray, we are in Long 64-bit mode */ SYM_FUNC_START(wakeup_long64) ANNOTATE_NOENDBR
arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46
Quoted 7 lines
arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64) movq saved_rsi(%rip), %rsi movq saved_rbp(%rip), %rbp - movq saved_rip(%rip), %rax + leaq .Lresume_point(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%rax
If this is already running on the virtual mapping, this can simply be changed to a direct jump.
Brian Gerst
Ard Biesheuvel
maintainerRe: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S
Quoted 32 lines · Brian Gerst
Quoted 30 lines · Ard Biesheuvel
Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.
Note that
wakeup_long64()runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/acpi/wakeup_64.S | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -14,7 +14,7 @@ .code64 /* - * Hooray, we are in Long 64-bit mode (but still running in low memory) + * Hooray, we are in Long 64-bit mode */ SYM_FUNC_START(wakeup_long64) ANNOTATE_NOENDBR@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64) movq saved_rsi(%rip), %rsi movq saved_rbp(%rip), %rbp
Quoted 8 lines
arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46
Quoted 4 lines
arch/x86/kernel/acpi/wakeup_64.S +1
-1- movq saved_rip(%rip), %rax + leaq .Lresume_point(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%raxIf this is already running on the virtual mapping, this can simply be changed to a direct jump.
Indeed, but I couldn't figure out how to do so without making objtool unhappy.
Ard Biesheuvel
maintainerQuoted 9 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel
This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of
x86_64vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].Kristin Accardi had fg-kasrl running without that, didn't she?
Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.
One could argue that this example is just a Clang bug, but it is very difficult to make that case with the toolchain developers, given that --emit-relocs (which is what tells the linker to emit the relocations that it received as input) has no specification, and some linker relaxations are not representable as static relocations to begin with (but to be fair, that currently mostly affects other architectures, but there is no reason this could never happen on x86)
Doing fgkaslr properly (IMHO) means supporting things like live patch and debug seamlessly, and in a portable manner. Toolchain support is critical, and securing that for a one-off x86 implementation rather than one that can be used across architectures and other bare-metal projects is going to be difficult.
Quoted 4 lines
From your footnotes, it looks like what you are *really* asking for is to pessimize x86 code to benefit other architectures. That isn't inherently wrong, but stating it as you have above is dishonest.
I was hoping to save the ad-hominems for later in the thread, when things *really* heat up.
The point is not to benefit other architectures. The point is to implement something once, and deploy it on all architectures in the same way. ELF is the greatest common denominator across the entire ecosystem, and so using idiomatic ELF to describe how to load the image and how to move it around in the virtual address space is on obvious choice.
Quoted 6 lines
Quoted 6 lines
The main sticking point is the fact that PIE linking on
x86_64requires PIE codegen, and that was shot down before on the basis that
a) GOTs in fully linked binaries are stupid
b) the code size increase would be prohibitive
c) the performance would suffer.
Quoted 8 lines
Quoted 5 lines
This series implements PIE codegen without permitting the use of GOT slots. The code size increase is between 0.2% (clang) and 0.5% (gcc), and I could not identify any performance regressions (using hackbench) on various different micro-architectures that I tried it on. (Suggestions for other benchmarks/test cases are welcome)
Could you show some examples of how the code changes?
Taking the address of a symbol (same code size)
0: 48 c7 c0 00 00 00 00 mov $0x0,%rax
3: R_X86_64_32S sym
7: 48 8d 05 00 00 00 00 lea 0x0(%rip),%rax # 0xe
a: R_X86_64_PC32
Loading a global variable from memory (one byte shorter in PIC)
e: 48 8b 04 25 00 00 00 mov 0x0,%rax
15: 00
12: R_X86_64_32S sym
16: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 0x1d
19: R_X86_64_PC32 sym-0x4
Indexing a global array (3 bytes longer in PIC, needs an additional GPR if source and destination are the same)
1d: 48 8b 04 c5 00 00 00 mov 0x0(,%rax,8),%rax
24: 00
21: R_X86_64_32S array
25: 48 8d 15 00 00 00 00 lea 0x0(%rip),%rdx # 0x2c
28: R_X86_64_PC32 array-0x4
2c: 48 8b 04 c2 mov (%rdx,%rax,8),%rax
Pushing the address of a symbol to the stack ((3 bytes longer in PIC, needs an additional GPR)
30: 68 00 00 00 00 push $0x0
31: R_X86_64_32S sym
35: 48 8d 05 00 00 00 00 lea 0x0(%rip),%rax # 0x3c
38: R_X86_64_PC32 sym-0x4
3c: 50 push %rax
Jump tables look completely different, but the table itself is only half the size. Even for non-PIC, jump tables are problematic for objtool, and so these need to be annotated by the compiler. I have some unfinished Clang patches that implement this, which I hope to get back to soon.
The asm patches in the series should give a good impression of how the code changes.
Quoted 20 lines
Quoted 13 lines
[1] There have been a few attempts at landing fine grained KASLR for x86, but the main problem is that it was tied to the x86 relocation format, which deviates from how fully linked relocatable ELF binaries are generally constructed (using PIE). Implementing fgkaslr in the ELF domain would make it suitable for other architectures too, as well as other use cases (bare metal or hosted) where no dynamic linking is performed (firmware, hypervisors). In order to implement this properly, i.e., with debugging support etc, it needs support from the tooling side. (Fine grained KASLR in combination with execute-only code mappings makes it extremely difficult for an attacker to subvert the control flow in the kernel in a way that can be meaningfully exploited).
Quoted 6 lines
[2] EFI zboot is already used by various architectures that have no decompressor stage at all (arm64, RISC-V, LoongArch), and this format can be combined with an ELF payload too. EFI zboot accommodates non-EFI boot chains by describing the size, offset, payload type and compression type in its header, so that it can be extracted and booted by other means.
Quoted 8 lines
The bzImage format already have that for all practical purposes. We *really* don't want to introduce a new binary format for the x86 kernel. A bunch of such attempts have been done in the past, and it is nothing but a mess that breaks things, because now you are encouraging different bootloaders to support a non-overlapping set of binary formats.
STRONG NAK on that one.
I think it should be feasible to implement a hybrid bzImage/EFI zboot format. There is already prior art in loaders that decompress the ELF payload directly (Xen).
Given that a x86_64 bootloader running in long mode needs to do very little beyond loading the ELF at some arbitrary 2M aligned offset and calling the entrypoint with a struct bootparams in %RDI, most of the logic in the decompressor is really only needed when booting in 32-bit mode.
So I think there is value in having a generic boot format that can be consumed by EFI directly, or by a generic ELF vmlinux loader (library) that understands the EFI zboot format and knows how to extract the ELF payload. I'd strongly prefer only a single idiom for describing the relocations in the image.
On other architectures (i.e., without decompressor), EFI zboot would be a prerequisite for fgkaslr, but it is up to the platform to decide whether to boot via EFI or load the ELF and apply the relocations. On x86_64, the same tooling would work seamlessly, but the decompressor could apply the relocations itself as well.
Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S
In reply to Ard BiesheuvelQuoted 43 lines · Ard Biesheuvel
Quoted 32 lines · Brian Gerst
Quoted 30 lines · Ard Biesheuvel
Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.
Note that
wakeup_long64()runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/acpi/wakeup_64.S | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -14,7 +14,7 @@ .code64 /* - * Hooray, we are in Long 64-bit mode (but still running in low memory) + * Hooray, we are in Long 64-bit mode */ SYM_FUNC_START(wakeup_long64) ANNOTATE_NOENDBR@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64) movq saved_rsi(%rip), %rsi movq saved_rbp(%rip), %rbpQuoted 8 lines
arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46
Quoted 4 lines
arch/x86/kernel/acpi/wakeup_64.S +1
-1- movq saved_rip(%rip), %rax + leaq .Lresume_point(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%raxIf this is already running on the virtual mapping, this can simply be changed to a direct jump.
Quoted 1 line
Indeed, but I couldn't figure out how to do so without making objtool unhappy.
I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?
Ard Biesheuvel
maintainerRe: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S
Quoted 45 lines · Brian Gerst
Quoted 43 lines · Ard Biesheuvel
Quoted 40 lines · Brian Gerst
Quoted 27 lines · Ard Biesheuvel
Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.
Note that
wakeup_long64()runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/acpi/wakeup_64.S | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -14,7 +14,7 @@ .code64 /* - * Hooray, we are in Long 64-bit mode (but still running in low memory) + * Hooray, we are in Long 64-bit mode */ SYM_FUNC_START(wakeup_long64) ANNOTATE_NOENDBRarch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46
Quoted 7 lines
arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64) movq saved_rsi(%rip), %rsi movq saved_rbp(%rip), %rbp - movq saved_rip(%rip), %rax + leaq .Lresume_point(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%raxIf this is already running on the virtual mapping, this can simply be changed to a direct jump.
Quoted 4 lines
Quoted 1 line
Indeed, but I couldn't figure out how to do so without making objtool unhappy.
I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?
arch/x86/kernel/acpi/wakeup_64.o: warning: objtool: wakeup_long64() falls through to next function do_suspend_lowlevel()
Ard Biesheuvel
maintainerRe: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S
Quoted 47 lines · Ard Biesheuvel
Quoted 45 lines · Brian Gerst
Quoted 43 lines · Ard Biesheuvel
Quoted 40 lines · Brian Gerst
Quoted 27 lines · Ard Biesheuvel
Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.
Note that
wakeup_long64()runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/acpi/wakeup_64.S | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -14,7 +14,7 @@ .code64 /* - * Hooray, we are in Long 64-bit mode (but still running in low memory) + * Hooray, we are in Long 64-bit mode */ SYM_FUNC_START(wakeup_long64) ANNOTATE_NOENDBRarch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46
Quoted 7 lines
arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64) movq saved_rsi(%rip), %rsi movq saved_rbp(%rip), %rbp - movq saved_rip(%rip), %rax + leaq .Lresume_point(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%raxIf this is already running on the virtual mapping, this can simply be changed to a direct jump.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Indeed, but I couldn't figure out how to do so without making objtool unhappy.
I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?
arch/x86/kernel/acpi/wakeup_64.o: warning: objtool:wakeup_long64()falls through to next functiondo_suspend_lowlevel()
Note that this is x86_64_defconfig with CONFIG_X86_KERNEL_IBT disabled.
Re: [RFC/RFT PATCH 15/19] x64/acpi: Use PIC-compatible references in wakeup_64.S
Quoted 51 lines · Ard Biesheuvel
Quoted 49 lines · Ard Biesheuvel
Quoted 47 lines · Brian Gerst
Quoted 44 lines · Ard Biesheuvel
Quoted 32 lines · Brian Gerst
Quoted 30 lines · Ard Biesheuvel
Use ordinary RIP-relative references to make the code compatible with running the linker in PIE mode.
Note that
wakeup_long64()runs in the kernel's ordinary virtual mapping so there is no need to record the address of .Lresume_point in a global variable. And fix the comment while at it.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/acpi/wakeup_64.S | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)arch/x86/kernel/acpi/wakeup_64.S +1
-1@@ -14,7 +14,7 @@ .code64 /* - * Hooray, we are in Long 64-bit mode (but still running in low memory) + * Hooray, we are in Long 64-bit mode */ SYM_FUNC_START(wakeup_long64) ANNOTATE_NOENDBR@@ -41,7 +41,7 @@ SYM_FUNC_START(wakeup_long64) movq saved_rsi(%rip), %rsi movq saved_rbp(%rip), %rbpQuoted 8 lines
arch/x86/kernel/acpi/wakeup_64.S SYM_FUNC_START(wakeup_long64) · line 46
Quoted 4 lines
arch/x86/kernel/acpi/wakeup_64.S +1
-1- movq saved_rip(%rip), %rax + leaq .Lresume_point(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%raxIf this is already running on the virtual mapping, this can simply be changed to a direct jump.
Indeed, but I couldn't figure out how to do so without making objtool unhappy.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I replaced it with a simple "jmp .Lresume_point" and objtool seemed fine with it on a defconfig build. What error did you see?
arch/x86/kernel/acpi/wakeup_64.o: warning: objtool:wakeup_long64()falls through to next functiondo_suspend_lowlevel()Note that this is
x86_64_defconfigwithCONFIG_X86_KERNEL_IBTdisabled.
I do see the error now. I had missed it when building the .o file and thought it would show when linking vmlinux.
I was able to make objtool happy by embedding wakeup_long64() into do_suspend_lowlevel() and using SYM_INNER_LABEL_ALIGN().
Re: [RFC/RFT PATCH 08/19] x86: Use PIE codegen for the relocatable 64-bit kernel
In reply to Ard Biesheuvelarch/x86/Makefile endif · line 184
Quoted 12 lines · Ard Biesheuvel
As an intermediate step towards enabling PIE linking for the
x86_64KASLR kernel, enable PIE codegen for all C and Rust objects that are linked into the kernel proper. Add aKconfigoptionRELOCATABLE_PIEfor this, depending on RELR support in the linker, as the relocation tables will blow up the kernel image otherwise.
KBUILD_CFLAGS+= $(call cc-option,-fcf-protection=branch -fno-jump-tables)
...arch/x86/Makefile +2
-0+ + pie-ccflags-$(CONFIG_CC_IS_GCC) += $(call cc-option.-mdirect-extern-access)
I think that dot there next to -mdirect-extern-accesss should have been a comma.
Ard Biesheuvel
maintainerRe: [RFC/RFT PATCH 08/19] x86: Use PIE codegen for the relocatable 64-bit kernel
Quoted 11 lines · Jan Engelhardt
Quoted 6 lines · Ard Biesheuvel
As an intermediate step towards enabling PIE linking for the
x86_64KASLR kernel, enable PIE codegen for all C and Rust objects that are linked into the kernel proper. Add aKconfigoptionRELOCATABLE_PIEfor this, depending on RELR support in the linker, as the relocation tables will blow up the kernel image otherwise.
Quoted 8 lines
arch/x86/Makefile endif · line 184
Quoted 4 lines
KBUILD_CFLAGS+= $(call cc-option,-fcf-protection=branch -fno-jump-tables)
...arch/x86/Makefile +2
-0+ + pie-ccflags-$(CONFIG_CC_IS_GCC) += $(call cc-option.-mdirect-extern-access)I think that dot there next to -mdirect-extern-accesss should have been a comma.
Indeed, thanks for spotting that!
H. Peter Anvin
reviewerRe: [RFC/RFT PATCH 04/19] x86: Make the 64-bit bzImage always physically relocatable
In reply to Ard BiesheuvelQuoted 11 lines · Ard Biesheuvel
On
x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.
Quoted 7 lines
Clarify this in
Kconfigand in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whetherCONFIG_RELOCATABLEis set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite forEFI_STUBandRANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.
I don't see any reason to support non-relocatable kernels anymore. In fact, in a patchset I am working on I have already removed it.
-hpa
Re: [RFC/RFT PATCH 04/19] x86: Make the 64-bit bzImage always physically relocatable
Quoted 20 lines · On Sun, 11 Jan 2026 20:01:02 -0800 "H. Peter Anvin" <[email protected]> wrote:
Quoted 11 lines · Ard Biesheuvel
On
x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.
Quoted 7 lines
Clarify this in
Kconfigand in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whetherCONFIG_RELOCATABLEis set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite forEFI_STUBandRANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.
Quoted 2 lines
I don't see any reason to support non-relocatable kernels anymore. In fact, in a patchset I am working on I have already removed it.
For just 64bit, or 32bit as well?
The 'bloat' for 32bit will be higher due to the lack of pc-relative addressing.
David
Quoted 4 lines
-hpa
H. Peter Anvin
reviewerRe: [RFC/RFT PATCH 04/19] x86: Make the 64-bit bzImage always physically relocatable
Quoted 23 lines · David Laight
Quoted 20 lines · On Sun, 11 Jan 2026 20:01:02 -0800 "H. Peter Anvin" <[email protected]> wrote:
Quoted 18 lines · Ard Biesheuvel
On
x86_64, the physical placement of the kernel is independent from its mapping in the 'High Kernel Mapping' range. This means that even a position dependent kernel built without boot-time relocation support can run from any suitably aligned physical address, and there is no need to make this behavior dependent on whether or not the kernel is virtually relocatable.On i386, the situation is different, given that the physical and virtual load offsets must be equal, and so only a relocatable kernel can be loaded at a physical address that deviates from its build-time default.
Clarify this in
Kconfigand in the code, and advertise the 64-bit bzImage as loadable at any physical offset regardless of whetherCONFIG_RELOCATABLEis set. In practice, this makes little difference, given that it defaults to 'y' and is a prerequisite forEFI_STUBandRANDOMIZE_BASE, but it will help with some future refactoring of the relocation code.
Quoted 14 lines
Quoted 2 lines
I don't see any reason to support non-relocatable kernels anymore. In fact, in a patchset I am working on I have already removed it.
For just 64bit, or 32bit as well?
The 'bloat' for 32bit will be higher due to the lack of pc-relative addressing.David
Quoted 4 lines
-hpa
Either. The bloat is strictly boot time.
Kees Cook
maintainerQuoted 9 lines · Ard Biesheuvel
Quoted 8 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel
This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of
x86_64vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].Kristin Accardi had fg-kasrl running without that, didn't she?
I understand "such as fg-kaslr" to have been just a terse way of saying "such as a complete multi-architectural fg-kaslr"
Quoted 11 lines
Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.
Another historical bit of context is that one of the main reasons Kristen's fg-kaslr got stuck was the linker support needed for (the 65k worth of) section pass-through. That never got resolved, and the solutions either required huge linker files (that tickled performance flaws in the linkers) that resulted in 10 minute linking times, or to disable all the orphan section handling, which was a regression in our sanity checking and bug-finding.
So, getting a well-behaved fg-kaslr still needs toolchain support, and getting there is going to need further design work. As far as PIE, this just makes the fg-kaslr toolchain work easier (fewer special cases), along with all the other benefits of moving to PIE.
-Kees
Signature · 1 line
Kees Cook
Re: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing
In reply to Ard BiesheuvelQuoted 15 lines · Ard Biesheuvel
Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/kvm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)arch/x86/kernel/kvm.c +0
-0
arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810
Quoted 6 lines
arch/x86/kernel/kvm.c +1
-2@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);* restoring to/from the stack. */ #define PV_VCPU_PREEMPTED_ASM \ - "movq __per_cpu_offset(,%rdi,8), %rax\n\t" \ - "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \ + "0:leaq 0b(%rip), %rax\n\t" \
Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.
It might also be worth calling out in the changelog that this function is called across CPUs, e.g. from kvm_smp_send_call_func_ipi(), and thus can't use gs:
or any other "normal" method for accessing per-CPU data.
Quoted 8 lines
arch/x86/kernel/kvm.c +2
-0+"addq __per_cpu_offset - 0b(%rax,%rdi,8), %rax\n\t" \ + "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time-0b(%rax)\n\t" \"setne %al\n\t" DEFINE_ASM_FUNC(__raw_callee_save___kvm_vcpu_is_preempted,Signature · 1 line
2.47.3
Re: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing
Quoted 11 lines · On Tue, 20 Jan 2026 09:04:26 -0800 Sean Christopherson <[email protected]>…
Quoted 10 lines · Ard Biesheuvel
Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/kvm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Quoted 15 lines
arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810
Quoted 11 lines
arch/x86/kernel/kvm.c +1
-2@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);* restoring to/from the stack. */ #define PV_VCPU_PREEMPTED_ASM \ - "movq __per_cpu_offset(,%rdi,8), %rax\n\t" \ - "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \ + "0:leaq 0b(%rip), %rax\n\t" \Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.
I remember taking a while to grok that as well.
Can't you just use . as in:
leaq .(%rip), %rax
shame the assembler doesn't understand:
movq %rip, %rax
(maybe it does...)
David
H. Peter Anvin
reviewerQuoted 34 lines · Kees Cook
Quoted 9 lines · Ard Biesheuvel
Quoted 8 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel
This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of
x86_64vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].Kristin Accardi had fg-kasrl running without that, didn't she?
I understand "such as fg-kaslr" to have been just a terse way of saying "such as a complete multi-architectural fg-kaslr"
Quoted 11 lines
Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.
Another historical bit of context is that one of the main reasons Kristen's fg-kaslr got stuck was the linker support needed for (the 65k worth of) section pass-through. That never got resolved, and the solutions either required huge linker files (that tickled performance flaws in the linkers) that resulted in 10 minute linking times, or to disable all the orphan section handling, which was a regression in our sanity checking and bug-finding.
Quoted 5 lines
So, getting a well-behaved fg-kaslr still needs toolchain support, and getting there is going to need further design work. As far as PIE, this just makes the fg-kaslr toolchain work easier (fewer special cases), along with all the other benefits of moving to PIE.
As I *explicitly* stated earlier, there isn't anything inherently wrong with putting a small onus on x86 in order to make the general Linux code better -- but please, be honest about it *so we know what the actual tradeoffs are*.
For x86, we really do want to maintain the kernel memory model, which allows us to directly reference symbols in complex address expressions and to directly jump across modules. This means the "PIE" will need to be different from the way PIE works in user space, which is in part designed to avoid needing to dirty readonly pages, which would inhibit sharing -- which is explicitly NOT a concern for the kernel.
So that is one thing that the toolchain needs to be able to do.
I fully expect that we will continue to need to have some kinds of overrides for specific symbols, too, because there aren't any really sane ways to express them to the toolchain; this especially applies to linker-script and some assembly symbols. For example, the real-mode code (which uses the reloc tool as well) has to support segment and segbase-relative relocations, which are something that ELF simply has no concept of.
I have a lot more of an issue with trying to change the x86 boot protocol, simply because the way booting works in x86 has been incredibly successful; yes, the bzImage file format is ugly as ****, but that is a direct result of 34 years of continuous backwards compatibility. One of the reasons we have been able to do that is that we have *explicitly* rejected other boot models, such as Grub's self-declared Multiboot "standard" (which they have had to revise multiple times by now) and the early Xen boot model of booting vmlinux directly. We have added *many* capabilities to bzImage as needed, and it has turned out to be quite flexible in the end.
That, in turn, has been possible exactly *because* the Linux kernel provides a "prekernel". I don't even really like calling it the "decompressor" anymore; it really has developed far beyond that.
Every time you introduce a new boot model you take a serious risk that a boot loader author will say "hey, I'll just support this new model", and you *also* take the serious risk that the boot model isn't adequate. When the Grub authors says "we started using the 'modern' Linux entry point" -- meaning they would bypass the entry stub and call the kernel entry point directly -- they *reduced* the overall functionality, because history has shown that:
a. The kernel image is far easier for the end user to update than the potentially many bootloaders, because the bootloader depends on the deployment model (consider network booting, for example, or when the bootloader is in firmware.)
b. The kernel image provides ONE CENTRAL PLACE to add features, fix bugs, and develop workarounds for strange systems.
Thus, I will do anything I can to continue to veto changes to the x86 boot model, unless they come with a VERY VERY good motivation.
We originally made the mistake with EFI to leave too much to the bootloader; because of the downsides above we really needed to backpedal and take over control much earlier in the flow, just as with BIOS -- apparently to the Grub developers' utterly inexplicable chagrin.
-hpa
Ard Biesheuvel
maintainerRe: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing
In reply to David LaightQuoted 31 lines · David Laight
Quoted 11 lines · On Tue, 20 Jan 2026 09:04:26 -0800 Sean Christopherson <[email protected]>…
Quoted 10 lines · Ard Biesheuvel
Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/kvm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Quoted 15 lines
arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810
Quoted 11 lines
arch/x86/kernel/kvm.c +1
-2@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);* restoring to/from the stack. */ #define PV_VCPU_PREEMPTED_ASM \ - "movq __per_cpu_offset(,%rdi,8), %rax\n\t" \ - "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \ + "0:leaq 0b(%rip), %rax\n\t" \Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.
Quoted 5 lines
I remember taking a while to grok that as well.
Can't you just use . as in:
leaq .(%rip), %rax
How would the other two instructions referring to '0b' in their immediates refer to '.' in that case?
Re: [RFC/RFT PATCH 10/19] x86/kvm: Use RIP-relative addressing
Quoted 32 lines · On Tue, 20 Jan 2026 21:54:30 +0100 Ard Biesheuvel <[email protected]> wrote:
Quoted 31 lines · David Laight
Quoted 26 lines · On Tue, 20 Jan 2026 09:04:26 -0800 Sean Christopherson <[email protected]>…
Quoted 15 lines · Ard Biesheuvel
Replace absolute references in inline asm with RIP-relative ones, to avoid the need for relocation fixups at boot time. This is a prerequisite for PIE linking, which only permits 64-bit wide loader-visible absolute references.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/kvm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)arch/x86/kernel/kvm.c +0
-0arch/x86/kernel/kvm.c extern bool __raw_callee_save___kvm_vcpu_is_preempted(long); · line 810
Quoted 6 lines
arch/x86/kernel/kvm.c +1
-2@@ -807,8 +807,9 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);* restoring to/from the stack. */ #define PV_VCPU_PREEMPTED_ASM \ - "movq __per_cpu_offset(,%rdi,8), %rax\n\t" \ - "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \ + "0:leaq 0b(%rip), %rax\n\t" \Please use something other than '0' for the label, it took me forever (and looking at disassembly) to realize "0b" was just a backwards label and not some fancy syntax I didn't know.
Quoted 8 lines
Quoted 5 lines
I remember taking a while to grok that as well.
Can't you just use . as in:
leaq .(%rip), %raxHow would the other two instructions referring to '0b' in their immediates refer to '.' in that case?
I'd forgotten about those, not in the quoted bit of patch :-(
Traditionally (going back to MACRO-11) numeric labels would start at 10 and go up in 10s, TECO had a nice macro to renumber them. (Showing my age again)
David
Ard Biesheuvel
maintainerQuoted 42 lines · H. Peter Anvin
Quoted 34 lines · Kees Cook
Quoted 9 lines · Ard Biesheuvel
Quoted 8 lines · H. Peter Anvin
Quoted 4 lines · Ard Biesheuvel
This series is a follow-up to a series I sent a bit more than a year ago, to switch to PIE linking of
x86_64vmlinux, which is a prerequisite for further hardening measures, such as fg-kaslr [1], as well as further harmonization of the boot protocols between architectures [2].Kristin Accardi had fg-kasrl running without that, didn't she?
I understand "such as fg-kaslr" to have been just a terse way of saying "such as a complete multi-architectural fg-kaslr"
Quoted 11 lines
Yes, as a proof of concept. But it is tied to the x86 approach of performing runtime relocations based on build time relocation data, which is problematic now that linkers have started to perform relaxations, as these cannot always be translated 1:1. For instance, we already have a latent bug in the x86 relocs tool, which ignores GOTPCREL relocations on the basis that the relocation is relative. However, this is only true for Clang/lld, which does not update the static relocation tables after performing relaxations. ld.bfd does attempt to keep those tables in sync, and so a GOTPCREL relocation should be flagged as a bug when encountered, because it means there is a GOT slot somewhere with no relocation associated with it.
Another historical bit of context is that one of the main reasons Kristen's fg-kaslr got stuck was the linker support needed for (the 65k worth of) section pass-through. That never got resolved, and the solutions either required huge linker files (that tickled performance flaws in the linkers) that resulted in 10 minute linking times, or to disable all the orphan section handling, which was a regression in our sanity checking and bug-finding.
Quoted 5 lines
So, getting a well-behaved fg-kaslr still needs toolchain support, and getting there is going to need further design work. As far as PIE, this just makes the fg-kaslr toolchain work easier (fewer special cases), along with all the other benefits of moving to PIE.
Quoted 4 lines
As I *explicitly* stated earlier, there isn't anything inherently wrong with putting a small onus on x86 in order to make the general Linux code better -- but please, be honest about it *so we know what the actual tradeoffs are*.
It is not just about the general Linux code. The x86 fgkaslr implementation was never merged because the toolchain side needs changes. And convincing the toolchain maintainers to take our changes is difficult if we keep using relocation tables that are not fit for purpose to perform runtime fixups on code that was built using the 'kernel' code model, which is explicitly position dependent.
Quoted 3 lines
For x86, we really do want to maintain the kernel memory model, which allows us to directly reference symbols in complex address expressions and to directly jump across modules.
AIUI, those complex address expressions are mostly indexed loads from global arrays, which do get slightly less efficient, but not in a way that was noticeable in any benchmarking I did (or LKP for that matter, which generally sniffs out any performance regressions). This includes jump tables, but as I already explained, RIP-relative jump tables have an upside too, given that the table itself is only half the size.
The ability to directly jump across modules is not affected at all by these changes.
Quoted 5 lines
This means the "PIE" will need to be different
from the way PIE works in user space, which is in part designed to avoid needing to dirty readonly pages, which would inhibit sharing -- which is explicitly NOT a concern for the kernel.
This is already implemented in this series: no GOT entries are permitted, and text relocations are allowed.
Quoted 2 lines
So that is one thing that the toolchain needs to be able to do.
It already can, and this series makes use of it.
Note that the size of the relocation table taken from an allmodconfig bzImage drops from 7.3 M to 2.4 M (defconfig goes from 800k to 45k), so there is a minor intrinsic benefit to these changes as well. But it is mostly about moving away from bespoke tooling and formats that are becoming more of a maintenance burden as the number of supported toolchains and languages increases.
Quoted 7 lines
I fully expect that we will continue to need to have some kinds of overrides for specific symbols, too, because there aren't any really sane ways to express them to the toolchain; this especially applies to linker-script and some assembly symbols. For example, the real-mode code (which uses the reloc tool as well) has to support segment and segbase-relative relocations, which are something that ELF simply has no concept of.
The real mode trampoline is not affected at all by these changes, given that it is built as a separate executable. Using a bespoke relocation format there is fine, because it is internal ABI.
Quoted 10 lines
I have a lot more of an issue with trying to change the x86 boot protocol, simply because the way booting works in x86 has been incredibly successful; yes, the bzImage file format is ugly as ****, but that is a direct result of 34 years of continuous backwards compatibility. One of the reasons we have been able to do that is that we have *explicitly* rejected other boot models, such as Grub's self-declared Multiboot "standard" (which they have had to revise multiple times by now) and the early Xen boot model of booting vmlinux directly. We have added *many* capabilities to bzImage as needed, and it has turned out to be quite flexible in the end.
Quoted 4 lines
That, in turn, has been possible exactly *because* the Linux kernel provides a "prekernel". I don't even really like calling it the "decompressor" anymore; it really has developed far beyond that.
The decompressor is needed when booting the 64-bit kernel from a boot loader that calls it in 32-bit mode.
When entering in long mode, with all memory mapped 1:1 (or at least, the kernel image itself, and all assets in memory that the bootloader exposes to the kernel), the decompressor does nothing useful, and all the problems it solves (by doing demand paging etc) only exist because it created them in the first place.
SEV-SNP confidential compute made an even bigger mess of this, because it can trigger #VC exceptions too, which also need to be handled.
Note that the EFI stub does not bother with the decompressor anymore, and unpacks and boots vmlinux directly. This was needed because the decompressor fundamentally relies on memory that is both writable and executable (as it moves its own executable image around in memory), which is difficult to reconcile with recent PC firmware implementations that are pedantic about mapping memory RWX.
But actually, I am not proposing to get rid of bzImage. I am proposing to make it more transparent so generic bootloader components can be constructed that consume the ELF directly.
Borislav Petkov
maintainerRe: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section
In reply to Ard BiesheuvelQuoted 32 lines · Ard Biesheuvel
Currently,
idt_tableis allocated as page-aligned .bss, and remapped read-only after init. This breaks a 2 MiB large page into 4k page mappings, which defeats some of the effort done at boot to map the kernel image using large pages, for improved TLB efficiency.Mark this allocation as
__ro_after_initinstead, so it will be made read-only automatically after boot, without breaking up large page mappings.This also fixes a latent bug on i386, where the size of
idt_tableis less than a page, and so remapping it read-only could potentially affect other read-write variables too, if those are not page-aligned as well.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/idt.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)arch/x86/kernel/idt.c +1
-1@@ -170,7 +170,7 @@ static const __initconst struct idt_data apic_idts[] = { };/* Must be page-aligned because the real IDT is used in the cpu entry area */ -static gate_desc idt_table[IDT_ENTRIES] __page_aligned_bss; +static gate_desc idt_table[IDT_ENTRIES ]__aligned(PAGE_SIZE) __ro_after_init ;static struct desc_ptr idt_descr__ro_after_init = { .size =IDT_TABLE_SIZE -1 ,
arch/x86/kernel/idt.c void __init idt_setup_apic_and_irq_gates(void) · line 312
Quoted 8 lines
arch/x86/kernel/idt.c +0
-3@@ -308,9 +308,6 @@ void __init idt_setup_apic_and_irq_gates(void)idt_map_in_cea ();load_idt (&idt_descr); -/* Make the IDT table read only */ -set_memory_ro ((unsigned long )&idt_table,1 ); - idt_setup_done = true; }
Good idea, except my guest shows me something else:
before:
[ 0.186281] IDT table: 0xffffffff89c7f000
0xffffffff89c00000-0xffffffff89c7f000 508K RW GLB NX pte 0xffffffff89c7f000-0xffffffff89c80000 4K ro GLB NX pte 0xffffffff89c80000-0xffffffff89e00000 1536K RW GLB NX pte 0xffffffff89e00000-0xffffffff8be00000 32M RW PSE GLB NX pmd
This is clearly a single, 4K RO pageframe right in the middle of a splintered 2M page.
after:
[ 0.180635] IDT table: 0xffffffff822cf000
0xffffffff81e00000-0xffffffff82200000 4M ro PSE GLB NX pmd 0xffffffff82200000-0xffffffff8236f000 1468K ro GLB NX pte 0xffffffff8236f000-0xffffffff82400000 580K RW GLB NX pte 0xffffffff82400000-0xffffffff89800000 116M RW PSE GLB NX pmd
but after applying your patch it looks like it still broke the 2M mapping as the remaining piece is RW.
If I do this:
static gate_desc idt_table[IDT_ENTRIES] __aligned(PMD_SIZE) __ro_after_init;
it still doesn't help:
[ 0.197808] IDT table: 0xffffffff82800000
0xffffffff81e00000-0xffffffff82800000 10M ro PSE GLB NX pmd 0xffffffff82800000-0xffffffff828a0000 640K ro GLB NX pte 0xffffffff828a0000-0xffffffff82a00000 1408K RW GLB NX pte 0xffffffff82a00000-0xffffffff89e00000 116M RW PSE GLB NX pmd
because that trailing piece of the 2M page is still RW.
And who knows what else am I breaking when doing this:
[ 2.368601] ------------[ cut here ]------------ [ 2.389816] [CRTC:35:crtc-0] vblank wait timed out [ 2.396676] WARNING: drivers/gpu/drm/drm_atomic_helper.c:1920 at drm_atomic_helper_wait_for_vblanks.part.0+0x1ba/0x1e0, CPU#1: kworker/1:0/57 [ 2.406715] Modules linked in: [ 2.408462] CPU: 1 UID: 0 PID: 57 Comm: kworker/1:0 Not tainted 6.19.0-rc6+ #4 PREEMPT(full)
...
I don't know, sacrificing a 2M page just for the idt_table and so that it doesn't get splintered, not sure it is worth it.
Hmmm.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section
Quoted 61 lines · Borislav Petkov
Quoted 32 lines · Ard Biesheuvel
Currently,
idt_tableis allocated as page-aligned .bss, and remapped read-only after init. This breaks a 2 MiB large page into 4k page mappings, which defeats some of the effort done at boot to map the kernel image using large pages, for improved TLB efficiency.Mark this allocation as
__ro_after_initinstead, so it will be made read-only automatically after boot, without breaking up large page mappings.This also fixes a latent bug on i386, where the size of
idt_tableis less than a page, and so remapping it read-only could potentially affect other read-write variables too, if those are not page-aligned as well.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/idt.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)arch/x86/kernel/idt.c +1
-1@@ -170,7 +170,7 @@ static const __initconst struct idt_data apic_idts[] = { };/* Must be page-aligned because the real IDT is used in the cpu entry area */ -static gate_desc idt_table[IDT_ENTRIES] __page_aligned_bss; +static gate_desc idt_table[IDT_ENTRIES ]__aligned(PAGE_SIZE) __ro_after_init ;static struct desc_ptr idt_descr__ro_after_init = { .size =IDT_TABLE_SIZE -1 ,arch/x86/kernel/idt.c void __init idt_setup_apic_and_irq_gates(void) · line 312
Quoted 8 lines
arch/x86/kernel/idt.c +0
-3@@ -308,9 +308,6 @@ void __init idt_setup_apic_and_irq_gates(void)idt_map_in_cea ();load_idt (&idt_descr); -/* Make the IDT table read only */ -set_memory_ro ((unsigned long )&idt_table,1 ); - idt_setup_done = true; }Good idea, except my guest shows me something else:
before:
[ 0.186281] IDT table: 0xffffffff89c7f0000xffffffff89c00000-0xffffffff89c7f000 508K RW GLB NX pte 0xffffffff89c7f000-0xffffffff89c80000 4K ro GLB NX pte 0xffffffff89c80000-0xffffffff89e00000 1536K RW GLB NX pte 0xffffffff89e00000-0xffffffff8be00000 32M RW PSE GLB NX pmdThis is clearly a single, 4K RO pageframe right in the middle of a splintered 2M page.
after:
[ 0.180635] IDT table: 0xffffffff822cf000
Quoted 8 lines
0xffffffff81e00000-0xffffffff82200000 4M ro PSE GLB NX pmd 0xffffffff82200000-0xffffffff8236f000 1468K ro GLB NX pte 0xffffffff8236f000-0xffffffff82400000 580K RW GLB NX pte 0xffffffff82400000-0xffffffff89800000 116M RW PSE GLB NX pmdbut after applying your patch it looks like it still broke the 2M mapping as the remaining piece is RW.
That is because the init region is between .data and .bss, which are not 2M aligned, and so freeing/remapping the init region breaks the hugepage mapping of the region between them. This is addressed in patch #3.
Borislav Petkov
maintainerRe: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section
Quoted 4 lines · Ard Biesheuvel
That is because the init region is between .data and .bss, which are not 2M aligned, and so freeing/remapping the init region breaks the hugepage mapping of the region between them. This is addressed in patch #3.
Ok, I'll continue looking but this commit message should not mislead in stating that it prevents the "breaking up large page mappings." I'll fix it up if I end up picking up this or you fix it up in your next revision please.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section
Quoted 9 lines · Borislav Petkov
Quoted 4 lines · Ard Biesheuvel
That is because the init region is between .data and .bss, which are not 2M aligned, and so freeing/remapping the init region breaks the hugepage mapping of the region between them. This is addressed in patch #3.
Ok, I'll continue looking but this commit message should not mislead in stating that it prevents the "breaking up large page mappings."
It does. It just doesn't prevent if from happening for other reasons.
Quoted 3 lines
I'll fix it up
if I end up picking up this or you fix it up in your next revision please.
Ack.
Borislav Petkov
maintainerRe: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section
Quoted 1 line · Ard Biesheuvel
It does. It just doesn't prevent if from happening for other reasons.
So this reads to me like "it does but it doesn't". Huh?!?
C'mon Ard, let's be more precise pls.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section
Quoted 8 lines · Borislav Petkov
Quoted 1 line · Ard Biesheuvel
It does. It just doesn't prevent if from happening for other reasons.
So this reads to me like "it does but it doesn't". Huh?!?
C'mon Ard, let's be more precise pls.
Fair enough. What I meant to say is that if your BSS placed on a 2M boundary or is sufficiently larger than that, some of it will be mapped using PMDs, and this change will prevent those from being broken up into page mappings. If your BSS is too small to be covered by huge page mappings in the first place, this change obviously does nothing (except fixing a bug on i386)
Borislav Petkov
maintainerRe: [RFC/RFT PATCH 01/19] x86/idt: Move idt_table to __ro_after_init section
Quoted 6 lines · Ard Biesheuvel
Fair enough. What I meant to say is that if your BSS placed on a 2M boundary or is sufficiently larger than that, some of it will be mapped using PMDs, and this change will prevent those from being broken up into page mappings. If your BSS is too small to be covered by huge page mappings in the first place, this change obviously does nothing (except fixing a bug on i386)
That makes more sense. :)
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette[tip: x86/sev] x86/sev: Don't emit BSS_DECRYPTED section unless it is in use
In reply to Ard BiesheuvelThe following commit has been merged into the x86/sev branch of tip:
The following commit has been merged into the x86/sev branch of tip:
Commit-ID: 8c89d3ad3095808ac130c535ad7ed3d1344d5986 Gitweb: https://git.kernel.org/tip/8c89d3ad3095808ac130c535ad7ed3d1344d5986 Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 08 Jan 2026 09:25:29 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Sat, 31 Jan 2026 14:42:53 +01:00
x86/sev: Don't emit BSS_DECRYPTED section unless it is in use
The BSS_DECRYPTED section that gets emitted into .bss will be empty if CONFIG_AMD_MEM_ENCRYPT is not defined. However, due to the fact that it is injected into .bss rather than emitted as a separate section, the 2 MiB alignment that it specifies is still taken into account unconditionally, pushing .bss out to the next 2 MiB boundary, leaving a gap that is never freed.
So only emit a non-empty BSS_DECRYPTED section if it is going to be used. In that case, it would still be nice to free the padding, but that is left for later.
--- arch/x86/kernel/vmlinux.lds.S | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
arch/x86/kernel/vmlinux.lds.S +12 -9
@@ -67,7 +67,18 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define ALIGN_ENTRY_TEXT_BEGIN . = ALIGN(PMD_SIZE); #define ALIGN_ENTRY_TEXT_END . = ALIGN(PMD_SIZE); +#else + +#define X86_ALIGN_RODATA_BEGIN +#define X86_ALIGN_RODATA_END \ + . = ALIGN(PAGE_SIZE); \ + __end_rodata_aligned = .; +#define ALIGN_ENTRY_TEXT_BEGIN +#define ALIGN_ENTRY_TEXT_END +#endif + +#ifdef CONFIG_AMD_MEM_ENCRYPT /* * This section contains data which will be mapped as decrypted. Memory * encryption operates on a page basis. Make this section PMD-aligned
@@ -88,17 +99,9 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; __pi___end_bss_decrypted = .; \ #else - -#define X86_ALIGN_RODATA_BEGIN -#define X86_ALIGN_RODATA_END \ - . = ALIGN(PAGE_SIZE); \ - __end_rodata_aligned = .; - -#define ALIGN_ENTRY_TEXT_BEGIN -#define ALIGN_ENTRY_TEXT_END #define BSS_DECRYPTED - #endif + #if defined(CONFIG_X86_64) && defined(CONFIG_KEXEC_CORE) #define KEXEC_RELOCATE_KERNEL \ . = ALIGN(0x100); \
Borislav Petkov
maintainerRe: [RFC/RFT PATCH 03/19] x86: Combine .data with .bss in kernel mapping
In reply to Ard BiesheuvelQuoted 76 lines · Ard Biesheuvel
The primary mapping of the kernel image is made using huge pages where possible, mostly to minimize TLB pressure (Only the entry text section requires alignment to 2 MiB). This involves some rounding and padding of the .text and .rodata sections, resulting in gaps. These gaps are smaller than a huge page, and are remapped using different permissions, resulting in fragmentation of the huge page mappings at the edges of those regions.
Similarly, there is a gap between .data and .bss, where the init text and data regions reside. This means that the end of the .data region and the start of the .bss region are not covered by huge page mappings either, even though both regions use the same permissions (RW+NX).
Improve the situation, by placing .data and .bss adjacently in the linker map, and putting the init text and data regions after .rodata, taking the place of the rodata/data gap. This results in one fewer gap, and a more efficient mapping of the .data and .bss regions.
To preserve the
x86_64ELF layout withPT_LOADregions aligned to 2 MiB, start the second ELF segment at .init.data and align it to 2 MiB. The resulting padding will be covered by the init region and will be freed along with it after boot.defconfig + Clang 19:
Before:
0xffffffff81000000-0xffffffff82200000 18M ro PSE GLB x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro GLB x pte 0xffffffff8231c000-0xffffffff82400000 912K RW GLB NX pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE GLB NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro GLB NX pte 0xffffffff82b40000-0xffffffff82c00000 768K RW GLB NX pte 0xffffffff82c00000-0xffffffff83400000 8M RW PSE GLB NX pmd 0xffffffff83400000-0xffffffff83800000 4M RW GLB NX pteAfter:
0xffffffff81000000-0xffffffff82200000 18M ro PSE GLB x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro GLB x pte 0xffffffff8231c000-0xffffffff82400000 912K RW GLB NX pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE GLB NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro GLB NX pte 0xffffffff82b40000-0xffffffff82c00000 768K RW GLB NX pte 0xffffffff82c00000-0xffffffff82e00000 2M RW PSE GLB NX pmd 0xffffffff82e00000-0xffffffff83000000 2M RW GLB NX pte 0xffffffff83000000-0xffffffff83800000 8M RW PSE GLB NX pmdWith the gaps removed/unmapped (pti=on)
Before:
0xffffffff81000000-0xffffffff81200000 2M ro PSE GLB x pmd 0xffffffff81200000-0xffffffff82200000 16M ro PSE x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro x pte 0xffffffff8231c000-0xffffffff82400000 912K pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro NX pte 0xffffffff82b40000-0xffffffff82c00000 768K pte 0xffffffff82c00000-0xffffffff83400000 8M RW PSE NX pmd 0xffffffff83400000-0xffffffff8342a000 168K RW NX pte0xffffffff8342a000-0xffffffff836f3000 2852K pte
0xffffffff836f3000-0xffffffff83800000 1076K RW NX pteAfter:
0xffffffff81000000-0xffffffff81200000 2M ro PSE GLB x pmd 0xffffffff81200000-0xffffffff82200000 16M ro PSE x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro x pte 0xffffffff8231c000-0xffffffff82400000 912K pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro NX pte 0xffffffff82b40000-0xffffffff82e3d000 3060K pte 0xffffffff82e3d000-0xffffffff83000000 1804K RW NX pte 0xffffffff83000000-0xffffffff83800000 8M RW PSE NX pmd
Quoted 6 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/vmlinux.lds.S | 91 +++++++++++--------- arch/x86/mm/init_64.c | 5 +- arch/x86/mm/pat/set_memory.c | 2 +- 3 files changed, 52 insertions(+), 46 deletions(-)
I guess we could do this - I don't see why not... we'll have to take it for a longer spin tho.
arch/x86/kernel/vmlinux.lds.S const_cpu_current_top_of_stack = cpu_current_top_of_stack; · line 65
Quoted 10 lines
arch/x86/kernel/vmlinux.lds.S +0
-2@@ -61,12 +61,15 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define X86_ALIGN_RODATA_BEGIN . = ALIGN(HPAGE_SIZE); #define X86_ALIGN_RODATA_END \ - . = ALIGN(HPAGE_SIZE); \ - __end_rodata_hpage_align = .; \
$ git grep __end_rodata_hpage_align arch/x86/include/asm/sections.h:13:extern char __end_rodata_hpage_align[]; arch/x86/tools/relocs.c:93: "__end_rodata_hpage_align|"
I guess you wanna remove those too and say that that marker is unused. Better yet do that in a pre-patch.
Quoted 13 lines
arch/x86/kernel/vmlinux.lds.S +6
-1- __end_rodata_aligned = .; + . = ALIGN(PAGE_SIZE); \ + __end_rodata_aligned = ALIGN(HPAGE_SIZE); #define ALIGN_ENTRY_TEXT_BEGIN . = ALIGN(PMD_SIZE); #define ALIGN_ENTRY_TEXT_END . = ALIGN(PMD_SIZE); + +#define DATA_SEGMENT_START \ + . = ALIGN(HPAGE_SIZE); \ + __data_segment_start = .; #else #define X86_ALIGN_RODATA_BEGIN
...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [RFC/RFT PATCH 03/19] x86: Combine .data with .bss in kernel mapping
Quoted 102 lines · Borislav Petkov
Quoted 76 lines · Ard Biesheuvel
The primary mapping of the kernel image is made using huge pages where possible, mostly to minimize TLB pressure (Only the entry text section requires alignment to 2 MiB). This involves some rounding and padding of the .text and .rodata sections, resulting in gaps. These gaps are smaller than a huge page, and are remapped using different permissions, resulting in fragmentation of the huge page mappings at the edges of those regions.
Similarly, there is a gap between .data and .bss, where the init text and data regions reside. This means that the end of the .data region and the start of the .bss region are not covered by huge page mappings either, even though both regions use the same permissions (RW+NX).
Improve the situation, by placing .data and .bss adjacently in the linker map, and putting the init text and data regions after .rodata, taking the place of the rodata/data gap. This results in one fewer gap, and a more efficient mapping of the .data and .bss regions.
To preserve the
x86_64ELF layout withPT_LOADregions aligned to 2 MiB, start the second ELF segment at .init.data and align it to 2 MiB. The resulting padding will be covered by the init region and will be freed along with it after boot.defconfig + Clang 19:
Before:
0xffffffff81000000-0xffffffff82200000 18M ro PSE GLB x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro GLB x pte 0xffffffff8231c000-0xffffffff82400000 912K RW GLB NX pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE GLB NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro GLB NX pte 0xffffffff82b40000-0xffffffff82c00000 768K RW GLB NX pte 0xffffffff82c00000-0xffffffff83400000 8M RW PSE GLB NX pmd 0xffffffff83400000-0xffffffff83800000 4M RW GLB NX pteAfter:
0xffffffff81000000-0xffffffff82200000 18M ro PSE GLB x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro GLB x pte 0xffffffff8231c000-0xffffffff82400000 912K RW GLB NX pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE GLB NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro GLB NX pte 0xffffffff82b40000-0xffffffff82c00000 768K RW GLB NX pte 0xffffffff82c00000-0xffffffff82e00000 2M RW PSE GLB NX pmd 0xffffffff82e00000-0xffffffff83000000 2M RW GLB NX pte 0xffffffff83000000-0xffffffff83800000 8M RW PSE GLB NX pmdWith the gaps removed/unmapped (pti=on)
Before:
0xffffffff81000000-0xffffffff81200000 2M ro PSE GLB x pmd 0xffffffff81200000-0xffffffff82200000 16M ro PSE x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro x pte 0xffffffff8231c000-0xffffffff82400000 912K pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro NX pte 0xffffffff82b40000-0xffffffff82c00000 768K pte 0xffffffff82c00000-0xffffffff83400000 8M RW PSE NX pmd 0xffffffff83400000-0xffffffff8342a000 168K RW NX pte0xffffffff8342a000-0xffffffff836f3000 2852K pte
0xffffffff836f3000-0xffffffff83800000 1076K RW NX pteAfter:
0xffffffff81000000-0xffffffff81200000 2M ro PSE GLB x pmd 0xffffffff81200000-0xffffffff82200000 16M ro PSE x pmd 0xffffffff82200000-0xffffffff8231c000 1136K ro x pte 0xffffffff8231c000-0xffffffff82400000 912K pte 0xffffffff82400000-0xffffffff82a00000 6M ro PSE NX pmd 0xffffffff82a00000-0xffffffff82b40000 1280K ro NX pte 0xffffffff82b40000-0xffffffff82e3d000 3060K pte 0xffffffff82e3d000-0xffffffff83000000 1804K RW NX pte 0xffffffff83000000-0xffffffff83800000 8M RW PSE NX pmdQuoted 6 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/vmlinux.lds.S | 91 +++++++++++--------- arch/x86/mm/init_64.c | 5 +- arch/x86/mm/pat/set_memory.c | 2 +- 3 files changed, 52 insertions(+), 46 deletions(-)I guess we could do this - I don't see why not... we'll have to take it for a longer spin tho.
arch/x86/kernel/vmlinux.lds.S const_cpu_current_top_of_stack = cpu_current_top_of_stack; · line 65
Quoted 10 lines
arch/x86/kernel/vmlinux.lds.S +0
-2@@ -61,12 +61,15 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define X86_ALIGN_RODATA_BEGIN . = ALIGN(HPAGE_SIZE); #define X86_ALIGN_RODATA_END \ - . = ALIGN(HPAGE_SIZE); \ - __end_rodata_hpage_align = .; \$ git grep __end_rodata_hpage_align arch/x86/include/asm/sections.h:13:extern char __end_rodata_hpage_align[]; arch/x86/tools/relocs.c:93: "__end_rodata_hpage_align|"
Quoted 3 lines
I guess you wanna remove those too and say that that marker is unused. Better yet do that in a pre-patch.
Indeed. When __end_rodata_hpage_align exists, it is always equal to __end_rodata_aligned, so it can just be dropped entirely.