x86/pie: Make kernel image's virtual address flexible
Subsystems (32)
cover x86/pie: Make kernel image's virtual address flexible
Purpose:
These patches make the changes necessary to build the kernel as Position Independent Executable (PIE) on x86_64. A PIE kernel can be relocated below the top 2G of the virtual address space. And this patchset provides an example to allow kernel image to be relocated in top 512G of the address space.
The ultimate purpose for PIE kernel is to increase the security of the the kernel and also the fleixbility of the kernel image's virtual address, which can be even in the low half of the address space. More locations the kernel can fit in, this means an attacker could guess harder.
The patchset is based on Thomas Garnier's X86 PIE patchset v6[1] and v11[2]. However, some design changes are made and some bugs are fixed by testing with different configurations and compilers.
Important changes:
- For fixmap area, move vsyscall page out of fixmap area and unify __FIXADDR_TOP for x86. Then fixmap area could be relocated together with kernel image.
- For compile-time base address of kernel image, keep it in top 2G of address space. Introduce a new variable to store the run-time base address and adapt for VA/PA transition during runtime.
- For percpu section, keep it as zero mapping for SMP. Because compile-time base address of kernel image still resides in top 2G of address space, then RIP-relative reference can still be used when percpu section is zero mapping. However, when do relocation for percpu variable references, percpu variable should be treated as normal variable and absolute references should be relocated accordingly. In addition, the relocation offset should be subtracted from the GS base in order to ensure correct operation.
- For x86/boot/head64.c, don't build it as mcmodel=large. Instead, use data relocation to acqiure global symbol's value and make fixup_pointer() as a nop when running in identity mapping. This is because not all global symbol references in the code use fixup_pointer(), e.g. variables in macro related to 5-level paging, which can be optimized by GCC as relative referencs. If build it as mcmodel=large, there will be more fixup_pointer() calls, resulting in uglier code. Actually, if build it as PIE even when CONFIG_X86_PIE is disabled, then all fixup_pointer() could be dropped. However stack protector would be broken if per-cpu stack protector is not supported.
Limitations:
- Since I am not familiar with XEN, it has been disabled for now as it is not adapted for PIE. This is due to the assignment of wrong pointers (with low address values) to x86_ini_ops when running in identity mapping. This issue can be resolved by building pagetable eraly and jumping to high kernel address as soon as possible.
- It is not allowed to reference global variables in an alternative section since RIP-relative addressing is not fixed in apply_alternatives(). Fortunately, all disallowed relocations in the alternative section can be captured by objtool. I believe that this issue can also be fixed by using objtool.
- For module loading, only allow to load module without GOT for simplicity. Only weak global variable referencs are using GOT.
Tests:
I only have tested booting with GCC 5.1.0 (min version), GCC 12.2.0 and CLANG 15.0.7. And I have also run the following tests for both default configuration and Ubuntu configuration.
Performance/Size impact (GCC 12.2.0):
Size of vmlinux (Default configuration):
File size:
- PIE disabled: +0.012%
- PIE enabled: -2.219%
instructions:
- PIE disabled: same
- PIE enabled: +1.383%
.text section:
- PIE disabled: same
- PIE enabled: +0.589%
Size of vmlinux (Ubuntu configuration):
File size:
- PIE disabled: same
- PIE enabled: +2.391%
instructions:
- PIE disabled: +0.013%
- PIE enabled: +1.566%
.text section:
- PIE disabled: same
- PIE enabled: +0.055%
The .text section size increase is due to more instructions required for PIE code. There are two reasons that have been mentioned in previous mailist. Firstly, switch folding is disabled under PIE [3]. Secondly, two instructions are needed for PIE to represent a single instruction with sign extension, such as when accessing an array element. While only one instruction is required when using mcmode=kernel, for PIE, it needs to use lea to get the base of the array first.
Hackbench (50% and 1600% on thread/process for pipe/sockets):
- PIE disabled: no significant change (avg -/+ 0.5% on default config).
- PIE enabled: -2% to +2% in average (default config).
Kernbench (average of 10 Half and Optimal runs):
Elapsed Time:
- PIE disabled: no significant change (avg -0.2% on ubuntu config)
- PIE enabled: average -0.2% to +0.2%
System Time:
- PIE disabled: no significant change (avg -0.5% on ubuntu config)
- PIE enabled: average -0.5% to +0.5%
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82303
Brian Gerst (1):
x86-64: Use per-cpu stack canary if supported by compiler
Hou Wenlong (29):
x86/irq: Adapt assembly for PIE support
x86,rethook: Adapt assembly for PIE support
x86/paravirt: Use relative reference for original instruction x86/Kconfig: Introduce new Kconfig for PIE kernel building x86/PVH: Use fixed_percpu_data to set up GS base
x86/pie: Enable stack protector only if per-cpu stack canary is
supported
x86/percpu: Use PC-relative addressing for percpu variable references x86/tools: Explicitly include autoconf.h for hostprogs x86/percpu: Adapt percpu references relocation for PIE support x86/ftrace: Adapt assembly for PIE support
x86/pie: Force hidden visibility for all symbol references x86/boot/compressed: Adapt sed command to generate voffset.h when PIE
is enabled
x86/pie: Add .data.rel.* sections into link script
KVM: x86: Adapt assembly for PIE support
x86/PVH: Adapt PVH booting for PIE support
x86/bpf: Adapt BPF_CALL JIT codegen for PIE support x86/modules: Adapt module loading for PIE support x86/boot/64: Use data relocation to get absloute address when PIE is
enabled
objtool: Add validation for x86 PIE support
objtool: Adapt indirect call of __fentry__() for PIE support x86/pie: Build the kernel as PIE
x86/vsyscall: Don't use set_fixmap() to map vsyscall page x86/xen: Pin up to VSYSCALL_ADDR when vsyscall page is out of fixmap
area
x86/fixmap: Move vsyscall page out of fixmap area x86/fixmap: Unify FIXADDR_TOP
x86/boot: Fill kernel image puds dynamically
x86/mm: Sort address_markers array when X86 PIE is enabled x86/pie: Allow kernel image to be relocated in top 512G x86/boot: Extend relocate range for PIE kernel image
Thomas Garnier (13):
x86/crypto: Adapt assembly for PIE support
x86: Add macro to get symbol address for PIE support
x86: relocate_kernel - Adapt assembly for PIE support x86/entry/64: Adapt assembly for PIE support
x86: pm-trace: Adapt assembly for PIE support x86/CPU: Adapt assembly for PIE support x86/acpi: Adapt assembly for PIE support x86/boot/64: Adapt assembly for PIE support x86/power/64: Adapt assembly for PIE support x86/alternatives: Adapt assembly for PIE support x86/ftrace: Adapt ftrace nop patching for PIE support x86/mm: Make the x86 GOT read-only x86/relocs: Handle PIE relocations
Documentation/x86/x86_64/mm.rst | 4 + arch/x86/Kconfig | 36 +++++- arch/x86/Makefile | 33 +++-- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/kaslr.c | 55 +++++++++ arch/x86/boot/compressed/misc.c | 4 +- arch/x86/boot/compressed/misc.h | 9 ++ arch/x86/crypto/aegis128-aesni-asm.S | 6 +- arch/x86/crypto/aesni-intel_asm.S | 2 +- arch/x86/crypto/aesni-intel_avx-x86_64.S | 3 +- arch/x86/crypto/aria-aesni-avx-asm_64.S | 30 ++--- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 30 ++--- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 30 ++--- arch/x86/crypto/camellia-x86_64-asm_64.S | 8 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 50 ++++---- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 44 ++++--- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 3 +- arch/x86/crypto/des3_ede-asm_64.S | 96 ++++++++++----- arch/x86/crypto/ghash-clmulni-intel_asm.S | 4 +- arch/x86/crypto/sha256-avx2-asm.S | 18 ++- arch/x86/entry/calling.h | 17 ++- arch/x86/entry/entry_64.S | 22 +++- arch/x86/entry/vdso/Makefile | 2 +- arch/x86/entry/vsyscall/vsyscall_64.c | 7 +- arch/x86/include/asm/alternative.h | 6 +- arch/x86/include/asm/asm.h | 1 + arch/x86/include/asm/fixmap.h | 28 +---- arch/x86/include/asm/irq_stack.h | 2 +- arch/x86/include/asm/kmsan.h | 6 +- arch/x86/include/asm/nospec-branch.h | 10 +- arch/x86/include/asm/page_64.h | 8 +- arch/x86/include/asm/page_64_types.h | 8 ++ arch/x86/include/asm/paravirt.h | 17 ++- arch/x86/include/asm/paravirt_types.h | 12 +- arch/x86/include/asm/percpu.h | 29 ++++- arch/x86/include/asm/pgtable_64_types.h | 10 +- arch/x86/include/asm/pm-trace.h | 2 +- arch/x86/include/asm/processor.h | 17 ++- arch/x86/include/asm/sections.h | 5 + arch/x86/include/asm/stackprotector.h | 16 ++- arch/x86/include/asm/sync_core.h | 6 +- arch/x86/include/asm/vsyscall.h | 13 ++ arch/x86/kernel/acpi/wakeup_64.S | 31 ++--- arch/x86/kernel/alternative.c | 8 +- arch/x86/kernel/asm-offsets_64.c | 2 +- arch/x86/kernel/callthunks.c | 2 +- arch/x86/kernel/cpu/common.c | 15 ++- arch/x86/kernel/ftrace.c | 46 ++++++- arch/x86/kernel/ftrace_64.S | 9 +- arch/x86/kernel/head64.c | 77 +++++++++--- arch/x86/kernel/head_64.S | 68 ++++++++--- arch/x86/kernel/kvm.c | 21 +++- arch/x86/kernel/module.c | 27 +++++ arch/x86/kernel/paravirt.c | 4 + arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kernel/rethook.c | 8 ++ arch/x86/kernel/setup.c | 6 + arch/x86/kernel/vmlinux.lds.S | 10 +- arch/x86/kvm/svm/vmenter.S | 10 +- arch/x86/kvm/vmx/vmenter.S | 2 +- arch/x86/lib/cmpxchg16b_emu.S | 8 +- arch/x86/mm/dump_pagetables.c | 36 +++++- arch/x86/mm/fault.c | 1 - arch/x86/mm/init_64.c | 10 +- arch/x86/mm/ioremap.c | 5 +- arch/x86/mm/kasan_init_64.c | 4 +- arch/x86/mm/pat/set_memory.c | 2 +- arch/x86/mm/pgtable.c | 13 ++ arch/x86/mm/pgtable_32.c | 3 - arch/x86/mm/physaddr.c | 14 +-- arch/x86/net/bpf_jit_comp.c | 17 ++- arch/x86/platform/efi/efi_thunk_64.S | 4 + arch/x86/platform/pvh/head.S | 29 ++++- arch/x86/power/hibernate_asm_64.S | 4 +- arch/x86/tools/Makefile | 4 +- arch/x86/tools/relocs.c | 113 ++++++++++++++++- arch/x86/xen/mmu_pv.c | 32 +++-- arch/x86/xen/xen-asm.S | 10 +- arch/x86/xen/xen-head.S | 14 ++- include/asm-generic/vmlinux.lds.h | 12 ++ scripts/Makefile.lib | 1 + scripts/recordmcount.c | 81 ++++++++----- tools/objtool/arch/x86/decode.c | 10 +- tools/objtool/builtin-check.c | 4 +- tools/objtool/check.c | 121 +++++++++++++++++++ tools/objtool/include/objtool/builtin.h | 1 + 86 files changed, 1202 insertions(+), 410 deletions(-)
Patchset is based on tip/master.
base-commit: 01cbd032298654fe4c85e153dd9a224e5bc10194 --
2.31.1
RFC PATCH 1/43 x86/crypto: Adapt assembly for PIE support
From: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
[Hou Wenlong: Adapt new assembly code in x86/crypto]
--- arch/x86/crypto/aegis128-aesni-asm.S | 6 +- arch/x86/crypto/aesni-intel_asm.S | 2 +- arch/x86/crypto/aesni-intel_avx-x86_64.S | 3 +- arch/x86/crypto/aria-aesni-avx-asm_64.S | 30 +++--- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 30 +++--- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 30 +++--- arch/x86/crypto/camellia-x86_64-asm_64.S | 8 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 50 +++++----- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 44 +++++---- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 3 +- arch/x86/crypto/des3_ede-asm_64.S | 96 +++++++++++++------- arch/x86/crypto/ghash-clmulni-intel_asm.S | 4 +- arch/x86/crypto/sha256-avx2-asm.S | 18 ++-- 13 files changed, 187 insertions(+), 137 deletions(-)
arch/x86/crypto/aegis128-aesni-asm.S +3 -3
@@ -201,8 +201,8 @@ SYM_FUNC_START(crypto_aegis128_aesni_init) movdqa KEY, STATE4 /* load the constants: */ - movdqa .Laegis128_const_0, STATE2 - movdqa .Laegis128_const_1, STATE1 + movdqa .Laegis128_const_0(%rip), STATE2 + movdqa .Laegis128_const_1(%rip), STATE1 pxor STATE2, STATE3 pxor STATE1, STATE4
@@ -682,7 +682,7 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec_tail) punpcklbw T0, T0 punpcklbw T0, T0 punpcklbw T0, T0 - movdqa .Laegis128_counter, T1 + movdqa .Laegis128_counter(%rip), T1 pcmpgtb T1, T0 pand T0, MSG
arch/x86/crypto/aesni-intel_asm.S +1 -1
@@ -2717,7 +2717,7 @@ SYM_FUNC_END(aesni_cts_cbc_dec) * BSWAP_MASK == endian swapping mask */ SYM_FUNC_START_LOCAL(_aesni_inc_init) - movaps .Lbswap_mask, BSWAP_MASK + movaps .Lbswap_mask(%rip), BSWAP_MASK movaps IV, CTR pshufb BSWAP_MASK, CTR mov $1, TCTR_LOW
arch/x86/crypto/aesni-intel_avx-x86_64.S +2 -1
@@ -649,7 +649,8 @@ _get_AAD_rest0\@: vpshufb and an array of shuffle masks */ movq %r12, %r11 salq $4, %r11 - vmovdqu aad_shift_arr(%r11), \T1 + leaq aad_shift_arr(%rip), %rax + vmovdqu (%rax,%r11,), \T1 vpshufb \T1, \T7, \T7 _get_AAD_rest_final\@: vpshufb SHUF_MASK(%rip), \T7, \T7
arch/x86/crypto/aria-aesni-avx-asm_64.S +15 -15
@@ -80,7 +80,7 @@ transpose_4x4(c0, c1, c2, c3, a0, a1); \ transpose_4x4(d0, d1, d2, d3, a0, a1); \ \ - vmovdqu .Lshufb_16x16b, a0; \ + vmovdqu .Lshufb_16x16b(%rip), a0; \ vmovdqu st1, a1; \ vpshufb a0, a2, a2; \ vpshufb a0, a3, a3; \
@@ -132,7 +132,7 @@ transpose_4x4(c0, c1, c2, c3, a0, a1); \ transpose_4x4(d0, d1, d2, d3, a0, a1); \ \ - vmovdqu .Lshufb_16x16b, a0; \ + vmovdqu .Lshufb_16x16b(%rip), a0; \ vmovdqu st1, a1; \ vpshufb a0, a2, a2; \ vpshufb a0, a3, a3; \
@@ -300,11 +300,11 @@ x4, x5, x6, x7, \ t0, t1, t2, t3, \ t4, t5, t6, t7) \ - vmovdqa .Ltf_s2_bitmatrix, t0; \ - vmovdqa .Ltf_inv_bitmatrix, t1; \ - vmovdqa .Ltf_id_bitmatrix, t2; \ - vmovdqa .Ltf_aff_bitmatrix, t3; \ - vmovdqa .Ltf_x2_bitmatrix, t4; \ + vmovdqa .Ltf_s2_bitmatrix(%rip), t0; \ + vmovdqa .Ltf_inv_bitmatrix(%rip), t1; \ + vmovdqa .Ltf_id_bitmatrix(%rip), t2; \ + vmovdqa .Ltf_aff_bitmatrix(%rip), t3; \ + vmovdqa .Ltf_x2_bitmatrix(%rip), t4; \ vgf2p8affineinvqb $(tf_s2_const), t0, x1, x1; \ vgf2p8affineinvqb $(tf_s2_const), t0, x5, x5; \ vgf2p8affineqb $(tf_inv_const), t1, x2, x2; \
@@ -324,13 +324,13 @@ x4, x5, x6, x7, \ t0, t1, t2, t3, \ t4, t5, t6, t7) \ - vmovdqa .Linv_shift_row, t0; \ - vmovdqa .Lshift_row, t1; \ - vbroadcastss .L0f0f0f0f, t6; \ - vmovdqa .Ltf_lo__inv_aff__and__s2, t2; \ - vmovdqa .Ltf_hi__inv_aff__and__s2, t3; \ - vmovdqa .Ltf_lo__x2__and__fwd_aff, t4; \ - vmovdqa .Ltf_hi__x2__and__fwd_aff, t5; \ + vmovdqa .Linv_shift_row(%rip), t0; \ + vmovdqa .Lshift_row(%rip), t1; \ + vpbroadcastd .L0f0f0f0f(%rip), t6; \ + vmovdqa .Ltf_lo__inv_aff__and__s2(%rip), t2; \ + vmovdqa .Ltf_hi__inv_aff__and__s2(%rip), t3; \ + vmovdqa .Ltf_lo__x2__and__fwd_aff(%rip), t4; \ + vmovdqa .Ltf_hi__x2__and__fwd_aff(%rip), t5; \ \ vaesenclast t7, x0, x0; \ vaesenclast t7, x4, x4; \
@@ -1035,7 +1035,7 @@ SYM_FUNC_START_LOCAL(__aria_aesni_avx_ctr_gen_keystream_16way) /* load IV and byteswap */ vmovdqu (%r8), %xmm8; - vmovdqa .Lbswap128_mask (%rip), %xmm1; + vmovdqa .Lbswap128_mask(%rip), %xmm1; vpshufb %xmm1, %xmm8, %xmm3; /* be => le */ vpcmpeqd %xmm0, %xmm0, %xmm0;
arch/x86/crypto/camellia-aesni-avx-asm_64.S +15 -15
@@ -52,10 +52,10 @@ /* \ * S-function with AES subbytes \ */ \ - vmovdqa .Linv_shift_row, t4; \ - vbroadcastss .L0f0f0f0f, t7; \ - vmovdqa .Lpre_tf_lo_s1, t0; \ - vmovdqa .Lpre_tf_hi_s1, t1; \ + vmovdqa .Linv_shift_row(%rip), t4; \ + vbroadcastss .L0f0f0f0f(%rip), t7; \ + vmovdqa .Lpre_tf_lo_s1(%rip), t0; \ + vmovdqa .Lpre_tf_hi_s1(%rip), t1; \ \ /* AES inverse shift rows */ \ vpshufb t4, x0, x0; \
@@ -68,8 +68,8 @@ vpshufb t4, x6, x6; \ \ /* prefilter sboxes 1, 2 and 3 */ \ - vmovdqa .Lpre_tf_lo_s4, t2; \ - vmovdqa .Lpre_tf_hi_s4, t3; \ + vmovdqa .Lpre_tf_lo_s4(%rip), t2; \ + vmovdqa .Lpre_tf_hi_s4(%rip), t3; \ filter_8bit(x0, t0, t1, t7, t6); \ filter_8bit(x7, t0, t1, t7, t6); \ filter_8bit(x1, t0, t1, t7, t6); \
@@ -83,8 +83,8 @@ filter_8bit(x6, t2, t3, t7, t6); \ \ /* AES subbytes + AES shift rows */ \ - vmovdqa .Lpost_tf_lo_s1, t0; \ - vmovdqa .Lpost_tf_hi_s1, t1; \ + vmovdqa .Lpost_tf_lo_s1(%rip), t0; \ + vmovdqa .Lpost_tf_hi_s1(%rip), t1; \ vaesenclast t4, x0, x0; \ vaesenclast t4, x7, x7; \ vaesenclast t4, x1, x1; \
@@ -95,16 +95,16 @@ vaesenclast t4, x6, x6; \ \ /* postfilter sboxes 1 and 4 */ \ - vmovdqa .Lpost_tf_lo_s3, t2; \ - vmovdqa .Lpost_tf_hi_s3, t3; \ + vmovdqa .Lpost_tf_lo_s3(%rip), t2; \ + vmovdqa .Lpost_tf_hi_s3(%rip), t3; \ filter_8bit(x0, t0, t1, t7, t6); \ filter_8bit(x7, t0, t1, t7, t6); \ filter_8bit(x3, t0, t1, t7, t6); \ filter_8bit(x6, t0, t1, t7, t6); \ \ /* postfilter sbox 3 */ \ - vmovdqa .Lpost_tf_lo_s2, t4; \ - vmovdqa .Lpost_tf_hi_s2, t5; \ + vmovdqa .Lpost_tf_lo_s2(%rip), t4; \ + vmovdqa .Lpost_tf_hi_s2(%rip), t5; \ filter_8bit(x2, t2, t3, t7, t6); \ filter_8bit(x5, t2, t3, t7, t6); \ \
@@ -443,7 +443,7 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) transpose_4x4(c0, c1, c2, c3, a0, a1); \ transpose_4x4(d0, d1, d2, d3, a0, a1); \ \ - vmovdqu .Lshufb_16x16b, a0; \ + vmovdqu .Lshufb_16x16b(%rip), a0; \ vmovdqu st1, a1; \ vpshufb a0, a2, a2; \ vpshufb a0, a3, a3; \
@@ -482,7 +482,7 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) #define inpack16_pre(x0, x1, x2, x3, x4, x5, x6, x7, y0, y1, y2, y3, y4, y5, \ y6, y7, rio, key) \ vmovq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor 0 * 16(rio), x0, y7; \ vpxor 1 * 16(rio), x0, y6; \
@@ -533,7 +533,7 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) vmovdqu x0, stack_tmp0; \ \ vmovq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor x0, y7, y7; \ vpxor x0, y6, y6; \
arch/x86/crypto/camellia-aesni-avx2-asm_64.S +15 -15
@@ -64,12 +64,12 @@ /* \ * S-function with AES subbytes \ */ \ - vbroadcasti128 .Linv_shift_row, t4; \ - vpbroadcastd .L0f0f0f0f, t7; \ - vbroadcasti128 .Lpre_tf_lo_s1, t5; \ - vbroadcasti128 .Lpre_tf_hi_s1, t6; \ - vbroadcasti128 .Lpre_tf_lo_s4, t2; \ - vbroadcasti128 .Lpre_tf_hi_s4, t3; \ + vbroadcasti128 .Linv_shift_row(%rip), t4; \ + vpbroadcastd .L0f0f0f0f(%rip), t7; \ + vbroadcasti128 .Lpre_tf_lo_s1(%rip), t5; \ + vbroadcasti128 .Lpre_tf_hi_s1(%rip), t6; \ + vbroadcasti128 .Lpre_tf_lo_s4(%rip), t2; \ + vbroadcasti128 .Lpre_tf_hi_s4(%rip), t3; \ \ /* AES inverse shift rows */ \ vpshufb t4, x0, x0; \
@@ -115,8 +115,8 @@ vinserti128 $1, t2##_x, x6, x6; \ vextracti128 $1, x1, t3##_x; \ vextracti128 $1, x4, t2##_x; \ - vbroadcasti128 .Lpost_tf_lo_s1, t0; \ - vbroadcasti128 .Lpost_tf_hi_s1, t1; \ + vbroadcasti128 .Lpost_tf_lo_s1(%rip), t0; \ + vbroadcasti128 .Lpost_tf_hi_s1(%rip), t1; \ vaesenclast t4##_x, x2##_x, x2##_x; \ vaesenclast t4##_x, t6##_x, t6##_x; \ vinserti128 $1, t6##_x, x2, x2; \
@@ -131,16 +131,16 @@ vinserti128 $1, t2##_x, x4, x4; \ \ /* postfilter sboxes 1 and 4 */ \ - vbroadcasti128 .Lpost_tf_lo_s3, t2; \ - vbroadcasti128 .Lpost_tf_hi_s3, t3; \ + vbroadcasti128 .Lpost_tf_lo_s3(%rip), t2; \ + vbroadcasti128 .Lpost_tf_hi_s3(%rip), t3; \ filter_8bit(x0, t0, t1, t7, t6); \ filter_8bit(x7, t0, t1, t7, t6); \ filter_8bit(x3, t0, t1, t7, t6); \ filter_8bit(x6, t0, t1, t7, t6); \ \ /* postfilter sbox 3 */ \ - vbroadcasti128 .Lpost_tf_lo_s2, t4; \ - vbroadcasti128 .Lpost_tf_hi_s2, t5; \ + vbroadcasti128 .Lpost_tf_lo_s2(%rip), t4; \ + vbroadcasti128 .Lpost_tf_hi_s2(%rip), t5; \ filter_8bit(x2, t2, t3, t7, t6); \ filter_8bit(x5, t2, t3, t7, t6); \ \
@@ -475,7 +475,7 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) transpose_4x4(c0, c1, c2, c3, a0, a1); \ transpose_4x4(d0, d1, d2, d3, a0, a1); \ \ - vbroadcasti128 .Lshufb_16x16b, a0; \ + vbroadcasti128 .Lshufb_16x16b(%rip), a0; \ vmovdqu st1, a1; \ vpshufb a0, a2, a2; \ vpshufb a0, a3, a3; \
@@ -514,7 +514,7 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) #define inpack32_pre(x0, x1, x2, x3, x4, x5, x6, x7, y0, y1, y2, y3, y4, y5, \ y6, y7, rio, key) \ vpbroadcastq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor 0 * 32(rio), x0, y7; \ vpxor 1 * 32(rio), x0, y6; \
@@ -565,7 +565,7 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) vmovdqu x0, stack_tmp0; \ \ vpbroadcastq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor x0, y7, y7; \ vpxor x0, y6, y6; \
arch/x86/crypto/camellia-x86_64-asm_64.S +5 -3
@@ -77,11 +77,13 @@ #define RXORbl %r9b #define xor2ror16(T0, T1, tmp1, tmp2, ab, dst) \ + leaq T0(%rip), tmp1; \ movzbl ab ## bl, tmp2 ## d; \ + xorq (tmp1, tmp2, 8), dst; \ + leaq T1(%rip), tmp2; \ movzbl ab ## bh, tmp1 ## d; \ - rorq $16, ab; \ - xorq T0(, tmp2, 8), dst; \ - xorq T1(, tmp1, 8), dst; + xorq (tmp2, tmp1, 8), dst; \ + rorq $16, ab; /********************************************************************** 1-way camellia
arch/x86/crypto/cast5-avx-x86_64-asm_64.S +27 -23
98 lines not on this page · raw message
arch/x86/crypto/cast6-avx-x86_64-asm_64.S +24 -20
82 lines not on this page · raw message
arch/x86/crypto/crc32c-pcl-intel-asm_64.S +2 -1
@@ -168,7 +168,8 @@ continue_block: xor crc2, crc2 ## branch into array - mov jump_table(,%rax,8), %bufp + leaq jump_table(%rip), %bufp + mov (%bufp,%rax,8), %bufp JMP_NOSPEC bufp ################################################################
arch/x86/crypto/des3_ede-asm_64.S +64 -32
150 lines not on this page · raw message
arch/x86/crypto/ghash-clmulni-intel_asm.S +2 -2
@@ -93,7 +93,7 @@ SYM_FUNC_START(clmul_ghash_mul) FRAME_BEGIN movups (%rdi), DATA movups (%rsi), SHASH - movaps .Lbswap_mask, BSWAP + movaps .Lbswap_mask(%rip), BSWAP pshufb BSWAP, DATA call __clmul_gf128mul_ble pshufb BSWAP, DATA
@@ -110,7 +110,7 @@ SYM_FUNC_START(clmul_ghash_update) FRAME_BEGIN cmp $16, %rdx jb .Lupdate_just_ret # check length - movaps .Lbswap_mask, BSWAP + movaps .Lbswap_mask(%rip), BSWAP movups (%rdi), DATA movups (%rcx), SHASH pshufb BSWAP, DATA
arch/x86/crypto/sha256-avx2-asm.S +12 -6
@@ -589,19 +589,23 @@ last_block_enter: .align 16 loop1: - vpaddd K256+0*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 0*32(INP, SRND), X0, XFER vmovdqa XFER, 0*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 0*32 - vpaddd K256+1*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 1*32(INP, SRND), X0, XFER vmovdqa XFER, 1*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 1*32 - vpaddd K256+2*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 2*32(INP, SRND), X0, XFER vmovdqa XFER, 2*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 2*32 - vpaddd K256+3*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 3*32(INP, SRND), X0, XFER vmovdqa XFER, 3*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 3*32
@@ -611,11 +615,13 @@ loop1: loop2: ## Do last 16 rounds with no scheduling - vpaddd K256+0*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 0*32(INP, SRND), X0, XFER vmovdqa XFER, 0*32+_XFER(%rsp, SRND) DO_4ROUNDS _XFER + 0*32 - vpaddd K256+1*32(SRND), X1, XFER + leaq K256(%rip), INP + vpaddd 1*32(INP, SRND), X1, XFER vmovdqa XFER, 1*32+_XFER(%rsp, SRND) DO_4ROUNDS _XFER + 1*32 add $2*32, SRND
Signature · 1 line
2.31.1
RFC PATCH 2/43 x86: Add macro to get symbol address for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Add a new _ASM_MOVABS macro to fetch a symbol address. Replace "_ASM_MOV $<symbol>, %dst" code construct that are not compatible with PIE.
--- arch/x86/include/asm/asm.h | 1 + 1 file changed, 1 insertion(+)
arch/x86/include/asm/asm.h +1 -0
@@ -35,6 +35,7 @@#define _ASM_ALIGN __ASM_SEL(.balign 4, .balign 8)#define _ASM_MOV __ASM_SIZE(mov) +#define _ASM_MOVABS __ASM_SEL(movl, movabsq)#define _ASM_INC __ASM_SIZE(inc)#define _ASM_DEC __ASM_SIZE(dec)#define _ASM_ADD __ASM_SIZE(add)
Signature · 1 line
2.31.1
RFC PATCH 3/43 x86: relocate_kernel - Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly code to use only absolute references of symbols for the kernel to be PIE compatible.
--- arch/x86/kernel/relocate_kernel_64.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/relocate_kernel_64.S +1 -1
@@ -223,7 +223,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) movq %rax, %cr3 lea PAGE_SIZE(%r8), %rsp call swap_pages - movq $virtual_mapped, %rax + movabsq $virtual_mapped, %rax pushq %rax ANNOTATE_UNRET_SAFE ret
Signature · 1 line
2.31.1
RFC PATCH 4/43 x86/entry/64: Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
--- arch/x86/entry/entry_64.S | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
arch/x86/entry/entry_64.S +11 -5
@@ -1089,7 +1089,8 @@ SYM_CODE_START(error_entry) movl %ecx, %eax /* zero extend */ cmpq %rax, RIP+8(%rsp) je .Lbstep_iret - cmpq $.Lgs_change, RIP+8(%rsp) + leaq .Lgs_change(%rip), %rcx + cmpq %rcx, RIP+8(%rsp) jne .Lerror_entry_done_lfence /*
@@ -1302,10 +1303,10 @@ SYM_CODE_START(asm_exc_nmi) * resume the outer NMI. */ - movq $repeat_nmi, %rdx + leaq repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja 1f - movq $end_repeat_nmi, %rdx + leaq end_repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja nested_nmi_out 1:
@@ -1359,7 +1360,8 @@ nested_nmi: pushq %rdx pushfq pushq $__KERNEL_CS - pushq $repeat_nmi + leaq repeat_nmi(%rip), %rdx + pushq %rdx /* Put stack back */ addq $(6*8), %rsp
@@ -1398,7 +1400,11 @@ first_nmi: addq $8, (%rsp) /* Fix up RSP */ pushfq /* RFLAGS */ pushq $__KERNEL_CS /* CS */ - pushq $1f /* RIP */ + pushq $0 /* Space for RIP */ + pushq %rdx /* Save RDX */ + leaq 1f(%rip), %rdx /* Put the address of 1f label into RDX */ + movq %rdx, 8(%rsp) /* Store it in RIP field */ + popq %rdx /* Restore RDX */ iretq /* continues at repeat_nmi below */ UNWIND_HINT_IRET_REGS 1:
Signature · 1 line
2.31.1
RFC PATCH 5/43 x86: pm-trace: Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change assembly to use the new _ASM_MOVABS macro instead of _ASM_MOV for the assembly to be PIE compatible.
--- arch/x86/include/asm/pm-trace.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/pm-trace.h +1 -1
@@ -8,7 +8,7 @@ do { \ if (pm_trace_enabled) { \ const void *tracedata; \ - asm volatile(_ASM_MOV " $1f,%0\n" \ + asm volatile(_ASM_MOVABS " $1f,%0\n" \ ".section .tracedata,\"a\"\n" \ "1:\t.word %c1\n\t" \ _ASM_PTR " %c2\n" \
Signature · 1 line
2.31.1
RFC PATCH 6/43 x86/CPU: Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
--- arch/x86/include/asm/sync_core.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/include/asm/sync_core.h +4 -2
@@ -31,10 +31,12 @@ static inline void iret_to_self(void)"pushfq\n\t" "mov %%cs, %0\n\t" "pushq %q0\n\t" -"pushq $1f\n\t" +"leaq 1f(%%rip), %q0\n\t" +"pushq %q0\n\t" "iretq\n\t" "1:" - : "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory"); + : "=&r" (tmp), ASM_CALL_CONSTRAINT + : : "cc", "memory"); } #endif /* CONFIG_X86_32 */
Signature · 1 line
2.31.1
RFC PATCH 7/43 x86/acpi: Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
--- arch/x86/kernel/acpi/wakeup_64.S | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)
arch/x86/kernel/acpi/wakeup_64.S +16 -15
@@ -17,7 +17,7 @@ * Hooray, we are in Long 64-bit mode (but still running in low memory) */ SYM_FUNC_START(wakeup_long64) - movq saved_magic, %rax + movq saved_magic(%rip), %rax movq $0x123456789abcdef0, %rdx cmpq %rdx, %rax je 2f
@@ -33,14 +33,14 @@ SYM_FUNC_START(wakeup_long64) movw %ax, %es movw %ax, %fs movw %ax, %gs - movq saved_rsp, %rsp + movq saved_rsp(%rip), %rsp - movq saved_rbx, %rbx - movq saved_rdi, %rdi - movq saved_rsi, %rsi - movq saved_rbp, %rbp + movq saved_rbx(%rip), %rbx + movq saved_rdi(%rip), %rdi + movq saved_rsi(%rip), %rsi + movq saved_rbp(%rip), %rbp - movq saved_rip, %rax + movq saved_rip(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%rax SYM_FUNC_END(wakeup_long64)
@@ -51,7 +51,7 @@ SYM_FUNC_START(do_suspend_lowlevel) xorl %eax, %eax call save_processor_state - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax) movq %rsi, pt_regs_si(%rax)
@@ -70,13 +70,14 @@ SYM_FUNC_START(do_suspend_lowlevel) pushfq popq pt_regs_flags(%rax) - movq $.Lresume_point, saved_rip(%rip) + leaq .Lresume_point(%rip), %rax + movq %rax, saved_rip(%rip) - movq %rsp, saved_rsp - movq %rbp, saved_rbp - movq %rbx, saved_rbx - movq %rdi, saved_rdi - movq %rsi, saved_rsi + movq %rsp, saved_rsp(%rip) + movq %rbp, saved_rbp(%rip) + movq %rbx, saved_rbx(%rip) + movq %rdi, saved_rdi(%rip) + movq %rsi, saved_rsi(%rip) addq $8, %rsp movl $3, %edi
@@ -88,7 +89,7 @@ SYM_FUNC_START(do_suspend_lowlevel) .align 4 .Lresume_point: /* We don't restore %rax, it must be 0 anyway */ - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq saved_context_cr4(%rax), %rbx movq %rbx, %cr4 movq saved_context_cr3(%rax), %rbx
Signature · 1 line
2.31.1
RFC PATCH 9/43 x86/power/64: Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
--- arch/x86/power/hibernate_asm_64.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/power/hibernate_asm_64.S +2 -2
@@ -39,7 +39,7 @@ SYM_FUNC_START(restore_registers) movq %rax, %cr4; # turn PGE back on /* We don't restore %rax, it must be 0 anyway */ - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq pt_regs_sp(%rax), %rsp movq pt_regs_bp(%rax), %rbp movq pt_regs_si(%rax), %rsi
@@ -70,7 +70,7 @@ SYM_FUNC_START(restore_registers) SYM_FUNC_END(restore_registers) SYM_FUNC_START(swsusp_arch_suspend) - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax) movq %rsi, pt_regs_si(%rax)
Signature · 1 line
2.31.1
RFC PATCH 8/43 x86/boot/64: Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly code to use absolute reference for transition between address spaces and relative references when referencing global variables in the same address space. Ensure the kernel built with PIE references the correct addresses based on context.
[Hou Wenlong: Adapt new assembly code and remove change for initial_code(%rip)]
--- arch/x86/kernel/head_64.S | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
arch/x86/kernel/head_64.S +14 -8
@@ -114,7 +114,8 @@ SYM_CODE_START_NOALIGN(startup_64) popq %rsi /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(early_top_pgt - __START_KERNEL_map), %rax + movabs $(early_top_pgt - __START_KERNEL_map), %rcx + addq %rcx, %rax jmp 1f SYM_CODE_END(startup_64)
@@ -156,13 +157,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * added to the initial pgdir entry that will be programmed into CR3. */ #ifdef CONFIG_AMD_MEM_ENCRYPT - movq sme_me_mask, %rax + movq sme_me_mask(%rip), %rax #else xorq %rax, %rax #endif /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(init_top_pgt - __START_KERNEL_map), %rax + movabs $(init_top_pgt - __START_KERNEL_map), %rcx + addq %rcx, %rax 1: #ifdef CONFIG_X86_MCE
@@ -226,7 +228,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movq %rax, %cr4 /* Ensure I am executing from virtual addresses */ - movq $1f, %rax + movabs $1f, %rax ANNOTATE_RETPOLINE_SAFE jmp *%rax 1:
@@ -237,7 +239,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movl smpboot_control(%rip), %ecx /* Get the per cpu offset for the given CPU# which is in ECX */ - movq __per_cpu_offset(,%rcx,8), %rdx + leaq __per_cpu_offset(%rip), %rdx + movq (%rdx,%rcx,8), %rdx #else xorl %edx, %edx /* zero-extended to clear all of RDX */ #endif /* CONFIG_SMP */
@@ -248,7 +251,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * * RDX contains the per-cpu offset */ - movq pcpu_hot + X86_current_task(%rdx), %rax + leaq (pcpu_hot + X86_current_task)(%rip), %rax + movq (%rdx,%rax,1), %rax movq TASK_threadsp(%rax), %rsp /*
@@ -259,7 +263,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) */ subq $16, %rsp movw $(GDT_SIZE-1), (%rsp) - leaq gdt_page(%rdx), %rax + leaq gdt_page(%rip), %rax + addq %rdx, %rax movq %rax, 2(%rsp) lgdt (%rsp) addq $16, %rsp
@@ -362,7 +367,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * REX.W + FF /5 JMP m16:64 Jump far, absolute indirect, * address given in m16:64. */ - pushq $.Lafter_lret # put return address on stack for unwinder + movabs $.Lafter_lret, %rax + pushq %rax # put return address on stack for unwinder xorl %ebp, %ebp # clear frame pointer movq initial_code(%rip), %rax pushq $__KERNEL_CS # set correct cs
Signature · 1 line
2.31.1
RFC PATCH 16/43 x86-64: Use per-cpu stack canary if supported by compiler
In reply to Hou WenlongFrom: Brian Gerst <[email protected]>
From: Brian Gerst <[email protected]>
If the compiler supports it, use a standard per-cpu variable for the stack protector instead of the old fixed location. Keep the fixed location code for compatibility with older compilers.
[Hou Wenlong: Disable it on Clang, adapt new code change and adapt missing GS set up path in pvh_start_xen()]
--- arch/x86/Kconfig | 12 ++++++++++++ arch/x86/Makefile | 21 ++++++++++++++------- arch/x86/entry/entry_64.S | 6 +++++- arch/x86/include/asm/processor.h | 17 ++++++++++++----- arch/x86/include/asm/stackprotector.h | 16 +++++++--------- arch/x86/kernel/asm-offsets_64.c | 2 +- arch/x86/kernel/cpu/common.c | 15 +++++++-------- arch/x86/kernel/head_64.S | 16 ++++++++++------ arch/x86/kernel/vmlinux.lds.S | 4 +++- arch/x86/platform/pvh/head.S | 8 ++++++++ arch/x86/xen/xen-head.S | 14 +++++++++----- 11 files changed, 88 insertions(+), 43 deletions(-)
arch/x86/Kconfig +12 -0
@@ -410,6 +410,18 @@ config CC_HAS_SANE_STACKPROTECTOR the compiler produces broken code or if it does not let us control the segment on 32-bit kernels. +config CC_HAS_CUSTOMIZED_STACKPROTECTOR + bool + # Although clang supports -mstack-protector-guard-reg option, it + # would generate GOT reference for __stack_chk_guard even with + # -fno-PIE flag. + default y if (!CC_IS_CLANG && $(cc-option,-mstack-protector-guard-reg=gs)) + +config STACKPROTECTOR_FIXED + bool + depends on X86_64 && STACKPROTECTOR + default !CC_HAS_CUSTOMIZED_STACKPROTECTOR + menu "Processor type and features" config SMP
arch/x86/Makefile +14 -7
@@ -111,13 +111,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64
@@ -167,6 +161,19 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs +endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifneq ($(CONFIG_STACKPROTECTOR_FIXED),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) \ + -mstack-protector-guard-symbol=__stack_chk_guard + else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif + endif endif #
arch/x86/entry/entry_64.S +5 -1
@@ -229,6 +229,10 @@ SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) int3 SYM_CODE_END(entry_SYSCALL_64) +#ifdef CONFIG_STACKPROTECTOR_FIXED +#define __stack_chk_guard fixed_percpu_data + FIXED_stack_canary +#endif + /* * %rdi: prev task * %rsi: next task
@@ -252,7 +256,7 @@ SYM_FUNC_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(fixed_percpu_data) + FIXED_stack_canary + movq %rbx, PER_CPU_VAR(__stack_chk_guard) #endif /*
arch/x86/include/asm/processor.h +12 -5
@@ -379,6 +379,8 @@ struct irq_stack { }__aligned(IRQ_STACK_SIZE) ;#ifdef CONFIG_X86_64 + +#ifdef CONFIG_STACKPROTECTOR_FIXED struct fixed_percpu_data {/* * GCC hardcodes the stack canary as %gs:40. Since the
@@ -394,21 +396,26 @@ struct fixed_percpu_data { DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;DECLARE_INIT_PER_CPU (fixed_percpu_data); +#endif /* CONFIG_STACKPROTECTOR_FIXED */ static inline unsigned long cpu_kernelmode_gs_base (int cpu) { +#ifdef CONFIG_STACKPROTECTOR_FIXED return (unsigned long )per_cpu (fixed_percpu_data.gs_base, cpu); +#else +#ifdef CONFIG_SMP +return per_cpu_offset (cpu); +#else +return 0 ; +#endif +#endif } extern asmlinkage void ignore_sysret(void);/* Save actual FS/GS selectors and bases to current->thread */ void current_save_fsgs (void ); -#else /* X86_64 */ -#ifdef CONFIG_STACKPROTECTOR -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); -#endif -#endif /* !X86_64 */ +#endif /* X86_64 */ struct perf_event ;
arch/x86/include/asm/stackprotector.h +7 -9
@@ -36,6 +36,12 @@#include <linux/sched.h> +#ifdef CONFIG_STACKPROTECTOR_FIXED +#define __stack_chk_guard fixed_percpu_data.stack_canary +#else +DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +#endif +/* * Initialize the stackprotector canary value. *
@@ -51,25 +57,17 @@ static __always_inline void boot_init_stack_canary(void) {unsigned long canary =get_random_canary (); -#ifdef CONFIG_X86_64 +#ifdef CONFIG_STACKPROTECTOR_FIXED BUILD_BUG_ON (offsetof(struct fixed_percpu_data , stack_canary) !=40 );#endif current->stack_canary = canary; -#ifdef CONFIG_X86_64 -this_cpu_write (fixed_percpu_data.stack_canary, canary); -#else this_cpu_write (__stack_chk_guard, canary); -#endif }static inline void cpu_init_stack_canary (int cpu,struct task_struct *idle) { -#ifdef CONFIG_X86_64 -per_cpu (fixed_percpu_data.stack_canary, cpu) = idle->stack_canary; -#else per_cpu (__stack_chk_guard, cpu) = idle->stack_canary; -#endif }#else /* STACKPROTECTOR */
arch/x86/kernel/asm-offsets_64.c +1 -1
@@ -56,7 +56,7 @@ int main(void)BLANK (); -#ifdef CONFIG_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR_FIXED OFFSET (FIXED_stack_canary, fixed_percpu_data, stack_canary);BLANK ();#endif
arch/x86/kernel/cpu/common.c +7 -8
@@ -2051,10 +2051,6 @@ DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) = {EXPORT_PER_CPU_SYMBOL (pcpu_hot);#ifdef CONFIG_X86_64 -DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, - fixed_percpu_data)__aligned(PAGE_SIZE) __visible; -EXPORT_PER_CPU_SYMBOL_GPL (fixed_percpu_data); -static void wrmsrl_cstar (unsigned long val) {/*
@@ -2102,15 +2098,18 @@ void syscall_init(void)X86_EFLAGS_IOPL |X86_EFLAGS_NT |X86_EFLAGS_RF | X86_EFLAGS_AC|X86_EFLAGS_ID); } - -#else /* CONFIG_X86_64 */ +#endif /* CONFIG_X86_64 */ #ifdef CONFIG_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR_FIXED +DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, + fixed_percpu_data)__aligned(PAGE_SIZE) __visible; +EXPORT_PER_CPU_SYMBOL_GPL (fixed_percpu_data); +#else DEFINE_PER_CPU(unsigned long, __stack_chk_guard);EXPORT_PER_CPU_SYMBOL (__stack_chk_guard);#endif - -#endif /* CONFIG_X86_64 */ +#endif /* * Clear all 6 debug registers:
arch/x86/kernel/head_64.S +10 -6
@@ -68,7 +68,13 @@ SYM_CODE_START_NOALIGN(startup_64) /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx +#if defined(CONFIG_STACKPROTECTOR_FIXED) leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#elif defined(CONFIG_SMP) + movabs $__per_cpu_load, %rdx +#else + xorl %edx, %edx +#endif movl %edx, %eax shrq $32, %rdx wrmsr
@@ -283,16 +289,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movl %eax,%fs movl %eax,%gs - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + /* + * Set up GS base. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifndef CONFIG_SMP - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#if !defined(CONFIG_SMP) && defined(CONFIG_STACKPROTECTOR_FIXED) + leaq __per_cpu_load(%rip), %rdx #endif movl %edx, %eax shrq $32, %rdx
arch/x86/kernel/vmlinux.lds.S +3 -1
@@ -500,12 +500,14 @@ SECTIONS */ #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load INIT_PER_CPU(gdt_page); -INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); +#ifdef CONFIG_STACKPROTECTOR_FIXED +INIT_PER_CPU(fixed_percpu_data); #ifdef CONFIG_SMP . = ASSERT((fixed_percpu_data == 0), "fixed_percpu_data is not at start of per-cpu area"); #endif +#endif #endif /* CONFIG_X86_64 */
arch/x86/platform/pvh/head.S +8 -0
@@ -96,8 +96,16 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx +#if defined(CONFIG_STACKPROTECTOR_FIXED) mov $_pa(INIT_PER_CPU_VAR(fixed_percpu_data)), %eax xor %edx, %edx +#elif defined(CONFIG_SMP) + mov $__per_cpu_load, %rax + cdq +#else + xor %eax, %eax + xor %edx, %edx +#endif wrmsr call xen_prepare_pvh
arch/x86/xen/xen-head.S +9 -5
@@ -51,15 +51,19 @@ SYM_CODE_START(startup_xen) leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + /* + * Set up GS base. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax +#if defined(CONFIG_STACKPROTECTOR_FIXED) + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#elif defined(CONFIG_SMP) + movabs $__per_cpu_load, %rdx +#else + xorl %eax, %eax +#endif cdq wrmsr
Signature · 1 line
2.31.1
RFC PATCH 18/43 x86/percpu: Use PC-relative addressing for percpu variable references
In reply to Hou WenlongFor PIE binary, all symbol references are PC-relative addressing, even for percpu variable. So to keep compatible with PIE, add %rip suffix in percpu assembly macros if PIE is enabled. However, relocation of percpu variable references is broken now for PIE. It would be fixed later.
--- arch/x86/entry/calling.h | 17 ++++++++++++---- arch/x86/include/asm/nospec-branch.h | 10 +++++----- arch/x86/include/asm/percpu.h | 29 +++++++++++++++++++++++++--- arch/x86/kernel/head_64.S | 2 +- arch/x86/kernel/kvm.c | 21 ++++++++++++++++---- arch/x86/lib/cmpxchg16b_emu.S | 8 ++++---- arch/x86/xen/xen-asm.S | 10 +++++----- 7 files changed, 71 insertions(+), 26 deletions(-)
arch/x86/entry/calling.h +13 -4
@@ -173,7 +173,7 @@ For 32-bit we have the following conventions - kernel is built with .endm#define THIS_CPU_user_pcid_flush_mask \ - PER_CPU_VAR(cpu_tlbstate) + TLB_STATE_user_pcid_flush_mask + PER_CPU_VAR(cpu_tlbstate + TLB_STATE_user_pcid_flush_mask) .macro SWITCH_TO_USER_CR3_NOSTACK scratch_reg:req scratch_reg2:req ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI
@@ -370,8 +370,8 @@ For 32-bit we have the following conventions - kernel is built with .endm .macro SAVE_AND_SET_GSBASE scratch_reg:req save_reg:req + GET_PERCPU_BASE \scratch_reg \save_reg rdgsbase \save_reg - GET_PERCPU_BASE \scratch_reg wrgsbase \scratch_reg .endm
@@ -407,15 +407,24 @@ For 32-bit we have the following conventions - kernel is built with* Thus the kernel would consume a guest's TSC_AUX if an NMI arrives * while running KVM's run loop. */ -.macro GET_PERCPU_BASE reg:req +#ifdef CONFIG_X86_PIE +.macro GET_PERCPU_BASE reg:req scratch_reg:req + LOAD_CPU_AND_NODE_SEG_LIMIT \reg + andq $VDSO_CPUNODE_MASK, \reg + leaq __per_cpu_offset(%rip), \scratch_reg + movq (\scratch_reg, \reg, 8), \reg +.endm +#else +.macro GET_PERCPU_BASE reg:req scratch_reg:req LOAD_CPU_AND_NODE_SEG_LIMIT \reg andq $VDSO_CPUNODE_MASK, \reg movq __per_cpu_offset(, \reg, 8), \reg .endm +#endif /* CONFIG_X86_PIE */ #else -.macro GET_PERCPU_BASE reg:req +.macro GET_PERCPU_BASE reg:req scratch_reg:req movq pcpu_unit_offsets(%rip), \reg .endm
arch/x86/include/asm/nospec-branch.h +5 -5
@@ -59,13 +59,13 @@#ifdef CONFIG_CALL_THUNKS_DEBUG # define CALL_THUNKS_DEBUG_INC_CALLS \ - incq %gs:__x86_call_count; + incq %gs:(__x86_call_count)__percpu_rel;# define CALL_THUNKS_DEBUG_INC_RETS \ - incq %gs:__x86_ret_count; + incq %gs:(__x86_ret_count)__percpu_rel;# define CALL_THUNKS_DEBUG_INC_STUFFS \ - incq %gs:__x86_stuffs_count; + incq %gs:(__x86_stuffs_count)__percpu_rel;# define CALL_THUNKS_DEBUG_INC_CTXSW \ - incq %gs:__x86_ctxsw_count; + incq %gs:(__x86_ctxsw_count)__percpu_rel;#else # define CALL_THUNKS_DEBUG_INC_CALLS # define CALL_THUNKS_DEBUG_INC_RETS
@@ -95,7 +95,7 @@ CALL_THUNKS_DEBUG_INC_CALLS#define INCREMENT_CALL_DEPTH \ - sarq $5, %gs:pcpu_hot + X86_call_depth; \ + sarq $5, %gs:(pcpu_hot + X86_call_depth)__percpu_rel;\ CALL_THUNKS_DEBUG_INC_CALLS#define ASM_INCREMENT_CALL_DEPTH \
arch/x86/include/asm/percpu.h +26 -3
@@ -4,16 +4,26 @@#ifdef CONFIG_X86_64 #define __percpu_seg gs +#ifdef CONFIG_X86_PIE +#define __percpu_rel (%rip) +#else +#define __percpu_rel +#endif /* CONFIG_X86_PIE */ #else #define __percpu_seg fs +#define __percpu_rel#endif #ifdef __ASSEMBLY__#ifdef CONFIG_SMP -#define PER_CPU_VAR (var) %__percpu_seg:var +/* Compatible with Position Independent Code */ +#define PER_CPU_VAR (var) %__percpu_seg:(var)##__percpu_rel +/* Rare absolute reference */ +#define PER_CPU_VAR_ABS (var) %__percpu_seg:var#else /* ! SMP */ -#define PER_CPU_VAR (var) var +#define PER_CPU_VAR (var) (var)##__percpu_rel +#define PER_CPU_VAR_ABS (var) var#endif /* SMP */ #ifdef CONFIG_X86_64_SMP
@@ -148,10 +158,23 @@ do { \ (typeof (_var ))(unsigned long ) pfo_val__; \ }) +/* +* Position Independent code uses relative addresses only. +* The 'P' modifier prevents RIP-relative addressing in GCC, +* so use 'a' modifier instead. Howerver, 'P' modifier allows +* RIP-relative addressing in Clang but Clang doesn't support +* 'a' modifier. +*/ +#if defined(CONFIG_X86_PIE ) && defined(CONFIG_CC_IS_GCC ) +#define __percpu_stable_arg __percpu_arg(a[var]) +#else +#define __percpu_stable_arg __percpu_arg(P[var]) +#endif +#define percpu_stable_op (size, op, _var) \ ({ \ __pcpu_type_##size pfo_val__; \ - asm(__pcpu_op2_##size(op, __percpu_arg(P[var]), "%[val]") \ + asm(__pcpu_op2_##size(op, __percpu_stable_arg, "%[val]") \ : [val] __pcpu_reg_##size("=", pfo_val__) \ : [var] "p" (&(_var))); \ (typeof(_var))(unsigned long) pfo_val__; \
arch/x86/kernel/head_64.S +1 -1
@@ -396,7 +396,7 @@ SYM_CODE_START(start_cpu0) UNWIND_HINT_END_OF_STACK /* Find the idle task stack */ - movq PER_CPU_VAR(pcpu_hot) + X86_current_task, %rcx + movq PER_CPU_VAR(pcpu_hot + X86_current_task), %rcx movq TASK_threadsp(%rcx), %rsp jmp .Ljump_to_C_code
arch/x86/kernel/kvm.c +17 -4
@@ -794,14 +794,27 @@ PV_CALLEE_SAVE_REGS_THUNK(__kvm_vcpu_is_preempted);extern bool __raw_callee_save___kvm_vcpu_is_preempted (long ); +#ifndef CONFIG_X86_PIE +#define KVM_CHECK_VCPU_PREEMPTED \ + "movq __per_cpu_offset(,%rdi,8), %rax;" \ + "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax);" +#else +#define KVM_CHECK_VCPU_PREEMPTED \ + "pushq %rdi;" \ + "leaq __per_cpu_offset(%rip), %rax;" \ + "movq (%rax,%rdi,8), %rax;" \ + "leaq steal_time(%rip), %rdi;" \ + "cmpb $0, (%rax, %rdi, 1);" \ + "popq %rdi;" +#endif +/* * Hand-optimize version for x86-64 to avoid 8 64-bit register saving and * restoring to/from the stack. */ -#define PV_VCPU_PREEMPTED_ASM \ - "movq __per_cpu_offset(,%rdi,8), %rax\n\t" \ - "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax)\n\t" \ - "setne %al\n\t" +#define PV_VCPU_PREEMPTED_ASM \ + KVM_CHECK_VCPU_PREEMPTED \ + "setne %al\n\t"DEFINE_PARAVIRT_ASM (__raw_callee_save___kvm_vcpu_is_preempted, PV_VCPU_PREEMPTED_ASM, .text);
arch/x86/lib/cmpxchg16b_emu.S +4 -4
@@ -27,13 +27,13 @@ SYM_FUNC_START(this_cpu_cmpxchg16b_emu) pushfq cli - cmpq PER_CPU_VAR((%rsi)), %rax + cmpq PER_CPU_VAR_ABS((%rsi)), %rax jne .Lnot_same - cmpq PER_CPU_VAR(8(%rsi)), %rdx + cmpq PER_CPU_VAR_ABS(8(%rsi)), %rdx jne .Lnot_same - movq %rbx, PER_CPU_VAR((%rsi)) - movq %rcx, PER_CPU_VAR(8(%rsi)) + movq %rbx, PER_CPU_VAR_ABS((%rsi)) + movq %rcx, PER_CPU_VAR_ABS(8(%rsi)) popfq mov $1, %al
arch/x86/xen/xen-asm.S +5 -5
@@ -28,7 +28,7 @@ * non-zero. */ SYM_FUNC_START(xen_irq_disable_direct) - movb $1, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + movb $1, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) RET SYM_FUNC_END(xen_irq_disable_direct)
@@ -69,7 +69,7 @@ SYM_FUNC_END(check_events) SYM_FUNC_START(xen_irq_enable_direct) FRAME_BEGIN /* Unmask events */ - movb $0, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + movb $0, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) /* * Preempt here doesn't matter because that will deal with any
@@ -78,7 +78,7 @@ SYM_FUNC_START(xen_irq_enable_direct) */ /* Test for pending */ - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_pending + testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_pending) jz 1f call check_events
@@ -97,7 +97,7 @@ SYM_FUNC_END(xen_irq_enable_direct) * x86 use opposite senses (mask vs enable). */ SYM_FUNC_START(xen_save_fl_direct) - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) setz %ah addb %ah, %ah RET
@@ -113,7 +113,7 @@ SYM_FUNC_END(xen_read_cr2); SYM_FUNC_START(xen_read_cr2_direct) FRAME_BEGIN - _ASM_MOV PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_arch_cr2, %_ASM_AX + _ASM_MOV PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_arch_cr2), %_ASM_AX FRAME_END RET SYM_FUNC_END(xen_read_cr2_direct);
Signature · 1 line
2.31.1
RFC PATCH 10/43 x86/alternatives: Adapt assembly for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the assembly options to work with pointers instead of integers. The generated code is the same PIE just ensures input is a pointer.
--- arch/x86/include/asm/alternative.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/alternative.h +3 -3
@@ -307,7 +307,7 @@ static inline int alternatives_text_reserved(void *start, void *end)/* Like alternative_io, but for replacing a direct call with another one. */ #define alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ asm_inline volatile (ALTERNATIVE("call %P[old]", "call %P[new]", ft_flags) \ - : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) + : output : [old] "X" (oldfunc), [new] "X" (newfunc), ## input)/* * Like alternative_call, but there are two features and respective functions.
@@ -320,8 +320,8 @@ static inline int alternatives_text_reserved(void *start, void *end) asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", ft_flags1,\ "call %P[new2]", ft_flags2) \ : output, ASM_CALL_CONSTRAINT \ - : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ -[new2] " i" (newfunc2), ## input) + : [old] "X" (oldfunc), [new1] "X" (newfunc1), \ +[new2] " X " (newfunc2), ## input) /* * use this macro(s) if you need more than one output parameter
Signature · 1 line
2.31.1
RFC PATCH 11/43 x86/irq: Adapt assembly for PIE support
In reply to Hou WenlongChange the assembly options to work with pointers instead of integers. The generated code is the same PIE just ensures input is a pointer.
--- arch/x86/include/asm/irq_stack.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/irq_stack.h +1 -1
@@ -93,7 +93,7 @@"popq %%rsp \n" \ \ : "+r" (tos), ASM_CALL_CONSTRAINT \ -: [__func] " i" (func), [tos] " r" (tos) argconstr \ +: [__func] " X " (func), [tos] " r" (tos) argconstr \ : " cc", " rax", " rcx", " rdx", " rsi", " rdi", " r8", " r9", " r10", \ " memory" \ ); \
Signature · 1 line
2.31.1
RFC PATCH 12/43 x86,rethook: Adapt assembly for PIE support
In reply to Hou WenlongChange the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
--- arch/x86/kernel/rethook.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kernel/rethook.c +8 -0
@@ -27,7 +27,15 @@ asm(#ifdef CONFIG_X86_64 ANNOTATE_NOENDBR /* This is only jumped from ret instruction */ /* Push a fake return address to tell the unwinder it's a rethook. */ +#ifdef CONFIG_X86_PIE +" pushq $0\n" +" pushq %rdi\n" +" leaq arch_rethook_trampoline(%rip), %rdi\n" +" movq %rdi, 8(%rsp)\n" + " popq %rdi\n" +#else " pushq $arch_rethook_trampoline\n" +#endifUNWIND_HINT_FUNC " pushq $" __stringify(__KERNEL_DS) "\n"/* Save the 'sp - 16', this will be fixed later. */
Signature · 1 line
2.31.1
RFC PATCH 13/43 x86/paravirt: Use relative reference for original instruction
In reply to Hou WenlongSimilar to the alternative patching, use relative reference for original instruction rather than absolute one, which saves 8 bytes for one entry on x86_64. And it could generate R_X86_64_PC32 relocation instead of R_X86_64_64 relocation, which also reduces relocation metadata on relocatable builds. And the alignment could be hard coded to be 4 now.
--- arch/x86/include/asm/paravirt.h | 10 +++++----- arch/x86/include/asm/paravirt_types.h | 8 ++++---- arch/x86/kernel/alternative.c | 8 +++++--- arch/x86/kernel/callthunks.c | 2 +- 4 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/include/asm/paravirt.h +5 -5
@@ -742,16 +742,16 @@ extern void default_banner(void);#else /* __ASSEMBLY__ */ -#define _PVSITE (ptype, ops, word, algn) \ +#define _PVSITE (ptype, ops) \ 771:; \ ops; \ 772:; \ .pushsection .parainstructions,"a"; \ - .align algn; \ - word 771b; \ + .align 4; \ + .long 771b-.; \ .byte ptype; \ .byte 772b-771b; \ - _ASM_ALIGN; \ + .align 4; \ .popsection
@@ -759,7 +759,7 @@ extern void default_banner(void);#ifdef CONFIG_PARAVIRT_XXL #define PARA_PATCH (off) ((off) / 8) -#define PARA_SITE (ptype, ops) _PVSITE(ptype, ops, .quad, 8) +#define PARA_SITE (ptype, ops) _PVSITE(ptype, ops)#define PARA_INDIRECT (addr) *addr(%rip)#ifdef CONFIG_DEBUG_ENTRY
arch/x86/include/asm/paravirt_types.h +4 -4
@@ -5,7 +5,7 @@#ifndef __ASSEMBLY__/* These all sit in the .parainstructions section to tell us what to patch. */ struct paravirt_patch_site { -u8 *instr;/* original instructions */ +s32 instr_offset;/* original instructions */ u8 type;/* type of this instruction */ u8 len;/* length of original instruction */ };
@@ -270,11 +270,11 @@ extern struct paravirt_patch_template pv_ops;#define _paravirt_alt (insn_string, type) \ "771:\n\t" insn_string "\n" "772:\n" \ ".pushsection .parainstructions,\"a\"\n" \ - _ASM_ALIGN "\n" \ - _ASM_PTR " 771b\n" \ + " .align 4\n" \ + " .long 771b-.\n" \ " .byte " type "\n" \ " .byte 772b-771b\n" \ - _ASM_ALIGN "\n" \ + " .align 4\n" \ ".popsection\n"/* Generate patchable code, with the default asm parameters. */
arch/x86/kernel/alternative.c +5 -3
@@ -1230,20 +1230,22 @@ void __init_or_module apply_paravirt(struct paravirt_patch_site *start, {struct paravirt_patch_site *p;char insn_buff[MAX_PATCH_LEN ]; +u8 *instr;for (p = start; p < end; p++) {unsigned int used; + instr = (u8 *)&p->instr_offset + p->instr_offset;BUG_ON (p->len >MAX_PATCH_LEN );/* prep the buffer with the original instructions */ -memcpy (insn_buff, p->instr, p->len); - used =paravirt_patch (p->type, insn_buff, (unsigned long )p->instr, p->len); +memcpy (insn_buff, instr, p->len); + used =paravirt_patch (p->type, insn_buff, (unsigned long )instr, p->len);BUG_ON (used > p->len);/* Pad the rest with nops */ add_nops (insn_buff + used, p->len - used); -text_poke_early (p->instr, insn_buff, p->len); +text_poke_early (instr, insn_buff, p->len); } } extern struct paravirt_patch_site __start_parainstructions[],
arch/x86/kernel/callthunks.c +1 -1
@@ -245,7 +245,7 @@ patch_paravirt_call_sites(struct paravirt_patch_site *start,struct paravirt_patch_site *p;for (p = start; p < end; p++) -patch_call (p->instr, ct); +patch_call ((void *)&p->instr_offset + p->instr_offset, ct); }static __init_or_module void
Signature · 1 line
2.31.1
RFC PATCH 14/43 x86/Kconfig: Introduce new Kconfig for PIE kernel building
In reply to Hou WenlongAdd a new Kconfig to control the behaviour of PIE building, and disable it now.
--- arch/x86/Kconfig | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/Kconfig +4 -0
@@ -2208,6 +2208,10 @@ config RELOCATABLE it has been loaded at and the compile time physical address (CONFIG_PHYSICAL_START) is used as the minimum location. +config X86_PIE + def_bool n + depends on X86_64 + config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)" depends on RELOCATABLE
Signature · 1 line
2.31.1
RFC PATCH 15/43 x86/PVH: Use fixed_percpu_data to set up GS base
In reply to Hou Wenlongstartup_64() and startup_xen() both use fixed_percpu_data to set up GS base. So for consitency, use it too in PVH entry.
--- arch/x86/platform/pvh/head.S | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/x86/platform/pvh/head.S +1 -3
@@ -96,7 +96,7 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx - mov $_pa(canary), %eax + mov $_pa(INIT_PER_CPU_VAR(fixed_percpu_data)), %eax xor %edx, %edx wrmsr
@@ -156,8 +156,6 @@ SYM_DATA_START_LOCAL(gdt_start) SYM_DATA_END_LABEL(gdt_start, SYM_L_LOCAL, gdt_end) .balign 16 -SYM_DATA_LOCAL(canary, .fill 48, 1, 0) - SYM_DATA_START_LOCAL(early_stack) .fill BOOT_STACK_SIZE, 1, 0 SYM_DATA_END_LABEL(early_stack, SYM_L_LOCAL, early_stack_end)
Signature · 1 line
2.31.1
RFC PATCH 17/43 x86/pie: Enable stack protector only if per-cpu stack canary is supported
In reply to Hou WenlongSince -fPIE option is not incompatible with -mcmode=kernel option, PIE kernel would drop -mcmodel=kernel option. However, GCC would use %fs as segment register for stack protector when -mcmodel=kernel option is dropped. So only enable stack protector for PIE kernel if per-cpu stack canary is supported.
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -403,6 +403,7 @@ config PGTABLE_LEVELS config CC_HAS_SANE_STACKPROTECTOR bool + default CC_HAS_CUSTOMIZED_STACKPROTECTOR if X86_PIE default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) help
Signature · 1 line
2.31.1
RFC PATCH 19/43 x86/tools: Explicitly include autoconf.h for hostprogs
In reply to Hou WenlongThe relocs tool needs access to the CONFIG_* symbols found in include/generated/autoconf.h, however, the header file is not included. so the #if CONFIG_FW_LOADER code in arch/x86/tool/relocs.c is never compiled.
--- arch/x86/tools/Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/tools/Makefile +3 -1
@@ -38,7 +38,9 @@ $(obj)/insn_decoder_test.o: $(srctree)/tools/arch/x86/lib/insn.c $(srctree)/tool $(obj)/insn_sanity.o: $(srctree)/tools/arch/x86/lib/insn.c $(srctree)/tools/arch/x86/lib/inat.c $(srctree)/tools/arch/x86/include/asm/inat_types.h $(srctree)/tools/arch/x86/include/asm/inat.h $(srctree)/tools/arch/x86/include/asm/insn.h $(objtree)/arch/x86/lib/inat-tables.c -HOST_EXTRACFLAGS += -I$(srctree)/tools/include +HOST_EXTRACFLAGS += -I$(srctree)/tools/include \ + -include include/generated/autoconf.h + hostprogs += relocs relocs-objs := relocs_32.o relocs_64.o relocs_common.o PHONY += relocs
Signature · 1 line
2.31.1
RFC PATCH 21/43 x86/ftrace: Adapt assembly for PIE support
In reply to Hou WenlongChange the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
--- arch/x86/kernel/ftrace_64.S | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
arch/x86/kernel/ftrace_64.S +8 -1
@@ -315,7 +315,14 @@ STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller) SYM_FUNC_START(__fentry__) CALL_DEPTH_ACCOUNT +#ifdef CONFIG_X86_PIE + pushq %r8 + leaq ftrace_stub(%rip), %r8 + cmpq %r8, ftrace_trace_function(%rip) + popq %r8 +#else cmpq $ftrace_stub, ftrace_trace_function +#endif jnz trace RET
@@ -329,7 +336,7 @@ trace: * ip and parent ip are used and the list function is called when * function tracing is enabled. */ - movq ftrace_trace_function, %r8 + movq ftrace_trace_function(%rip), %r8 CALL_NOSPEC r8 restore_mcount_regs
Signature · 1 line
2.31.1
RFC PATCH 30/43 x86/bpf: Adapt BPF_CALL JIT codegen for PIE support
In reply to Hou WenlongIf image is NULL, ip calculated is in bottom address and func is in kernel image address, then the offset is valid when the kernel stays in the top 2G address space. However, PIE kernel image could be below top 2G, which makes the offset out of range. Since the length of PC-relative call instruction is fixed, it's pointless to calculate the offset without the proper image base (it has been zero until the last pass). Use 1 as the dummy offset to generate the instruction in the first pass.
--- arch/x86/net/bpf_jit_comp.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
arch/x86/net/bpf_jit_comp.c +15 -2
@@ -1549,8 +1549,21 @@ st: if (is_imm8(insn->off))return -EINVAL ; offs =x86_call_depth_emit_accounting (&prog, func); } -if (emit_call (&prog, func, image + addrs[i -1 ] + offs)) -return -EINVAL ; +/* +* If image is NULL, ip is in bottom address and func +* is in kernel image address (top 2G), so the offset +* is valid. However, PIE kernel image could be below +* top 2G, then the offset would be out of range. Since +* the length of PC-relative call(0xe8) is fixed, so it's +* pointless to calculate the offset until the last pass. +* Use 1 as the dummy offset if image is NULL. +*/ +if (image) + err =emit_call (&prog, func, image + addrs[i -1 ] + offs); +else + err =emit_call (&prog, (void *)(X86_PATCH_SIZE +1UL ),0 ); +if (err) +return err;break ; }
Signature · 1 line
2.31.1
RFC PATCH 31/43 x86/modules: Adapt module loading for PIE support
In reply to Hou WenlongAdapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for __fentry__().
--- arch/x86/include/asm/sections.h | 5 +++++ arch/x86/kernel/module.c | 27 +++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)
arch/x86/include/asm/sections.h +5 -0
@@ -12,6 +12,11 @@ extern char __end_rodata_aligned[];#if defined(CONFIG_X86_64 )extern char __end_rodata_hpage_align[]; + +#ifdef CONFIG_X86_PIE +extern char __start_got[], __end_got[]; +#endif +#endif extern char __end_of_kernel_reserve[];
arch/x86/kernel/module.c +27 -0
@@ -129,6 +129,18 @@ int apply_relocate(Elf32_Shdr *sechdrs,return 0 ; }#else /*X86_64*/ +#ifdef CONFIG_X86_PIE +static u64 find_got_kernel_entry (Elf64_Sym *sym,const Elf64_Rela *rela) +{ +u64 *pos; + +for (pos = (u64 *)__start_got; pos < (u64 *)__end_got; pos++) +if (*pos == sym->st_value) +return (u64 )pos + rela->r_addend; +return 0 ; +} +#endif + static int __write_relocate_add(Elf64_Shdr *sechdrs, const char *strtab, unsigned int symindex,
@@ -171,6 +183,7 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,case R_X86_64_64 : size =8 ;break ; +#ifndef CONFIG_X86_PIE case R_X86_64_32 :if (val != *(u32 *)&val) goto overflow;
@@ -181,6 +194,13 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#else +case R_X86_64_GOTPCREL : + val =find_got_kernel_entry (sym, rel); +if (!val) + goto unexpected_got_reference; + fallthrough; +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;
@@ -214,11 +234,18 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, }return 0 ; +#ifdef CONFIG_X86_PIE +unexpected_got_reference: +pr_err ("Target got entry doesn't exist in kernel got, loc %p\n" , loc); +return -ENOEXEC ; +#else overflow:pr_err ("overflow in relocation type %d val %Lx\n" , (int )ELF64_R_TYPE (rel[i].r_info), val);pr_err ("`%s' likely not compiled with -mcmodel=kernel\n" , me->name); +#endif +return -ENOEXEC ; }
Signature · 1 line
2.31.1
RFC PATCH 25/43 x86/mm: Make the x86 GOT read-only
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
The GOT is changed during early boot when relocations are applied. Make it read-only directly. This table exists only for PIE binary. Since weak symbol reference would always be GOT reference, there are 8 entries in GOT, but only one entry for __fentry__() is in use. Other GOT references have been optimized by linker.
[Hou Wenlong: Change commit message and skip GOT size check]
--- arch/x86/kernel/vmlinux.lds.S | 2 ++ include/asm-generic/vmlinux.lds.h | 12 ++++++++++++ 2 files changed, 14 insertions(+)
arch/x86/kernel/vmlinux.lds.S +2 -0
@@ -462,6 +462,7 @@ SECTIONS #endif "Unexpected GOT/PLT entries detected!") +#ifndef CONFIG_X86_PIE /* * Sections that should stay zero sized, which is safer to * explicitly check instead of blindly discarding.
@@ -470,6 +471,7 @@ SECTIONS *(.got) *(.igot.*) } ASSERT(SIZEOF(.got) == 0, "Unexpected GOT entries detected!") +#endif .plt : { *(.plt) *(.plt.*) *(.iplt)
include/asm-generic/vmlinux.lds.h +12 -0
@@ -441,6 +441,17 @@ __end_ro_after_init = .;#endif +#ifdef CONFIG_X86_PIE +#define RO_GOT_X86 \ + .got : AT(ADDR(.got) - LOAD_OFFSET) { \ + __start_got = .; \ + *(.got) *(.igot.*); \ + __end_got = .; \ + } +#else +#define RO_GOT_X86 +#endif +/* * .kcfi_traps contains a list KCFI trap locations. */
@@ -486,6 +497,7 @@ BOUNDED_SECTION_PRE_LABEL(.pci_fixup_suspend_late, _pci_fixups_suspend_late, __start, __end) \ } \ \ +RO_GOT_X86 \FW_LOADER_BUILT_IN_DATA \ TRACEDATA \ \
Signature · 1 line
2.31.1
RFC PATCH 28/43 KVM: x86: Adapt assembly for PIE support
In reply to Hou WenlongChange the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
--- arch/x86/kvm/svm/vmenter.S | 10 +++++----- arch/x86/kvm/vmx/vmenter.S | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/vmenter.S +5 -5
@@ -270,16 +270,16 @@ SYM_FUNC_START(__svm_vcpu_run) RESTORE_GUEST_SPEC_CTRL_BODY RESTORE_HOST_SPEC_CTRL_BODY -10: cmpb $0, kvm_rebooting +10: cmpb $0, _ASM_RIP(kvm_rebooting) jne 2b ud2 -30: cmpb $0, kvm_rebooting +30: cmpb $0, _ASM_RIP(kvm_rebooting) jne 4b ud2 -50: cmpb $0, kvm_rebooting +50: cmpb $0, _ASM_RIP(kvm_rebooting) jne 6b ud2 -70: cmpb $0, kvm_rebooting +70: cmpb $0, _ASM_RIP(kvm_rebooting) jne 8b ud2
@@ -381,7 +381,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) RESTORE_GUEST_SPEC_CTRL_BODY RESTORE_HOST_SPEC_CTRL_BODY -3: cmpb $0, kvm_rebooting +3: cmpb $0, _ASM_RIP(kvm_rebooting) jne 2b ud2
arch/x86/kvm/vmx/vmenter.S +1 -1
@@ -289,7 +289,7 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) RET .Lfixup: - cmpb $0, kvm_rebooting + cmpb $0, _ASM_RIP(kvm_rebooting) jne .Lvmfail ud2 .Lvmfail:
Signature · 1 line
2.31.1
RFC PATCH 26/43 x86/pie: Add .data.rel.* sections into link script
In reply to Hou WenlongThe .data.rel.local and .data.rel.ro.local sections are generated when PIE is enabled, merge them into .data section, since no dynamic loader is used.
--- arch/x86/kernel/vmlinux.lds.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kernel/vmlinux.lds.S +4 -0
@@ -177,6 +177,10 @@ SECTIONS /* rarely changed data like cpu maps */ READ_MOSTLY_DATA(INTERNODE_CACHE_BYTES) +#ifdef CONFIG_X86_PIE + *(.data.rel) + *(.data.rel.*) +#endif /* End of data section */ _edata = .; } :data
Signature · 1 line
2.31.1
RFC PATCH 24/43 x86/boot/compressed: Adapt sed command to generate voffset.h when PIE is enabled
In reply to Hou WenlongWhen PIE is enabled, all symbols would be set as hidden to reduce GOT references. According to generic ABI, a hidden symbol contained in a relocatable object must be either removed or converted to STB_LOCAL binding by the link-editor when the relocatable object is included in an executable file or shared object. Both gold and ld.lld change the binding of a STV_HIDDEND symbol to STB_LOCAL. But For GNU ld, it will keep global hidden. However, sed command to generate voffset.h only captures global symbol, then empty voffset.h would be generated when PIE is enabled with lld. So capture local symbol too in sed command.
--- arch/x86/boot/compressed/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/boot/compressed/Makefile +1 -1
@@ -79,7 +79,7 @@ LDFLAGS_vmlinux += -T hostprogs := mkpiggy HOST_EXTRACFLAGS += -I$(srctree)/tools/include -sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABCDGRSTVW] \(_text\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' +sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABCDGRSTVWabcdgrstvw] \(_text\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' quiet_cmd_voffset = VOFFSET $@ cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
Signature · 1 line
2.31.1
RFC PATCH 23/43 x86/pie: Force hidden visibility for all symbol references
In reply to Hou WenlongEliminate all GOT entries in the kernel, by forcing hidden visibility for all symbol references, which informs the compiler that such references will be resolved at link time without the need for allocating GOT entries. However, there are still some GOT entries after this, one for __fentry__() indirect call, and others are due to global weak symbol references.
--- arch/x86/Makefile | 7 +++++++ arch/x86/entry/vdso/Makefile | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-)
arch/x86/Makefile +7 -0
@@ -158,6 +158,11 @@ else KBUILD_RUSTFLAGS += $(rustflags-y) KBUILD_CFLAGS += -mno-red-zone + +ifdef CONFIG_X86_PIE + PIE_CFLAGS := -include $(srctree)/include/linux/hidden.h + KBUILD_CFLAGS += $(PIE_CFLAGS) +endif KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel
@@ -176,6 +181,8 @@ ifeq ($(CONFIG_STACKPROTECTOR),y) endif endif +export PIE_CFLAGS + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug
arch/x86/entry/vdso/Makefile +1 -1
@@ -92,7 +92,7 @@ ifneq ($(RETPOLINE_VDSO_CFLAGS),) endif endif -$(vobjs): KBUILD_CFLAGS := $(filter-out $(PADDING_CFLAGS) $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(RANDSTRUCT_CFLAGS) $(GCC_PLUGINS_CFLAGS) $(RETPOLINE_CFLAGS),$(KBUILD_CFLAGS)) $(CFL) +$(vobjs): KBUILD_CFLAGS := $(filter-out $(PIE_CFLAGS) $(PADDING_CFLAGS) $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(RANDSTRUCT_CFLAGS) $(GCC_PLUGINS_CFLAGS) $(RETPOLINE_CFLAGS),$(KBUILD_CFLAGS)) $(CFL) $(vobjs): KBUILD_AFLAGS += -DBUILD_VDSO #
Signature · 1 line
2.31.1
RFC PATCH 38/43 x86/fixmap: Move vsyscall page out of fixmap area
In reply to Hou WenlongAfter mapping vsyscall page individually, vsyscall page could be moved out of fixmap area.
--- arch/x86/entry/vsyscall/vsyscall_64.c | 4 ---- arch/x86/include/asm/fixmap.h | 17 +++++------------ arch/x86/kernel/head_64.S | 6 +++--- arch/x86/mm/fault.c | 1 - arch/x86/mm/init_64.c | 2 +- 5 files changed, 9 insertions(+), 21 deletions(-)
arch/x86/entry/vsyscall/vsyscall_64.c +0 -4
@@ -35,7 +35,6 @@#include <asm/vsyscall.h> #include <asm/unistd.h> -#include <asm/fixmap.h> #include <asm/traps.h> #include <asm/paravirt.h>
@@ -391,7 +390,4 @@ void __init map_vsyscall(void)if (vsyscall_mode ==XONLY )vm_flags_init (&gate_vma,VM_EXEC ); - -BUILD_BUG_ON ((unsigned long )__fix_to_virt (VSYSCALL_PAGE ) != - (unsigned long )VSYSCALL_ADDR ); }
arch/x86/include/asm/fixmap.h +5 -12
@@ -23,13 +23,13 @@* covered fully. */ #ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP -# define FIXMAP_PMD_NUM 2 +# define FIXMAP_PMD_NUM 1#else # define KM_PMDS (KM_MAX_IDX * ((CONFIG_NR_CPUS + 511) / 512)) -# define FIXMAP_PMD_NUM (KM_PMDS + 2) +# define FIXMAP_PMD_NUM (KM_PMDS + 1)#endif -/* fixmap starts downwards from the 507th entry in level2_fixmap_pgt */ -#define FIXMAP_PMD_TOP 507 +/* fixmap starts downwards from the 506th entry in level2_fixmap_pgt */ +#define FIXMAP_PMD_TOP 506#ifndef __ASSEMBLY__#include <linux/kernel.h>
@@ -38,8 +38,6 @@#include <asm/pgtable_types.h> #ifdef CONFIG_X86_32 #include <linux/threads.h> -#else -#include <uapi/asm/vsyscall.h> #endif /*
@@ -55,8 +53,7 @@extern unsigned long __FIXADDR_TOP;#define FIXADDR_TOP ((unsigned long)__FIXADDR_TOP)#else -#define FIXADDR_TOP (round_up(VSYSCALL_ADDR + PAGE_SIZE, 1<<PMD_SHIFT) - \ - PAGE_SIZE) +#define FIXADDR_TOP (0xffffffffff600000UL - PAGE_SIZE)#endif /*
@@ -81,10 +78,6 @@ extern unsigned long __FIXADDR_TOP;enum fixed_addresses {#ifdef CONFIG_X86_32 FIX_HOLE , -#else -#ifdef CONFIG_X86_VSYSCALL_EMULATION -VSYSCALL_PAGE = (FIXADDR_TOP -VSYSCALL_ADDR ) >>PAGE_SHIFT , -#endif #endif FIX_DBGP_BASE ,FIX_EARLYCON_MEM_BASE ,
arch/x86/kernel/head_64.S +3 -3
@@ -659,15 +659,15 @@ SYM_DATA_START_PAGE_ALIGNED(level2_kernel_pgt) SYM_DATA_END(level2_kernel_pgt) SYM_DATA_START_PAGE_ALIGNED(level2_fixmap_pgt) - .fill (512 - 4 - FIXMAP_PMD_NUM),8,0 + .fill (512 - 5 - FIXMAP_PMD_NUM),8,0 pgtno = 0 .rept (FIXMAP_PMD_NUM) .quad level1_fixmap_pgt + (pgtno << PAGE_SHIFT) - __START_KERNEL_map \ + _PAGE_TABLE_NOENC; pgtno = pgtno + 1 .endr - /* 6 MB reserved space + a 2MB hole */ - .fill 4,8,0 + /* 2MB (with 4KB vsyscall page inside) + 6 MB reserved space + a 2MB hole */ + .fill 5,8,0 SYM_DATA_END(level2_fixmap_pgt) SYM_DATA_START_PAGE_ALIGNED(level1_fixmap_pgt)
arch/x86/mm/fault.c +0 -1
@@ -22,7 +22,6 @@#include <asm/cpufeature.h> /* boot_cpu_has, ... */ #include <asm/traps.h> /* dotraplinkage, ... */ -#include <asm/fixmap.h> /* VSYSCALL_ADDR */ #include <asm/vsyscall.h> /* emulate_vsyscall */ #include <asm/vm86.h> /* struct vm86 */ #include <asm/mmu_context.h> /* vma_pkey() */
arch/x86/mm/init_64.c +1 -1
@@ -40,7 +40,7 @@#include <linux/uaccess.h> #include <asm/pgalloc.h> #include <asm/dma.h> -#include <asm/fixmap.h> +#include <asm/vsyscall.h> #include <asm/e820/api.h> #include <asm/apic.h> #include <asm/tlb.h>
Signature · 1 line
2.31.1
RFC PATCH 34/43 objtool: Adapt indirect call of __fentry__() for PIE support
In reply to Hou WenlongWhen using PIE with function tracing, the compiler generates a call through the GOT (call *__fentry__@GOTPCREL). This instruction is an indirect call (INSN_CALL_DYNAMIC) and wouldn't be collected by add_call_destinations(). So collect those indirect calls of __fentry__() individually for PIE support. And replace the 6th byte of the GOT call by a 1-byte nop so ftrace can handle the previous 5-bytes as before.
When RETPOLINE is enabled, __fentry__() is still an indirect call, which generates warnings in objtool. For simplicity, select DYNAMIC_FTRACE to patch it as NOPs. And regard it as INSN_CALL to omit warnings for jump table and retpoline checks in ojbtool.
--- arch/x86/Kconfig | 1 + tools/objtool/arch/x86/decode.c | 10 +++++++-- tools/objtool/check.c | 39 +++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 2 deletions(-)
arch/x86/Kconfig +1 -0
@@ -2225,6 +2225,7 @@ config X86_PIE def_bool n depends on X86_64 select OBJTOOL if HAVE_OBJTOOL + select DYNAMIC_FTRACE if FUNCTION_TRACER && RETPOLINE config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)"
tools/objtool/arch/x86/decode.c +8 -2
@@ -747,15 +747,21 @@ void arch_initial_func_cfi_state(struct cfi_init_state *state)const char *arch_nop_insn (int len) { -static const char nops[5 ][5 ] = { +static const char nops[6 ][6 ] = { {BYTES_NOP1 }, {BYTES_NOP2 }, {BYTES_NOP3 }, {BYTES_NOP4 }, {BYTES_NOP5 }, +/* +* For PIE kernel, use a 5-byte nop +* and 1-byte nop to keep the frace +* hooking algorithm working correct. +*/ + {BYTES_NOP5 ,BYTES_NOP1 }, }; -if (len <1 || len >5 ) { +if (len <1 || len >6 ) {WARN ("invalid NOP size: %d\n" , len);return NULL ; }
tools/objtool/check.c +39 -0
@@ -1785,6 +1785,38 @@ static int add_call_destinations(struct objtool_file *file)return 0 ; } +static int add_indirect_mcount_calls (struct objtool_file *file) +{ +struct instruction *insn; +struct reloc *reloc; + +for_each_insn (file, insn) { +if (insn->type !=INSN_CALL_DYNAMIC ) +continue ; + + reloc =insn_reloc (file, insn); +if (!reloc) +continue ; +if (!reloc->sym->fentry) +continue ; + +/* +* __fentry__() is an indirect call even in RETPOLINE builiding +* when X86_PIE is enabled, so DYNAMIC_FTRACE is selected. Then +* all indirect calls of __fentry__() would be patched as NOP +* later, so regard it as retpoline safe as a hack here. Also +* regard it as a direct call, otherwise, it would be treat as +* a jump to jump table in insn_jump_table(), because +* _jump_table and _call_dest share the same memory. +*/ + insn->type =INSN_CALL ; + insn->retpoline_safe = true; +add_call_dest (file, insn, reloc->sym, false); + } + +return 0 ; +} +/* * The .alternatives section requires some extra special care over and above * other special sections because alternatives are patched in place.
@@ -2668,6 +2700,13 @@ static int decode_sections(struct objtool_file *file)if (ret)return ret; +/* +* For X86 PIE kernel, __fentry__ call is an indirect call instead +* of direct call. +*/ +if (opts.pie) +add_indirect_mcount_calls (file); +/* * Must be after add_call_destinations() such that it can override * dead_end_function() marks.
Signature · 1 line
2.31.1
RFC PATCH 39/43 x86/fixmap: Unify FIXADDR_TOP
In reply to Hou WenlongNow FIXADDR_TOP is nothing to do with vsyscall page, it can be declared as variable too for x86_64, so unify it for x86.
--- arch/x86/include/asm/fixmap.h | 13 ------------- arch/x86/kernel/head64.c | 1 - arch/x86/mm/dump_pagetables.c | 3 ++- arch/x86/mm/ioremap.c | 5 ++--- arch/x86/mm/pgtable.c | 13 +++++++++++++ arch/x86/mm/pgtable_32.c | 3 --- 6 files changed, 17 insertions(+), 21 deletions(-)
arch/x86/include/asm/fixmap.h +0 -13
@@ -40,21 +40,8 @@#include <linux/threads.h> #endif -/* -* We can't declare FIXADDR_TOP as variable for x86_64 because vsyscall -* uses fixmaps that relies on FIXADDR_TOP for proper address calculation. -* Because of this, FIXADDR_TOP x86 integration was left as later work. -*/ -#ifdef CONFIG_X86_32 -/* -* Leave one empty page between vmalloc'ed areas and -* the start of the fixmap. -*/ extern unsigned long __FIXADDR_TOP;#define FIXADDR_TOP ((unsigned long)__FIXADDR_TOP) -#else -#define FIXADDR_TOP (0xffffffffff600000UL - PAGE_SIZE) -#endif /* * Here we define all the compile-time 'special' virtual
arch/x86/kernel/head64.c +0 -1
@@ -499,7 +499,6 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_modeBUILD_BUG_ON (!(MODULES_VADDR > __START_KERNEL));MAYBE_BUILD_BUG_ON (!(((MODULES_END -1 ) &PGDIR_MASK ) == (__START_KERNEL &PGDIR_MASK ))); -BUILD_BUG_ON (__fix_to_virt (__end_of_fixed_addresses) <=MODULES_END );cr4_init_shadow ();
arch/x86/mm/dump_pagetables.c +2 -1
@@ -104,7 +104,7 @@ static struct addr_marker address_markers[] = { [HIGH_KERNEL_NR] = { __START_KERNEL_map, "High Kernel Mapping" }, [MODULES_VADDR_NR] = { MODULES_VADDR, "Modules" }, [MODULES_END_NR] = { MODULES_END, "End Modules" }, - [FIXADDR_START_NR] = { FIXADDR_START, "Fixmap Area" }, + [FIXADDR_START_NR] = { 0UL, "Fixmap Area" }, [END_OF_SPACE_NR ] = {-1 ,NULL } };
@@ -453,6 +453,7 @@ static int __init pt_dump_init(void) address_markers[KASAN_SHADOW_START_NR ].start_address =KASAN_SHADOW_START ; address_markers[KASAN_SHADOW_END_NR ].start_address =KASAN_SHADOW_END ;#endif + address_markers[FIXADDR_START_NR ].start_address =FIXADDR_START ;#endif #ifdef CONFIG_X86_32 address_markers[VMALLOC_START_NR ].start_address =VMALLOC_START ;
arch/x86/mm/ioremap.c +2 -3
@@ -879,10 +879,9 @@ void __init early_ioremap_init(void)pmd_t *pmd;#ifdef CONFIG_X86_64 -BUILD_BUG_ON ((fix_to_virt (0 ) +PAGE_SIZE ) & ((1 <<PMD_SHIFT ) -1 )); -#else -WARN_ON ((fix_to_virt (0 ) +PAGE_SIZE ) & ((1 <<PMD_SHIFT ) -1 )); +BUG_ON (__fix_to_virt (__end_of_fixed_addresses) <=MODULES_END );#endif +WARN_ON ((fix_to_virt (0 ) +PAGE_SIZE ) & ((1 <<PMD_SHIFT ) -1 ));early_ioremap_setup ();
arch/x86/mm/pgtable.c +13 -0
@@ -627,6 +627,19 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, }#endif +#ifdef CONFIG_X86_32 +/* +* Leave one empty page between vmalloc'ed areas and +* the start of the fixmap. +*/ +#define __FIXADDR_TOP_BASE 0xfffff000 +#else +#define __FIXADDR_TOP_BASE (0xffffffffff600000UL - PAGE_SIZE) +#endif + +unsigned long __FIXADDR_TOP = __FIXADDR_TOP_BASE; +EXPORT_SYMBOL (__FIXADDR_TOP); +/** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve
arch/x86/mm/pgtable_32.c +0 -3
@@ -65,9 +65,6 @@ void set_pte_vaddr(unsigned long vaddr, pte_t pteval)flush_tlb_one_kernel (vaddr); } -unsigned long __FIXADDR_TOP =0xfffff000 ; -EXPORT_SYMBOL (__FIXADDR_TOP); -/* * vmalloc=size forces the vmalloc area to be exactly 'size' * bytes. This can be used to increase (or decrease) the
Signature · 1 line
2.31.1
RFC PATCH 36/43 x86/vsyscall: Don't use set_fixmap() to map vsyscall page
In reply to Hou WenlongIn order to unify FIXADDR_TOP for x86 and allow fixmap area to be moveable, vsyscall page should be mapped individually. However, for XENPV guest, vsyscall page needs to be mapped into user pagetable too. So introduce a new PVMMU op to help to map vsyscall page.
--- arch/x86/entry/vsyscall/vsyscall_64.c | 3 +-- arch/x86/include/asm/paravirt.h | 7 +++++++ arch/x86/include/asm/paravirt_types.h | 4 ++++ arch/x86/include/asm/vsyscall.h | 13 +++++++++++++ arch/x86/kernel/paravirt.c | 4 ++++ arch/x86/xen/mmu_pv.c | 20 ++++++++++++++------ 6 files changed, 43 insertions(+), 8 deletions(-)
arch/x86/entry/vsyscall/vsyscall_64.c +1 -2
@@ -385,8 +385,7 @@ void __init map_vsyscall(void)* page. */ if (vsyscall_mode ==EMULATE ) { -__set_fixmap (VSYSCALL_PAGE , physaddr_vsyscall, -PAGE_KERNEL_VVAR ); +__set_vsyscall_page (physaddr_vsyscall,PAGE_KERNEL_VVAR );set_vsyscall_pgtable_user_bits (swapper_pg_dir); }
arch/x86/include/asm/paravirt.h +7 -0
@@ -576,6 +576,13 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx, { pv_ops.mmu.set_fixmap (idx, phys, flags); } + +#ifdef CONFIG_X86_VSYSCALL_EMULATION +static inline void __set_vsyscall_page (phys_addr_t phys,pgprot_t flags) +{ + pv_ops.mmu.set_vsyscall_page (phys, flags); +} +#endif #endif #if defined(CONFIG_SMP ) && defined(CONFIG_PARAVIRT_SPINLOCKS )
arch/x86/include/asm/paravirt_types.h +4 -0
@@ -224,6 +224,10 @@ struct pv_mmu_ops {an mfn. We can tell which is which from the index. */ void (*set_fixmap)(unsigned /* enum fixed_addresses */ idx,phys_addr_t phys,pgprot_t flags); + +#ifdef CONFIG_X86_VSYSCALL_EMULATION +void (*set_vsyscall_page)(phys_addr_t phys,pgprot_t flags); +#endif #endif } __no_randomize_layout;
arch/x86/include/asm/vsyscall.h +13 -0
@@ -2,6 +2,7 @@#ifndef _ASM_X86_VSYSCALL_H#define _ASM_X86_VSYSCALL_H +#include <asm/pgtable.h> #include <linux/seqlock.h> #include <uapi/asm/vsyscall.h>
@@ -15,6 +16,18 @@ extern void set_vsyscall_pgtable_user_bits(pgd_t *root);*/ extern bool emulate_vsyscall (unsigned long error_code,struct pt_regs *regs,unsigned long address); +static inline void native_set_vsyscall_page (phys_addr_t phys,pgprot_t flags) +{ +pgprot_val (flags) &= __default_kernel_pte_mask; +set_pte_vaddr (VSYSCALL_ADDR ,pfn_pte (phys >>PAGE_SHIFT , flags)); +} + +#ifndef CONFIG_PARAVIRT_XXL +#define __set_vsyscall_page native_set_vsyscall_page +#else +#include <asm/paravirt.h> +#endif +#else static inline void map_vsyscall (void ) {} static inline bool emulate_vsyscall(unsigned long error_code,
arch/x86/kernel/paravirt.c +4 -0
@@ -33,6 +33,7 @@#include <asm/tlb.h> #include <asm/io_bitmap.h> #include <asm/gsseg.h> +#include <asm/vsyscall.h> /* * nop stub, which must not clobber anything *including the stack* to
@@ -357,6 +358,9 @@ struct paravirt_patch_template pv_ops = { }, .mmu.set_fixmap = native_set_fixmap, +#ifdef CONFIG_X86_VSYSCALL_EMULATION + .mmu.set_vsyscall_page = native_set_vsyscall_page, +#endif #endif /* CONFIG_PARAVIRT_XXL */ #if defined(CONFIG_PARAVIRT_SPINLOCKS )
arch/x86/xen/mmu_pv.c +14 -6
@@ -59,6 +59,7 @@#include <asm/tlbflush.h> #include <asm/fixmap.h> +#include <asm/vsyscall.h> #include <asm/mmu_context.h> #include <asm/setup.h> #include <asm/paravirt.h>
@@ -2020,9 +2021,6 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)switch (idx) { case FIX_BTMAP_END ... FIX_BTMAP_BEGIN: -#ifdef CONFIG_X86_VSYSCALL_EMULATION -case VSYSCALL_PAGE : -#endif /* All local page mappings */ pte =pfn_pte (phys, prot);break ;
@@ -2058,14 +2056,21 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot) vaddr =__fix_to_virt (idx);if (HYPERVISOR_update_va_mapping (vaddr, pte,UVMF_INVLPG ))BUG (); +}#ifdef CONFIG_X86_VSYSCALL_EMULATION +static void xen_set_vsyscall_page (phys_addr_t phys,pgprot_t prot) +{ +pte_t pte =pfn_pte (phys >>PAGE_SHIFT , prot); + +if (HYPERVISOR_update_va_mapping (VSYSCALL_ADDR , pte,UVMF_INVLPG )) +BUG (); +/* Replicate changes to map the vsyscall page into the user pagetable vsyscall mapping. */ -if (idx ==VSYSCALL_PAGE ) -set_pte_vaddr_pud (level3_user_vsyscall, vaddr, pte); -#endif +set_pte_vaddr_pud (level3_user_vsyscall,VSYSCALL_ADDR , pte); } +#endif static void __init xen_post_allocator_init (void ) {
@@ -2156,6 +2161,9 @@ static const typeof(pv_ops) xen_mmu_ops __initconst = { }, .set_fixmap = xen_set_fixmap, +#ifdef CONFIG_X86_VSYSCALL_EMULATION + .set_vsyscall_page = xen_set_vsyscall_page, +#endif }, };
Signature · 1 line
2.31.1
RFC PATCH 32/43 x86/boot/64: Use data relocation to get absloute address when PIE is enabled
In reply to Hou WenlongWhen PIE is enabled, all symbol references are RIP-relative, so there is no need to fixup global symbol references when in low address. However, in order to acquire absloute virtual address of symbol, introduce a macro to use data relocation to get it.
--- arch/x86/kernel/head64.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
arch/x86/kernel/head64.c +22 -8
@@ -86,10 +86,22 @@ static struct desc_ptr startup_gdt_descr = {#define __head __section(".head.text") +#ifdef CONFIG_X86_PIE +#define SYM_ABS_VAL (sym) \ + ({ static unsigned long __initdata __##sym = (unsigned long)sym; __##sym; }) + +static void __head *fixup_pointer(void *ptr, unsigned long physaddr) +{ +return ptr; +} +#else +#define SYM_ABS_VAL (sym) ((unsigned long)sym) + static void __head *fixup_pointer(void *ptr, unsigned long physaddr) {return ptr - (void *)_text + (void *)physaddr; } +#endif /* CONFIG_X86_PIE */ static unsigned long __head *fixup_long (void *ptr,unsigned long physaddr) {
@@ -142,8 +154,8 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv* attribute. */ if (sme_get_me_mask ()) { - vaddr = (unsigned long )__start_bss_decrypted; - vaddr_end = (unsigned long )__end_bss_decrypted; + vaddr =SYM_ABS_VAL (__start_bss_decrypted); + vaddr_end =SYM_ABS_VAL (__end_bss_decrypted);for (; vaddr < vaddr_end; vaddr +=PMD_SIZE ) {/*
@@ -189,6 +201,8 @@ unsigned long __head __startup_64(unsigned long physaddr,bool la57;int i;unsigned int *next_pgt_ptr; +unsigned long text_base =SYM_ABS_VAL (_text); +unsigned long end_base =SYM_ABS_VAL (_end); la57 =check_la57_support (physaddr);
@@ -200,7 +214,7 @@ unsigned long __head __startup_64(unsigned long physaddr,* Compute the delta between the address I am compiled to run at * and the address I am actually running at. */ - load_delta = physaddr - (unsigned long )(_text - __START_KERNEL_map); + load_delta = physaddr - (text_base - __START_KERNEL_map);/* Is the address not 2M aligned? */ if (load_delta & ~PMD_MASK )
@@ -214,9 +228,9 @@ unsigned long __head __startup_64(unsigned long physaddr, pgd =fixup_pointer (&early_top_pgt, physaddr); p = pgd +pgd_index (__START_KERNEL_map);if (la57) - *p = (unsigned long )level4_kernel_pgt; + *p =SYM_ABS_VAL (level4_kernel_pgt);else - *p = (unsigned long )level3_kernel_pgt; + *p =SYM_ABS_VAL (level3_kernel_pgt); *p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta;if (la57) {
@@ -273,7 +287,7 @@ unsigned long __head __startup_64(unsigned long physaddr, pmd_entry +=sme_get_me_mask (); pmd_entry += physaddr; -for (i =0 ; i <DIV_ROUND_UP (_end - _text,PMD_SIZE ); i++) { +for (i =0 ; i <DIV_ROUND_UP (end_base - text_base,PMD_SIZE ); i++) {int idx = i + (physaddr >>PMD_SHIFT ); pmd[idx %PTRS_PER_PMD ] = pmd_entry + i *PMD_SIZE ;
@@ -298,11 +312,11 @@ unsigned long __head __startup_64(unsigned long physaddr, pmd =fixup_pointer (level2_kernel_pgt, physaddr);/* invalidate pages before the kernel image */ -for (i =0 ; i <pmd_index ((unsigned long )_text); i++) +for (i =0 ; i <pmd_index (text_base); i++) pmd[i] &= ~_PAGE_PRESENT;/* fixup pages that are part of the kernel image */ -for (; i <=pmd_index ((unsigned long )_end); i++) +for (; i <=pmd_index (end_base); i++)if (pmd[i] & _PAGE_PRESENT) pmd[i] += load_delta;
Signature · 1 line
2.31.1
RFC PATCH 29/43 x86/PVH: Adapt PVH booting for PIE support
In reply to Hou WenlongIf PIE is enabled, all symbol references would be RIP-relative. However, PVH booting runs in low address space, which could cause wrong x86_init callbacks assignment. Since init_top_pgt has building high kernel address mapping, let PVH booting runs in high address space to make all things right.
PVH booting assumes that no relocation happened. Since the kernel compile address is still in top 2G, so it is allowed to use R_X86_64_32S for symbol references in pvh_start_xen().
--- arch/x86/platform/pvh/head.S | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
arch/x86/platform/pvh/head.S +15 -2
@@ -94,6 +94,13 @@ SYM_CODE_START_LOCAL(pvh_start_xen) /* 64-bit entry point. */ .code64 1: +#ifdef CONFIG_X86_PIE + movabs $2f, %rax + ANNOTATE_RETPOLINE_SAFE + jmp *%rax +2: + ANNOTATE_NOENDBR // above +#endif /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx #if defined(CONFIG_STACKPROTECTOR_FIXED)
@@ -149,9 +156,15 @@ SYM_CODE_END(pvh_start_xen) .section ".init.data","aw" .balign 8 SYM_DATA_START_LOCAL(gdt) + /* + * Use an ASM_PTR (quad on x64) for _pa(gdt_start) because PIE requires + * a pointer size storage value before applying the relocation. On + * 32-bit _ASM_PTR will be a long which is aligned the space needed for + * relocation. + */ .word gdt_end - gdt_start - .long _pa(gdt_start) - .word 0 + _ASM_PTR _pa(gdt_start) + .balign 8 SYM_DATA_END(gdt) SYM_DATA_START_LOCAL(gdt_start) .quad 0x0000000000000000 /* NULL descriptor */
Signature · 1 line
2.31.1
RFC PATCH 42/43 x86/pie: Allow kernel image to be relocated in top 512G
In reply to Hou WenlongFor PIE kernel image, it could be relocated at any address. To be simplified, treat the 2G area which including kernel image, modules area and fixmap area as a whole, allow it to be relocated in top 512G. After that, the relocated kernel address may be below than __START_KERNEL_map, so use a global variable to store the base of relocated kernel image. And pa/va transformation of kernel image address is adapted.
--- arch/x86/include/asm/kmsan.h | 6 ++--- arch/x86/include/asm/page_64.h | 8 +++---- arch/x86/include/asm/page_64_types.h | 8 +++++++ arch/x86/include/asm/pgtable_64_types.h | 10 ++++---- arch/x86/kernel/head64.c | 32 ++++++++++++++++++------- arch/x86/kernel/head_64.S | 12 ++++++++++ arch/x86/kernel/setup.c | 6 +++++ arch/x86/mm/dump_pagetables.c | 9 ++++--- arch/x86/mm/init_64.c | 8 +++---- arch/x86/mm/kasan_init_64.c | 4 ++-- arch/x86/mm/pat/set_memory.c | 2 +- arch/x86/mm/physaddr.c | 14 +++++------ arch/x86/platform/efi/efi_thunk_64.S | 4 ++++ 13 files changed, 87 insertions(+), 36 deletions(-)
arch/x86/include/asm/kmsan.h +3 -3
@@ -63,16 +63,16 @@ static inline bool kmsan_phys_addr_valid(unsigned long addr)static inline bool kmsan_virt_addr_valid (void *addr) {unsigned long x = (unsigned long )addr; -unsigned long y = x - __START_KERNEL_map; +unsigned long y = x -KERNEL_MAP_BASE ; -/* use the carry flag to determine if x was < __START_KERNEL_map */ +/* use the carry flag to determine if x was < KERNEL_MAP_BASE */ if (unlikely (x > y)) { x = y + phys_base;if (y >=KERNEL_IMAGE_SIZE )return false; }else { - x = y + (__START_KERNEL_map -PAGE_OFFSET ); + x = y + (KERNEL_MAP_BASE -PAGE_OFFSET );/* carry flag will be set if starting x was >= PAGE_OFFSET */ if ((x > y) || !kmsan_phys_addr_valid (x))
arch/x86/include/asm/page_64.h +4 -4
@@ -20,10 +20,10 @@ extern unsigned long vmemmap_base;static __always_inline unsigned long __phys_addr_nodebug (unsigned long x) { -unsigned long y = x - __START_KERNEL_map; +unsigned long y = x -KERNEL_MAP_BASE ; -/* use the carry flag to determine if x was < __START_KERNEL_map */ - x = y + ((x > y) ? phys_base : (__START_KERNEL_map -PAGE_OFFSET )); +/* use the carry flag to determine if x was < KERNEL_MAP_BASE */ + x = y + ((x > y) ? phys_base : (KERNEL_MAP_BASE -PAGE_OFFSET ));return x; }
@@ -34,7 +34,7 @@ extern unsigned long __phys_addr_symbol(unsigned long);#else #define __phys_addr (x) __phys_addr_nodebug(x)#define __phys_addr_symbol (x) \ - ((unsigned long)(x) - __START_KERNEL_map + phys_base) + ((unsigned long)(x) - KERNEL_MAP_BASE + phys_base)#endif #define __phys_reloc_hide (x) (x)
arch/x86/include/asm/page_64_types.h +8 -0
@@ -4,6 +4,8 @@#ifndef __ASSEMBLY__#include <asm/kaslr.h> + +extern unsigned long kernel_map_base;#endif #ifdef CONFIG_KASAN
@@ -49,6 +51,12 @@#define __START_KERNEL_map _AC(0xffffffff80000000, UL) +#ifdef CONFIG_X86_PIE +#define KERNEL_MAP_BASE kernel_map_base +#else +#define KERNEL_MAP_BASE __START_KERNEL_map +#endif /* CONFIG_X86_PIE */ +/* See Documentation/x86/x86_64/mm.rst for a description of the memory map. */ #define __PHYSICAL_MASK_SHIFT 52
arch/x86/include/asm/pgtable_64_types.h +6 -4
@@ -187,14 +187,16 @@ extern unsigned int ptrs_per_p4d;#define KMSAN_MODULES_ORIGIN_START (KMSAN_MODULES_SHADOW_START + MODULES_LEN)#endif /* CONFIG_KMSAN */ -#define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) +#define RAW_MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) +#define MODULES_VADDR (KERNEL_MAP_BASE + KERNEL_IMAGE_SIZE)/* The module sections ends with the start of the fixmap */ #ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP -# define MODULES_END _AC(0xffffffffff000000, UL) +# define RAW_MODULES_END _AC(0xffffffffff000000, UL)#else -# define MODULES_END _AC(0xfffffffffe000000, UL) +# define RAW_MODULES_END _AC(0xfffffffffe000000, UL)#endif -#define MODULES_LEN (MODULES_END - MODULES_VADDR) +#define MODULES_LEN (RAW_MODULES_END - RAW_MODULES_VADDR) +#define MODULES_END (MODULES_VADDR + MODULES_LEN)#define ESPFIX_PGD_ENTRY _AC(-2, UL)#define ESPFIX_BASE_ADDR (ESPFIX_PGD_ENTRY << P4D_SHIFT)
arch/x86/kernel/head64.c +24 -8
@@ -66,6 +66,11 @@ unsigned long vmemmap_base __ro_after_init = __VMEMMAP_BASE_L4;EXPORT_SYMBOL (vmemmap_base);#endif +#ifdef CONFIG_X86_PIE +unsigned long kernel_map_base__ro_after_init = __START_KERNEL_map; +EXPORT_SYMBOL (kernel_map_base); +#endif +/* * GDT used on the boot CPU before switching to virtual addresses. */
@@ -193,6 +198,7 @@ unsigned long __head __startup_64(unsigned long physaddr, {unsigned long load_delta, *p;unsigned long pgtable_flags; +unsigned long kernel_map_base_offset =0 ;pgdval_t *pgd;p4dval_t *p4d;pudval_t *pud;
@@ -252,6 +258,13 @@ unsigned long __head __startup_64(unsigned long physaddr, pud[511 ] += load_delta; } +#ifdef CONFIG_X86_PIE + kernel_map_base_offset = text_base &PUD_MASK ; + *fixup_long (&kernel_map_base, physaddr) = kernel_map_base_offset; + kernel_map_base_offset -= __START_KERNEL_map; + *fixup_long (&__FIXADDR_TOP, physaddr) += kernel_map_base_offset; +#endif + pmd =fixup_pointer (level2_fixmap_pgt, physaddr);for (i =FIXMAP_PMD_TOP ; i >FIXMAP_PMD_TOP -FIXMAP_PMD_NUM ; i--) pmd[i] += load_delta;
@@ -328,7 +341,7 @@ unsigned long __head __startup_64(unsigned long physaddr,/* fixup pages that are part of the kernel image */ for (; i <=pmd_index (end_base); i++)if (pmd[i] & _PAGE_PRESENT) - pmd[i] += load_delta; + pmd[i] += load_delta + kernel_map_base_offset;/* invalidate pages after the kernel image */ for (; i <PTRS_PER_PMD ; i++)
@@ -338,7 +351,8 @@ unsigned long __head __startup_64(unsigned long physaddr,* Fixup phys_base - remove the memory encryption mask to obtain * the true physical address. */ - *fixup_long (&phys_base, physaddr) += load_delta -sme_get_me_mask (); + *fixup_long (&phys_base, physaddr) += load_delta + kernel_map_base_offset - +sme_get_me_mask ();return sme_postprocess_startup (bp, pmd); }
@@ -376,7 +390,7 @@ bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd)if (!pgtable_l5_enabled ()) p4d_p = pgd_p;else if (pgd) - p4d_p = (p4dval_t *)((pgd &PTE_PFN_MASK ) + __START_KERNEL_map - phys_base); + p4d_p = (p4dval_t *)((pgd &PTE_PFN_MASK ) +KERNEL_MAP_BASE - phys_base);else {if (next_early_pgt >=EARLY_DYNAMIC_PAGE_TABLES ) {reset_early_page_tables ();
@@ -385,13 +399,13 @@ bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd) p4d_p = (p4dval_t *)early_dynamic_pgts[next_early_pgt++];memset (p4d_p,0 ,sizeof (*p4d_p) *PTRS_PER_P4D ); - *pgd_p = (pgdval_t )p4d_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; + *pgd_p = (pgdval_t )p4d_p -KERNEL_MAP_BASE + phys_base + _KERNPG_TABLE; } p4d_p +=p4d_index (address); p4d = *p4d_p;if (p4d) - pud_p = (pudval_t *)((p4d &PTE_PFN_MASK ) + __START_KERNEL_map - phys_base); + pud_p = (pudval_t *)((p4d &PTE_PFN_MASK ) +KERNEL_MAP_BASE - phys_base);else {if (next_early_pgt >=EARLY_DYNAMIC_PAGE_TABLES ) {reset_early_page_tables ();
@@ -400,13 +414,13 @@ bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd) pud_p = (pudval_t *)early_dynamic_pgts[next_early_pgt++];memset (pud_p,0 ,sizeof (*pud_p) *PTRS_PER_PUD ); - *p4d_p = (p4dval_t )pud_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; + *p4d_p = (p4dval_t )pud_p -KERNEL_MAP_BASE + phys_base + _KERNPG_TABLE; } pud_p +=pud_index (address); pud = *pud_p;if (pud) - pmd_p = (pmdval_t *)((pud &PTE_PFN_MASK ) + __START_KERNEL_map - phys_base); + pmd_p = (pmdval_t *)((pud &PTE_PFN_MASK ) +KERNEL_MAP_BASE - phys_base);else {if (next_early_pgt >=EARLY_DYNAMIC_PAGE_TABLES ) {reset_early_page_tables ();
@@ -415,7 +429,7 @@ bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd) pmd_p = (pmdval_t *)early_dynamic_pgts[next_early_pgt++];memset (pmd_p,0 ,sizeof (*pmd_p) *PTRS_PER_PMD ); - *pud_p = (pudval_t )pmd_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; + *pud_p = (pudval_t )pmd_p -KERNEL_MAP_BASE + phys_base + _KERNPG_TABLE; } pmd_p[pmd_index (address)] = pmd;
@@ -497,6 +511,7 @@ static void __init copy_bootdata(char *real_mode_data) asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode_data) { +#ifndef CONFIG_X86_PIE /* * Build-time sanity checks on the kernel image and module * area mappings. (these are purely build-time and produce no code)
@@ -509,6 +524,7 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_modeBUILD_BUG_ON (!(MODULES_VADDR > __START_KERNEL));MAYBE_BUILD_BUG_ON (!(((MODULES_END -1 ) &PGDIR_MASK ) == (__START_KERNEL &PGDIR_MASK ))); +#endif cr4_init_shadow ();
arch/x86/kernel/head_64.S +12 -0
@@ -130,7 +130,13 @@ SYM_CODE_START_NOALIGN(startup_64) popq %rsi /* Form the CR3 value being sure to include the CR3 modifier */ +#ifdef CONFIG_X86_PIE + movq kernel_map_base(%rip), %rdi + movabs $early_top_pgt, %rcx + subq %rdi, %rcx +#else movabs $(early_top_pgt - __START_KERNEL_map), %rcx +#endif addq %rcx, %rax jmp 1f SYM_CODE_END(startup_64)
@@ -179,7 +185,13 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #endif /* Form the CR3 value being sure to include the CR3 modifier */ +#ifdef CONFIG_X86_PIE + movq kernel_map_base(%rip), %rdi + movabs $init_top_pgt, %rcx + subq %rdi, %rcx +#else movabs $(init_top_pgt - __START_KERNEL_map), %rcx +#endif addq %rcx, %rax 1:
arch/x86/kernel/setup.c +6 -0
@@ -808,11 +808,17 @@ static int dump_kernel_offset(struct notifier_block *self, unsigned long v, void *p) { if (kaslr_enabled()) { +#ifdef CONFIG_X86_PIE + pr_emerg("Kernel Offset: 0x%lx from 0x%lx\n", +kaslr_offset (), + __START_KERNEL); +#else pr_emerg ("Kernel Offset: 0x%lx from 0x%lx (relocation range: 0x%lx-0x%lx)\n" ,kaslr_offset (), __START_KERNEL, __START_KERNEL_map,MODULES_VADDR -1 ); +#endif } else {pr_emerg ("Kernel Offset: disabled\n" ); }
arch/x86/mm/dump_pagetables.c +6 -3
@@ -102,9 +102,9 @@ static struct addr_marker address_markers[] = { #ifdef CONFIG_EFI [EFI_END_NR ] = {EFI_VA_END ,"EFI Runtime Services" }, #endif - [HIGH_KERNEL_NR] = { __START_KERNEL_map, "High Kernel Mapping" }, - [MODULES_VADDR_NR] = { MODULES_VADDR, "Modules" }, - [MODULES_END_NR] = { MODULES_END, "End Modules" }, + [HIGH_KERNEL_NR] = { 0UL, "High Kernel Mapping" }, + [MODULES_VADDR_NR] = { 0UL, "Modules" }, + [MODULES_END_NR] = { 0UL, "End Modules" }, [FIXADDR_START_NR] = { 0UL, "Fixmap Area" }, [END_OF_SPACE_NR ] = {-1 ,NULL } };
@@ -475,6 +475,9 @@ static int __init pt_dump_init(void) address_markers[KASAN_SHADOW_START_NR ].start_address =KASAN_SHADOW_START ; address_markers[KASAN_SHADOW_END_NR ].start_address =KASAN_SHADOW_END ;#endif + address_markers[HIGH_KERNEL_NR ].start_address =KERNEL_MAP_BASE ; + address_markers[MODULES_VADDR_NR ].start_address =MODULES_VADDR ; + address_markers[MODULES_END_NR ].start_address =MODULES_END ; address_markers[FIXADDR_START_NR ].start_address =FIXADDR_START ;#endif #ifdef CONFIG_X86_32
arch/x86/mm/init_64.c +4 -4
@@ -413,7 +413,7 @@ void __init init_extra_mapping_uc(unsigned long phys, unsigned long size)/* * The head.S code sets up the kernel high mapping: * -* from __START_KERNEL_map to __START_KERNEL_map + size (== _end-_text) +* from KERNEL_MAP_BASE to KERNEL_MAP_BASE + size (== _end-_text) * * phys_base holds the negative offset to the kernel, which is added * to the compile time generated pmds. This results in invalid pmds up
@@ -425,8 +425,8 @@ void __init init_extra_mapping_uc(unsigned long phys, unsigned long size)*/ void __init cleanup_highmap (void ) { -unsigned long vaddr = __START_KERNEL_map; -unsigned long vaddr_end = __START_KERNEL_map +KERNEL_IMAGE_SIZE ; +unsigned long vaddr =KERNEL_MAP_BASE ; +unsigned long vaddr_end =KERNEL_MAP_BASE +KERNEL_IMAGE_SIZE ;unsigned long end =roundup ((unsigned long )_brk_end,PMD_SIZE ) -1 ;pmd_t *pmd = level2_kernel_pgt;
@@ -436,7 +436,7 @@ void __init cleanup_highmap(void)* arch/x86/xen/mmu.c:xen_setup_kernel_pagetable(). */ if (max_pfn_mapped) - vaddr_end = __START_KERNEL_map + (max_pfn_mapped <<PAGE_SHIFT ); + vaddr_end =KERNEL_MAP_BASE + (max_pfn_mapped <<PAGE_SHIFT );for (; vaddr +PMD_SIZE -1 < vaddr_end; pmd++, vaddr +=PMD_SIZE ) {if (pmd_none (*pmd))
arch/x86/mm/kasan_init_64.c +2 -2
@@ -197,7 +197,7 @@ static inline p4d_t *early_p4d_offset(pgd_t *pgd, unsigned long addr)return (p4d_t *)pgd; p4d =pgd_val (*pgd) &PTE_PFN_MASK ; - p4d += __START_KERNEL_map - phys_base; + p4d +=KERNEL_MAP_BASE - phys_base;return (p4d_t *)p4d +p4d_index (addr); }
@@ -420,7 +420,7 @@ void __init kasan_init(void) shadow_cea_per_cpu_begin, 0);kasan_populate_early_shadow ((void *)shadow_cea_end, -kasan_mem_to_shadow ((void *)__START_KERNEL_map)); +kasan_mem_to_shadow ((void *)KERNEL_MAP_BASE )); kasan_populate_shadow((unsigned long)kasan_mem_to_shadow(_stext), (unsigned long)kasan_mem_to_shadow(_end),
arch/x86/mm/pat/set_memory.c +1 -1
@@ -1709,7 +1709,7 @@ static int cpa_process_alias(struct cpa_data *cpa)if (!within (vaddr, (unsigned long )_text, _brk_end) &&__cpa_pfn_in_highmap (cpa->pfn)) {unsigned long temp_cpa_vaddr = (cpa->pfn <<PAGE_SHIFT ) + - __START_KERNEL_map - phys_base; +KERNEL_MAP_BASE - phys_base; alias_cpa = *cpa; alias_cpa.vaddr = &temp_cpa_vaddr; alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY );
arch/x86/mm/physaddr.c +7 -7
@@ -14,15 +14,15 @@#ifdef CONFIG_DEBUG_VIRTUAL unsigned long __phys_addr (unsigned long x) { -unsigned long y = x - __START_KERNEL_map; +unsigned long y = x -KERNEL_MAP_BASE ; -/* use the carry flag to determine if x was < __START_KERNEL_map */ +/* use the carry flag to determine if x was < KERNEL_MAP_BASE */ if (unlikely (x > y)) { x = y + phys_base;VIRTUAL_BUG_ON (y >=KERNEL_IMAGE_SIZE ); }else { - x = y + (__START_KERNEL_map -PAGE_OFFSET ); + x = y + (KERNEL_MAP_BASE -PAGE_OFFSET );/* carry flag will be set if starting x was >= PAGE_OFFSET */ VIRTUAL_BUG_ON ((x > y) || !phys_addr_valid (x));
@@ -34,7 +34,7 @@ EXPORT_SYMBOL(__phys_addr);unsigned long __phys_addr_symbol (unsigned long x) { -unsigned long y = x - __START_KERNEL_map; +unsigned long y = x -KERNEL_MAP_BASE ;/* only check upper bounds since lower bounds will trigger carry */ VIRTUAL_BUG_ON (y >=KERNEL_IMAGE_SIZE );
@@ -46,16 +46,16 @@ EXPORT_SYMBOL(__phys_addr_symbol);bool __virt_addr_valid (unsigned long x) { -unsigned long y = x - __START_KERNEL_map; +unsigned long y = x -KERNEL_MAP_BASE ; -/* use the carry flag to determine if x was < __START_KERNEL_map */ +/* use the carry flag to determine if x was < KERNEL_MAP_BASE */ if (unlikely (x > y)) { x = y + phys_base;if (y >=KERNEL_IMAGE_SIZE )return false; }else { - x = y + (__START_KERNEL_map -PAGE_OFFSET ); + x = y + (KERNEL_MAP_BASE -PAGE_OFFSET );/* carry flag will be set if starting x was >= PAGE_OFFSET */ if ((x > y) || !phys_addr_valid (x))
arch/x86/platform/efi/efi_thunk_64.S +4 -0
@@ -52,7 +52,11 @@ STACK_FRAME_NON_STANDARD __efi64_thunk /* * Calculate the physical address of the kernel text. */ +#ifdef CONFIG_X86_PIE + movq kernel_map_base(%rip), %rax +#else movq $__START_KERNEL_map, %rax +#endif subq phys_base(%rip), %rax leaq 1f(%rip), %rbp
Signature · 1 line
2.31.1
RFC PATCH 33/43 objtool: Add validation for x86 PIE support
In reply to Hou WenlongFor x86 PIE binary, only RIP-relative addressing is allowed, however, there are still a little absolute references of R_X86_64_64 relocation type for data section and a little absolute references of R_X86_64_32S relocation type in pvh_start_xen() function.
--- arch/x86/Kconfig | 1 + scripts/Makefile.lib | 1 + tools/objtool/builtin-check.c | 4 +- tools/objtool/check.c | 82 +++++++++++++++++++++++++ tools/objtool/include/objtool/builtin.h | 1 + 5 files changed, 88 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -2224,6 +2224,7 @@ config RELOCATABLE config X86_PIE def_bool n depends on X86_64 + select OBJTOOL if HAVE_OBJTOOL config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)"
scripts/Makefile.lib +1 -0
@@ -270,6 +270,7 @@ objtool-args-$(CONFIG_HAVE_STATIC_CALL_INLINE) += --static-call objtool-args-$(CONFIG_HAVE_UACCESS_VALIDATION) += --uaccess objtool-args-$(CONFIG_GCOV_KERNEL) += --no-unreachable objtool-args-$(CONFIG_PREFIX_SYMBOLS) += --prefix=$(CONFIG_FUNCTION_PADDING_BYTES) +objtool-args-$(CONFIG_X86_PIE) += --pie objtool-args = $(objtool-args-y) \ $(if $(delay-objtool), --link) \
tools/objtool/builtin-check.c +3 -1
@@ -81,6 +81,7 @@ static const struct option check_options[] = {OPT_BOOLEAN ('t' ,"static-call" , &opts.static_call,"annotate static calls" ),OPT_BOOLEAN ('u' ,"uaccess" , &opts.uaccess,"validate uaccess rules for SMAP" ),OPT_BOOLEAN (0 ,"cfi" , &opts.cfi,"annotate kernel control flow integrity (kCFI) function preambles" ), +OPT_BOOLEAN (0 ,"pie" , &opts.pie,"validate addressing rules for PIE" ),OPT_CALLBACK_OPTARG (0 ,"dump" ,NULL ,NULL ,"orc" ,"dump metadata" , parse_dump), OPT_GROUP("Options:"),
@@ -137,7 +138,8 @@ static bool opts_valid(void) opts.sls || opts.stackval || opts.static_call || - opts.uaccess) { + opts.uaccess || + opts.pie) {if (opts.dump_orc) {ERROR ("--dump can't be combined with other options" );return false;
tools/objtool/check.c +82 -0
100 lines not on this page · raw message
tools/objtool/include/objtool/builtin.h +1 -0
@@ -26,6 +26,7 @@ struct opts {bool uaccess;int prefix;bool cfi; +bool pie;/* options: */ bool backtrace;
Signature · 1 line
2.31.1
RFC PATCH 43/43 x86/boot: Extend relocate range for PIE kernel image
In reply to Hou WenlongAllow PIE kernel image to be relocated in unused holes in top 512G of address space.
--- Documentation/x86/x86_64/mm.rst | 4 +++ arch/x86/Kconfig | 11 +++++++ arch/x86/boot/compressed/kaslr.c | 55 ++++++++++++++++++++++++++++++++ arch/x86/boot/compressed/misc.c | 4 ++- arch/x86/boot/compressed/misc.h | 9 ++++++ 5 files changed, 82 insertions(+), 1 deletion(-)
Documentation/x86/x86_64/mm.rst +4 -0
@@ -149,6 +149,10 @@ Note that if CONFIG_RANDOMIZE_MEMORY is enabled, the direct mapping of all physical memory, vmalloc/ioremap space and virtual memory map are randomized. Their order is preserved but their base will be offset early at boot time. +Note that if EXTENDED_RANDOMIZE_BASE is enabled, the kernel image area +including kernel image, module area and fixmap area is randomized as a whole +in top 512G of address space. + Be very careful vs. KASLR when changing anything here. The KASLR address range must not overlap with anything except the KASAN shadow area, which is correct as KASAN disables KASLR.
arch/x86/Kconfig +11 -0
@@ -2266,6 +2266,17 @@ config RANDOMIZE_BASE If unsure, say Y. +config EXTENDED_RANDOMIZE_BASE + bool "Randomize the address of the kernel image (PIE)" + default y + depends on X86_PIE && RANDOMIZE_BASE + help + This packs kernel image, module area and fixmap area as a + whole, and allows it to be randomized in top 512G of virtual + address space when PIE is enabled. + + If unsure, say Y. + # Relocation on x86 needs some additional build support config X86_NEED_RELOCS def_bool y
arch/x86/boot/compressed/kaslr.c +55 -0
58 lines not on this page · raw message
arch/x86/boot/compressed/misc.c +3 -1
@@ -210,8 +210,10 @@ static void handle_relocations(void *output, unsigned long output_len,* needed if KASLR has chosen a different starting address offset * from __START_KERNEL_map. */ -if (IS_ENABLED (CONFIG_X86_64 )) +if (IS_ENABLED (CONFIG_X86_64 )) { delta = virt_addr -LOAD_PHYSICAL_ADDR ; + delta +=pie_randomize (); + }if (!delta) {debug_putstr ("No relocation needed... " );
arch/x86/boot/compressed/misc.h +9 -0
@@ -113,6 +113,15 @@ static inline void choose_random_location(unsigned long input, }#endif +#ifdef CONFIG_EXTENDED_RANDOMIZE_BASE +unsigned long pie_randomize (void ); +#else +static inline unsigned long pie_randomize (void ) +{ +return 0 ; +} +#endif +/* cpuflags.c */ bool has_cpuflag (int flag);
Signature · 1 line
2.31.1
RFC PATCH 37/43 x86/xen: Pin up to VSYSCALL_ADDR when vsyscall page is out of fixmap area
In reply to Hou WenlongIf vsyscall page is moved out of fixmap area, then FIXADDR_TOP would be below vsyscall page. So it should pin up to VSYSCALL_ADDR if vsyscall is enabled.
--- arch/x86/xen/mmu_pv.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/x86/xen/mmu_pv.c +9 -3
@@ -587,6 +587,12 @@ static void xen_p4d_walk(struct mm_struct *mm, p4d_t *p4d,xen_pud_walk (mm, pud, func, last, limit); } +#ifdef CONFIG_X86_VSYSCALL_EMULATION +#define __KERNEL_MAP_TOP (VSYSCALL_ADDR + PAGE_SIZE) +#else +#define __KERNEL_MAP_TOP FIXADDR_TOP +#endif +/* * (Yet another) pagetable walker. This one is intended for pinning a * pagetable. This means that it walks a pagetable and calls the
@@ -594,7 +600,7 @@ static void xen_p4d_walk(struct mm_struct *mm, p4d_t *p4d,* at every level. It walks the entire pagetable, but it only bothers * pinning pte pages which are below limit. In the normal case this * will be STACK_TOP_MAX, but at boot we need to pin up to -* FIXADDR_TOP. +* __KERNEL_MAP_TOP. * * We must skip the Xen hole in the middle of the address space, just after * the big x86-64 virtual hole.
@@ -609,7 +615,7 @@ static void __xen_pgd_walk(struct mm_struct *mm, pgd_t *pgd,/* The limit is the last byte to be touched */ limit--; -BUG_ON (limit >=FIXADDR_TOP ); +BUG_ON (limit >= __KERNEL_MAP_TOP);/* * 64-bit has a great big hole in the middle of the address
@@ -797,7 +803,7 @@ static void __init xen_after_bootmem(void)#ifdef CONFIG_X86_VSYSCALL_EMULATION SetPagePinned (virt_to_page (level3_user_vsyscall));#endif -xen_pgd_walk (&init_mm, xen_mark_pinned,FIXADDR_TOP ); +xen_pgd_walk (&init_mm, xen_mark_pinned, __KERNEL_MAP_TOP); } static void xen_unpin_page(struct mm_struct *mm, struct page *page,
Signature · 1 line
2.31.1
RFC PATCH 27/43 x86/relocs: Handle PIE relocations
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Change the relocation tool to correctly handle relocations generated by -fPIE option:
- Add relocation for each entry of the .got section given the linker does not generate R_X86_64_GLOB_DAT on a simple link.
- Ignore R_X86_64_GOTPCREL.
--- arch/x86/tools/relocs.c | 96 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+), 1 deletion(-)
arch/x86/tools/relocs.c +95 -1
@@ -42,6 +42,7 @@ struct section {Elf32_Word *xsymtab;Elf_Rel *reltab;char *strtab; +Elf_Addr *got; };static struct section *secs;
@@ -308,6 +309,36 @@ static Elf_Sym *sym_lookup(const char *symname)return 0 ; } +static Elf_Sym *sym_lookup_addr (Elf_Addr addr,const char **name) +{ +int i; + +for (i =0 ; i < ehdr.e_shnum; i++) { +struct section *sec = &secs[i]; +long nsyms; +Elf_Sym *symtab; +Elf_Sym *sym; + +if (sec->shdr.sh_type !=SHT_SYMTAB ) +continue ; + + nsyms = sec->shdr.sh_size/sizeof (Elf_Sym); + symtab = sec->symtab; + +for (sym = symtab; --nsyms >=0 ; sym++) { +if (sym->st_value == addr) { +if (name) { + *name =sym_name (sec->link->strtab, + sym); + } +return sym; + } + } + } +return 0 ; +} + +#if BYTE_ORDER ==LITTLE_ENDIAN #define le16_to_cpu (val) (val)#define le32_to_cpu (val) (val)
@@ -588,6 +619,35 @@ static void read_relocs(FILE *fp) } } +static void read_got (FILE *fp) +{ +int i; + +for (i =0 ; i < ehdr.e_shnum; i++) { +struct section *sec = &secs[i]; + + sec->got =NULL ; +if (sec->shdr.sh_type !=SHT_PROGBITS || +strcmp (sec_name (i),".got" )) { +continue ; + } + sec->got =malloc (sec->shdr.sh_size); +if (!sec->got) { +die ("malloc of %" FMT " bytes for got failed\n" , + sec->shdr.sh_size); + } +if (fseek (fp, sec->shdr.sh_offset,SEEK_SET ) <0 ) { +die ("Seek to %" FMT " failed: %s\n" , + sec->shdr.sh_offset,strerror (errno)); + } +if (fread (sec->got,1 , sec->shdr.sh_size, fp) + != sec->shdr.sh_size) { +die ("Cannot read got: %s\n" , +strerror (errno)); + } + } +} +static void print_absolute_symbols (void ) {
@@ -718,6 +778,32 @@ static void add_reloc(struct relocs *r, uint32_t offset) r->offset[r->count++] = offset; } +/* +* The linker does not generate relocations for the GOT for the kernel. +* If a GOT is found, simulate the relocations that should have been included. +*/ +static void walk_got_table (int (*process)(struct section *sec,Elf_Rel *rel, +Elf_Sym *sym,const char *symname), +struct section *sec) +{ +int i; +Elf_Addr entry; +Elf_Sym *sym; +const char *symname; +Elf_Rel rel; + +for (i =0 ; i < sec->shdr.sh_size/sizeof (Elf_Addr); i++) { + entry = sec->got[i]; + sym =sym_lookup_addr (entry, &symname); +if (!sym) +die ("Could not found got symbol for entry %d\n" , i); + rel.r_offset = sec->shdr.sh_addr + i *sizeof (Elf_Addr); + rel.r_info =ELF_BITS ==64 ?R_X86_64_GLOB_DAT + :R_386_GLOB_DAT ; +process (sec, &rel, sym, symname); + } +} +static void walk_relocs (int (*process)(struct section *sec,Elf_Rel *rel,Elf_Sym *sym,const char *symname)) {
@@ -731,6 +817,8 @@ static void walk_relocs(int (*process)(struct section *sec, Elf_Rel *rel,struct section *sec = &secs[i];if (sec->shdr.sh_type !=SHT_REL_TYPE ) { +if (sec->got) +walk_got_table (process, sec);continue ; } sec_symtab = sec->link;
@@ -842,6 +930,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, offset += per_cpu_load_addr;switch (r_type) { +case R_X86_64_GOTPCREL :case R_X86_64_NONE :/* NONE can be ignored. */ break ;
@@ -905,7 +994,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,* the relocations are processed. * Make sure that the offset will fit. */ -if ((int32_t )offset != (int64_t )offset) +if (r_type !=R_X86_64_64 && (int32_t )offset != (int64_t )offset)die ("Relocation offset doesn't fit in 32 bits\n" );if (r_type ==R_X86_64_64 )
@@ -914,6 +1003,10 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,add_reloc (&relocs32, offset);break ; +case R_X86_64_GLOB_DAT : +add_reloc (&relocs64, offset); +break ; +default :die ("Unsupported relocation type: %s (%d)\n" ,rel_type (r_type), r_type);
@@ -1188,6 +1281,7 @@ void process(FILE *fp, int use_real_mode, int as_text,read_strtabs (fp);read_symtabs (fp);read_relocs (fp); +read_got (fp);if (ELF_BITS ==64 )percpu_init ();if (show_absolute_syms) {
Signature · 1 line
2.31.1
RFC PATCH 20/43 x86/percpu: Adapt percpu references relocation for PIE support
In reply to Hou WenlongThe original design of percpu references relocation only handles relative references and ignores absolute references. Because percpu variable had already been relative based on segment. And .percpu ELF section has a virtual address of zero and absolute references can be kept if kaslr is enabled. As for a little relative references, they needs to be relocated by negative offset.
However, it is not compatible with PIE, because almost all percpu references would be RIP-relative. But RIP-relative addressing could only support -2G ~ +2G. In order to move kernel address below top 2G, percpu relative references wouldn't be relocated, instead, percpu base could be adjusted. As for absolute references, they would be relocated like normal variable. After that, percpu references in .altinstr_replacement section couldn't work right, because no fixups are applied for percpu references in apply_alternatives(). However, it could be caught by objtool. Currently, only call depth tracking uses it, so disable it if X86_PIE is enabled.
--- arch/x86/Kconfig | 2 +- arch/x86/kernel/head_64.S | 10 ++++++++++ arch/x86/tools/relocs.c | 17 ++++++++++++++--- 3 files changed, 25 insertions(+), 4 deletions(-)
arch/x86/Kconfig +1 -1
@@ -2635,7 +2635,7 @@ config CPU_UNRET_ENTRY config CALL_DEPTH_TRACKING bool "Mitigate RSB underflow with call depth tracking" - depends on CPU_SUP_INTEL && HAVE_CALL_THUNKS + depends on CPU_SUP_INTEL && HAVE_CALL_THUNKS && !X86_PIE select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE select CALL_THUNKS default y
arch/x86/kernel/head_64.S +10 -0
@@ -72,6 +72,11 @@ SYM_CODE_START_NOALIGN(startup_64) leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx #elif defined(CONFIG_SMP) movabs $__per_cpu_load, %rdx +#ifdef CONFIG_X86_PIE + movabs $__per_cpu_start, %rax + subq %rax, %rdx + movq %rdx, __per_cpu_offset(%rip) +#endif #else xorl %edx, %edx #endif
@@ -79,6 +84,11 @@ SYM_CODE_START_NOALIGN(startup_64) shrq $32, %rdx wrmsr +#if defined(CONFIG_X86_PIE) && defined(CONFIG_SMP) + movq __per_cpu_offset(%rip), %rdx + movq %rdx, PER_CPU_VAR(this_cpu_off) +#endif + pushq %rsi call startup_64_setup_env popq %rsi
arch/x86/tools/relocs.c +14 -3
@@ -848,6 +848,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,case R_X86_64_PC32 :case R_X86_64_PLT32 : +#ifndef CONFIG_X86_PIE /* * PC relative relocations don't need to be adjusted unless * referencing a percpu symbol.
@@ -856,6 +857,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,*/ if (is_percpu_sym (sym, symname))add_reloc (&relocs32neg, offset); +#endif break ;case R_X86_64_PC64 :
@@ -871,10 +873,18 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,case R_X86_64_32S :case R_X86_64_64 :/* -* References to the percpu area don't need to be adjusted. +* References to the percpu area don't need to be adjusted when +* CONFIG_X86_PIE is not enabled. */ -if (is_percpu_sym (sym, symname)) +if (is_percpu_sym (sym, symname)) { +#if CONFIG_X86_PIE +if (r_type !=R_X86_64_64 ) +die ("Invalid absolute reference against per-CPU symbol %s\n" , + symname); +add_reloc (&relocs64, offset); +#endif break ; + }if (shn_abs) {/*
@@ -1044,7 +1054,8 @@ static int cmp_relocs(const void *va, const void *vb)static void sort_relocs (struct relocs *r) { -qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); +if (r->count) +qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); }static int write32 (uint32_t v,FILE *f)
Signature · 1 line
2.31.1
RFC PATCH 41/43 x86/mm: Sort address_markers array when X86 PIE is enabled
In reply to Hou WenlongWhen X86 PIE is enabled, kernel image is allowed to relocated in top 512G, then kernel image address could be below EFI range address. So sort address_markers array to make the order right.
--- arch/x86/mm/dump_pagetables.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/x86/mm/dump_pagetables.c +24 -0
@@ -17,6 +17,7 @@#include <linux/highmem.h> #include <linux/pci.h> #include <linux/ptdump.h> +#include <linux/sort.h> #include <asm/e820/types.h>
@@ -436,6 +437,27 @@ void ptdump_walk_pgd_level_checkwx(void)ptdump_walk_pgd_level_core (NULL , &init_mm,INIT_PGD , true, false); } +#ifdef CONFIG_X86_PIE +static int __init address_markers_sort_cmp (const void *pa,const void *pb) +{ +struct addr_marker *a = (struct addr_marker *)pa; +struct addr_marker *b = (struct addr_marker *)pb; + +return (a->start_address > b->start_address) - + (a->start_address < b->start_address); +} + +static void __init address_markers_sort (void ) +{ +sort (&address_markers[0 ],ARRAY_SIZE (address_markers),sizeof (address_markers[0 ]), + address_markers_sort_cmp,NULL ); +} +#else +static void __init address_markers_sort (void ) +{ +} +#endif +static int __init pt_dump_init (void ) {/*
@@ -467,6 +489,8 @@ static int __init pt_dump_init(void) address_markers[LDT_NR ].start_address =LDT_BASE_ADDR ;# endif #endif +address_markers_sort (); +return 0 ; }__initcall (pt_dump_init);
Signature · 1 line
2.31.1
RFC PATCH 40/43 x86/boot: Fill kernel image puds dynamically
In reply to Hou WenlongFor PIE kernel, it could be randomized in any address. Later, kernel image would be moved down the top 2G, so fille kernel image puds dynamically.
--- arch/x86/kernel/head64.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
arch/x86/kernel/head64.c +12 -2
@@ -239,8 +239,18 @@ unsigned long __head __startup_64(unsigned long physaddr, } pud =fixup_pointer (&level3_kernel_pgt, physaddr); - pud[510 ] += load_delta; - pud[511 ] += load_delta; +if (IS_ENABLED (CONFIG_X86_PIE )) { + pud[510 ] =0 ; + pud[511 ] =0 ; + + i =pud_index (text_base); + pgtable_flags = _KERNPG_TABLE_NOENC - __START_KERNEL_map + load_delta; + pud[i] = pgtable_flags +SYM_ABS_VAL (level2_kernel_pgt); + pud[i +1 ] = pgtable_flags +SYM_ABS_VAL (level2_fixmap_pgt); + }else { + pud[510 ] += load_delta; + pud[511 ] += load_delta; + } pmd =fixup_pointer (level2_fixmap_pgt, physaddr);for (i =FIXMAP_PMD_TOP ; i >FIXMAP_PMD_TOP -FIXMAP_PMD_NUM ; i--)
Signature · 1 line
2.31.1
RFC PATCH 22/43 x86/ftrace: Adapt ftrace nop patching for PIE support
In reply to Hou WenlongFrom: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
When using PIE with function tracing, the compiler generates a call through the GOT (call *__fentry__@GOTPCREL). This instruction takes 6-bytes instead of 5-bytes with a relative call. And -mnop-mcount option is not implemented for -fPIE now.
If PIE is enabled, replace the 6th byte of the GOT call by a 1-byte nop so ftrace can handle the previous 5-bytes as before.
[Hou Wenlong: Adapt code change and fix wrong offset calculation in make_nop_x86()]
--- arch/x86/kernel/ftrace.c | 46 ++++++++++++++++++++++- scripts/recordmcount.c | 81 ++++++++++++++++++++++++++-------------- 2 files changed, 98 insertions(+), 29 deletions(-)
arch/x86/kernel/ftrace.c +45 -1
@@ -124,6 +124,50 @@ ftrace_modify_code_direct(unsigned long ip, const char *old_code,return 0 ; } +/* Bytes before call GOT offset */ +static const unsigned char got_call_preinsn[] = {0xff ,0x15 }; + +static int __ref +ftrace_modify_initial_code (unsigned long ip,unsigned const char *old_code, +unsigned const char *new_code) +{ +unsigned char replaced[MCOUNT_INSN_SIZE +1 ]; + +/* +* If PIE is not enabled default to the original approach to code +* modification. +*/ +if (!IS_ENABLED (CONFIG_X86_PIE )) +return ftrace_modify_code_direct (ip, old_code, new_code); + + ftrace_expected = old_code; + +/* Ensure the instructions point to a call to the GOT */ +if (copy_from_kernel_nofault (replaced, (void *)ip,sizeof (replaced))) { +WARN_ONCE (1 ,"invalid function" ); +return -EFAULT ; + } + +if (memcmp (replaced, got_call_preinsn,sizeof (got_call_preinsn))) { +WARN_ONCE (1 ,"invalid function call" ); +return -EINVAL ; + } + +/* +* Build a nop slide with a 5-byte nop and 1-byte nop to keep the ftrace +* hooking algorithm working with the expected 5 bytes instruction. +*/ +memset (replaced, x86_nops[1 ][0 ],sizeof (replaced)); +memcpy (replaced, new_code,MCOUNT_INSN_SIZE ); + +/* replace the text with the new text */ +if (ftrace_poke_late) +text_poke_queue ((void *)ip, replaced,MCOUNT_INSN_SIZE +1 ,NULL ); +else +text_poke_early ((void *)ip, replaced,MCOUNT_INSN_SIZE +1 ); +return 0 ; +} +int ftrace_make_nop (struct module *mod,struct dyn_ftrace *rec,unsigned long addr) {unsigned long ip = rec->ip;
@@ -141,7 +185,7 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long ad* just modify the code directly. */ if (addr ==MCOUNT_ADDR ) -return ftrace_modify_code_direct (ip, old, new); +return ftrace_modify_initial_code (ip, old, new);/* * x86 overrides ftrace_replace_code -- this function will never be used
scripts/recordmcount.c +53 -28
110 lines not on this page · raw message
Signature · 1 line
2.31.1
RFC PATCH 35/43 x86/pie: Build the kernel as PIE
In reply to Hou WenlongThe kernel is currently build with mcmode=kernel option which forces it to stay on the top 2G of the virtual address space. For PIE, use -fPIE option to build the kernel as a Position Independent Executable (PIE), which uses RIP-relative addressing and could be able to move below the top 2G.
The --emit-relocs linker option was kept instead of using -pie to limit the impact on mapped sections. Any incompatible relocation will be catch by the objtool at compile time.
--- arch/x86/Kconfig | 8 ++++++-- arch/x86/Makefile | 9 +++++++-- 2 files changed, 13 insertions(+), 4 deletions(-)
arch/x86/Kconfig +6 -2
@@ -2222,10 +2222,14 @@ config RELOCATABLE (CONFIG_PHYSICAL_START) is used as the minimum location. config X86_PIE - def_bool n - depends on X86_64 + bool "Build a PIE kernel" + default n + depends on X86_64 && !XEN select OBJTOOL if HAVE_OBJTOOL select DYNAMIC_FTRACE if FUNCTION_TRACER && RETPOLINE + help + This builds a PIE kernel image that could be put at any + virtual address. config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)"
arch/x86/Makefile +7 -2
@@ -160,10 +160,15 @@ else KBUILD_CFLAGS += -mno-red-zone ifdef CONFIG_X86_PIE - PIE_CFLAGS := -include $(srctree)/include/linux/hidden.h + PIE_CFLAGS := -fPIE -include $(srctree)/include/linux/hidden.h KBUILD_CFLAGS += $(PIE_CFLAGS) -endif + # Relax relocation in both CFLAGS and LDFLAGS to support older compilers + KBUILD_CFLAGS += $(call cc-option,-Wa$(comma)-mrelax-relocations=no) + LDFLAGS_vmlinux += $(call ld-option,--no-relax) + KBUILD_LDFLAGS_MODULE += $(call ld-option,--no-relax) +else KBUILD_CFLAGS += -mcmodel=kernel +endif KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel
Signature · 1 line
2.31.1
Re: [PATCH RFC 33/43] objtool: Add validation for x86 PIE support
In reply to Hou WenlongLe 28/04/2023 à 11:51, Hou Wenlong a écrit :
Quoted 122 lines
[Vous ne recevez pas souvent de courriers de [email protected]. Découvrez pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification ]
For x86 PIE binary, only RIP-relative addressing is allowed, however, there are still a little absolute references of
R_X86_64_64relocation type for data section and a little absolute references ofR_X86_64_32Srelocation type inpvh_start_xen()function.Suggested-by Lai Jiangshan <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/Kconfig | 1 + scripts/Makefile.lib | 1 + tools/objtool/builtin-check.c | 4 +- tools/objtool/check.c | 82 +++++++++++++++++++++++++ tools/objtool/include/objtool/builtin.h | 1 + 5 files changed, 88 insertions(+), 1 deletion(-)arch/x86/Kconfig +1
-0@@ -2224,6 +2224,7 @@ config RELOCATABLE config X86_PIE def_bool n depends on X86_64 + select OBJTOOL if HAVE_OBJTOOL config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)"scripts/Makefile.lib +1
-0@@ -270,6 +270,7 @@ objtool-args-$(CONFIG_HAVE_STATIC_CALL_INLINE) += --static-call objtool-args-$(CONFIG_HAVE_UACCESS_VALIDATION) += --uaccess objtool-args-$(CONFIG_GCOV_KERNEL) += --no-unreachable objtool-args-$(CONFIG_PREFIX_SYMBOLS) += --prefix=$(CONFIG_FUNCTION_PADDING_BYTES) +objtool-args-$(CONFIG_X86_PIE) += --pie objtool-args = $(objtool-args-y) \ $(if $(delay-objtool), --link) \tools/objtool/builtin-check.c +3
-1@@ -81,6 +81,7 @@ static const struct option check_options[] = {OPT_BOOLEAN ('t' ,"static-call" , &opts.static_call,"annotate static calls" ),OPT_BOOLEAN ('u' ,"uaccess" , &opts.uaccess,"validate uaccess rules for SMAP" ),OPT_BOOLEAN (0 ,"cfi" , &opts.cfi,"annotate kernel control flow integrity (kCFI) function preambles" ), +OPT_BOOLEAN (0 ,"pie" , &opts.pie,"validate addressing rules for PIE" ),OPT_CALLBACK_OPTARG (0 ,"dump" ,NULL ,NULL ,"orc" ,"dump metadata" , parse_dump), OPT_GROUP("Options:"),@@ -137,7 +138,8 @@ static bool opts_valid(void) opts.sls || opts.stackval || opts.static_call || - opts.uaccess) { + opts.uaccess || + opts.pie) {if (opts.dump_orc) {ERROR ("--dump can't be combined with other options" );return false;tools/objtool/check.c +42
-0@@ -131,6 +131,27 @@ static struct instruction *prev_insn_same_sym(struct objtool_file *file,for (insn =next_insn_same_sec (file, insn); insn; \ insn =next_insn_same_sec (file, insn)) +static struct instruction *find_insn_containing(struct objtool_file *file, +struct section *sec, +unsigned long offset) +{ +struct instruction *insn; + + insn =find_insn (file, sec,0 ); +if (!insn) +return NULL ; + +sec_for_each_insn_from (file, insn) { +if (insn->offset > offset) +return NULL ; +if (insn->offset <= offset && (insn->offset + insn->len) > offset) +return insn; + } + +return NULL ; +} + +static inline struct symbol *insn_call_dest (struct instruction *insn) { if (insn->type == INSN_JUMP_DYNAMIC ||@@ -4529,6 +4550,61 @@ static int validate_reachable_instructions(struct objtool_file *file)return 0 ; } +static int is_in_pvh_code (struct instruction *insn) +{ +struct symbol *sym = insn->sym; + +return sym && !strcmp (sym->name,"pvh_start_xen" ); +} + +static int validate_pie (struct objtool_file *file) +{ +struct section *sec; +struct reloc *reloc; +struct instruction *insn; +int warnings =0 ; + +for_each_sec (file, sec) { +if (!sec->reloc) +continue ; +if (!(sec->sh.sh_flags &SHF_ALLOC )) +continue ; + +list_for_each_entry (reloc, &sec->reloc->reloc_list, list) {
tools/objtool/check.c static int validate_reachable_instructions(struct objtool_file *file) · line 4583
Quoted 10 lines
tools/objtool/check.c +10
-0+switch (reloc->type) { +case R_X86_64_NONE : +case R_X86_64_PC32 : +case R_X86_64_PLT32 : +case R_X86_64_64 : +case R_X86_64_PC64 : +case R_X86_64_GOTPCREL : +break ; +case R_X86_64_32 : +case R_X86_64_32S :
That looks very specific to X86, should it go at another place ?
If it can work for any architecture, can you add generic macros, just like commit c1449735211d ("objtool: Use macros to define arch specific reloc types") then commit c984aef8c832 ("objtool/powerpc: Add --mcount specific implementation") ?
Quoted 55 lines
tools/objtool/check.c +30
-0+ insn =find_insn_containing (file, sec, reloc->offset); +if (!insn) { +WARN ("can't find relocate insn near %s+0x%lx" , + sec->name, reloc->offset); + }else { +if (is_in_pvh_code (insn)) +break ; +WARN ("insn at %s+0x%lx is not compatible with PIE" , + sec->name, insn->offset); + } + warnings++; +break ; +default : +WARN ("unexpected relocation type %d at %s+0x%lx" , + reloc->type, sec->name, reloc->offset); + warnings++; +break ; + } + } + } + +return warnings; +} +int check (struct objtool_file *file) {int ret, warnings =0 ;@@ -4673,6 +4749,12 @@ int check(struct objtool_file *file) warnings += ret; } +if (opts.pie) { + ret =validate_pie (file); +if (ret <0 ) +return ret; + warnings += ret; + }if (opts.stats) {printf ("nr_insns_visited: %ld\n" , nr_insns_visited);tools/objtool/include/objtool/builtin.h +1
-1@@ -26,6 +26,7 @@ struct opts {bool uaccess;int prefix;bool cfi; +bool pie;/* options: */ bool backtrace; --2.31.1
Re: [PATCH RFC 07/43] x86/acpi: Adapt assembly for PIE support
In reply to Hou WenlongQuoted 5 lines · Hou Wenlong
From: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
Quoted 7 lines
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
Signed-off-by Thomas Garnier <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>
Quoted 80 lines
--- arch/x86/kernel/acpi/wakeup_64.S | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)arch/x86/kernel/acpi/wakeup_64.S +16
-16@@ -17,7 +17,7 @@ * Hooray, we are in Long 64-bit mode (but still running in low memory) */ SYM_FUNC_START(wakeup_long64) - movq saved_magic, %rax + movq saved_magic(%rip), %rax movq $0x123456789abcdef0, %rdx cmpq %rdx, %rax je 2f@@ -33,14 +33,14 @@ SYM_FUNC_START(wakeup_long64) movw %ax, %es movw %ax, %fs movw %ax, %gs - movq saved_rsp, %rsp + movq saved_rsp(%rip), %rsp - movq saved_rbx, %rbx - movq saved_rdi, %rdi - movq saved_rsi, %rsi - movq saved_rbp, %rbp + movq saved_rbx(%rip), %rbx + movq saved_rdi(%rip), %rdi + movq saved_rsi(%rip), %rsi + movq saved_rbp(%rip), %rbp - movq saved_rip, %rax + movq saved_rip(%rip), %rax ANNOTATE_RETPOLINE_SAFE jmp *%rax SYM_FUNC_END(wakeup_long64)@@ -51,7 +51,7 @@ SYM_FUNC_START(do_suspend_lowlevel) xorl %eax, %eax call save_processor_state - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax) movq %rsi, pt_regs_si(%rax)@@ -70,13 +70,14 @@ SYM_FUNC_START(do_suspend_lowlevel) pushfq popq pt_regs_flags(%rax) - movq $.Lresume_point, saved_rip(%rip) + leaq .Lresume_point(%rip), %rax + movq %rax, saved_rip(%rip) - movq %rsp, saved_rsp - movq %rbp, saved_rbp - movq %rbx, saved_rbx - movq %rdi, saved_rdi - movq %rsi, saved_rsi + movq %rsp, saved_rsp(%rip) + movq %rbp, saved_rbp(%rip) + movq %rbx, saved_rbx(%rip) + movq %rdi, saved_rdi(%rip) + movq %rsi, saved_rsi(%rip) addq $8, %rsp movl $3, %edi@@ -88,7 +89,7 @@ SYM_FUNC_START(do_suspend_lowlevel) .align 4 .Lresume_point: /* We don't restore %rax, it must be 0 anyway */ - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq saved_context_cr4(%rax), %rbx movq %rbx, %cr4 movq saved_context_cr3(%rax), %rbx --2.31.1
Peter Zijlstra
maintainerRe: [PATCH RFC 33/43] objtool: Add validation for x86 PIE support
In reply to Christophe LeroyQuoted 19 lines · Christophe Leroy
Quoted 19 lines
tools/objtool/check.c +11
-0@@ -131,6 +131,27 @@ static struct instruction *prev_insn_same_sym(struct objtool_file *file,for (insn =next_insn_same_sec (file, insn); insn; \ insn =next_insn_same_sec (file, insn)) +static struct instruction *find_insn_containing(struct objtool_file *file, +struct section *sec, +unsigned long offset) +{ +struct instruction *insn; + + insn =find_insn (file, sec,0 ); +if (!insn) +return NULL ; + +sec_for_each_insn_from (file, insn) {
Quoted 8 lines
tools/objtool/check.c static struct instruction *prev_insn_same_sym(struct objtool_file *file, · line 152
Quoted 8 lines
tools/objtool/check.c +8
-0+if (insn->offset > offset) +return NULL ; +if (insn->offset <= offset && (insn->offset + insn->len) > offset) +return insn; + } + +return NULL ; +}
Urgh, this is horrendous crap. Yes you're only using it in case of a warning, but adding a function like this makes it appear like it's actually sane to use.
A far better implementation -- but still not stellar -- would be something like:
sym = find_symbol_containing(sec, offset);
if (!sym)
// fail
sym_for_each_insn(file, sym, insn) {
...
}But given insn_hash uses sec_offset_hash() you can do something similar to find_reloc_by_dest_range()
start = offset - (INSN_MAX_SIZE - 1);
for_offset_range(o, start, start + INSN_MAX_SIZE) {
hash_for_each_possible(file->insn_hash, insn, hash, sec_offset_hash(sec, o)) {
if (insn->sec != sec)
continue;
if (insn->offset <= offset &&
insn->offset + inns->len > offset)
return insn;
}
}
return NULL;Quoted 43 lines
Quoted 30 lines
tools/objtool/check.c +23
-0+ +static inline struct symbol *insn_call_dest (struct instruction *insn) { if (insn->type == INSN_JUMP_DYNAMIC ||@@ -4529,6 +4550,61 @@ static int validate_reachable_instructions(struct objtool_file *file)return 0 ; } +static int is_in_pvh_code (struct instruction *insn) +{ +struct symbol *sym = insn->sym; + +return sym && !strcmp (sym->name,"pvh_start_xen" ); +} + +static int validate_pie (struct objtool_file *file) +{ +struct section *sec; +struct reloc *reloc; +struct instruction *insn; +int warnings =0 ; + +for_each_sec (file, sec) { +if (!sec->reloc) +continue ; +if (!(sec->sh.sh_flags &SHF_ALLOC )) +continue ; + +list_for_each_entry (reloc, &sec->reloc->reloc_list, list) {tools/objtool/check.c static int validate_reachable_instructions(struct objtool_file *file) · line 4583
Quoted 10 lines
tools/objtool/check.c +10
-0+switch (reloc->type) { +case R_X86_64_NONE : +case R_X86_64_PC32 : +case R_X86_64_PLT32 : +case R_X86_64_64 : +case R_X86_64_PC64 : +case R_X86_64_GOTPCREL : +break ; +case R_X86_64_32 : +case R_X86_64_32S :That looks very specific to X86, should it go at another place ?
Quoted 4 lines
If it can work for any architecture, can you add generic macros, just like commit
c1449735211d("objtool: Use macros to define arch specific reloc types") then commitc984aef8c832("objtool/powerpc: Add --mcount specific implementation") ?
Yes, this should be something like arch_PIE_reloc() or so. Similar to arch_pc_relative_reloc().
Steven Rostedt
maintainerRe: [PATCH RFC 21/43] x86/ftrace: Adapt assembly for PIE support
In reply to Hou WenlongQuoted 31 lines · On Fri, 28 Apr 2023 17:51:01 +0800 "Hou Wenlong" <[email protected]>…
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/kernel/ftrace_64.S | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)arch/x86/kernel/ftrace_64.S +7
-0@@ -315,7 +315,14 @@ STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller) SYM_FUNC_START(__fentry__) CALL_DEPTH_ACCOUNT +#ifdef CONFIG_X86_PIE + pushq %r8 + leaq ftrace_stub(%rip), %r8 + cmpq %r8, ftrace_trace_function(%rip) + popq %r8 +#else cmpq $ftrace_stub, ftrace_trace_function +#endif jnz trace RET
arch/x86/kernel/ftrace_64.S trace: · line 342
Quoted 8 lines
arch/x86/kernel/ftrace_64.S +1
-1@@ -329,7 +336,7 @@ trace: * ip and parent ip are used and the list function is called when * function tracing is enabled. */ - movq ftrace_trace_function, %r8 + movq ftrace_trace_function(%rip), %r8 CALL_NOSPEC r8 restore_mcount_regs
I really don't want to add more updates to !DYNAMIC_FTRACE. This code only exists to make sure I don't break it for other architectures.
How about
arch/x86/Kconfig +1 -1
@@ -37,7 +37,7 @@ config X86_64 config FORCE_DYNAMIC_FTRACE def_bool y - depends on X86_32 + depends on X86_32 || X86_PIE depends on FUNCTION_TRACER select DYNAMIC_FTRACE help
?
-- Steve
Steven Rostedt
maintainerRe: [PATCH RFC 22/43] x86/ftrace: Adapt ftrace nop patching for PIE support
In reply to Hou WenlongQuoted 11 lines · On Fri, 28 Apr 2023 17:51:02 +0800 "Hou Wenlong" <[email protected]>…
From: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
When using PIE with function tracing, the compiler generates a call through the GOT (call *__fentry__@GOTPCREL). This instruction takes 6-bytes instead of 5-bytes with a relative call. And -mnop-mcount option is not implemented for -fPIE now.
If PIE is enabled, replace the 6th byte of the GOT call by a 1-byte nop so ftrace can handle the previous 5-bytes as before.
Wait! This won't work!
You can't just append another nop to fill in the blanks here. We must either have a single 6 byte nop, or we need to refactor the entire logic to something that other archs have.
The two nops means that the CPU can take it as two separate commands. There's nothing stopping the computer from preempting a task between the two. If that happens, and you modify the 1byte nop and 5byte nop with a single 6 byte command, when the task get's rescheduled, it will execute the last 5 bytes of that 6 byte command and take a general protection fault, and likely crash the machine.
NACK on this. It needs a better solution.
-- Steve
Quoted 4 lines
[Hou Wenlong: Adapt code change and fix wrong offset calculation in
make_nop_x86()]
Peter Zijlstra
maintainerRe: [PATCH RFC 34/43] objtool: Adapt indirect call of __fentry__() for PIE support
In reply to Hou WenlongQuoted 15 lines · Hou Wenlong
tools/objtool/arch/x86/decode.c +3
-1@@ -747,15 +747,21 @@ void arch_initial_func_cfi_state(struct cfi_init_state *state)const char *arch_nop_insn (int len) { -static const char nops[5 ][5 ] = { +static const char nops[6 ][6 ] = { {BYTES_NOP1 }, {BYTES_NOP2 }, {BYTES_NOP3 }, {BYTES_NOP4 }, {BYTES_NOP5 }, +/* +* For PIE kernel, use a 5-byte nop
tools/objtool/arch/x86/decode.c void arch_initial_func_cfi_state(struct cfi_init_state *state) · line 766
Quoted 10 lines
tools/objtool/arch/x86/decode.c +5
-1+* and 1-byte nop to keep the frace +* hooking algorithm working correct. +*/ + { BYTES_NOP5, BYTES_NOP1 }, }; -if (len <1 || len >5 ) { +if (len <1 || len >6 ) {WARN ("invalid NOP size: %d\n" , len);return NULL ; }
Like Steve already said, this is broken, we hard rely on these things being single instructions, this must absolutely be BYTES_NOP6.
And yes, then you get to fix a whole lot more.
Peter Zijlstra
maintainerFor some raison I didn't get 0/n but did get all of the others. Please keep your Cc list consistent.
Quoted 5 lines · Hou Wenlong
- It is not allowed to reference global variables in an alternative section since RIP-relative addressing is not fixed in
apply_alternatives(). Fortunately, all disallowed relocations in the alternative section can be captured by objtool. I believe that this issue can also be fixed by using objtool.
Ard Biesheuvel
maintainerRe: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
In reply to Hou WenlongQuoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().
I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define PV_VCPU_PREEMPTED_ASM \
"leaq __per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "(%rax) \n\t" \
"setne %al\n\t"
or
+#ifdef CONFIG_X86_PIE
+ " pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else
" pushq $arch_rethook_trampoline\n"
+#endif
instead of having these kludgy push/pop sequences to free up temp registers.
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
[0] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=x86-pie
Quoted 92 lines
Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/include/asm/sections.h | 5 +++++ arch/x86/kernel/module.c | 27 +++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)arch/x86/include/asm/sections.h +5
-0@@ -12,6 +12,11 @@ extern char __end_rodata_aligned[];#if defined(CONFIG_X86_64 )extern char __end_rodata_hpage_align[]; + +#ifdef CONFIG_X86_PIE +extern char __start_got[], __end_got[]; +#endif +#endif extern char __end_of_kernel_reserve[];arch/x86/kernel/module.c +27
-1@@ -129,6 +129,18 @@ int apply_relocate(Elf32_Shdr *sechdrs,return 0 ; }#else /*X86_64*/ +#ifdef CONFIG_X86_PIE +static u64 find_got_kernel_entry (Elf64_Sym *sym,const Elf64_Rela *rela) +{ +u64 *pos; + +for (pos = (u64 *)__start_got; pos < (u64 *)__end_got; pos++) +if (*pos == sym->st_value) +return (u64 )pos + rela->r_addend; +return 0 ; +} +#endif + static int __write_relocate_add(Elf64_Shdr *sechdrs, const char *strtab, unsigned int symindex,@@ -171,6 +183,7 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,case R_X86_64_64 : size =8 ;break ; +#ifndef CONFIG_X86_PIE case R_X86_64_32 :if (val != *(u32 *)&val) goto overflow;@@ -181,6 +194,13 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#else +case R_X86_64_GOTPCREL : + val =find_got_kernel_entry (sym, rel); +if (!val) + goto unexpected_got_reference; + fallthrough; +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;@@ -214,11 +234,18 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, }return 0 ; +#ifdef CONFIG_X86_PIE +unexpected_got_reference: +pr_err ("Target got entry doesn't exist in kernel got, loc %p\n" , loc); +return -ENOEXEC ; +#else overflow:pr_err ("overflow in relocation type %d val %Lx\n" , (int )ELF64_R_TYPE (rel[i].r_info), val);pr_err ("`%s' likely not compiled with -mcmodel=kernel\n" , me->name); +#endif +return -ENOEXEC ; } --2.31.1
Re: [PATCH RFC 22/43] x86/ftrace: Adapt ftrace nop patching for PIE support
In reply to Steven RostedtQuoted 28 lines · Steven Rostedt
Quoted 11 lines · On Fri, 28 Apr 2023 17:51:02 +0800 "Hou Wenlong" <[email protected]>…
From: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
When using PIE with function tracing, the compiler generates a call through the GOT (call *__fentry__@GOTPCREL). This instruction takes 6-bytes instead of 5-bytes with a relative call. And -mnop-mcount option is not implemented for -fPIE now.
If PIE is enabled, replace the 6th byte of the GOT call by a 1-byte nop so ftrace can handle the previous 5-bytes as before.
Wait! This won't work!
You can't just append another nop to fill in the blanks here. We must either have a single 6 byte nop, or we need to refactor the entire logic to something that other archs have.
The two nops means that the CPU can take it as two separate commands. There's nothing stopping the computer from preempting a task between the two. If that happens, and you modify the 1byte nop and 5byte nop with a single 6 byte command, when the task get's rescheduled, it will execute the last 5 bytes of that 6 byte command and take a general protection fault, and likely crash the machine.
Quoted 5 lines
NACK on this. It needs a better solution.
-- Steve
Hi Steve,
Sorry for not providing the original patch link:
https://lore.kernel.org/all/[email protected]/
I drop the Reviewed-by tag due to the change described in commit message.
This nop patching is only used for the first time (addr = MCOUNT) before SMP or executing code in module. And ftrace_make_call() is not modified, then we would use 5 byte direct call to replace the first 5 byte nop when tracepoint is enabled like before, it's still one instruction. So, the logic is same like before, patch the first 5 byte when tracepoint is enabled or disabled during running.
Quoted 4 lines
Quoted 4 lines
[Hou Wenlong: Adapt code change and fix wrong offset calculation in
make_nop_x86()]
Re: [PATCH RFC 21/43] x86/ftrace: Adapt assembly for PIE support
In reply to Steven RostedtQuoted 63 lines · Steven Rostedt
Quoted 31 lines · On Fri, 28 Apr 2023 17:51:01 +0800 "Hou Wenlong" <[email protected]>…
Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible.
Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/kernel/ftrace_64.S | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)arch/x86/kernel/ftrace_64.S +7
-0@@ -315,7 +315,14 @@ STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller) SYM_FUNC_START(__fentry__) CALL_DEPTH_ACCOUNT +#ifdef CONFIG_X86_PIE + pushq %r8 + leaq ftrace_stub(%rip), %r8 + cmpq %r8, ftrace_trace_function(%rip) + popq %r8 +#else cmpq $ftrace_stub, ftrace_trace_function +#endif jnz trace RETarch/x86/kernel/ftrace_64.S trace: · line 342
Quoted 8 lines
arch/x86/kernel/ftrace_64.S +1
-1@@ -329,7 +336,7 @@ trace: * ip and parent ip are used and the list function is called when * function tracing is enabled. */ - movq ftrace_trace_function, %r8 + movq ftrace_trace_function(%rip), %r8 CALL_NOSPEC r8 restore_mcount_regsI really don't want to add more updates to !DYNAMIC_FTRACE. This code only exists to make sure I don't break it for other architectures.
How about
arch/x86/Kconfig +1
-1@@ -37,7 +37,7 @@ config X86_64 config FORCE_DYNAMIC_FTRACE def_bool y - depends on X86_32 + depends on X86_32 || X86_PIE depends on FUNCTION_TRACER select DYNAMIC_FTRACE help
Quoted 2 lines
?
OK, I'll drop it. Actually, I select DYNAMIC_FTRACE when CONFIG_RETPOLINE is enabled for PIE due to the indirect call for __fentry__() in patch 34.
Thanks.
Quoted 1 line
-- Steve
Re: [PATCH RFC 33/43] objtool: Add validation for x86 PIE support
In reply to Christophe LeroyQuoted 138 lines · Christophe Leroy
Le 28/04/2023 à 11:51, Hou Wenlong a écrit :
Quoted 122 lines
[Vous ne recevez pas souvent de courriers de [email protected]. Découvrez pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification ]
For x86 PIE binary, only RIP-relative addressing is allowed, however, there are still a little absolute references of
R_X86_64_64relocation type for data section and a little absolute references ofR_X86_64_32Srelocation type inpvh_start_xen()function.Suggested-by Lai Jiangshan <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/Kconfig | 1 + scripts/Makefile.lib | 1 + tools/objtool/builtin-check.c | 4 +- tools/objtool/check.c | 82 +++++++++++++++++++++++++ tools/objtool/include/objtool/builtin.h | 1 + 5 files changed, 88 insertions(+), 1 deletion(-)arch/x86/Kconfig +1
-0@@ -2224,6 +2224,7 @@ config RELOCATABLE config X86_PIE def_bool n depends on X86_64 + select OBJTOOL if HAVE_OBJTOOL config RANDOMIZE_BASE bool "Randomize the address of the kernel image (KASLR)"scripts/Makefile.lib +1
-0@@ -270,6 +270,7 @@ objtool-args-$(CONFIG_HAVE_STATIC_CALL_INLINE) += --static-call objtool-args-$(CONFIG_HAVE_UACCESS_VALIDATION) += --uaccess objtool-args-$(CONFIG_GCOV_KERNEL) += --no-unreachable objtool-args-$(CONFIG_PREFIX_SYMBOLS) += --prefix=$(CONFIG_FUNCTION_PADDING_BYTES) +objtool-args-$(CONFIG_X86_PIE) += --pie objtool-args = $(objtool-args-y) \ $(if $(delay-objtool), --link) \tools/objtool/builtin-check.c +3
-1@@ -81,6 +81,7 @@ static const struct option check_options[] = {OPT_BOOLEAN ('t' ,"static-call" , &opts.static_call,"annotate static calls" ),OPT_BOOLEAN ('u' ,"uaccess" , &opts.uaccess,"validate uaccess rules for SMAP" ),OPT_BOOLEAN (0 ,"cfi" , &opts.cfi,"annotate kernel control flow integrity (kCFI) function preambles" ), +OPT_BOOLEAN (0 ,"pie" , &opts.pie,"validate addressing rules for PIE" ),OPT_CALLBACK_OPTARG (0 ,"dump" ,NULL ,NULL ,"orc" ,"dump metadata" , parse_dump), OPT_GROUP("Options:"),@@ -137,7 +138,8 @@ static bool opts_valid(void) opts.sls || opts.stackval || opts.static_call || - opts.uaccess) { + opts.uaccess || + opts.pie) {if (opts.dump_orc) {ERROR ("--dump can't be combined with other options" );return false;tools/objtool/check.c +42
-0@@ -131,6 +131,27 @@ static struct instruction *prev_insn_same_sym(struct objtool_file *file,for (insn =next_insn_same_sec (file, insn); insn; \ insn =next_insn_same_sec (file, insn)) +static struct instruction *find_insn_containing(struct objtool_file *file, +struct section *sec, +unsigned long offset) +{ +struct instruction *insn; + + insn =find_insn (file, sec,0 ); +if (!insn) +return NULL ; + +sec_for_each_insn_from (file, insn) { +if (insn->offset > offset) +return NULL ; +if (insn->offset <= offset && (insn->offset + insn->len) > offset) +return insn; + } + +return NULL ; +} + +static inline struct symbol *insn_call_dest (struct instruction *insn) { if (insn->type == INSN_JUMP_DYNAMIC ||@@ -4529,6 +4550,61 @@ static int validate_reachable_instructions(struct objtool_file *file)return 0 ; } +static int is_in_pvh_code (struct instruction *insn) +{ +struct symbol *sym = insn->sym; + +return sym && !strcmp (sym->name,"pvh_start_xen" ); +} + +static int validate_pie (struct objtool_file *file) +{ +struct section *sec; +struct reloc *reloc; +struct instruction *insn; +int warnings =0 ; + +for_each_sec (file, sec) { +if (!sec->reloc) +continue ; +if (!(sec->sh.sh_flags &SHF_ALLOC )) +continue ; + +list_for_each_entry (reloc, &sec->reloc->reloc_list, list) {tools/objtool/check.c static int validate_reachable_instructions(struct objtool_file *file) · line 4583
Quoted 10 lines
tools/objtool/check.c +10
-0+switch (reloc->type) { +case R_X86_64_NONE : +case R_X86_64_PC32 : +case R_X86_64_PLT32 : +case R_X86_64_64 : +case R_X86_64_PC64 : +case R_X86_64_GOTPCREL : +break ; +case R_X86_64_32 : +case R_X86_64_32S :That looks very specific to X86, should it go at another place ?
Quoted 5 lines
If it can work for any architecture, can you add generic macros, just like commit
c1449735211d("objtool: Use macros to define arch specific reloc types") then commitc984aef8c832("objtool/powerpc: Add --mcount specific implementation") ?
Get it, I'll refactor it and move code into X86 directory.
Thanks.
Quoted 55 lines
Quoted 55 lines
tools/objtool/check.c +30
-0+ insn =find_insn_containing (file, sec, reloc->offset); +if (!insn) { +WARN ("can't find relocate insn near %s+0x%lx" , + sec->name, reloc->offset); + }else { +if (is_in_pvh_code (insn)) +break ; +WARN ("insn at %s+0x%lx is not compatible with PIE" , + sec->name, insn->offset); + } + warnings++; +break ; +default : +WARN ("unexpected relocation type %d at %s+0x%lx" , + reloc->type, sec->name, reloc->offset); + warnings++; +break ; + } + } + } + +return warnings; +} +int check (struct objtool_file *file) {int ret, warnings =0 ;@@ -4673,6 +4749,12 @@ int check(struct objtool_file *file) warnings += ret; } +if (opts.pie) { + ret =validate_pie (file); +if (ret <0 ) +return ret; + warnings += ret; + }if (opts.stats) {printf ("nr_insns_visited: %ld\n" , nr_insns_visited);tools/objtool/include/objtool/builtin.h +1
-1@@ -26,6 +26,7 @@ struct opts {bool uaccess;int prefix;bool cfi; +bool pie;/* options: */ bool backtrace; --2.31.1
Re: [PATCH RFC 33/43] objtool: Add validation for x86 PIE support
In reply to Peter ZijlstraQuoted 48 lines · Peter Zijlstra
Quoted 19 lines · Christophe Leroy
Quoted 19 lines
tools/objtool/check.c +11
-0@@ -131,6 +131,27 @@ static struct instruction *prev_insn_same_sym(struct objtool_file *file,for (insn =next_insn_same_sec (file, insn); insn; \ insn =next_insn_same_sec (file, insn)) +static struct instruction *find_insn_containing(struct objtool_file *file, +struct section *sec, +unsigned long offset) +{ +struct instruction *insn; + + insn =find_insn (file, sec,0 ); +if (!insn) +return NULL ; + +sec_for_each_insn_from (file, insn) {Quoted 8 lines
tools/objtool/check.c static struct instruction *prev_insn_same_sym(struct objtool_file *file, · line 152
Quoted 8 lines
tools/objtool/check.c +8
-0+if (insn->offset > offset) +return NULL ; +if (insn->offset <= offset && (insn->offset + insn->len) > offset) +return insn; + } + +return NULL ; +}Urgh, this is horrendous crap. Yes you're only using it in case of a warning, but adding a function like this makes it appear like it's actually sane to use.
A far better implementation -- but still not stellar -- would be something like:
sym = find_symbol_containing(sec, offset); if (!sym) // fail sym_for_each_insn(file, sym, insn) { ... }But given
insn_hashusessec_offset_hash()you can do something similar tofind_reloc_by_dest_range()
Quoted 13 lines
start = offset - (INSN_MAX_SIZE - 1);
for_offset_range(o, start, start + INSN_MAX_SIZE) {
hash_for_each_possible(file->insn_hash, insn, hash, sec_offset_hash(sec, o)) {
if (insn->sec != sec)
continue;
if (insn->offset <= offset &&
insn->offset + inns->len > offset)
return insn;
}
}
return NULL;Thanks for your suggestion, I'll pick it in the next version.
Quoted 50 lines
Quoted 43 lines
Quoted 30 lines
tools/objtool/check.c +23
-0+ +static inline struct symbol *insn_call_dest (struct instruction *insn) { if (insn->type == INSN_JUMP_DYNAMIC ||@@ -4529,6 +4550,61 @@ static int validate_reachable_instructions(struct objtool_file *file)return 0 ; } +static int is_in_pvh_code (struct instruction *insn) +{ +struct symbol *sym = insn->sym; + +return sym && !strcmp (sym->name,"pvh_start_xen" ); +} + +static int validate_pie (struct objtool_file *file) +{ +struct section *sec; +struct reloc *reloc; +struct instruction *insn; +int warnings =0 ; + +for_each_sec (file, sec) { +if (!sec->reloc) +continue ; +if (!(sec->sh.sh_flags &SHF_ALLOC )) +continue ; + +list_for_each_entry (reloc, &sec->reloc->reloc_list, list) {tools/objtool/check.c static int validate_reachable_instructions(struct objtool_file *file) · line 4583
Quoted 10 lines
tools/objtool/check.c +10
-0+switch (reloc->type) { +case R_X86_64_NONE : +case R_X86_64_PC32 : +case R_X86_64_PLT32 : +case R_X86_64_64 : +case R_X86_64_PC64 : +case R_X86_64_GOTPCREL : +break ; +case R_X86_64_32 : +case R_X86_64_32S :That looks very specific to X86, should it go at another place ?
Quoted 4 lines
If it can work for any architecture, can you add generic macros, just like commit
c1449735211d("objtool: Use macros to define arch specific reloc types") then commitc984aef8c832("objtool/powerpc: Add --mcount specific implementation") ?Yes, this should be something like
arch_PIE_reloc()or so. Similar toarch_pc_relative_reloc().
Ard Biesheuvel
maintainerRe: [PATCH RFC 25/43] x86/mm: Make the x86 GOT read-only
In reply to Hou WenlongQuoted 47 lines · Hou Wenlong
From: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
The GOT is changed during early boot when relocations are applied. Make it read-only directly. This table exists only for PIE binary. Since weak symbol reference would always be GOT reference, there are 8 entries in GOT, but only one entry for
__fentry__()is in use. Other GOT references have been optimized by linker.[Hou Wenlong: Change commit message and skip GOT size check]
Signed-off-by Thomas Garnier <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/kernel/vmlinux.lds.S | 2 ++ include/asm-generic/vmlinux.lds.h | 12 ++++++++++++ 2 files changed, 14 insertions(+)arch/x86/kernel/vmlinux.lds.S +2
-0@@ -462,6 +462,7 @@ SECTIONS #endif "Unexpected GOT/PLT entries detected!") +#ifndef CONFIG_X86_PIE /* * Sections that should stay zero sized, which is safer to * explicitly check instead of blindly discarding.@@ -470,6 +471,7 @@ SECTIONS *(.got) *(.igot.*) } ASSERT(SIZEOF(.got) == 0, "Unexpected GOT entries detected!") +#endif .plt : { *(.plt) *(.plt.*) *(.iplt)include/asm-generic/vmlinux.lds.h +0
-0
include/asm-generic/vmlinux.lds.h line 445
Quoted 5 lines
include/asm-generic/vmlinux.lds.h +2
-0@@ -441,6 +441,17 @@ __end_ro_after_init = .;#endif +#ifdef CONFIG_X86_PIE +#define RO_GOT_X86
Please don't put X86 specific stuff in generic code.
include/asm-generic/vmlinux.lds.h line 454
Quoted 9 lines
include/asm-generic/vmlinux.lds.h +9
-0+ .got : AT(ADDR(.got) - LOAD_OFFSET) { \ + __start_got = .; \ + *(.got) *(.igot.*); \ + __end_got = .; \ + } +#else +#define RO_GOT_X86 +#endif +
I don't think it makes sense for this definition to be conditional. You can include it conditionally from the x86 code, but even that seems unnecessary, given that it will be empty otherwise.
Quoted 14 lines
include/asm-generic/vmlinux.lds.h +1
-1/* * .kcfi_traps contains a list KCFI trap locations. */ @@ -486,6 +497,7 @@ BOUNDED_SECTION_PRE_LABEL(.pci_fixup_suspend_late, _pci_fixups_suspend_late, __start, __end) \ } \ \ +RO_GOT_X86 \FW_LOADER_BUILT_IN_DATA \ TRACEDATA \ \ --2.31.1
Re: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler
In reply to Hou WenlongQuoted 37 lines · Hou Wenlong
From: Brian Gerst <[email protected]>
From: Brian Gerst <[email protected]>
If the compiler supports it, use a standard per-cpu variable for the stack protector instead of the old fixed location. Keep the fixed location code for compatibility with older compilers.
[Hou Wenlong: Disable it on Clang, adapt new code change and adapt missing GS set up path in
pvh_start_xen()]Signed-off-by Brian Gerst <[email protected]>Co-developed-by Hou Wenlong <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/Kconfig | 12 ++++++++++++ arch/x86/Makefile | 21 ++++++++++++++------- arch/x86/entry/entry_64.S | 6 +++++- arch/x86/include/asm/processor.h | 17 ++++++++++++----- arch/x86/include/asm/stackprotector.h | 16 +++++++--------- arch/x86/kernel/asm-offsets_64.c | 2 +- arch/x86/kernel/cpu/common.c | 15 +++++++-------- arch/x86/kernel/head_64.S | 16 ++++++++++------ arch/x86/kernel/vmlinux.lds.S | 4 +++- arch/x86/platform/pvh/head.S | 8 ++++++++ arch/x86/xen/xen-head.S | 14 +++++++++----- 11 files changed, 88 insertions(+), 43 deletions(-)arch/x86/Kconfig +0
-0
arch/x86/Kconfig config CC_HAS_SANE_STACKPROTECTOR · line 418
Quoted 9 lines
arch/x86/Kconfig +6
-0@@ -410,6 +410,18 @@ config CC_HAS_SANE_STACKPROTECTOR the compiler produces broken code or if it does not let us control the segment on 32-bit kernels. +config CC_HAS_CUSTOMIZED_STACKPROTECTOR + bool + # Although clang supports -mstack-protector-guard-reg option, it + # would generate GOT reference for __stack_chk_guard even with + # -fno-PIE flag. + default y if (!CC_IS_CLANG && $(cc-option,-mstack-protector-guard-reg=gs))
Hi Hou,
I've filed this bug against LLVM and will work with LLVM folks at Intel to resolve:
https://github.com/llvm/llvm-project/issues/62481
Can you please review that report and let me know here or there if I missed anything? Would you also mind including a link to that in the comments in the next version of this patch?
Less relevant issues I filed looking at some related codegen:
https://github.com/llvm/llvm-project/issues/62482
https://github.com/llvm/llvm-project/issues/62480
And we should probably look into:
https://github.com/llvm/llvm-project/issues/22476
Signature · 2 lines
Thanks, ~Nick Desaulniers
Juergen Gross
maintainerRe: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler
In reply to Hou WenlongQuoted 31 lines · Hou Wenlong
From: Brian Gerst <[email protected]>
From: Brian Gerst <[email protected]>
If the compiler supports it, use a standard per-cpu variable for the stack protector instead of the old fixed location. Keep the fixed location code for compatibility with older compilers.
[Hou Wenlong: Disable it on Clang, adapt new code change and adapt missing GS set up path in
pvh_start_xen()]Signed-off-by Brian Gerst <[email protected]>Co-developed-by Hou Wenlong <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/Kconfig | 12 ++++++++++++ arch/x86/Makefile | 21 ++++++++++++++------- arch/x86/entry/entry_64.S | 6 +++++- arch/x86/include/asm/processor.h | 17 ++++++++++++----- arch/x86/include/asm/stackprotector.h | 16 +++++++--------- arch/x86/kernel/asm-offsets_64.c | 2 +- arch/x86/kernel/cpu/common.c | 15 +++++++-------- arch/x86/kernel/head_64.S | 16 ++++++++++------ arch/x86/kernel/vmlinux.lds.S | 4 +++- arch/x86/platform/pvh/head.S | 8 ++++++++ arch/x86/xen/xen-head.S | 14 +++++++++----- 11 files changed, 88 insertions(+), 43 deletions(-)
...
Quoted 13 lines
arch/x86/xen/xen-head.S +1
-4@@ -51,15 +51,19 @@ SYM_CODE_START(startup_xen) leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + /*
arch/x86/xen/xen-head.S SYM_CODE_START(startup_xen) · line 63
Quoted 10 lines
arch/x86/xen/xen-head.S +5
-1+ * Set up GS base. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax +#if defined(CONFIG_STACKPROTECTOR_FIXED) + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#elif defined(CONFIG_SMP) + movabs $__per_cpu_load, %rdx
Shouldn't above 2 targets be %rax?
arch/x86/xen/xen-head.S SYM_CODE_START(startup_xen) · line 69
Quoted 6 lines
arch/x86/xen/xen-head.S +3
-0+#else + xorl %eax, %eax +#endif cdq wrmsr
Juergen
Re: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler
Quoted 50 lines · Juergen Gross
Quoted 31 lines · Hou Wenlong
From: Brian Gerst <[email protected]>
From: Brian Gerst <[email protected]>
If the compiler supports it, use a standard per-cpu variable for the stack protector instead of the old fixed location. Keep the fixed location code for compatibility with older compilers.
[Hou Wenlong: Disable it on Clang, adapt new code change and adapt missing GS set up path in
pvh_start_xen()]Signed-off-by Brian Gerst <[email protected]>Co-developed-by Hou Wenlong <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/Kconfig | 12 ++++++++++++ arch/x86/Makefile | 21 ++++++++++++++------- arch/x86/entry/entry_64.S | 6 +++++- arch/x86/include/asm/processor.h | 17 ++++++++++++----- arch/x86/include/asm/stackprotector.h | 16 +++++++--------- arch/x86/kernel/asm-offsets_64.c | 2 +- arch/x86/kernel/cpu/common.c | 15 +++++++-------- arch/x86/kernel/head_64.S | 16 ++++++++++------ arch/x86/kernel/vmlinux.lds.S | 4 +++- arch/x86/platform/pvh/head.S | 8 ++++++++ arch/x86/xen/xen-head.S | 14 +++++++++----- 11 files changed, 88 insertions(+), 43 deletions(-)...
Quoted 15 lines
arch/x86/xen/xen-head.S +2
-4@@ -51,15 +51,19 @@ SYM_CODE_START(startup_xen) leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + /* + * Set up GS base. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */
Quoted 9 lines
arch/x86/xen/xen-head.S SYM_CODE_START(startup_xen) · line 63
Quoted 6 lines
arch/x86/xen/xen-head.S +4
-1movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax +#if defined(CONFIG_STACKPROTECTOR_FIXED) + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#elif defined(CONFIG_SMP) + movabs $__per_cpu_load, %rdxShouldn't above 2 targets be %rax?
Ah yes, my mistake. I didn't test it on XEN guest, sorry, I'll test XEN guest before the next submission.
Thanks.
Quoted 14 lines
arch/x86/xen/xen-head.S SYM_CODE_START(startup_xen) · line 68
Quoted 5 lines
arch/x86/xen/xen-head.S +3
-0+#else + xorl %eax, %eax +#endif cdq wrmsrJuergen
pub 2048R/28BF132F 2014-06-02 Juergen Gross <[email protected]>
uid Juergen Gross <[email protected]> uid Juergen Gross <[email protected]> uid Juergen Gross <[email protected]>sub 2048R/16375B53 2014-06-02
Re: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler
In reply to Nick DesaulniersQuoted 41 lines · Nick Desaulniers
Quoted 40 lines · Hou Wenlong
From: Brian Gerst <[email protected]>
From: Brian Gerst <[email protected]>
If the compiler supports it, use a standard per-cpu variable for the stack protector instead of the old fixed location. Keep the fixed location code for compatibility with older compilers.
[Hou Wenlong: Disable it on Clang, adapt new code change and adapt missing GS set up path in
pvh_start_xen()]Signed-off-by Brian Gerst <[email protected]>Co-developed-by Hou Wenlong <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/Kconfig | 12 ++++++++++++ arch/x86/Makefile | 21 ++++++++++++++------- arch/x86/entry/entry_64.S | 6 +++++- arch/x86/include/asm/processor.h | 17 ++++++++++++----- arch/x86/include/asm/stackprotector.h | 16 +++++++--------- arch/x86/kernel/asm-offsets_64.c | 2 +- arch/x86/kernel/cpu/common.c | 15 +++++++-------- arch/x86/kernel/head_64.S | 16 ++++++++++------ arch/x86/kernel/vmlinux.lds.S | 4 +++- arch/x86/platform/pvh/head.S | 8 ++++++++ arch/x86/xen/xen-head.S | 14 +++++++++----- 11 files changed, 88 insertions(+), 43 deletions(-)arch/x86/Kconfig +0
-0@@ -410,6 +410,18 @@ config CC_HAS_SANE_STACKPROTECTOR the compiler produces broken code or if it does not let us control the segment on 32-bit kernels.
Quoted 15 lines
arch/x86/Kconfig config CC_HAS_SANE_STACKPROTECTOR · line 418
Quoted 6 lines
arch/x86/Kconfig +6
-0+config CC_HAS_CUSTOMIZED_STACKPROTECTOR + bool + # Although clang supports -mstack-protector-guard-reg option, it + # would generate GOT reference for __stack_chk_guard even with + # -fno-PIE flag. + default y if (!CC_IS_CLANG && $(cc-option,-mstack-protector-guard-reg=gs))Hi Hou,
I've filed this bug against LLVM and will work with LLVM folks at Intel to resolve:
https://github.com/llvm/llvm-project/issues/62481
Can you please review that report and let me know here or there if I missed anything? Would you also mind including a link to that in the comments in the next version of this patch?
Hi Nick,
Thanks for your help, I'll include the link in the next version. Actually, I had post an issue on github too when I test the patch on LLVM. But no replies. :(.
https://github.com/llvm/llvm-project/issues/60116
There is another problem I met for this patch, some unexpected code are generated:
do_one_initcall: (init/main.o)
......
movq __stack_chk_guard(%rip), %rax
movq %rax,0x2b0(%rsp)
The complier generates wrong instruction, no GOT reference and gs register. I only see it in init/main.c file. I have tried to move the function into another file and compiled it with same cflags. It could generate right instruction for the function in another file.
The LLVM chain toolsare built by myself:
clang version 15.0.7 (https://github.com/llvm/llvm-project.git8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
Quoted 8 lines
Less relevant issues I filed looking at some related codegen:
https://github.com/llvm/llvm-project/issues/62482
https://github.com/llvm/llvm-project/issues/62480And we should probably look into:
https://github.com/llvm/llvm-project/issues/22476
Except for per-cpu stack canary patch, there is another issue I post on
github: https://github.com/llvm/llvm-project/issues/60096
The related patch is:
https://lore.kernel.org/lkml/175116f75c38c15d8d73a03301eab805fea13a0a.1682673543.git.houwenlong.hwl@antgroup.com/
I couldn't find the related documentation about that, hope you can help me too.
One more problem that I didn't post is:
https://lore.kernel.org/lkml/8d6bbaf66b90cf1a8fd2c5da98f5e094b9ffcb27.1682673543.git.houwenlong.hwl@antgroup.com/
Quoted 3 lines
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler
Quoted 19 lines · Hou Wenlong
Quoted 9 lines · Nick Desaulniers
arch/x86/Kconfig config CC_HAS_SANE_STACKPROTECTOR · line 418
Quoted 7 lines · Hou Wenlong
arch/x86/Kconfig +6
-0+config CC_HAS_CUSTOMIZED_STACKPROTECTOR + bool + # Although clang supports -mstack-protector-guard-reg option, it + # would generate GOT reference for __stack_chk_guard even with + # -fno-PIE flag. + default y if (!CC_IS_CLANG && $(cc-option,-mstack-protector-guard-reg=gs))Quoted 8 lines
Hi Hou,
I've filed this bug against LLVM and will work with LLVM folks at Intel to resolve:
https://github.com/llvm/llvm-project/issues/62481
Can you please review that report and let me know here or there if I missed anything? Would you also mind including a link to that in the comments in the next version of this patch?
Quoted 5 lines
Hi Nick,
Thanks for your help, I'll include the link in the next version. Actually, I had post an issue on github too when I test the patch on LLVM. But no replies. :(.
Ah, sorry about that. The issue tracker is pretty high volume and stuff gets missed. With many users comes many bug reports. We could be better about triage though. If it's specific to the Linux kernel,
https://github.com/ClangBuiltLinux/linux/issues is a better issue tracker to use; we can move bug reports upstream to
https://github.com/llvm/llvm-project/issues/ when necessary. It's linked off of clangbuiltlinux.github.io if you lose it.
Quoted 10 lines
https://github.com/llvm/llvm-project/issues/60116
There is another problem I met for this patch, some unexpected code are generated:
do_one_initcall: (init/main.o)
......
movq__stack_chk_guard(%rip), %rax
movq %rax,0x2b0(%rsp)
Quoted 4 lines
The complier generates wrong instruction, no GOT reference and gs register. I only see it in
init/main.cfile. I have tried to move the function into another file and compiled it with same cflags. It could generate right instruction for the function in another file.
The wrong instruction or the wrong operand? This is loading the canary into the stack slot in the fn prolog. Perhaps the expected cflag is not getting set (or being removed) from init/main.c? You should be able to do:
$ make LLVM=1 init/main.o V=1
to see how clang was invoked to see if the expected flag was there, or not.
Quoted 4 lines
The LLVM chain toolsare built by myself:
clang version 15.0.7 (https://github.com/llvm/llvm-project.git8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
Perhaps worth rebuilding with top of tree, which is clang 17.
Quoted 5 lines
Quoted 5 lines
Less relevant issues I filed looking at some related codegen:
https://github.com/llvm/llvm-project/issues/62482
https://github.com/llvm/llvm-project/issues/62480
Quoted 7 lines
Quoted 4 lines
And we should probably look into:
https://github.com/llvm/llvm-project/issues/22476Except for per-cpu stack canary patch, there is another issue I post on
github: https://github.com/llvm/llvm-project/issues/60096
Thanks, I'll bring that up with Intel, too.
Quoted 9 lines
The related patch is:
https://lore.kernel.org/lkml/175116f75c38c15d8d73a03301eab805fea13a0a.1682673543.git.houwenlong.hwl@antgroup.com/I couldn't find the related documentation about that, hope you can help me too.
One more problem that I didn't post is:
https://lore.kernel.org/lkml/8d6bbaf66b90cf1a8fd2c5da98f5e094b9ffcb27.1682673543.git.houwenlong.hwl@antgroup.com/
Mind filing another bug for this in llvm's issue tracker? We can discuss there if LLD needs to be doing something different.
Thanks for uncovering these and helping us get them fixed up!
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler
Quoted 91 lines · Nick Desaulniers
Quoted 19 lines · Hou Wenlong
Quoted 9 lines · Nick Desaulniers
arch/x86/Kconfig config CC_HAS_SANE_STACKPROTECTOR · line 418
Quoted 7 lines · Hou Wenlong
arch/x86/Kconfig +6
-0+config CC_HAS_CUSTOMIZED_STACKPROTECTOR + bool + # Although clang supports -mstack-protector-guard-reg option, it + # would generate GOT reference for __stack_chk_guard even with + # -fno-PIE flag. + default y if (!CC_IS_CLANG && $(cc-option,-mstack-protector-guard-reg=gs))Quoted 8 lines
Hi Hou,
I've filed this bug against LLVM and will work with LLVM folks at Intel to resolve:
https://github.com/llvm/llvm-project/issues/62481
Can you please review that report and let me know here or there if I missed anything? Would you also mind including a link to that in the comments in the next version of this patch?Quoted 5 lines
Hi Nick,
Thanks for your help, I'll include the link in the next version. Actually, I had post an issue on github too when I test the patch on LLVM. But no replies. :(.
Ah, sorry about that. The issue tracker is pretty high volume and stuff gets missed. With many users comes many bug reports. We could be better about triage though. If it's specific to the Linux kernel,
https://github.com/ClangBuiltLinux/linux/issues is a better issue tracker to use; we can move bug reports upstream to
https://github.com/llvm/llvm-project/issues/ when necessary. It's linked off of clangbuiltlinux.github.io if you lose it.Quoted 10 lines
https://github.com/llvm/llvm-project/issues/60116
There is another problem I met for this patch, some unexpected code are generated:
do_one_initcall: (init/main.o)
......
movq__stack_chk_guard(%rip), %rax
movq %rax,0x2b0(%rsp)Quoted 4 lines
The complier generates wrong instruction, no GOT reference and gs register. I only see it in
init/main.cfile. I have tried to move the function into another file and compiled it with same cflags. It could generate right instruction for the function in another file.The wrong instruction or the wrong operand? This is loading the canary into the stack slot in the fn prolog. Perhaps the expected cflag is not getting set (or being removed) from
init/main.c? You should be able to do:$ make LLVM=1 init/main.o V=1to see how clang was invoked to see if the expected flag was there, or not.
Quoted 4 lines
The LLVM chain toolsare built by myself:
clang version 15.0.7 (https://github.com/llvm/llvm-project.git8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)Perhaps worth rebuilding with top of tree, which is clang 17.
Quoted 5 lines
Quoted 5 lines
Less relevant issues I filed looking at some related codegen:
https://github.com/llvm/llvm-project/issues/62482
https://github.com/llvm/llvm-project/issues/62480Quoted 7 lines
Quoted 4 lines
And we should probably look into:
https://github.com/llvm/llvm-project/issues/22476Except for per-cpu stack canary patch, there is another issue I post on
github: https://github.com/llvm/llvm-project/issues/60096Thanks, I'll bring that up with Intel, too.
Quoted 9 lines
The related patch is:
https://lore.kernel.org/lkml/175116f75c38c15d8d73a03301eab805fea13a0a.1682673543.git.houwenlong.hwl@antgroup.com/I couldn't find the related documentation about that, hope you can help me too.
One more problem that I didn't post is:
https://lore.kernel.org/lkml/8d6bbaf66b90cf1a8fd2c5da98f5e094b9ffcb27.1682673543.git.houwenlong.hwl@antgroup.com/
Quoted 7 lines
Mind filing another bug for this in llvm's issue tracker? We can discuss there if LLD needs to be doing something different.
Thanks for uncovering these and helping us get them fixed up! --
Thanks,
~Nick Desaulniers
In my opinion, Clang's behavior is working as intended. The Linux kernel should support R_X86_64_REX_GOTPCRELX, and the solution is
straightforward: treat R_X86_64_REX_GOTPCRELX the same way as R_X86_64_PC32 (-shared -Bsymbolic), assuming that every symbol is defined, which means that every symbol is non-preemptible.
Clang's -fno-pic option chooses R_X86_64_REX_GOTPCRELX which is correct, although it differs from GCC's -fno-pic option.
The compiler doesn't know whether __stack_chk_guard will be provided by the main executable (libc.a) or a shared object (libc.so, available on some ports of glibc but not x86, on musl this is available for all ports).
(Also see __stack_chk_guard on
https://maskray.me/blog/2022-12-18-control-flow-integrity)
If an R_X86_64_32 relocation is used and __stack_chk_guard is
defined by a shared object, copy relocation.
We will need an ELF hack called [copy relocation](https://maskray.me/blog/2021-01-09-copy-relocations-canonical-plt-entries-and-protected).
The instruction movq __stack_chk_guard@GOTPCREL(%rip), %rbx produces
an R_X86_64_REX_GOTPCRELX relocation.
If __stack_chk_guard is non-preemptible, linkers can [optimize the
access to be direct](https://maskray.me/blog/2021-08-29-all-about-global-offset-table#got-optimization).
Although we could technically use the -fno-direct-access-external-data option to switch between R_X86_64_REX_GOTPCRELX and R_X86_64_32, I think there is no justification to complicate the compiler.
Signature · 1 line
宋方睿
Quoted 4 lines · Peter Zijlstra
For some raison I didn't get 0/n but did get all of the others. Please keep your Cc list consistent.
Sorry, I'll pay attention next time.
Quoted 9 lines · Hou Wenlong
Quoted 6 lines
- It is not allowed to reference global variables in an alternative section since RIP-relative addressing is not fixed in
apply_alternatives(). Fortunately, all disallowed relocations in the alternative section can be captured by objtool. I believe that this issue can also be fixed by using objtool.
Thank you for your patch. However, it's more complicated for call depth tracking case. Although, the per-cpu variable in the alternative section is relocated, but the content of the "skl_call_thunk_template" is copied into another place, so the offset is still incorrect. I'm not sure if this case is common or not. Since the destination is clear, I could do relocation here as well, but it would make the code more complicated.
Thanks!
Re: [PATCH RFC 16/43] x86-64: Use per-cpu stack canary if supported by compiler
In reply to Nick DesaulniersQuoted 49 lines · Nick Desaulniers
Quoted 19 lines · Hou Wenlong
Quoted 9 lines · Nick Desaulniers
arch/x86/Kconfig config CC_HAS_SANE_STACKPROTECTOR · line 418
Quoted 7 lines · Hou Wenlong
arch/x86/Kconfig +6
-0+config CC_HAS_CUSTOMIZED_STACKPROTECTOR + bool + # Although clang supports -mstack-protector-guard-reg option, it + # would generate GOT reference for __stack_chk_guard even with + # -fno-PIE flag. + default y if (!CC_IS_CLANG && $(cc-option,-mstack-protector-guard-reg=gs))Quoted 8 lines
Hi Hou,
I've filed this bug against LLVM and will work with LLVM folks at Intel to resolve:
https://github.com/llvm/llvm-project/issues/62481
Can you please review that report and let me know here or there if I missed anything? Would you also mind including a link to that in the comments in the next version of this patch?Quoted 5 lines
Hi Nick,
Thanks for your help, I'll include the link in the next version. Actually, I had post an issue on github too when I test the patch on LLVM. But no replies. :(.
Ah, sorry about that. The issue tracker is pretty high volume and stuff gets missed. With many users comes many bug reports. We could be better about triage though. If it's specific to the Linux kernel,
https://github.com/ClangBuiltLinux/linux/issues is a better issue tracker to use; we can move bug reports upstream to
https://github.com/llvm/llvm-project/issues/ when necessary. It's linked off of clangbuiltlinux.github.io if you lose it.Quoted 10 lines
https://github.com/llvm/llvm-project/issues/60116
There is another problem I met for this patch, some unexpected code are generated:
do_one_initcall: (init/main.o)
......
movq__stack_chk_guard(%rip), %rax
movq %rax,0x2b0(%rsp)Quoted 4 lines
The complier generates wrong instruction, no GOT reference and gs register. I only see it in
init/main.cfile. I have tried to move the function into another file and compiled it with same cflags. It could generate right instruction for the function in another file.
Quoted 9 lines
The wrong instruction or the wrong operand? This is loading the canary into the stack slot in the fn prolog. Perhaps the expected cflag is not getting set (or being removed) from
init/main.c? You should be able to do:$ make LLVM=1 init/main.o V=1to see how clang was invoked to see if the expected flag was there, or not.
Hi Nick,
The ouput is:
clang -Wp,-MMD,init/.main.o.d -nostdinc -I./arch/x86/include
-I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Werror -fmacro-prefix-map=./= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -funsigned-char -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -Werror=option-ignored -Werror=unused-command-line-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=branch -fno-jump-tables -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mstack-protector-guard-reg=gs -mstack-protector-guard-symbol=__stack_chk_guard -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-thunk -mfunction-return=thunk-extern -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang -fno-stack-clash-protection -falign-functions=16 -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -Wimplicit-fallthrough -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wformat-extra-args -Wformat-invalid-specifier -Wformat-zero-length -Wnonnull -Wformat-insufficient-args -Wno-sign-compare -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access -fno-function-sections -fno-data-sections -DKBUILD_MODFILE='"init/main"' -DKBUILD_BASENAME='"main"' -DKBUILD_MODNAME='"main"' -D__KBUILD_MODNAME=kmod_main -c -o init/main.o init/main.c
I see the expected flags in the ouput. But the generated code is wrong:00000000000006e0 <do_one_initcall>: (init/main.o)
.......
6ff: 48 8b 05 00 00 00 00 movq (%rip), %rax # 0x706 <do_one_initcall+0x26>
0000000000000702: R_X86_64_PC32 __stack_chk_guard-0x4
706: 48 89 84 24 e0 02 00 00 movq %rax, 736(%rsp)The expected generated code should be:
0000000000000010 <name_to_dev_t>: (init/do_mounts.o)
......
2c: 4c 8b 25 00 00 00 00 movq (%rip), %r12 # 0x33 <name_to_dev_t+0x23>
000000000000002f: R_X86_64_REX_GOTPCRELX __stack_chk_guard-0x4
33: 65 49 8b 04 24 movq %gs:(%r12), %rax
38: 48 89 44 24 30 movq %rax, 48(%rsp)Actually, this is the main reason why I disable per-cpu stack canary on LLVM. This patch could be picked separately, if you have time to help me find out the reason.
Thanks.
Quoted 39 lines
Quoted 4 lines
The LLVM chain toolsare built by myself:
clang version 15.0.7 (https://github.com/llvm/llvm-project.git8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)Perhaps worth rebuilding with top of tree, which is clang 17.
Quoted 5 lines
Quoted 5 lines
Less relevant issues I filed looking at some related codegen:
https://github.com/llvm/llvm-project/issues/62482
https://github.com/llvm/llvm-project/issues/62480Quoted 7 lines
Quoted 4 lines
And we should probably look into:
https://github.com/llvm/llvm-project/issues/22476Except for per-cpu stack canary patch, there is another issue I post on
github: https://github.com/llvm/llvm-project/issues/60096Thanks, I'll bring that up with Intel, too.
Quoted 9 lines
The related patch is:
https://lore.kernel.org/lkml/175116f75c38c15d8d73a03301eab805fea13a0a.1682673543.git.houwenlong.hwl@antgroup.com/I couldn't find the related documentation about that, hope you can help me too.
One more problem that I didn't post is:
https://lore.kernel.org/lkml/8d6bbaf66b90cf1a8fd2c5da98f5e094b9ffcb27.1682673543.git.houwenlong.hwl@antgroup.com/Mind filing another bug for this in llvm's issue tracker? We can discuss there if LLD needs to be doing something different.
Thanks for uncovering these and helping us get them fixed up!
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
In reply to Ard BiesheuvelQuoted 45 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define
PV_VCPU_PREEMPTED_ASM\
"leaq__per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, "__stringify(KVM_STEAL_TIME_preempted)"(%rax) \n\t" \
"setne %al\n\t"or
arch/x86/kernel/module.c +3
-0+#ifdef CONFIG_X86_PIE +" pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else" pushq $arch_rethook_trampoline\n"
+#endifinstead of having these kludgy push/pop sequences to free up temp registers.
Quoted 9 lines
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
[0] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=x86-pie
Hi Ard,
Thanks for providing the link, it has been very helpful for me as I am new to the topic of compilers. One key difference I noticed is that you linked the kernel with "-pie" instead of "--emit-reloc". I also noticed that Thomas' initial patchset[0] used "-pie", but in RFC v3 [1], it switched to "--emit-reloc" in order to reduce dynamic relocation space on mapped memory.
The another issue is that it requires the addition of the "-mrelax-relocations=no" option to support older compilers and linkers. R_X86_64_GOTPCRELX and R_X86_64_REX_GOTPCRELX relocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option in arch/x86/boot/compressed/Makefile with the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Thanks.
[0] https://yhbt.net/lore/all/[email protected]
[1] https://yhbt.net/lore/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]/
Quoted 92 lines
Quoted 92 lines
Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/include/asm/sections.h | 5 +++++ arch/x86/kernel/module.c | 27 +++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)arch/x86/include/asm/sections.h +5
-0@@ -12,6 +12,11 @@ extern char __end_rodata_aligned[];#if defined(CONFIG_X86_64 )extern char __end_rodata_hpage_align[]; + +#ifdef CONFIG_X86_PIE +extern char __start_got[], __end_got[]; +#endif +#endif extern char __end_of_kernel_reserve[];arch/x86/kernel/module.c +27
-1@@ -129,6 +129,18 @@ int apply_relocate(Elf32_Shdr *sechdrs,return 0 ; }#else /*X86_64*/ +#ifdef CONFIG_X86_PIE +static u64 find_got_kernel_entry (Elf64_Sym *sym,const Elf64_Rela *rela) +{ +u64 *pos; + +for (pos = (u64 *)__start_got; pos < (u64 *)__end_got; pos++) +if (*pos == sym->st_value) +return (u64 )pos + rela->r_addend; +return 0 ; +} +#endif + static int __write_relocate_add(Elf64_Shdr *sechdrs, const char *strtab, unsigned int symindex,@@ -171,6 +183,7 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,case R_X86_64_64 : size =8 ;break ; +#ifndef CONFIG_X86_PIE case R_X86_64_32 :if (val != *(u32 *)&val) goto overflow;@@ -181,6 +194,13 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#else +case R_X86_64_GOTPCREL : + val =find_got_kernel_entry (sym, rel); +if (!val) + goto unexpected_got_reference; + fallthrough; +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;@@ -214,11 +234,18 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, }return 0 ; +#ifdef CONFIG_X86_PIE +unexpected_got_reference: +pr_err ("Target got entry doesn't exist in kernel got, loc %p\n" , loc); +return -ENOEXEC ; +#else overflow:pr_err ("overflow in relocation type %d val %Lx\n" , (int )ELF64_R_TYPE (rel[i].r_info), val);pr_err ("`%s' likely not compiled with -mcmodel=kernel\n" , me->name); +#endif +return -ENOEXEC ; } --2.31.1
Ard Biesheuvel
maintainerRe: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
Quoted 53 lines · Hou Wenlong
Quoted 51 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define
PV_VCPU_PREEMPTED_ASM\
"leaq__per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, "__stringify(KVM_STEAL_TIME_preempted)"(%rax) \n\t" \
"setne %al\n\t"or
arch/x86/kernel/module.c +3
-0+#ifdef CONFIG_X86_PIE +" pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else" pushq $arch_rethook_trampoline\n"
+#endifinstead of having these kludgy push/pop sequences to free up temp registers.
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
Quoted 6 lines
Hi Ard,
Thanks for providing the link, it has been very helpful for me as I am new to the topic of compilers.
Happy to hear that.
Quoted 6 lines
One key difference I noticed is that you
linked the kernel with "-pie" instead of "--emit-reloc". I also noticed that Thomas' initial patchset[0] used "-pie", but in RFC v3 [1], it switched to "--emit-reloc" in order to reduce dynamic relocation space on mapped memory.
The problem with --emit-relocs is that the relocations emitted into the binary may get out of sync with the actual code after the linker has applied relocations.
$ cat /tmp/a.s foo:movq foo@GOTPCREL(%rip), %rax
$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s ard@gambale:~/linux$ x86_64-linux-gnu-objdump -dr /tmp/a.o
/tmp/a.o: file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <foo>:
0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7>
3: R_X86_64_REX_GOTPCRELX foo-0x4
$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.o 0000000000000000 <foo>: 0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7> 3: R_X86_64_REX_GOTPCRELX foo-0x4
$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-no-pie,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000401000 <foo>: 401000: 48 c7 c0 00 10 40 00 mov $0x401000,%rax 401003: R_X86_64_32S foo
$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles
-Wl,-q,--defsym,_start=0x0 /tmp/a.s
$ x86_64-linux-gnu-objdump -dr /tmp/a.elf
0000000000001000 <foo>:
1000: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 1000 <foo>
1003: R_X86_64_PC32 foo-0x4This all looks as expected. However, when using Clang, we end up with
$ clang -target x86_64-linux-gnu -o /tmp/a.elf -nostartfiles
-fuse-ld=lld -Wl,--relax,-q,--defsym,_start=0x0 /tmp/a.s
$ x86_64-linux-gnu-objdump -dr /tmp/a.elf
00000000000012c0 <foo>:
12c0: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 12c0 <foo>
12c3: R_X86_64_REX_GOTPCRELX foo-0x4So in this case, what --emit-relocs gives us is not what is actually in the binary. We cannot just ignore these either, given that they are treated differently depending on whether the symbol is a per-CPU symbol or not - in the former case, we need to perform a fixup if the relaxed reference is RIP relative, and in the latter case, if the relaxed reference is absolute.
On top of that, --emit-relocs does not cover the GOT, so we'd still need to process that from the code explicitly.
In general, relying on --emit-relocs is kind of dodgy, and I think combining PIE linking with --emit-relocs is a bad idea.
Quoted 2 lines
The another issue is that it requires the addition of the "-mrelax-relocations=no" option to support older compilers and linkers.
Why? The decompressor is now linked in PIE mode so we should be able to drop that. Or do you need to add is somewhere else?
Quoted 10 lines
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELXrelocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option inarch/x86/boot/compressed/Makefilewith the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.
The stack canary references are under discussion here [3]. I have also sent a patch for kallsyms symbol references [4]. Beyond that, there should be very few cases where GOT entries are emitted, so I don't think this is fundamentally a problem.
I haven't run into the __fentry__ issue myself: do you think we should fix this in the compiler?
Quoted 3 lines
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Excellent, good luck with that.
However, you will still need to make a convincing case for why this is all worth the trouble. Especially given that you disable the depth tracking code, which I don't think should be mutually exclusive.
I am aware that this a rather tricky, and involves rewriting RIP-relative per-CPU variable accesses, but it would be good to get a discussion started on that topic, and figure out whether there is a way forward there. Ignoring it is not going to help.
Quoted 5 lines
[0] https://yhbt.net/lore/all/[email protected]
[1] https://yhbt.net/lore/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]/
[3] https://github.com/llvm/llvm-project/issues/60116
[4] [email protected]
Quoted 92 lines
Quoted 92 lines
Quoted 92 lines
Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/include/asm/sections.h | 5 +++++ arch/x86/kernel/module.c | 27 +++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)arch/x86/include/asm/sections.h +5
-0@@ -12,6 +12,11 @@ extern char __end_rodata_aligned[];#if defined(CONFIG_X86_64 )extern char __end_rodata_hpage_align[]; + +#ifdef CONFIG_X86_PIE +extern char __start_got[], __end_got[]; +#endif +#endif extern char __end_of_kernel_reserve[];arch/x86/kernel/module.c +27
-1@@ -129,6 +129,18 @@ int apply_relocate(Elf32_Shdr *sechdrs,return 0 ; }#else /*X86_64*/ +#ifdef CONFIG_X86_PIE +static u64 find_got_kernel_entry (Elf64_Sym *sym,const Elf64_Rela *rela) +{ +u64 *pos; + +for (pos = (u64 *)__start_got; pos < (u64 *)__end_got; pos++) +if (*pos == sym->st_value) +return (u64 )pos + rela->r_addend; +return 0 ; +} +#endif + static int __write_relocate_add(Elf64_Shdr *sechdrs, const char *strtab, unsigned int symindex,@@ -171,6 +183,7 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,case R_X86_64_64 : size =8 ;break ; +#ifndef CONFIG_X86_PIE case R_X86_64_32 :if (val != *(u32 *)&val) goto overflow;@@ -181,6 +194,13 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#else +case R_X86_64_GOTPCREL : + val =find_got_kernel_entry (sym, rel); +if (!val) + goto unexpected_got_reference; + fallthrough; +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;@@ -214,11 +234,18 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, }return 0 ; +#ifdef CONFIG_X86_PIE +unexpected_got_reference: +pr_err ("Target got entry doesn't exist in kernel got, loc %p\n" , loc); +return -ENOEXEC ; +#else overflow:pr_err ("overflow in relocation type %d val %Lx\n" , (int )ELF64_R_TYPE (rel[i].r_info), val);pr_err ("`%s' likely not compiled with -mcmodel=kernel\n" , me->name); +#endif +return -ENOEXEC ; } --2.31.1
Re: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
Quoted 54 lines · Ard Biesheuvel
Quoted 53 lines · Hou Wenlong
Quoted 51 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define
PV_VCPU_PREEMPTED_ASM\
"leaq__per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, "__stringify(KVM_STEAL_TIME_preempted)"(%rax) \n\t" \
"setne %al\n\t"or
arch/x86/kernel/module.c +3
-0+#ifdef CONFIG_X86_PIE +" pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else" pushq $arch_rethook_trampoline\n"
+#endifinstead of having these kludgy push/pop sequences to free up temp registers.
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
Quoted 9 lines
Quoted 6 lines
Hi Ard,
Thanks for providing the link, it has been very helpful for me as I am new to the topic of compilers.Happy to hear that.
Thanks for your prompt reply.
Quoted 55 lines
Quoted 6 lines
One key difference I noticed is that you
linked the kernel with "-pie" instead of "--emit-reloc". I also noticed that Thomas' initial patchset[0] used "-pie", but in RFC v3 [1], it switched to "--emit-reloc" in order to reduce dynamic relocation space on mapped memory.The problem with --emit-relocs is that the relocations emitted into the binary may get out of sync with the actual code after the linker has applied relocations.
$ cat /tmp/a.s foo:movq foo@GOTPCREL(%rip), %rax$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s ard@gambale:~/linux$ x86_64-linux-gnu-objdump -dr /tmp/a.o
/tmp/a.o: file format elf64-x86-64Disassembly of section .text:
0000000000000000 <foo>:
0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7>3:
R_X86_64_REX_GOTPCRELXfoo-0x4$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.o 0000000000000000 <foo>: 0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7> 3: R_X86_64_REX_GOTPCRELX foo-0x4$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-no-pie,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000401000 <foo>: 401000: 48 c7 c0 00 10 40 00 mov $0x401000,%rax 401003: R_X86_64_32S foo$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000001000 <foo>: 1000: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 1000 <foo> 1003: R_X86_64_PC32 foo-0x4This all looks as expected. However, when using Clang, we end up with
$ clang -target x86_64-linux-gnu -o /tmp/a.elf -nostartfiles -fuse-ld=lld -Wl,--relax,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 00000000000012c0 <foo>: 12c0: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 12c0 <foo> 12c3: R_X86_64_REX_GOTPCRELX foo-0x4
Quoted 7 lines
So in this case, what --emit-relocs gives us is not what is actually in the binary. We cannot just ignore these either, given that they are treated differently depending on whether the symbol is a per-CPU symbol or not - in the former case, we need to perform a fixup if the relaxed reference is RIP relative, and in the latter case, if the relaxed reference is absolute.
With symbols hidden and the compile-time address of the kernel image kept in the top 2G, is it possible for the relaxed reference to be absolute, even if I keep the percpu section zero-mapping for SMP? I didn't see absoulte relaxed reference after dropping "-mrelax-relocations=no" option.
Quoted 3 lines
On top of that, --emit-relocs does not cover the GOT, so we'd still need to process that from the code explicitly.
Yes, so the relocs tool would process GOT, and generate R_X86_64_GLOB_DAT relocation for GOT entries in patch 27:
https://lore.kernel.org/lkml/d25c7644249355785365914398bdba1ed2c52468.1682673543.git.houwenlong.hwl@antgroup.com
Quoted 9 lines
In general, relying on --emit-relocs is kind of dodgy, and I think combining PIE linking with --emit-relocs is a bad idea.
Quoted 2 lines
The another issue is that it requires the addition of the "-mrelax-relocations=no" option to support older compilers and linkers.
Why? The decompressor is now linked in PIE mode so we should be able to drop that. Or do you need to add is somewhere else?
I tried to use binutils 2.25 (mini version), it couldn't recognize R_X86_64_GOTPCRELX and R_X86_64_REX_GOTPCRELX.
Quoted 11 lines
Quoted 10 lines
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELXrelocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option inarch/x86/boot/compressed/Makefilewith the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.
Quoted 8 lines
The stack canary references are under discussion here [3]. I have also sent a patch for kallsyms symbol references [4]. Beyond that, there should be very few cases where GOT entries are emitted, so I don't think this is fundamentally a problem.
I haven't run into the
__fentry__issue myself: do you think we should fix this in the compiler?
The issue about __fentry__ is that the compiler would generate 6-bytes indirect call through GOT with "-fPIE" option. However, the original ftrace nop patching assumes it is a 5-bytes direct call. And "-mnop-mcount" option is not compatiable with "-fPIE" option, so the complier woudn't patch it as nop.
So we should patch it with one 5-bytes nop followed by one 1-byte nop, This way, ftrace can handle the previous 5-bytes as before. Also I have built PIE kernel with relocation relaxation on GCC, and the linker would relax it as following:ffffffff810018f0 <do_one_initcall>:
ffffffff810018f0: f3 0f 1e fa endbr64 ffffffff810018f4: 67 e8 a6 d6 05 00 addr32 call ffffffff8105efa0 <__fentry__> ffffffff810018f6: R_X86_64_PC32 __fentry__-0x4
It still requires a different nop patching for ftrace. I notice "Optimize GOTPCRELX Relocations" chapter in x86-64 psABI, which suggests that the GOT indirect call can be relaxed as "call fentry nop" or "nop call fentry", it appears that the latter is chosen. If the linker could generate the former, then no fixup would be necessary for ftrace with PIE.
Quoted 10 lines
Quoted 3 lines
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Excellent, good luck with that.
However, you will still need to make a convincing case for why this is all worth the trouble. Especially given that you disable the depth tracking code, which I don't think should be mutually exclusive.
Actually, I could do relocation for it when apply patching for the depth tracking code. I'm not sure such case is common or not.
Quoted 6 lines
I am aware that this a rather tricky, and involves rewriting RIP-relative per-CPU variable accesses, but it would be good to get a discussion started on that topic, and figure out whether there is a way forward there. Ignoring it is not going to help.
I see that your PIE linking chose to put the per-cpu section in high kernel image address, I still keep it as zero-mapping. However, both are in the RIP-relative addressing range.
Quoted 101 lines
Quoted 5 lines
[0] https://yhbt.net/lore/all/[email protected]
[1] https://yhbt.net/lore/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]/[3] https://github.com/llvm/llvm-project/issues/60116
[4] [email protected]Quoted 92 lines
Quoted 92 lines
Quoted 92 lines
Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/include/asm/sections.h | 5 +++++ arch/x86/kernel/module.c | 27 +++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)arch/x86/include/asm/sections.h +5
-0@@ -12,6 +12,11 @@ extern char __end_rodata_aligned[];#if defined(CONFIG_X86_64 )extern char __end_rodata_hpage_align[]; + +#ifdef CONFIG_X86_PIE +extern char __start_got[], __end_got[]; +#endif +#endif extern char __end_of_kernel_reserve[];arch/x86/kernel/module.c +27
-1@@ -129,6 +129,18 @@ int apply_relocate(Elf32_Shdr *sechdrs,return 0 ; }#else /*X86_64*/ +#ifdef CONFIG_X86_PIE +static u64 find_got_kernel_entry (Elf64_Sym *sym,const Elf64_Rela *rela) +{ +u64 *pos; + +for (pos = (u64 *)__start_got; pos < (u64 *)__end_got; pos++) +if (*pos == sym->st_value) +return (u64 )pos + rela->r_addend; +return 0 ; +} +#endif + static int __write_relocate_add(Elf64_Shdr *sechdrs, const char *strtab, unsigned int symindex,@@ -171,6 +183,7 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,case R_X86_64_64 : size =8 ;break ; +#ifndef CONFIG_X86_PIE case R_X86_64_32 :if (val != *(u32 *)&val) goto overflow;@@ -181,6 +194,13 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#else +case R_X86_64_GOTPCREL : + val =find_got_kernel_entry (sym, rel); +if (!val) + goto unexpected_got_reference; + fallthrough; +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;@@ -214,11 +234,18 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, }return 0 ; +#ifdef CONFIG_X86_PIE +unexpected_got_reference: +pr_err ("Target got entry doesn't exist in kernel got, loc %p\n" , loc); +return -ENOEXEC ; +#else overflow:pr_err ("overflow in relocation type %d val %Lx\n" , (int )ELF64_R_TYPE (rel[i].r_info), val);pr_err ("`%s' likely not compiled with -mcmodel=kernel\n" , me->name); +#endif +return -ENOEXEC ; } --2.31.1
Re: [PATCH RFC 25/43] x86/mm: Make the x86 GOT read-only
In reply to Ard BiesheuvelQuoted 56 lines · Ard Biesheuvel
Quoted 47 lines · Hou Wenlong
From: Thomas Garnier <[email protected]>
From: Thomas Garnier <[email protected]>
The GOT is changed during early boot when relocations are applied. Make it read-only directly. This table exists only for PIE binary. Since weak symbol reference would always be GOT reference, there are 8 entries in GOT, but only one entry for
__fentry__()is in use. Other GOT references have been optimized by linker.[Hou Wenlong: Change commit message and skip GOT size check]
Signed-off-by Thomas Garnier <[email protected]>Signed-off-by Hou Wenlong <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/kernel/vmlinux.lds.S | 2 ++ include/asm-generic/vmlinux.lds.h | 12 ++++++++++++ 2 files changed, 14 insertions(+)arch/x86/kernel/vmlinux.lds.S +2
-0@@ -462,6 +462,7 @@ SECTIONS #endif "Unexpected GOT/PLT entries detected!") +#ifndef CONFIG_X86_PIE /* * Sections that should stay zero sized, which is safer to * explicitly check instead of blindly discarding.@@ -470,6 +471,7 @@ SECTIONS *(.got) *(.igot.*) } ASSERT(SIZEOF(.got) == 0, "Unexpected GOT entries detected!") +#endif .plt : { *(.plt) *(.plt.*) *(.iplt)include/asm-generic/vmlinux.lds.h +0
-0include/asm-generic/vmlinux.lds.h line 445
Quoted 5 lines
include/asm-generic/vmlinux.lds.h +2
-0@@ -441,6 +441,17 @@ __end_ro_after_init = .;#endif +#ifdef CONFIG_X86_PIE +#define RO_GOT_X86 Please don't put X86 specific stuff in generic code.
Quoted 14 lines
include/asm-generic/vmlinux.lds.h line 454
Quoted 9 lines
include/asm-generic/vmlinux.lds.h +9
-0+ .got : AT(ADDR(.got) - LOAD_OFFSET) { \ + __start_got = .; \ + *(.got) *(.igot.*); \ + __end_got = .; \ + } +#else +#define RO_GOT_X86 +#endif +I don't think it makes sense for this definition to be conditional. You can include it conditionally from the x86 code, but even that seems unnecessary, given that it will be empty otherwise.
Hi Ard,
I know that X86 specific stuff should be in generic code. I notice that even relocation relaxation is enabled, the GOT would not be empty. I want it to be included in the read-only data section (RODATA) between the symbols __start_rodata and __end_rodata for safety. I noticed that you placed it in the data section during your PIE linking. Should it be marked as read-only if it is not empty?
Thanks.
Quoted 14 lines
Quoted 14 lines
include/asm-generic/vmlinux.lds.h +1
-1/* * .kcfi_traps contains a list KCFI trap locations. */ @@ -486,6 +497,7 @@ BOUNDED_SECTION_PRE_LABEL(.pci_fixup_suspend_late, _pci_fixups_suspend_late, __start, __end) \ } \ \ +RO_GOT_X86 \FW_LOADER_BUILT_IN_DATA \ TRACEDATA \ \ --2.31.1
Ard Biesheuvel
maintainerRe: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
In reply to Hou WenlongQuoted 123 lines · Hou Wenlong
Quoted 54 lines · Ard Biesheuvel
Quoted 53 lines · Hou Wenlong
Quoted 51 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define
PV_VCPU_PREEMPTED_ASM\
"leaq__per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, "__stringify(KVM_STEAL_TIME_preempted)"(%rax) \n\t" \
"setne %al\n\t"or
arch/x86/kernel/module.c +3
-0+#ifdef CONFIG_X86_PIE +" pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else" pushq $arch_rethook_trampoline\n"
+#endifinstead of having these kludgy push/pop sequences to free up temp registers.
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
Quoted 9 lines
Quoted 6 lines
Hi Ard,
Thanks for providing the link, it has been very helpful for me as I am new to the topic of compilers.Happy to hear that.
Thanks for your prompt reply.
Quoted 55 lines
Quoted 6 lines
One key difference I noticed is that you
linked the kernel with "-pie" instead of "--emit-reloc". I also noticed that Thomas' initial patchset[0] used "-pie", but in RFC v3 [1], it switched to "--emit-reloc" in order to reduce dynamic relocation space on mapped memory.The problem with --emit-relocs is that the relocations emitted into the binary may get out of sync with the actual code after the linker has applied relocations.
$ cat /tmp/a.s foo:movq foo@GOTPCREL(%rip), %rax$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s ard@gambale:~/linux$ x86_64-linux-gnu-objdump -dr /tmp/a.o
/tmp/a.o: file format elf64-x86-64Disassembly of section .text:
0000000000000000 <foo>:
0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7>3:
R_X86_64_REX_GOTPCRELXfoo-0x4$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.o 0000000000000000 <foo>: 0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7> 3: R_X86_64_REX_GOTPCRELX foo-0x4$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-no-pie,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000401000 <foo>: 401000: 48 c7 c0 00 10 40 00 mov $0x401000,%rax 401003: R_X86_64_32S foo$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000001000 <foo>: 1000: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 1000 <foo> 1003: R_X86_64_PC32 foo-0x4This all looks as expected. However, when using Clang, we end up with
$ clang -target x86_64-linux-gnu -o /tmp/a.elf -nostartfiles -fuse-ld=lld -Wl,--relax,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 00000000000012c0 <foo>: 12c0: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 12c0 <foo> 12c3: R_X86_64_REX_GOTPCRELX foo-0x4
Quoted 13 lines
Quoted 7 lines
So in this case, what --emit-relocs gives us is not what is actually in the binary. We cannot just ignore these either, given that they are treated differently depending on whether the symbol is a per-CPU symbol or not - in the former case, we need to perform a fixup if the relaxed reference is RIP relative, and in the latter case, if the relaxed reference is absolute.
With symbols hidden and the compile-time address of the kernel image kept in the top 2G, is it possible for the relaxed reference to be absolute, even if I keep the percpu section zero-mapping for SMP? I didn't see absoulte relaxed reference after dropping "-mrelax-relocations=no" option.
If you link in PIE mode, you should never see absolute references after relaxation.
Quoted 7 lines
Quoted 3 lines
On top of that, --emit-relocs does not cover the GOT, so we'd still need to process that from the code explicitly.
Yes, so the relocs tool would process GOT, and generate
R_X86_64_GLOB_DATrelocation for GOT entries in patch 27:
https://lore.kernel.org/lkml/d25c7644249355785365914398bdba1ed2c52468.1682673543.git.houwenlong.hwl@antgroup.com
Yes, something like that is needed. I'd lean towards generating the reloc data directly instead of creating an artifiical RELA section with GLOB_DAT relocations, but that is a minor detail.
Quoted 12 lines
Quoted 9 lines
In general, relying on --emit-relocs is kind of dodgy, and I think combining PIE linking with --emit-relocs is a bad idea.
Quoted 2 lines
The another issue is that it requires the addition of the "-mrelax-relocations=no" option to support older compilers and linkers.
Why? The decompressor is now linked in PIE mode so we should be able to drop that. Or do you need to add is somewhere else?
I tried to use binutils 2.25 (mini version), it couldn't recognize
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELX.
I'm not sure that matters. If the assembler accepts @GOTPCREL notation, it should generate the relocations that the linker can understand. If the toolchain is not internally consistent in this regard, I don't think it is our problem.
This might mean that we end up with more residual GOT entries than with a more recent toolchain, but I don't think that is a big deal.
Quoted 25 lines
Quoted 11 lines
Quoted 10 lines
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELXrelocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option inarch/x86/boot/compressed/Makefilewith the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.Quoted 8 lines
The stack canary references are under discussion here [3]. I have also sent a patch for kallsyms symbol references [4]. Beyond that, there should be very few cases where GOT entries are emitted, so I don't think this is fundamentally a problem.
I haven't run into the
__fentry__issue myself: do you think we should fix this in the compiler?The issue about
__fentry__is that the compiler would generate 6-bytes indirect call through GOT with "-fPIE" option. However, the original ftrace nop patching assumes it is a 5-bytes direct call. And "-mnop-mcount" option is not compatiable with "-fPIE" option, so the complier woudn't patch it as nop.
Quoted 9 lines
So we should patch it with one 5-bytes nop followed by one 1-byte nop, This way, ftrace can handle the previous 5-bytes as before. Also I have built PIE kernel with relocation relaxation on GCC, and the linker would relax it as following:
ffffffff810018f0<do_one_initcall>:ffffffff810018f0: f3 0f 1e fa endbr64 ffffffff810018f4: 67 e8 a6 d6 05 00 addr32 call ffffffff8105efa0 <__fentry__> ffffffff810018f6: R_X86_64_PC32 __fentry__-0x4
But if fentry is a function symbol, I would not expect the codegen to be different at all. Are you using -fno-plt?
Quoted 7 lines
It still requires a different nop patching for ftrace. I notice "Optimize GOTPCRELX Relocations" chapter in x86-64 psABI, which suggests that the GOT indirect call can be relaxed as "call fentry nop" or "nop call fentry", it appears that the latter is chosen. If the linker could generate the former, then no fixup would be necessary for ftrace with PIE.
Right. I think this may be a result of __fentry__ not being subject to the same rules wrt visibility etc, similar to __stack_chk_guard. These are arguably compiler issues that could qualify as bugs, given that these symbol references don't behave like ordinary symbol references.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Quoted 9 lines
Quoted 6 lines
Excellent, good luck with that.
However, you will still need to make a convincing case for why this is all worth the trouble. Especially given that you disable the depth tracking code, which I don't think should be mutually exclusive.
Actually, I could do relocation for it when apply patching for the depth tracking code. I'm not sure such case is common or not.
I think that alternatives patching in general would need to support RIP relative references in the alternatives. The depth tracking template is a bit different in this regard, and could be fixed more easily, I think.
Quoted 10 lines
Quoted 6 lines
I am aware that this a rather tricky, and involves rewriting RIP-relative per-CPU variable accesses, but it would be good to get a discussion started on that topic, and figure out whether there is a way forward there. Ignoring it is not going to help.
I see that your PIE linking chose to put the per-cpu section in high kernel image address, I still keep it as zero-mapping. However, both are in the RIP-relative addressing range.
Pure PIE linking cannot support the zero mapping - it can only work with --emit-relocs, which I was trying to avoid.
Re: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
Quoted 162 lines · Ard Biesheuvel
Quoted 123 lines · Hou Wenlong
Quoted 54 lines · Ard Biesheuvel
Quoted 53 lines · Hou Wenlong
Quoted 51 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define
PV_VCPU_PREEMPTED_ASM\
"leaq__per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, "__stringify(KVM_STEAL_TIME_preempted)"(%rax) \n\t" \
"setne %al\n\t"or
arch/x86/kernel/module.c +3
-0+#ifdef CONFIG_X86_PIE +" pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else" pushq $arch_rethook_trampoline\n"
+#endifinstead of having these kludgy push/pop sequences to free up temp registers.
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
Quoted 9 lines
Quoted 6 lines
Hi Ard,
Thanks for providing the link, it has been very helpful for me as I am new to the topic of compilers.Happy to hear that.
Thanks for your prompt reply.
Quoted 55 lines
Quoted 6 lines
One key difference I noticed is that you
linked the kernel with "-pie" instead of "--emit-reloc". I also noticed that Thomas' initial patchset[0] used "-pie", but in RFC v3 [1], it switched to "--emit-reloc" in order to reduce dynamic relocation space on mapped memory.The problem with --emit-relocs is that the relocations emitted into the binary may get out of sync with the actual code after the linker has applied relocations.
$ cat /tmp/a.s foo:movq foo@GOTPCREL(%rip), %rax$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s ard@gambale:~/linux$ x86_64-linux-gnu-objdump -dr /tmp/a.o
/tmp/a.o: file format elf64-x86-64Disassembly of section .text:
0000000000000000 <foo>:
0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7>3:
R_X86_64_REX_GOTPCRELXfoo-0x4$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.o 0000000000000000 <foo>: 0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7> 3: R_X86_64_REX_GOTPCRELX foo-0x4$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-no-pie,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000401000 <foo>: 401000: 48 c7 c0 00 10 40 00 mov $0x401000,%rax 401003: R_X86_64_32S foo$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000001000 <foo>: 1000: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 1000 <foo> 1003: R_X86_64_PC32 foo-0x4This all looks as expected. However, when using Clang, we end up with
$ clang -target x86_64-linux-gnu -o /tmp/a.elf -nostartfiles -fuse-ld=lld -Wl,--relax,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 00000000000012c0 <foo>: 12c0: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 12c0 <foo> 12c3: R_X86_64_REX_GOTPCRELX foo-0x4Quoted 13 lines
Quoted 7 lines
So in this case, what --emit-relocs gives us is not what is actually in the binary. We cannot just ignore these either, given that they are treated differently depending on whether the symbol is a per-CPU symbol or not - in the former case, we need to perform a fixup if the relaxed reference is RIP relative, and in the latter case, if the relaxed reference is absolute.
With symbols hidden and the compile-time address of the kernel image kept in the top 2G, is it possible for the relaxed reference to be absolute, even if I keep the percpu section zero-mapping for SMP? I didn't see absoulte relaxed reference after dropping "-mrelax-relocations=no" option.
If you link in PIE mode, you should never see absolute references after relaxation.
Quoted 7 lines
Quoted 3 lines
On top of that, --emit-relocs does not cover the GOT, so we'd still need to process that from the code explicitly.
Yes, so the relocs tool would process GOT, and generate
R_X86_64_GLOB_DATrelocation for GOT entries in patch 27:
https://lore.kernel.org/lkml/d25c7644249355785365914398bdba1ed2c52468.1682673543.git.houwenlong.hwl@antgroup.comYes, something like that is needed. I'd lean towards generating the reloc data directly instead of creating an artifiical RELA section with
GLOB_DATrelocations, but that is a minor detail.Quoted 9 lines
Quoted 9 lines
In general, relying on --emit-relocs is kind of dodgy, and I think combining PIE linking with --emit-relocs is a bad idea.
Quoted 2 lines
The another issue is that it requires the addition of the "-mrelax-relocations=no" option to support older compilers and linkers.
Why? The decompressor is now linked in PIE mode so we should be able to drop that. Or do you need to add is somewhere else?
Quoted 9 lines
Quoted 3 lines
I tried to use binutils 2.25 (mini version), it couldn't recognize
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELX.I'm not sure that matters. If the assembler accepts @GOTPCREL notation, it should generate the relocations that the linker can understand. If the toolchain is not internally consistent in this regard, I don't think it is our problem.
I get it. Thanks.
Quoted 38 lines
This might mean that we end up with more residual GOT entries than with a more recent toolchain, but I don't think that is a big deal.
Quoted 25 lines
Quoted 11 lines
Quoted 10 lines
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELXrelocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option inarch/x86/boot/compressed/Makefilewith the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.Quoted 8 lines
The stack canary references are under discussion here [3]. I have also sent a patch for kallsyms symbol references [4]. Beyond that, there should be very few cases where GOT entries are emitted, so I don't think this is fundamentally a problem.
I haven't run into the
__fentry__issue myself: do you think we should fix this in the compiler?The issue about
__fentry__is that the compiler would generate 6-bytes indirect call through GOT with "-fPIE" option. However, the original ftrace nop patching assumes it is a 5-bytes direct call. And "-mnop-mcount" option is not compatiable with "-fPIE" option, so the complier woudn't patch it as nop.Quoted 9 lines
So we should patch it with one 5-bytes nop followed by one 1-byte nop, This way, ftrace can handle the previous 5-bytes as before. Also I have built PIE kernel with relocation relaxation on GCC, and the linker would relax it as following:
ffffffff810018f0<do_one_initcall>:ffffffff810018f0: f3 0f 1e fa endbr64 ffffffff810018f4: 67 e8 a6 d6 05 00 addr32 call ffffffff8105efa0 <__fentry__> ffffffff810018f6: R_X86_64_PC32 __fentry__-0x4
Quoted 3 lines
But if fentry is a function symbol, I would not expect the codegen to be different at all. Are you using -fno-plt?
No, even with -fno-plt added, the compiler still generates a GOT reference for fentry. Therefore, the problem may be visibility, as you said.
Quoted 38 lines
Quoted 7 lines
It still requires a different nop patching for ftrace. I notice "Optimize GOTPCRELX Relocations" chapter in x86-64 psABI, which suggests that the GOT indirect call can be relaxed as "call fentry nop" or "nop call fentry", it appears that the latter is chosen. If the linker could generate the former, then no fixup would be necessary for ftrace with PIE.
Right. I think this may be a result of
__fentry__not being subject to the same rules wrt visibility etc, similar to__stack_chk_guard. These are arguably compiler issues that could qualify as bugs, given that these symbol references don't behave like ordinary symbol references.Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Quoted 9 lines
Quoted 6 lines
Excellent, good luck with that.
However, you will still need to make a convincing case for why this is all worth the trouble. Especially given that you disable the depth tracking code, which I don't think should be mutually exclusive.
Actually, I could do relocation for it when apply patching for the depth tracking code. I'm not sure such case is common or not.
I think that alternatives patching in general would need to support RIP relative references in the alternatives. The depth tracking template is a bit different in this regard, and could be fixed more easily, I think.
Quoted 6 lines
Quoted 6 lines
I am aware that this a rather tricky, and involves rewriting RIP-relative per-CPU variable accesses, but it would be good to get a discussion started on that topic, and figure out whether there is a way forward there. Ignoring it is not going to help.
Quoted 7 lines
Quoted 4 lines
I see that your PIE linking chose to put the per-cpu section in high kernel image address, I still keep it as zero-mapping. However, both are in the RIP-relative addressing range.
Pure PIE linking cannot support the zero mapping - it can only work with --emit-relocs, which I was trying to avoid.
Sorry, why doesn't PIE linking support zero mapping? I noticed in the commit message for your PIE linking that it stated, "if we randomize the kernel's VA by increasing it by X bytes, every RIP-relative per-CPU reference needs to be decreased by the same amount in order for the produced offset to remain correct." As a result, I decided to decrease the GS base and not relocate the RIP-relative per-CPU reference in the relocs. Consequently, all RIP-relative references, regardless of whether they are per-CPU variables or not, do not require relocation.
Furthermore, all symbols are hidden, which implies that all per-CPU references will not generate a GOT reference and will be relaxed as absolute reference due to zero mapping. However, the __stack_chk_guard on CLANG always generates a GOT reference, but I didn't see it being relaxed as absolute reference on LLVM.
Thanks!
Ard Biesheuvel
maintainerRe: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
Quoted 9 lines · Hou Wenlong
Quoted 7 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Quoted 4 lines · Ard Biesheuvel
Quoted 3 lines · Hou Wenlong
Quoted 1 line · Ard Biesheuvel
On Fri, 28 Apr 2023 at 10:53, Hou Wenlong <[email protected]> wrote:
...
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 11 lines
Quoted 10 lines
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELXrelocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option inarch/x86/boot/compressed/Makefilewith the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.Quoted 8 lines
The stack canary references are under discussion here [3]. I have also sent a patch for kallsyms symbol references [4]. Beyond that, there should be very few cases where GOT entries are emitted, so I don't think this is fundamentally a problem.
I haven't run into the
__fentry__issue myself: do you think we should fix this in the compiler?The issue about
__fentry__is that the compiler would generate 6-bytes indirect call through GOT with "-fPIE" option. However, the original ftrace nop patching assumes it is a 5-bytes direct call. And "-mnop-mcount" option is not compatiable with "-fPIE" option, so the complier woudn't patch it as nop.So we should patch it with one 5-bytes nop followed by one 1-byte nop, This way, ftrace can handle the previous 5-bytes as before. Also I have built PIE kernel with relocation relaxation on GCC, and the linker would relax it as following:
ffffffff810018f0<do_one_initcall>:ffffffff810018f0: f3 0f 1e fa endbr64 ffffffff810018f4: 67 e8 a6 d6 05 00 addr32 call ffffffff8105efa0 <__fentry__> ffffffff810018f6: R_X86_64_PC32 __fentry__-0x4
Quoted 7 lines
Quoted 3 lines
But if fentry is a function symbol, I would not expect the codegen to be different at all. Are you using -fno-plt?
No, even with -fno-plt added, the compiler still generates a GOT reference for fentry. Therefore, the problem may be visibility, as you said.
Yeah, I spotted this issue in GCC - I just sent them a patch this morning.
Quoted 45 lines
Quoted 38 lines
Quoted 7 lines
It still requires a different nop patching for ftrace. I notice "Optimize GOTPCRELX Relocations" chapter in x86-64 psABI, which suggests that the GOT indirect call can be relaxed as "call fentry nop" or "nop call fentry", it appears that the latter is chosen. If the linker could generate the former, then no fixup would be necessary for ftrace with PIE.
Right. I think this may be a result of
__fentry__not being subject to the same rules wrt visibility etc, similar to__stack_chk_guard. These are arguably compiler issues that could qualify as bugs, given that these symbol references don't behave like ordinary symbol references.Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Quoted 9 lines
Quoted 6 lines
Excellent, good luck with that.
However, you will still need to make a convincing case for why this is all worth the trouble. Especially given that you disable the depth tracking code, which I don't think should be mutually exclusive.
Actually, I could do relocation for it when apply patching for the depth tracking code. I'm not sure such case is common or not.
I think that alternatives patching in general would need to support RIP relative references in the alternatives. The depth tracking template is a bit different in this regard, and could be fixed more easily, I think.
Quoted 6 lines
Quoted 6 lines
I am aware that this a rather tricky, and involves rewriting RIP-relative per-CPU variable accesses, but it would be good to get a discussion started on that topic, and figure out whether there is a way forward there. Ignoring it is not going to help.
Quoted 7 lines
Quoted 4 lines
I see that your PIE linking chose to put the per-cpu section in high kernel image address, I still keep it as zero-mapping. However, both are in the RIP-relative addressing range.
Pure PIE linking cannot support the zero mapping - it can only work with --emit-relocs, which I was trying to avoid.
Quoted 9 lines
Sorry, why doesn't PIE linking support zero mapping? I noticed in the commit message for your PIE linking that it stated, "if we randomize the kernel's VA by increasing it by X bytes, every RIP-relative per-CPU reference needs to be decreased by the same amount in order for the produced offset to remain correct." As a result, I decided to decrease the GS base and not relocate the RIP-relative per-CPU reference in the relocs. Consequently, all RIP-relative references, regardless of whether they are per-CPU variables or not, do not require relocation.
Interesting. Does that work as expected with dynamically allocated per-CPU variables?
Quoted 6 lines
Furthermore, all symbols are hidden, which implies that all per-CPU references will not generate a GOT reference and will be relaxed as absolute reference due to zero mapping. However, the
__stack_chk_guardon CLANG always generates a GOT reference, but I didn't see it being relaxed as absolute reference on LLVM.
Yeah, we should fix that.
Re: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
Quoted 111 lines · Ard Biesheuvel
Quoted 9 lines · Hou Wenlong
Quoted 7 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Quoted 4 lines · Ard Biesheuvel
Quoted 3 lines · Hou Wenlong
Quoted 1 line · Ard Biesheuvel
On Fri, 28 Apr 2023 at 10:53, Hou Wenlong <[email protected]> wrote:
...
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 11 lines
Quoted 10 lines
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELXrelocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option inarch/x86/boot/compressed/Makefilewith the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.Quoted 8 lines
The stack canary references are under discussion here [3]. I have also sent a patch for kallsyms symbol references [4]. Beyond that, there should be very few cases where GOT entries are emitted, so I don't think this is fundamentally a problem.
I haven't run into the
__fentry__issue myself: do you think we should fix this in the compiler?The issue about
__fentry__is that the compiler would generate 6-bytes indirect call through GOT with "-fPIE" option. However, the original ftrace nop patching assumes it is a 5-bytes direct call. And "-mnop-mcount" option is not compatiable with "-fPIE" option, so the complier woudn't patch it as nop.So we should patch it with one 5-bytes nop followed by one 1-byte nop, This way, ftrace can handle the previous 5-bytes as before. Also I have built PIE kernel with relocation relaxation on GCC, and the linker would relax it as following:
ffffffff810018f0<do_one_initcall>:ffffffff810018f0: f3 0f 1e fa endbr64 ffffffff810018f4: 67 e8 a6 d6 05 00 addr32 call ffffffff8105efa0 <__fentry__> ffffffff810018f6: R_X86_64_PC32 __fentry__-0x4Quoted 7 lines
Quoted 3 lines
But if fentry is a function symbol, I would not expect the codegen to be different at all. Are you using -fno-plt?
No, even with -fno-plt added, the compiler still generates a GOT reference for fentry. Therefore, the problem may be visibility, as you said.
Yeah, I spotted this issue in GCC - I just sent them a patch this morning.
Quoted 45 lines
Quoted 38 lines
Quoted 7 lines
It still requires a different nop patching for ftrace. I notice "Optimize GOTPCRELX Relocations" chapter in x86-64 psABI, which suggests that the GOT indirect call can be relaxed as "call fentry nop" or "nop call fentry", it appears that the latter is chosen. If the linker could generate the former, then no fixup would be necessary for ftrace with PIE.
Right. I think this may be a result of
__fentry__not being subject to the same rules wrt visibility etc, similar to__stack_chk_guard. These are arguably compiler issues that could qualify as bugs, given that these symbol references don't behave like ordinary symbol references.Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Quoted 9 lines
Quoted 6 lines
Excellent, good luck with that.
However, you will still need to make a convincing case for why this is all worth the trouble. Especially given that you disable the depth tracking code, which I don't think should be mutually exclusive.
Actually, I could do relocation for it when apply patching for the depth tracking code. I'm not sure such case is common or not.
I think that alternatives patching in general would need to support RIP relative references in the alternatives. The depth tracking template is a bit different in this regard, and could be fixed more easily, I think.
Quoted 6 lines
Quoted 6 lines
I am aware that this a rather tricky, and involves rewriting RIP-relative per-CPU variable accesses, but it would be good to get a discussion started on that topic, and figure out whether there is a way forward there. Ignoring it is not going to help.
Quoted 7 lines
Quoted 4 lines
I see that your PIE linking chose to put the per-cpu section in high kernel image address, I still keep it as zero-mapping. However, both are in the RIP-relative addressing range.
Pure PIE linking cannot support the zero mapping - it can only work with --emit-relocs, which I was trying to avoid.
Quoted 9 lines
Sorry, why doesn't PIE linking support zero mapping? I noticed in the commit message for your PIE linking that it stated, "if we randomize the kernel's VA by increasing it by X bytes, every RIP-relative per-CPU reference needs to be decreased by the same amount in order for the produced offset to remain correct." As a result, I decided to decrease the GS base and not relocate the RIP-relative per-CPU reference in the relocs. Consequently, all RIP-relative references, regardless of whether they are per-CPU variables or not, do not require relocation.
Quoted 3 lines
Interesting. Does that work as expected with dynamically allocated per-CPU variables?
I didn't encounter any issues with the dynamically allocated per-CPU variables. Since the per_cpu_ptr macro uses the __per_cpu_offset array directly, it should work. In any case, I have tested loading the kvm module, which uses dynamically allocated per-CPU variables, and successfully booted a guest.
The related patch is:
https://lore.kernel.org/lkml/62d7e9e73467b711351a84ebce99372d3dccaa73.1682673543.git.houwenlong.hwl@antgroup.com
Thanks!
Quoted 8 lines
Quoted 6 lines
Furthermore, all symbols are hidden, which implies that all per-CPU references will not generate a GOT reference and will be relaxed as absolute reference due to zero mapping. However, the
__stack_chk_guardon CLANG always generates a GOT reference, but I didn't see it being relaxed as absolute reference on LLVM.Yeah, we should fix that.
Re: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
In reply to Ard BiesheuvelQuoted 128 lines · Ard Biesheuvel
Quoted 53 lines · Hou Wenlong
Quoted 51 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define
PV_VCPU_PREEMPTED_ASM\
"leaq__per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, "__stringify(KVM_STEAL_TIME_preempted)"(%rax) \n\t" \
"setne %al\n\t"or
arch/x86/kernel/module.c +3
-0+#ifdef CONFIG_X86_PIE +" pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else" pushq $arch_rethook_trampoline\n"
+#endifinstead of having these kludgy push/pop sequences to free up temp registers.
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
Quoted 6 lines
Hi Ard,
Thanks for providing the link, it has been very helpful for me as I am new to the topic of compilers.Happy to hear that.
Quoted 6 lines
One key difference I noticed is that you
linked the kernel with "-pie" instead of "--emit-reloc". I also noticed that Thomas' initial patchset[0] used "-pie", but in RFC v3 [1], it switched to "--emit-reloc" in order to reduce dynamic relocation space on mapped memory.The problem with --emit-relocs is that the relocations emitted into the binary may get out of sync with the actual code after the linker has applied relocations.
$ cat /tmp/a.s foo:movq foo@GOTPCREL(%rip), %rax$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s ard@gambale:~/linux$ x86_64-linux-gnu-objdump -dr /tmp/a.o
/tmp/a.o: file format elf64-x86-64Disassembly of section .text:
0000000000000000 <foo>:
0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7>3:
R_X86_64_REX_GOTPCRELXfoo-0x4$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.o 0000000000000000 <foo>: 0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7> 3: R_X86_64_REX_GOTPCRELX foo-0x4$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-no-pie,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000401000 <foo>: 401000: 48 c7 c0 00 10 40 00 mov $0x401000,%rax 401003: R_X86_64_32S foo$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000001000 <foo>: 1000: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 1000 <foo> 1003: R_X86_64_PC32 foo-0x4This all looks as expected. However, when using Clang, we end up with
$ clang -target x86_64-linux-gnu -o /tmp/a.elf -nostartfiles -fuse-ld=lld -Wl,--relax,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 00000000000012c0 <foo>: 12c0: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 12c0 <foo> 12c3: R_X86_64_REX_GOTPCRELX foo-0x4So in this case, what --emit-relocs gives us is not what is actually in the binary. We cannot just ignore these either, given that they are treated differently depending on whether the symbol is a per-CPU symbol or not - in the former case, we need to perform a fixup if the relaxed reference is RIP relative, and in the latter case, if the relaxed reference is absolute.
On top of that, --emit-relocs does not cover the GOT, so we'd still need to process that from the code explicitly.
Quoted 9 lines
In general, relying on --emit-relocs is kind of dodgy, and I think combining PIE linking with --emit-relocs is a bad idea.
Quoted 2 lines
The another issue is that it requires the addition of the "-mrelax-relocations=no" option to support older compilers and linkers.
Why? The decompressor is now linked in PIE mode so we should be able to drop that. Or do you need to add is somewhere else?
Hi Ard,
After removing the "-mrelax-relocations=no" option, I noticed that the linker was relaxing GOT references as absolute references for mov instructions, even if the symbol was in a high address, as long as I kept the compile-time base address of the kernel image in the top 2G. I consulted the "Optimize GOTPCRELX Relocations" chapter in x86-64 psABI, which stated that "When position-independent code is disabled and foo is defined locally in the lower 32-bit address space, memory operand in mov can be converted into immediate operand". However, it seemed that if the symbol was in the higher 32-bit address space, the memory operand in mov would also be converted into an immediate operand. If I decreased the compile-time base address of the kernel image, it would be relaxed as lea. Therefore, I believe that using "-mrelax-relocations=no" without "-pie" option is necessary. Is there a way to force the linker to relax it as lea without using the "-pie" option when linking?
Since all GOT references cannot be omitted, perhaps I should try linking the kernel with the "-pie" option.
Thanks!
Quoted 136 lines
Quoted 10 lines
R_X86_64_GOTPCRELXandR_X86_64_REX_GOTPCRELXrelocations are supported in binutils 2.26 and later, but the mini version required for the kernel is 2.25. This option disables relocation relaxation, which makes GOT not empty. I also noticed this option inarch/x86/boot/compressed/Makefilewith the reason given in [2]. Without relocation relaxation, GOT references would increase the size of GOT. Therefore, I do not want to use GOT reference in assembly directly. However, I realized that the compiler could still generate GOT references in some cases such as "fentry" calls and stack canary references.The stack canary references are under discussion here [3]. I have also sent a patch for kallsyms symbol references [4]. Beyond that, there should be very few cases where GOT entries are emitted, so I don't think this is fundamentally a problem.
I haven't run into the
__fentry__issue myself: do you think we should fix this in the compiler?Quoted 3 lines
Regarding module loading, I agree that we should support GOT reference for the module itself. I will refactor it according to your suggestion.
Excellent, good luck with that.
However, you will still need to make a convincing case for why this is all worth the trouble. Especially given that you disable the depth tracking code, which I don't think should be mutually exclusive.
I am aware that this a rather tricky, and involves rewriting RIP-relative per-CPU variable accesses, but it would be good to get a discussion started on that topic, and figure out whether there is a way forward there. Ignoring it is not going to help.
Quoted 5 lines
[0] https://yhbt.net/lore/all/[email protected]
[1] https://yhbt.net/lore/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]/[3] https://github.com/llvm/llvm-project/issues/60116
[4] [email protected]Quoted 92 lines
Quoted 92 lines
Quoted 92 lines
Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>--- arch/x86/include/asm/sections.h | 5 +++++ arch/x86/kernel/module.c | 27 +++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)arch/x86/include/asm/sections.h +5
-0@@ -12,6 +12,11 @@ extern char __end_rodata_aligned[];#if defined(CONFIG_X86_64 )extern char __end_rodata_hpage_align[]; + +#ifdef CONFIG_X86_PIE +extern char __start_got[], __end_got[]; +#endif +#endif extern char __end_of_kernel_reserve[];arch/x86/kernel/module.c +27
-1@@ -129,6 +129,18 @@ int apply_relocate(Elf32_Shdr *sechdrs,return 0 ; }#else /*X86_64*/ +#ifdef CONFIG_X86_PIE +static u64 find_got_kernel_entry (Elf64_Sym *sym,const Elf64_Rela *rela) +{ +u64 *pos; + +for (pos = (u64 *)__start_got; pos < (u64 *)__end_got; pos++) +if (*pos == sym->st_value) +return (u64 )pos + rela->r_addend; +return 0 ; +} +#endif + static int __write_relocate_add(Elf64_Shdr *sechdrs, const char *strtab, unsigned int symindex,@@ -171,6 +183,7 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,case R_X86_64_64 : size =8 ;break ; +#ifndef CONFIG_X86_PIE case R_X86_64_32 :if (val != *(u32 *)&val) goto overflow;@@ -181,6 +194,13 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#else +case R_X86_64_GOTPCREL : + val =find_got_kernel_entry (sym, rel); +if (!val) + goto unexpected_got_reference; + fallthrough; +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;@@ -214,11 +234,18 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, }return 0 ; +#ifdef CONFIG_X86_PIE +unexpected_got_reference: +pr_err ("Target got entry doesn't exist in kernel got, loc %p\n" , loc); +return -ENOEXEC ; +#else overflow:pr_err ("overflow in relocation type %d val %Lx\n" , (int )ELF64_R_TYPE (rel[i].r_info), val);pr_err ("`%s' likely not compiled with -mcmodel=kernel\n" , me->name); +#endif +return -ENOEXEC ; } --2.31.1
Ard Biesheuvel
maintainerRe: [PATCH RFC 31/43] x86/modules: Adapt module loading for PIE support
Quoted 141 lines · Hou Wenlong
Quoted 128 lines · Ard Biesheuvel
Quoted 53 lines · Hou Wenlong
Quoted 51 lines · Ard Biesheuvel
Quoted 6 lines · Hou Wenlong
Adapt module loading to support PIE relocations. No GOT is generared for module, all the GOT entry of got references in module should exist in kernel GOT. Currently, there is only one usable got reference for
__fentry__().I don't think this is the right approach. We should permit GOTPCREL relocations properly, which means making them point to a location in memory that carries the absolute address of the symbol. There are several ways to go about that, but perhaps the simplest way is to make the symbol address in ksymtab a 64-bit absolute value (but retain the PC32 references for the symbol name and the symbol namespace name). That way, you can always resolve such GOTPCREL relocations by pointing it to the ksymtab entry. Another option would be to take inspiration from the PLT code we have on ARM and arm64 (and other architectures, surely) and to count the GOT based relocations, allocate some extra r/o module space for each, and allocate slots and populate them with the right value as you fix up the relocations.
Then, many such relocations can be relaxed at module load time if the symbol is in range. IIUC, the module and kernel will still be inside the same 2G window even after widening the KASLR range to 512G, so most GOT loads can be converted into RIP relative LEA instructions.
Note that this will also permit you to do things like
#define
PV_VCPU_PREEMPTED_ASM\
"leaq__per_cpu_offset(%rip), %rax \n\t" \
"movq (%rax,%rdi,8), %rax \n\t" \
"addq steal_time@GOTPCREL(%rip), %rax \n\t" \
"cmpb $0, "__stringify(KVM_STEAL_TIME_preempted)"(%rax) \n\t" \
"setne %al\n\t"or
arch/x86/kernel/module.c +3
-0+#ifdef CONFIG_X86_PIE +" pushq arch_rethook_trampoline@GOTPCREL(%rip)\n" +#else" pushq $arch_rethook_trampoline\n"
+#endifinstead of having these kludgy push/pop sequences to free up temp registers.
(FYI I have looked into this PIE linking just a few weeks ago [0] so this is all rather fresh in my memory)
Quoted 6 lines
Hi Ard,
Thanks for providing the link, it has been very helpful for me as I am new to the topic of compilers.Happy to hear that.
Quoted 6 lines
One key difference I noticed is that you
linked the kernel with "-pie" instead of "--emit-reloc". I also noticed that Thomas' initial patchset[0] used "-pie", but in RFC v3 [1], it switched to "--emit-reloc" in order to reduce dynamic relocation space on mapped memory.The problem with --emit-relocs is that the relocations emitted into the binary may get out of sync with the actual code after the linker has applied relocations.
$ cat /tmp/a.s foo:movq foo@GOTPCREL(%rip), %rax$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s ard@gambale:~/linux$ x86_64-linux-gnu-objdump -dr /tmp/a.o
/tmp/a.o: file format elf64-x86-64Disassembly of section .text:
0000000000000000 <foo>:
0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7>3:
R_X86_64_REX_GOTPCRELXfoo-0x4$ x86_64-linux-gnu-gcc -c -o /tmp/a.o /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.o 0000000000000000 <foo>: 0: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 7 <foo+0x7> 3: R_X86_64_REX_GOTPCRELX foo-0x4$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-no-pie,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000401000 <foo>: 401000: 48 c7 c0 00 10 40 00 mov $0x401000,%rax 401003: R_X86_64_32S foo$ x86_64-linux-gnu-gcc -o /tmp/a.elf -nostartfiles -Wl,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 0000000000001000 <foo>: 1000: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 1000 <foo> 1003: R_X86_64_PC32 foo-0x4This all looks as expected. However, when using Clang, we end up with
$ clang -target x86_64-linux-gnu -o /tmp/a.elf -nostartfiles -fuse-ld=lld -Wl,--relax,-q,--defsym,_start=0x0 /tmp/a.s $ x86_64-linux-gnu-objdump -dr /tmp/a.elf 00000000000012c0 <foo>: 12c0: 48 8d 05 f9 ff ff ff lea -0x7(%rip),%rax # 12c0 <foo> 12c3: R_X86_64_REX_GOTPCRELX foo-0x4So in this case, what --emit-relocs gives us is not what is actually in the binary. We cannot just ignore these either, given that they are treated differently depending on whether the symbol is a per-CPU symbol or not - in the former case, we need to perform a fixup if the relaxed reference is RIP relative, and in the latter case, if the relaxed reference is absolute.
On top of that, --emit-relocs does not cover the GOT, so we'd still need to process that from the code explicitly.
Quoted 9 lines
In general, relying on --emit-relocs is kind of dodgy, and I think combining PIE linking with --emit-relocs is a bad idea.
Quoted 2 lines
The another issue is that it requires the addition of the "-mrelax-relocations=no" option to support older compilers and linkers.
Why? The decompressor is now linked in PIE mode so we should be able to drop that. Or do you need to add is somewhere else?
Hi Ard,
Quoted 13 lines
After removing the "-mrelax-relocations=no" option, I noticed that the linker was relaxing GOT references as absolute references for mov instructions, even if the symbol was in a high address, as long as I kept the compile-time base address of the kernel image in the top 2G. I consulted the "Optimize GOTPCRELX Relocations" chapter in x86-64 psABI, which stated that "When position-independent code is disabled and foo is defined locally in the lower 32-bit address space, memory operand in mov can be converted into immediate operand". However, it seemed that if the symbol was in the higher 32-bit address space, the memory operand in mov would also be converted into an immediate operand. If I decreased the compile-time base address of the kernel image, it would be relaxed as lea. Therefore, I believe that using "-mrelax-relocations=no" without "-pie" option is necessary.
Indeed. As you noted, the linker assumes that non-PIE linked binaries will always appear at their link time address, and relaxations will try to take advantage of that.
Currently, we use -pie linking only for the decompressor, and we should be able to drop -mrelax-relocations=no from its LDFLAGS. But position dependent linking should not use relaxations at all.
Quoted 3 lines
Is there a way to force the linker to relax
it as lea without using the "-pie" option when linking?
Not that I am aware of.
Quoted 3 lines
Since all GOT references cannot be omitted, perhaps I should try linking the kernel with the "-pie" option.
That way, we will end up with two sets of relocations, the static ones from --emit-relocs and the dynamic ones from -pie. This should be manageable, given that the difference between those sets should exactly cover the GOT.
However, relying on --emit-relocs and -pie at the same time seems clumsy to me. I'd prefer to only depend on -pie at /some/ point.
Signature · 1 line
Ard.
Re: [PATCH RFC 13/43] x86/paravirt: Use relative reference for original instruction
In reply to Hou WenlongQuoted 10 lines · Hou Wenlong
Similar to the alternative patching, use relative reference for original instruction rather than absolute one, which saves 8 bytes for one entry on
x86_64. And it could generateR_X86_64_PC32relocation instead ofR_X86_64_64relocation, which also reduces relocation metadata on relocatable builds. And the alignment could be hard coded to be 4 now.Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>
I think this patch should be taken even without the series.
Juergen
Re: [PATCH RFC 13/43] x86/paravirt: Use relative reference for original instruction
Quoted 15 lines · Juergen Gross
Quoted 9 lines · Hou Wenlong
Similar to the alternative patching, use relative reference for original instruction rather than absolute one, which saves 8 bytes for one entry on
x86_64. And it could generateR_X86_64_PC32relocation instead ofR_X86_64_64relocation, which also reduces relocation metadata on relocatable builds. And the alignment could be hard coded to be 4 now.Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>
Quoted 3 lines
Reviewed-by Juergen Gross <[email protected]>I think this patch should be taken even without the series.
It looks good to me, I am just not sure what the alignment is needed at all.
Why not to make the struct __packed (like struct alt_instr) and get rid of all the .align directives? Am I missing something?
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH RFC 13/43] x86/paravirt: Use relative reference for original instruction
Quoted 17 lines · Nadav Amit
Quoted 17 lines · Juergen Gross
Quoted 9 lines · Hou Wenlong
Similar to the alternative patching, use relative reference for original instruction rather than absolute one, which saves 8 bytes for one entry on
x86_64. And it could generateR_X86_64_PC32relocation instead ofR_X86_64_64relocation, which also reduces relocation metadata on relocatable builds. And the alignment could be hard coded to be 4 now.Signed-off-by Hou Wenlong <[email protected]>Cc Thomas Garnier <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Kees Cook <[email protected]>Reviewed-by Juergen Gross <[email protected]>
Quoted 7 lines
Quoted 1 line
I think this patch should be taken even without the series.
It looks good to me, I am just not sure what the alignment is needed at all.
Why not to make the struct
__packed(like structalt_instr) and get rid of all the .align directives? Am I missing something?
Yes, making the struct __packed can save more space. If I understand correctly, it could be done even without this patch but it may lead to misaligned memory access. However, it seems to not matter as I didn't find any related log for packing struct alt_instr. I can do such things if needed.
Thanks.