Function Granular KASLR
Subsystems (20)
v6 cover Function Granular KASLR
This is a massive rework and a respin of Kristen Accardi's marvellous FG-KASLR series (v5).
The major differences since v5 [0]:
* You can now tune the number of functions per each section to achieve the preferable vmlinux size or protection level. Default is still as one section per each function.
This can be handy for storage-constrained systems. 4-8 fps are still strong, but reduce the size of the final vmlinu{x,z} significantly;
* I don't use orphan sections anymore. It's not reliable at all / may differ from linker to linker, and also conflicts with CONFIG_LD_ORPHAN_WARN which is great for catching random bugs ->
* All the .text.* sections are now being described explicitly in the linker script. A Perl script is used to take the original LDS, the original object file, read a list of input sections from it and generate the resulting LDS.
This costs a bit of linking time as LD tends to think hard when processing scripts > 1 Mb. It adds about 40-60 seconds to the whole linking process (BTF step, 2-3 kallsyms steps and the final step), but "better safe than sorry".
In addition, that approach allows to reserve some space at the end and add some link assertions ->
* Input .text section now must be empty, otherwise the linkage will be stopped. This is implemented by the size assertion in the resulting LD script and is designed to plug the potentional layout leakage. This also means that ->
* "Regular" ASM functions are now being placed into unique separate functions the same way compiler does this for C functions. This is achieved by introducing and using several new macros which take the symbol name as a base for its new section name. This gives a better opportunity to both DCE and FG-KASLR, as ASM code now can also be randomized or garbage-collected;
* It's now fully compatible with ClangLTO, ClangCFI, CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last revision was published;
* Includes several fixes: relocations inside .altinstr_replacement code and minor issues found and/or suggested by LKP robot.
The series was compile-time and runtime tested on the following setups with no issues:
- x86_64, GCC 11, Binutils 2.35;
- x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).
The first 4 patches are from the linux-kbuild tree and included to avoid merge conflicts and non-intuitive resolving of them.
The series is also available here: [1]
[0] https://lore.kernel.org/kernel-hardening/[email protected]
[1] https://github.com/alobakin/linux/pull/3
The original v5 cover letter:
Function Granular Kernel Address Space Layout Randomization (fgkaslr)
---------------------------------------------------------------------
This patch set is an implementation of finer grained kernel address space randomization. It rearranges your kernel code at load time
on a per-function level granularity, with only around a second added to boot time.
Changes in v5:
--------------
* fixed a bug in the code which increases boot heap size for CONFIG_FG_KASLR which prevented the boot heap from being increased for CONFIG_FG_KASLR when using bzip2 compression. Thanks to Andy Lavr for finding the problem and identifying the solution.
* changed the adjustment of the orc_unwind_ip table at boot time to disregard relocs associated with this table, and instead inspect the entries separately. Relocs are not able to be used since they are no longer correct once the table is resorted at buildtime.
* changed how orc_unwind_ip addresses in randomized sections are identified to include the byte immediately after the end of the section.
* updated module code to use kvmalloc/kvfree based on suggestions from Evgenii Shatokhin <[email protected]>.
* changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to match the nokaslr option. fgkaslr="X" can be added at a later date if it is needed.
* Added a patch to force livepatch to require symbols to be unique if using while fgkaslr either for core or modules.
Changes in v4:
-------------
* dropped the patch to split out change to STATIC definition in x86/boot/compressed/misc.c and replaced with a patch authored by Kees Cook to avoid the duplicate malloc definitions
* Added a section to Documentation/admin-guide/kernel-parameters.txt to document the fgkaslr boot option.
* redesigned the patch to hide the new layout when reading /proc/kallsyms. The previous implementation utilized a dynamically allocated linked list to display the kernel and module symbols in alphabetical order. The new implementation uses a randomly shuffled index array to display the kernel and module symbols in a random order.
Changes in v3:
-------------
* Makefile changes to accommodate CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
* removal of extraneous ALIGN_PAGE from _etext changes
* changed variable names in x86/tools/relocs to be less confusing
* split out change to STATIC definition in x86/boot/compressed/misc.c
* Updates to Documentation to make it more clear what is preserved in .text
* much more detailed commit message for function granular KASLR patch
* minor tweaks and changes that make for more readable code
* this cover letter updated slightly to add additional details
Changes in v2:
--------------
* Fix to address i386 build failure
* Allow module reordering patch to be configured separately so that arm (or other non-x86_64 arches) can take advantage of module function reordering. This support has not be tested by me, but smoke tested by Ard Biesheuvel <[email protected]> on arm.
* Fix build issue when building on arm as reported by
Ard Biesheuvel <[email protected]>
Patches to objtool are included because they are dependencies for this patchset, however they have been submitted by their maintainer separately.
Background
----------
KASLR was merged into the kernel with the objective of increasing the difficulty of code reuse attacks. Code reuse attacks reused existing code snippets to get around existing memory protections. They exploit software bugs which expose addresses of useful code snippets to control the flow of execution for their own nefarious purposes. KASLR moves the entire kernel code text as a unit at boot time in order to make addresses less predictable. The order of the code within the segment is unchanged - only the base address is shifted. There are a few shortcomings to this algorithm.
1. Low Entropy - there are only so many locations the kernel can fit in. This means an attacker could guess without too much trouble.
2. Knowledge of a single address can reveal the offset of the base address, exposing all other locations for a published/known kernel image.
3. Info leaks abound.
Finer grained ASLR has been proposed as a way to make ASLR more resistant to info leaks. It is not a new concept at all, and there are many variations possible. Function reordering is an implementation of finer grained ASLR which randomizes the layout of an address space on a function level granularity. We use the term "fgkaslr" in this document to refer to the technique of function reordering when used with KASLR, as well as finer grained KASLR in general.
Proposed Improvement
--------------------
This patch set proposes adding function reordering on top of the existing KASLR base address randomization. The over-arching objective is incremental improvement over what we already have. It is designed to work in combination with the existing solution. The implementation is really pretty simple, and there are 2 main area where changes occur:
* Build time
GCC has had an option to place functions into individual .text sections for many years now. This option can be used to implement function reordering at load time. The final compiled vmlinux retains all the section headers, which can be used to help find the address ranges of each function. Using this information and an expanded table of relocation addresses, individual text sections can be suffled immediately after decompression. Some data tables inside the kernel that have assumptions about order require re-sorting after being updated when applying relocations. In order to modify these tables, a few key symbols are excluded from the objcopy symbol stripping process for use after shuffling the text segments.
Some highlights from the build time changes to look for:
The top level kernel Makefile was modified to add the gcc flag if it is supported. Currently, I am applying this flag to everything it is possible to randomize. Anything that is written in C and not present in a special input section is randomized. The final binary segment 0 retains a consolidated .text section, as well as all the individual .text.* sections. Future work could turn off this flags for selected files or even entire subsystems, although obviously at the cost of security.
The relocs tool is updated to add relative relocations. This information previously wasn't included because it wasn't necessary when moving the entire .text segment as a unit.
A new file was created to contain a list of symbols that objcopy should keep. We use those symbols at load time as described below.
* Load time
The boot kernel was modified to parse the vmlinux elf file after decompression to check for our interesting symbols that we kept, and to look for any .text.* sections to randomize. The consolidated .text section is skipped and not moved. The sections are shuffled randomly, and copied into memory following the .text section in a new random order. The existing code which updated relocation addresses was modified to account for not just a fixed delta from the load address, but the offset that the function section was moved to. This requires inspection of each address to see if it was impacted by a randomization. We use a bsearch to make this less horrible on performance. Any tables that need to be modified with new addresses or resorted are updated using the symbol addresses parsed from the elf symbol table.
In order to hide our new layout, symbols reported through /proc/kallsyms will be displayed in a random order.
Security Considerations
-----------------------
The objective of this patch set is to improve a technology that is already merged into the kernel (KASLR). This code will not prevent all attacks, but should instead be considered as one of several tools that can be used. In particular, this code is meant to make KASLR more effective in the presence of info leaks.
How much entropy we are adding to the existing entropy of standard KASLR will depend on a few variables. Firstly and most obviously, the number of functions that are randomized matters. This implementation keeps the existing .text section for code that cannot be randomized - for example, because it was assembly code. The less sections to randomize, the less entropy. In addition, due to alignment (16 bytes for x86_64), the number of bits in a address that the attacker needs to guess is reduced, as the lower bits are identical.
Performance Impact
------------------
There are two areas where function reordering can impact performance: boot time latency, and run time performance.
* Boot time latency
This implementation of finer grained KASLR impacts the boot time of the kernel in several places. It requires additional parsing of the kernel ELF file to obtain the section headers of the sections to be randomized. It calls the random number generator for each section to be randomized to determine that section's new memory location. It copies the decompressed kernel into a new area of memory to avoid corruption when laying out the newly randomized sections. It increases the number of relocations the kernel has to perform at boot time vs. standard KASLR, and it also requires a lookup on each address that needs to be relocated to see if it was in a randomized section and needs to be adjusted by a new offset. Finally, it re-sorts a few data tables that are required to be sorted by address.
Booting a test VM on a modern, well appointed system showed an increase in latency of approximately 1 second.
* Run time
The performance impact at run-time of function reordering varies by workload. Using kcbench, a kernel compilation benchmark, the performance of a kernel build with finer grained KASLR was about 1% slower than a kernel with standard KASLR. Analysis with perf showed a slightly higher percentage of L1-icache-load-misses. Other workloads were examined as well, with varied results. Some workloads performed significantly worse under FGKASLR, while others stayed the same or were mysteriously better. In general, it will depend on the code flow whether or not finer grained KASLR will impact your workload, and how the underlying code was designed. Because the layout changes per boot, each time a system is rebooted the performance of a workload may change.
Future work could identify hot areas that may not be randomized and either leave them in the .text section or group them together into a single section that may be randomized. If grouping things together helps, one other thing to consider is that if we could identify text blobs that should be grouped together to benefit a particular code flow, it could be interesting to explore whether this security feature could be also be used as a performance feature if you are interested in optimizing your kernel layout for a particular workload at boot time. Optimizing function layout for a particular workload has been researched and proven effective - for more information read the Facebook paper "Optimizing Function Placement for Large-Scale Data-Center Applications" (see references section below).
Image Size
----------
Adding additional section headers as a result of compiling with -ffunction-sections will increase the size of the vmlinux ELF file. With a standard distro config, the resulting vmlinux was increased by about 3%. The compressed image is also increased due to the header files, as well as the extra relocations that must be added. You can expect fgkaslr to increase the size of the compressed image by about 15%.
Memory Usage
------------
fgkaslr increases the amount of heap that is required at boot time, although this extra memory is released when the kernel has finished decompression. As a result, it may not be appropriate to use this feature on systems without much memory.
Building
--------
To enable fine grained KASLR, you need to have the following config options set (including all the ones you would use to build normal KASLR)
CONFIG_FG_KASLR=y
In addition, fgkaslr is only supported for the X86_64 architecture.
Modules
-------
Modules are randomized similarly to the rest of the kernel by shuffling the sections at load time prior to moving them into memory. The module must also have been build with the -ffunction-sections compiler option.
Although fgkaslr for the kernel is only supported for the X86_64 architecture, it is possible to use fgkaslr with modules on other architectures. To enable this feature, select
CONFIG_MODULE_FG_KASLR=y
This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.
Disabling
---------
Disabling normal KASLR using the nokaslr command line option also disables fgkaslr. It is also possible to disable fgkaslr separately by booting with nofgkaslr on the commandline.
References
----------
There are a lot of academic papers which explore finer grained ASLR. This paper in particular contributed the most to my implementation design as well as my overall understanding of the problem space:
Selfrando: Securing the Tor Browser against De-anonymization Exploits, M. Conti, S. Crane, T. Frassetto, et al.
For more information on how function layout impacts performance, see:
Optimizing Function Placement for Large-Scale Data-Center Applications, G. Ottoni, B. Maher
Alexander Lobakin (7):
linkage: add macros for putting ASM functions into own sections
x86: conditionally place regular ASM functions into separate sections
FG-KASLR: use a scripted approach to handle .text.* sections x86/boot: allow FG-KASLR to be selected
arm64/crypto: conditionally place ASM functions into separate sections
module: use a scripted approach for FG-KASLR
maintainers: add MAINTAINERS entry for FG-KASLR
Kees Cook (2):
x86/boot: Allow a "silent" kaslr random byte fetch x86/boot/compressed: Avoid duplicate malloc() implementations
Kristen Carlson Accardi (9):
x86: tools/relocs: Support >64K section headers
x86: Makefile: Add build and config option for CONFIG_FG_KASLR Make sure ORC lookup covers the entire _etext - _stext x86/tools: Add relative relocs for randomized functions
x86: Add support for function granular KASLR
kallsyms: Hide layout
livepatch: only match unique symbols when using fgkaslr
module: Reorder functions
Documentation: add a documentation for FG-KASLR
Masahiro Yamada (3):
kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
kbuild: merge vmlinux_link() between ARCH=um and other architectures
Sami Tolvanen (1):
kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG
.../admin-guide/kernel-parameters.txt | 6 + Documentation/security/fgkaslr.rst | 172 ++++ Documentation/security/index.rst | 1 + MAINTAINERS | 12 + Makefile | 17 +- arch/Kconfig | 3 + arch/arm64/crypto/aes-ce-ccm-core.S | 16 +- arch/arm64/crypto/aes-ce-core.S | 16 +- arch/arm64/crypto/aes-ce.S | 4 +- arch/arm64/crypto/aes-cipher-core.S | 8 +- arch/arm64/crypto/aes-modes.S | 16 +- arch/arm64/crypto/aes-neon.S | 4 +- arch/arm64/crypto/aes-neonbs-core.S | 38 +- arch/arm64/crypto/chacha-neon-core.S | 18 +- arch/arm64/crypto/crct10dif-ce-core.S | 14 +- arch/arm64/crypto/ghash-ce-core.S | 24 +- arch/arm64/crypto/nh-neon-core.S | 4 +- arch/arm64/crypto/poly1305-armv8.pl | 17 + arch/arm64/crypto/sha1-ce-core.S | 4 +- arch/arm64/crypto/sha2-ce-core.S | 4 +- arch/arm64/crypto/sha3-ce-core.S | 4 +- arch/arm64/crypto/sha512-armv8.pl | 11 + arch/arm64/crypto/sha512-ce-core.S | 4 +- arch/arm64/crypto/sm3-ce-core.S | 4 +- arch/arm64/crypto/sm4-ce-core.S | 4 +- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 9 +- arch/x86/boot/compressed/fgkaslr.c | 905 ++++++++++++++++++ arch/x86/boot/compressed/kaslr.c | 4 - arch/x86/boot/compressed/misc.c | 157 ++- arch/x86/boot/compressed/misc.h | 30 + arch/x86/boot/compressed/utils.c | 13 + arch/x86/boot/compressed/vmlinux.symbols | 19 + arch/x86/crypto/aegis128-aesni-asm.S | 36 +- arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 12 +- arch/x86/crypto/aesni-intel_asm.S | 116 ++- arch/x86/crypto/aesni-intel_avx-x86_64.S | 32 +- arch/x86/crypto/blake2s-core.S | 8 +- arch/x86/crypto/blowfish-x86_64-asm_64.S | 16 +- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 28 +- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 28 +- arch/x86/crypto/camellia-x86_64-asm_64.S | 16 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 24 +- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/chacha-avx2-x86_64.S | 12 +- arch/x86/crypto/chacha-avx512vl-x86_64.S | 12 +- arch/x86/crypto/chacha-ssse3-x86_64.S | 16 +- arch/x86/crypto/crc32-pclmul_asm.S | 4 +- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 4 +- arch/x86/crypto/crct10dif-pcl-asm_64.S | 4 +- arch/x86/crypto/des3_ede-asm_64.S | 8 +- arch/x86/crypto/ghash-clmulni-intel_asm.S | 12 +- arch/x86/crypto/nh-avx2-x86_64.S | 4 +- arch/x86/crypto/nh-sse2-x86_64.S | 4 +- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 8 +- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/serpent-avx2-asm_64.S | 20 +- arch/x86/crypto/serpent-sse2-i586-asm_32.S | 8 +- arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 8 +- arch/x86/crypto/sha1_avx2_x86_64_asm.S | 4 +- arch/x86/crypto/sha1_ni_asm.S | 4 +- arch/x86/crypto/sha1_ssse3_asm.S | 4 +- arch/x86/crypto/sha256-avx-asm.S | 4 +- arch/x86/crypto/sha256-avx2-asm.S | 4 +- arch/x86/crypto/sha256-ssse3-asm.S | 4 +- arch/x86/crypto/sha256_ni_asm.S | 4 +- arch/x86/crypto/sha512-avx-asm.S | 4 +- arch/x86/crypto/sha512-avx2-asm.S | 4 +- arch/x86/crypto/sha512-ssse3-asm.S | 4 +- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/twofish-i586-asm_32.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64.S | 8 +- arch/x86/entry/entry_32.S | 24 +- arch/x86/entry/entry_64.S | 18 +- arch/x86/entry/thunk_32.S | 4 +- arch/x86/entry/thunk_64.S | 8 +- arch/x86/include/asm/boot.h | 13 +- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 2 +- arch/x86/kernel/acpi/wakeup_32.S | 9 +- arch/x86/kernel/acpi/wakeup_64.S | 10 +- arch/x86/kernel/ftrace_32.S | 19 +- arch/x86/kernel/ftrace_64.S | 28 +- arch/x86/kernel/irqflags.S | 4 +- arch/x86/kernel/kprobes/core.c | 3 +- arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/relocate_kernel_32.S | 2 + arch/x86/kernel/relocate_kernel_64.S | 2 + arch/x86/kernel/vmlinux.lds.S | 6 +- arch/x86/kvm/emulate.c | 2 +- arch/x86/kvm/vmx/vmenter.S | 8 +- arch/x86/lib/clear_page_64.S | 12 +- arch/x86/lib/cmpxchg16b_emu.S | 4 +- arch/x86/lib/copy_mc_64.S | 8 +- arch/x86/lib/copy_page_64.S | 7 +- arch/x86/lib/copy_user_64.S | 18 +- arch/x86/lib/csum-copy_64.S | 4 +- arch/x86/lib/error-inject.c | 3 +- arch/x86/lib/getuser.S | 37 +- arch/x86/lib/hweight.S | 9 +- arch/x86/lib/iomap_copy_64.S | 4 +- arch/x86/lib/kaslr.c | 18 +- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 12 +- arch/x86/lib/msr-reg.S | 8 +- arch/x86/lib/putuser.S | 18 +- arch/x86/mm/mem_encrypt_boot.S | 8 +- arch/x86/platform/efi/efi_stub_64.S | 4 +- arch/x86/platform/efi/efi_thunk_64.S | 4 +- arch/x86/power/hibernate_asm_32.S | 14 +- arch/x86/power/hibernate_asm_64.S | 14 +- arch/x86/tools/relocs.c | 135 ++- arch/x86/tools/relocs.h | 4 +- arch/x86/tools/relocs_common.c | 15 +- arch/x86/xen/xen-asm.S | 49 +- arch/x86/xen/xen-head.S | 10 +- include/asm-generic/vmlinux.lds.h | 41 +- include/linux/decompress/mm.h | 12 +- include/linux/linkage.h | 76 ++ include/uapi/linux/elf.h | 1 + init/Kconfig | 51 + kernel/kallsyms.c | 158 ++- kernel/livepatch/core.c | 11 + kernel/module.c | 91 +- scripts/Makefile.build | 27 +- scripts/Makefile.lib | 7 + scripts/Makefile.modfinal | 36 +- scripts/Makefile.modpost | 22 +- scripts/gen_autoksyms.sh | 12 - scripts/generate_text_sections.pl | 149 +++ scripts/link-vmlinux.sh | 104 +- scripts/module.lds.S | 14 +- 133 files changed, 2771 insertions(+), 757 deletions(-) create mode 100644 Documentation/security/fgkaslr.rst create mode 100644 arch/x86/boot/compressed/fgkaslr.c create mode 100644 arch/x86/boot/compressed/utils.c create mode 100644 arch/x86/boot/compressed/vmlinux.symbols create mode 100755 scripts/generate_text_sections.pl
Signature · 1 line
2.31.1
PATCH v6 1/22 kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG
From: Sami Tolvanen <[email protected]>
With CONFIG_LTO_CLANG, we currently link modules into native code just before modpost, which means with TRIM_UNUSED_KSYMS enabled, we still look at the LLVM bitcode in the .o files when generating the list of used symbols. As the bitcode doesn't yet have calls to compiler intrinsics and llvm-nm doesn't see function references that only exist in function-level inline assembly, we currently need a whitelist for TRIM_UNUSED_KSYMS to work with LTO.
This change moves module LTO linking to happen earlier, and thus avoids the issue with LLVM bitcode and TRIM_UNUSED_KSYMS entirely, allowing us to also drop the whitelist from gen_autoksyms.sh.
--- scripts/Makefile.build | 27 ++++++++++++++++++++++++++- scripts/Makefile.lib | 7 +++++++ scripts/Makefile.modfinal | 21 ++------------------- scripts/Makefile.modpost | 22 +++------------------- scripts/gen_autoksyms.sh | 12 ------------ 5 files changed, 38 insertions(+), 51 deletions(-)
scripts/Makefile.build +26 -1
@@ -88,6 +88,10 @@ endif targets-for-modules := $(patsubst %.o, %.mod, $(filter %.o, $(obj-m))) +ifdef CONFIG_LTO_CLANG +targets-for-modules += $(patsubst %.o, %.lto.o, $(filter %.o, $(obj-m))) +endif + ifdef need-modorder targets-for-modules += $(obj)/modules.order endif
@@ -271,12 +275,33 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $$(objtool_dep) FORCE $(call if_changed_rule,cc_o_c) $(call cmd,force_checksrc) +ifdef CONFIG_LTO_CLANG +# Module .o files may contain LLVM bitcode, compile them into native code +# before ELF processing +quiet_cmd_cc_lto_link_modules = LTO [M] $@ +cmd_cc_lto_link_modules = \ + $(LD) $(ld_flags) -r -o $@ \ + $(shell [ -s $(@:.lto.o=.o.symversions) ] && \ + echo -T $(@:.lto.o=.o.symversions)) \ + --whole-archive $(filter-out FORCE,$^) + +ifdef CONFIG_STACK_VALIDATION +# objtool was skipped for LLVM bitcode, run it now that we have compiled +# modules into native code +cmd_cc_lto_link_modules += ; \ + $(objtree)/tools/objtool/objtool $(objtool_args) --module $@ +endif + +$(obj)/%.lto.o: $(obj)/%.o FORCE + $(call if_changed,cc_lto_link_modules) +endif + cmd_mod = { \ echo $(if $($*-objs)$($*-y)$($*-m), $(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), $(@:.mod=.o)); \ $(undefined_syms) echo; \ } > $@ -$(obj)/%.mod: $(obj)/%.o FORCE +$(obj)/%.mod: $(obj)/%$(mod-prelink-ext).o FORCE $(call if_changed,mod) quiet_cmd_cc_lst_c = MKLST $@
scripts/Makefile.lib +7 -0
@@ -225,6 +225,13 @@ dtc_cpp_flags = -Wp,-MMD,$(depfile).pre.tmp -nostdinc \ $(addprefix -I,$(DTC_INCLUDE)) \ -undef -D__DTS__ +ifeq ($(CONFIG_LTO_CLANG),y) +# With CONFIG_LTO_CLANG, .o files in modules might be LLVM bitcode, so we +# need to run LTO to compile them into native code (.lto.o) before further +# processing. +mod-prelink-ext := .lto +endif + # Objtool arguments are also needed for modfinal with LTO, so we define # then here to avoid duplication. objtool_args = \
scripts/Makefile.modfinal +2 -19
@@ -9,7 +9,7 @@ __modfinal: include include/config/auto.conf include $(srctree)/scripts/Kbuild.include -# for c_flags and objtool_args +# for c_flags and mod-prelink-ext include $(srctree)/scripts/Makefile.lib # find all modules listed in modules.order
@@ -30,23 +30,6 @@ quiet_cmd_cc_o_c = CC [M] $@ ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink) -ifdef CONFIG_LTO_CLANG -# With CONFIG_LTO_CLANG, reuse the object file we compiled for modpost to -# avoid a second slow LTO link -prelink-ext := .lto - -# ELF processing was skipped earlier because we didn't have native code, -# so let's now process the prelinked binary before we link the module. - -ifdef CONFIG_STACK_VALIDATION -cmd_ld_ko_o += \ - $(objtree)/tools/objtool/objtool $(objtool_args) \ - $(@:.ko=$(prelink-ext).o); - -endif # CONFIG_STACK_VALIDATION - -endif # CONFIG_LTO_CLANG - quiet_cmd_ld_ko_o = LD [M] $@ cmd_ld_ko_o += \ $(LD) -r $(KBUILD_LDFLAGS) \
@@ -72,7 +55,7 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \ # Re-generate module BTFs if either module's .ko or vmlinux changed -$(modules): %.ko: %$(prelink-ext).o %.mod.o scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE +$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE +$(call if_changed_except,ld_ko_o,vmlinux) ifdef CONFIG_DEBUG_INFO_BTF_MODULES +$(if $(newer-prereqs),$(call cmd,btf_ko))
scripts/Makefile.modpost +3 -19
@@ -41,7 +41,7 @@ __modpost: include include/config/auto.conf include $(srctree)/scripts/Kbuild.include -# for ld_flags +# for mod-prelink-ext include $(srctree)/scripts/Makefile.lib MODPOST = scripts/mod/modpost \
@@ -118,22 +118,6 @@ $(input-symdump): @echo >&2 ' Modules may not have dependencies or modversions.' @echo >&2 ' You may get many unresolved symbol warnings.' -ifdef CONFIG_LTO_CLANG -# With CONFIG_LTO_CLANG, .o files might be LLVM bitcode, so we need to run -# LTO to compile them into native code before running modpost -prelink-ext := .lto - -quiet_cmd_cc_lto_link_modules = LTO [M] $@ -cmd_cc_lto_link_modules = \ - $(LD) $(ld_flags) -r -o $@ \ - $(shell [ -s $(@:.lto.o=.o.symversions) ] && \ - echo -T $(@:.lto.o=.o.symversions)) \ - --whole-archive $^ - -%.lto.o: %.o - $(call if_changed,cc_lto_link_modules) -endif - modules := $(sort $(shell cat $(MODORDER))) # KBUILD_MODPOST_WARN can be set to avoid error out in case of undefined symbols
@@ -144,9 +128,9 @@ endif # Read out modules.order to pass in modpost. # Otherwise, allmodconfig would fail with "Argument list too long". quiet_cmd_modpost = MODPOST $@ - cmd_modpost = sed 's/\.ko$$/$(prelink-ext)\.o/' $< | $(MODPOST) -T - + cmd_modpost = sed 's/\.ko$$/$(mod-prelink-ext)\.o/' $< | $(MODPOST) -T - -$(output-symdump): $(MODORDER) $(input-symdump) $(modules:.ko=$(prelink-ext).o) FORCE +$(output-symdump): $(MODORDER) $(input-symdump) $(modules:.ko=$(mod-prelink-ext).o) FORCE $(call if_changed,modpost) targets += $(output-symdump)
scripts/gen_autoksyms.sh +0 -12
@@ -26,18 +26,6 @@ if [ -n "$CONFIG_MODVERSIONS" ]; then needed_symbols="$needed_symbols module_layout" fi -# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary -# when the .mod files are generated, which means they don't yet contain -# references to certain symbols that will be present in the final binaries. -if [ -n "$CONFIG_LTO_CLANG" ]; then - # intrinsic functions - needed_symbols="$needed_symbols memcpy memmove memset" - # ftrace - needed_symbols="$needed_symbols _mcount" - # stack protector symbols - needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard" -fi - ksym_wl= if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then # Use 'eval' to expand the whitelist path and check if it is relative
Signature · 1 line
2.31.1
PATCH v6 2/22 kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
In reply to Alexander LobakinFrom: Masahiro Yamada <[email protected]>
When Clang LTO is enabled, vmlinux_link() reuses vmlinux.o instead of re-linking ${KBUILD_VMLINUX_OBJS} and ${KBUILD_VMLINUX_LIBS}.
That is the only difference here, so merge the similar code.
--- scripts/link-vmlinux.sh | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)
scripts/link-vmlinux.sh +14 -16
@@ -154,12 +154,23 @@ vmlinux_link() local objects local strip_debug local map_option + local objs + local libs info LD ${output} # skip output file argument shift + if [ -n "${CONFIG_LTO_CLANG}" ]; then + # Use vmlinux.o instead of performing the slow LTO link again. + objs=vmlinux.o + libs= + else + objs="${KBUILD_VMLINUX_OBJS}" + libs="${KBUILD_VMLINUX_LIBS}" + fi + # The kallsyms linking does not need debug symbols included. if [ "$output" != "${output#.tmp_vmlinux.kallsyms}" ] ; then strip_debug=-Wl,--strip-debug
@@ -170,22 +181,9 @@ vmlinux_link() fi if [ "${SRCARCH}" != "um" ]; then - if [ -n "${CONFIG_LTO_CLANG}" ]; then - # Use vmlinux.o instead of performing the slow LTO - # link again. - objects="--whole-archive \ - vmlinux.o \ - --no-whole-archive \ - ${@}" - else - objects="--whole-archive \ - ${KBUILD_VMLINUX_OBJS} \ - --no-whole-archive \ - --start-group \ - ${KBUILD_VMLINUX_LIBS} \ - --end-group \ - ${@}" - fi + objects="--whole-archive ${objs} --no-whole-archive \ + --start-group ${libs} --end-group \ + $@" ${LD} ${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux} \ ${strip_debug#-Wl,} \
Signature · 1 line
2.31.1
PATCH v6 3/22 kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
In reply to Alexander LobakinFrom: Masahiro Yamada <[email protected]>
arch/um/Makefile passes the -f option to the ln command:
linux: vmlinux
@echo ' LINK $@'
$(Q)ln -f $< $@So, the hard link is always re-created, and the old one is removed anyway.
--- scripts/link-vmlinux.sh | 1 - 1 file changed, 1 deletion(-)
scripts/link-vmlinux.sh +0 -1
@@ -206,7 +206,6 @@ vmlinux_link() -Wl,-T,${lds} \ ${objects} \ -lutil -lrt -lpthread - rm -f linux fi }
Signature · 1 line
2.31.1
PATCH v6 4/22 kbuild: merge vmlinux_link() between ARCH=um and other architectures
In reply to Alexander LobakinFrom: Masahiro Yamada <[email protected]>
For ARCH=um, ${CC} is used as the linker driver. Hence, the linker options are prefixed with -Wl, .
Merge the similar code.
I replaced the -T option with the long option --script= so that it works well with/without ${wl}.
--- scripts/link-vmlinux.sh | 56 +++++++++++++++++------------------------ 1 file changed, 23 insertions(+), 33 deletions(-)
scripts/link-vmlinux.sh +23 -33
@@ -149,13 +149,12 @@ objtool_link() # ${2}, ${3}, ... - optional extra .o files vmlinux_link() { - local lds="${objtree}/${KBUILD_LDS}" local output=${1} - local objects - local strip_debug - local map_option local objs local libs + local ld + local ldflags + local ldlibs info LD ${output}
@@ -171,42 +170,33 @@ vmlinux_link() libs="${KBUILD_VMLINUX_LIBS}" fi + if [ "${SRCARCH}" = "um" ]; then + wl=-Wl, + ld="${CC}" + ldflags="${CFLAGS_vmlinux}" + ldlibs="-lutil -lrt -lpthread" + else + wl= + ld="${LD}" + ldflags="${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux}" + ldlibs= + fi + + ldflags="${ldflags} ${wl}--script=${objtree}/${KBUILD_LDS}" + # The kallsyms linking does not need debug symbols included. if [ "$output" != "${output#.tmp_vmlinux.kallsyms}" ] ; then - strip_debug=-Wl,--strip-debug + ldflags="${ldflags} ${wl}--strip-debug" fi if [ -n "${CONFIG_VMLINUX_MAP}" ]; then - map_option="-Map=${output}.map" + ldflags="${ldflags} ${wl}-Map=${output}.map" fi - if [ "${SRCARCH}" != "um" ]; then - objects="--whole-archive ${objs} --no-whole-archive \ - --start-group ${libs} --end-group \ - $@" - - ${LD} ${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux} \ - ${strip_debug#-Wl,} \ - -o ${output} \ - ${map_option} \ - -T ${lds} ${objects} - else - objects="-Wl,--whole-archive \ - ${KBUILD_VMLINUX_OBJS} \ - -Wl,--no-whole-archive \ - -Wl,--start-group \ - ${KBUILD_VMLINUX_LIBS} \ - -Wl,--end-group \ - ${@}" - - ${CC} ${CFLAGS_vmlinux} \ - ${strip_debug} \ - -o ${output} \ - ${map_option:+-Wl,${map_option}} \ - -Wl,-T,${lds} \ - ${objects} \ - -lutil -lrt -lpthread - fi + ${ld} ${ldflags} -o ${output} \ + ${wl}--whole-archive ${objs} ${wl}--no-whole-archive \ + ${wl}--start-group ${libs} ${wl}--end-group \ + $@ ${ldlibs} } # generate .BTF typeinfo from DWARF debuginfo
Signature · 1 line
2.31.1
PATCH v6 5/22 x86: tools/relocs: Support >64K section headers
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
While the relocs tool already supports finding the total number of section headers if vmlinux exceeds 64K sections, it fails to read the extended symbol table to get section header indexes for symbols, causing incorrect symbol table indexes to be used when there are > 64K symbols.
Parse the elf file to read the extended symbol table info, and then replace all direct references to st_shndx with calls to sym_index(), which will determine whether the value can be read directly or whether the value should be pulled out of the extended table.
--- arch/x86/tools/relocs.c | 103 ++++++++++++++++++++++++++++++---------- 1 file changed, 78 insertions(+), 25 deletions(-)
arch/x86/tools/relocs.c +78 -25
@@ -14,6 +14,10 @@static Elf_Ehdr ehdr;static unsigned long shnum;static unsigned int shstrndx; +static unsigned int shsymtabndx; +static unsigned int shxsymtabndx; + +static int sym_index (Elf_Sym *sym);struct relocs {uint32_t *offset;
@@ -32,6 +36,7 @@ struct section {Elf_Shdr shdr;struct section *link;Elf_Sym *symtab; +Elf32_Word *xsymtab;Elf_Rel *reltab;char *strtab; };
@@ -265,7 +270,7 @@ static const char *sym_name(const char *sym_strtab, Elf_Sym *sym) name = sym_strtab + sym->st_name; }else { - name =sec_name (sym->st_shndx); + name =sec_name (sym_index (sym)); }return name; }
@@ -335,6 +340,23 @@ static uint64_t elf64_to_cpu(uint64_t val)#define elf_xword_to_cpu (x) elf32_to_cpu(x)#endif +static int sym_index (Elf_Sym *sym) +{ +Elf_Sym *symtab = secs[shsymtabndx].symtab; +Elf32_Word *xsymtab = secs[shxsymtabndx].xsymtab; +unsigned long offset; +int index; + +if (sym->st_shndx !=SHN_XINDEX ) +return sym->st_shndx; + +/* calculate offset of sym from head of table. */ + offset = (unsigned long )sym - (unsigned long )symtab; + index = offset /sizeof (*sym); + +return elf32_to_cpu (xsymtab[index]); +} +static void read_ehdr (FILE *fp) {if (fread (&ehdr,sizeof (ehdr),1 , fp) !=1 ) {
@@ -468,31 +490,60 @@ static void read_strtabs(FILE *fp)static void read_symtabs (FILE *fp) {int i,j; +for (i =0 ; i < shnum; i++) {struct section *sec = &secs[i]; -if (sec->shdr.sh_type !=SHT_SYMTAB ) { +int num_syms; + +switch (sec->shdr.sh_type) { +case SHT_SYMTAB_SHNDX : + sec->xsymtab =malloc (sec->shdr.sh_size); +if (!sec->xsymtab) { +die ("malloc of %d bytes for xsymtab failed\n" , + sec->shdr.sh_size); + } +if (fseek (fp, sec->shdr.sh_offset,SEEK_SET ) <0 ) { +die ("Seek to %d failed: %s\n" , + sec->shdr.sh_offset,strerror (errno)); + } +if (fread (sec->xsymtab,1 , sec->shdr.sh_size, fp) + != sec->shdr.sh_size) { +die ("Cannot read extended symbol table: %s\n" , +strerror (errno)); + } + shxsymtabndx = i; +continue ; + +case SHT_SYMTAB : + num_syms = sec->shdr.sh_size /sizeof (Elf_Sym); + + sec->symtab =malloc (sec->shdr.sh_size); +if (!sec->symtab) { +die ("malloc of %d bytes for symtab failed\n" , + sec->shdr.sh_size); + } +if (fseek (fp, sec->shdr.sh_offset,SEEK_SET ) <0 ) { +die ("Seek to %d failed: %s\n" , + sec->shdr.sh_offset,strerror (errno)); + } +if (fread (sec->symtab,1 , sec->shdr.sh_size, fp) + != sec->shdr.sh_size) { +die ("Cannot read symbol table: %s\n" , +strerror (errno)); + } +for (j =0 ; j < num_syms; j++) { +Elf_Sym *sym = &sec->symtab[j]; + + sym->st_name =elf_word_to_cpu (sym->st_name); + sym->st_value =elf_addr_to_cpu (sym->st_value); + sym->st_size =elf_xword_to_cpu (sym->st_size); + sym->st_shndx =elf_half_to_cpu (sym->st_shndx); + } + shsymtabndx = i; +continue ; + +default :continue ; - } - sec->symtab =malloc (sec->shdr.sh_size); -if (!sec->symtab) { -die ("malloc of %d bytes for symtab failed\n" , - sec->shdr.sh_size); - } -if (fseek (fp, sec->shdr.sh_offset,SEEK_SET ) <0 ) { -die ("Seek to %d failed: %s\n" , - sec->shdr.sh_offset,strerror (errno)); - } -if (fread (sec->symtab,1 , sec->shdr.sh_size, fp) - != sec->shdr.sh_size) { -die ("Cannot read symbol table: %s\n" , -strerror (errno)); - } -for (j =0 ; j < sec->shdr.sh_size/sizeof (Elf_Sym); j++) { -Elf_Sym *sym = &sec->symtab[j]; - sym->st_name =elf_word_to_cpu (sym->st_name); - sym->st_value =elf_addr_to_cpu (sym->st_value); - sym->st_size =elf_xword_to_cpu (sym->st_size); - sym->st_shndx =elf_half_to_cpu (sym->st_shndx); } } }
@@ -759,7 +810,9 @@ static void percpu_init(void)*/ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname) { -return (sym->st_shndx == per_cpu_shndx) && +int shndx =sym_index (sym); + +return (shndx == per_cpu_shndx) &&strcmp (symname,"__init_begin" ) &&strcmp (symname,"__per_cpu_load" ) &&strncmp (symname,"init_per_cpu_" ,13 );
@@ -1092,7 +1145,7 @@ static int do_reloc_info(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,sec_name (sec->shdr.sh_info),rel_type (ELF_R_TYPE (rel->r_info)), symname, - sec_name(sym->st_shndx)); + sec_name(sym_index(sym)));return 0 ; }
Signature · 1 line
2.31.1
PATCH v6 6/22 x86/boot: Allow a "silent" kaslr random byte fetch
In reply to Alexander LobakinFrom: Kees Cook <[email protected]>
Under earlyprintk, each RNG call produces a debug report line. When shuffling hundreds of functions, this is not useful information (each line is identical and tells us nothing new). Instead, allow for a NULL "purpose" to suppress the debug reporting.
--- arch/x86/lib/kaslr.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
arch/x86/lib/kaslr.c +12 -6
@@ -56,11 +56,14 @@ unsigned long kaslr_get_random_long(const char *purpose)unsigned long raw, random =get_boot_seed ();bool use_i8254 = true; -debug_putstr (purpose); -debug_putstr (" KASLR using" ); +if (purpose) { +debug_putstr (purpose); +debug_putstr (" KASLR using" ); + }if (has_cpuflag (X86_FEATURE_RDRAND )) { -debug_putstr (" RDRAND" ); +if (purpose) +debug_putstr (" RDRAND" );if (rdrand_long (&raw)) { random ^= raw; use_i8254 = false;
@@ -68,7 +71,8 @@ unsigned long kaslr_get_random_long(const char *purpose) }if (has_cpuflag (X86_FEATURE_TSC )) { -debug_putstr (" RDTSC" ); +if (purpose) +debug_putstr (" RDTSC" ); raw =rdtsc (); random ^= raw;
@@ -76,7 +80,8 @@ unsigned long kaslr_get_random_long(const char *purpose) }if (use_i8254) { -debug_putstr (" i8254" ); +if (purpose) +debug_putstr (" i8254" ); random ^=i8254 (); }
@@ -86,7 +91,8 @@ unsigned long kaslr_get_random_long(const char *purpose) : "a" (random), "rm" (mix_const)); random += raw; -debug_putstr ("...\n" ); +if (purpose) +debug_putstr ("...\n" );return random; }
Signature · 1 line
2.31.1
PATCH v6 7/22 x86: Makefile: Add build and config option for CONFIG_FG_KASLR
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
Allow user to select CONFIG_FG_KASLR if dependencies are met. Change the make file to build with -ffunction-sections if CONFIG_FG_KASLR.
While the only architecture that supports CONFIG_FG_KASLR does not currently enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION, make sure these 2 features play nicely together for the future by ensuring that if CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is selected when used with CONFIG_FG_KASLR the function sections will not be consolidated back into .text. Thanks to Kees Cook for the dead code elimination changes.
[ alobakin:
- improve cflags management in the top Makefile
- move ARCH_HAS_FG_KASLR to the top arch/Kconfig
- add symtab_shndx to the list of known sections ]
--- Makefile | 13 ++++++++++++- arch/Kconfig | 3 +++ include/asm-generic/vmlinux.lds.h | 20 ++++++++++++++++++-- init/Kconfig | 12 ++++++++++++ 4 files changed, 45 insertions(+), 3 deletions(-)
Makefile +12 -1
@@ -918,8 +918,19 @@ ifdef CONFIG_DEBUG_SECTION_MISMATCH KBUILD_CFLAGS += $(call cc-option, -fno-inline-functions-called-once) endif +# ClangLTO implies -ffunction-sections -fdata-sections, no need +# to specify them manually and trigger a pointless full rebuild +ifndef CONFIG_LTO_CLANG +ifneq ($(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION)$(CONFIG_FG_KASLR),) +KBUILD_CFLAGS_KERNEL += -ffunction-sections +endif + +ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION +KBUILD_CFLAGS_KERNEL += -fdata-sections +endif +endif # CONFIG_LTO_CLANG + ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION -KBUILD_CFLAGS_KERNEL += -ffunction-sections -fdata-sections LDFLAGS_vmlinux += --gc-sections endif
arch/Kconfig +3 -0
@@ -1282,6 +1282,9 @@ config ARCH_SPLIT_ARG64 config ARCH_HAS_ELFCORE_COMPAT bool +config ARCH_HAS_FG_KASLR + bool + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig"
include/asm-generic/vmlinux.lds.h +18 -2
@@ -97,14 +97,12 @@* sections to be brought in with rodata. */ #if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION ) || defined(CONFIG_LTO_CLANG ) -#define TEXT_MAIN .text .text.[0-9a-zA-Z_]*#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliteral* .data.$__unnamed_* .data.$L*#define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]*#define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L*#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*#define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]*#else -#define TEXT_MAIN .text#define DATA_MAIN .data#define SDATA_MAIN .sdata#define RODATA_MAIN .rodata
@@ -112,6 +110,23 @@#define SBSS_MAIN .sbss#endif +/* +* LTO_CLANG, LD_DEAD_CODE_DATA_ELIMINATION and FG_KASLR options enable +* -ffunction-sections, which produces separately named .text sections. In +* the case of CONFIG_FG_KASLR, they need to stay distict so they can be +* separately randomized. Without CONFIG_FG_KASLR, the separate .text +* sections can be collected back into a common section, which makes the +* resulting image slightly smaller +*/ +#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION ) || \ + defined(CONFIG_LTO_CLANG )) && !defined(CONFIG_FG_KASLR ) +#define TEXT_MAIN .text .text.[0-9a-zA-Z_]* +#elif defined(CONFIG_FG_KASLR ) +#define TEXT_MAIN .text.__unused__ +#else +#define TEXT_MAIN .text +#endif +/* * GCC 4.5 and later have a 32 bytes section alignment for structures. * Except GCC 4.9, that feels the need to align on 64 bytes.
@@ -842,6 +857,7 @@#define ELF_DETAILS \ .comment 0 : { *(.comment) } \ .symtab 0 : { *(.symtab) } \ + .symtab_shndx 0 : { *(.symtab_shndx) } \ .strtab 0 : { *(.strtab) } \ .shstrtab 0 : { *(.shstrtab) }
init/Kconfig +12 -0
@@ -2021,6 +2021,18 @@ config PROFILING config TRACEPOINTS bool +config FG_KASLR + bool "Function Granular Kernel Address Space Layout Randomization" + depends on ARCH_HAS_FG_KASLR + help + This option improves the randomness of the kernel text + over basic Kernel Address Space Layout Randomization (KASLR) + by reordering the kernel text at boot time. This feature + uses information generated at compile time to re-layout the + kernel text section at boot time at function level granularity. + + If unsure, say N. + endmenu # General setup source "arch/Kconfig"
Signature · 1 line
2.31.1
PATCH v6 8/22 Make sure ORC lookup covers the entire _etext - _stext
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
When using -ffunction-sections to place each function in it's own text section so it can be randomized at load time, the linker will place most of the functions into separate .text.* sections. SIZEOF(.text) won't work here for calculating the ORC lookup table size, the total text size must be calculated to include .text AND all .text.*.
--- include/asm-generic/vmlinux.lds.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/asm-generic/vmlinux.lds.h +2 -1
@@ -887,10 +887,11 @@ KEEP(*(.orc_unwind)) \ __stop_orc_unwind = .; \ } \ + text_size = _etext - _stext; \ . = ALIGN(4); \ .orc_lookup : AT(ADDR(.orc_lookup) - LOAD_OFFSET) { \ orc_lookup = .; \ - . += (((SIZEOF(.text) + LOOKUP_BLOCK_SIZE - 1) / \ + . += (((text_size + LOOKUP_BLOCK_SIZE - 1) / \LOOKUP_BLOCK_SIZE ) +1 ) *4 ; \ orc_lookup_end = .; \ }
Signature · 1 line
2.31.1
PATCH v6 9/22 x86/tools: Add relative relocs for randomized functions
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
When reordering functions, the relative offsets for relocs that are either in the randomized sections, or refer to the randomized sections will need to be adjusted. Add code to detect whether a reloc satisfies these cases, and if so, add them to the appropriate reloc list.
--- arch/x86/boot/compressed/Makefile | 7 ++++++- arch/x86/tools/relocs.c | 32 +++++++++++++++++++++++++++---- arch/x86/tools/relocs.h | 4 ++-- arch/x86/tools/relocs_common.c | 15 ++++++++++----- 4 files changed, 46 insertions(+), 12 deletions(-)
arch/x86/boot/compressed/Makefile +6 -1
@@ -106,6 +106,11 @@ $(obj)/vmlinux: $(vmlinux-objs-y) $(efi-obj-y) FORCE $(call if_changed,ld) OBJCOPYFLAGS_vmlinux.bin := -R .comment -S + +ifdef CONFIG_FG_KASLR +RELOCS_ARGS += --fg-kaslr +endif + $(obj)/vmlinux.bin: vmlinux FORCE $(call if_changed,objcopy)
@@ -113,7 +118,7 @@ targets += $(patsubst $(obj)/%,%,$(vmlinux-objs-y)) vmlinux.bin.all vmlinux.relo CMD_RELOCS = arch/x86/tools/relocs quiet_cmd_relocs = RELOCS $@ - cmd_relocs = $(CMD_RELOCS) $< > $@;$(CMD_RELOCS) --abs-relocs $< + cmd_relocs = $(CMD_RELOCS) $(RELOCS_ARGS) $< > $@;$(CMD_RELOCS) $(RELOCS_ARGS) --abs-relocs $< $(obj)/vmlinux.relocs: vmlinux FORCE $(call if_changed,relocs)
arch/x86/tools/relocs.c +28 -4
@@ -42,6 +42,8 @@ struct section { };static struct section *secs; +static int fgkaslr_mode; +static const char *const sym_regex_kernel[S_NSYMTYPES ] = {/* * Following symbols have been audited. There values are constant and do
@@ -818,6 +820,24 @@ static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)strncmp (symname,"init_per_cpu_" ,13 ); } +static int is_function_section (struct section *sec) +{ +if (!fgkaslr_mode) +return 0 ; + +return !strncmp (sec_name (sec->shdr.sh_info),".text." ,6 ); +} + +static int is_randomized_sym (ElfW (Sym) *sym) +{ +if (!fgkaslr_mode) +return 0 ; + +if (sym->st_shndx > shnum) +return 0 ; + +return !strncmp (sec_name (sym_index (sym)),".text." ,6 ); +}static int do_reloc64 (struct section *sec,Elf_Rel *rel,ElfW (Sym) *sym,const char *symname)
@@ -843,12 +863,15 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,case R_X86_64_PC32 :case R_X86_64_PLT32 :/* -* PC relative relocations don't need to be adjusted unless -* referencing a percpu symbol. +* we need to keep pc relative relocations for sections which +* might be randomized, and for the percpu section. +* We also need to keep relocations for any offset which might +* reference an address in a section which has been randomized. * * NB: R_X86_64_PLT32 can be treated as R_X86_64_PC32. */ -if (is_percpu_sym (sym, symname)) +if (is_function_section (sec) ||is_randomized_sym (sym) || +is_percpu_sym (sym, symname))add_reloc (&relocs32neg, offset);break ;
@@ -1163,8 +1186,9 @@ static void print_reloc_info(void)void process (FILE *fp,int use_real_mode,int as_text,int show_absolute_syms,int show_absolute_relocs, -int show_reloc_info) +int show_reloc_info,int fgkaslr) { + fgkaslr_mode = fgkaslr;regex_init (use_real_mode);read_ehdr (fp);read_shdrs (fp);
arch/x86/tools/relocs.h +2 -2
@@ -31,8 +31,8 @@ enum symtype {void process_32 (FILE *fp,int use_real_mode,int as_text,int show_absolute_syms,int show_absolute_relocs, -int show_reloc_info); +int show_reloc_info,int fgkaslr);void process_64 (FILE *fp,int use_real_mode,int as_text,int show_absolute_syms,int show_absolute_relocs, -int show_reloc_info); +int show_reloc_info,int fgkaslr);#endif /* RELOCS_H */
arch/x86/tools/relocs_common.c +10 -5
@@ -12,14 +12,14 @@ void die(char *fmt, ...)static void usage (void ) { -die ("relocs [--abs-syms|--abs-relocs|--reloc-info|--text|--realmode]" \ -" vmlinux\n" ); +die ("relocs [--abs-syms|--abs-relocs|--reloc-info|--text|--realmode|" +"--fg-kaslr] vmlinux\n" ); }int main (int argc,char **argv) {int show_absolute_syms, show_absolute_relocs, show_reloc_info; -int as_text, use_real_mode; +int as_text, use_real_mode, fgkaslr_opt;const char *fname;FILE *fp;int i;
@@ -30,6 +30,7 @@ int main(int argc, char **argv) show_reloc_info =0 ; as_text =0 ; use_real_mode =0 ; + fgkaslr_opt =0 ; fname =NULL ;for (i =1 ; i < argc; i++) {char *arg = argv[i];
@@ -54,6 +55,10 @@ int main(int argc, char **argv) use_real_mode =1 ;continue ; } +if (strcmp (arg,"--fg-kaslr" ) ==0 ) { + fgkaslr_opt =1 ; +continue ; + } }else if (!fname) { fname = arg;
@@ -75,11 +80,11 @@ int main(int argc, char **argv)if (e_ident[EI_CLASS ] ==ELFCLASS64 )process_64 (fp, use_real_mode, as_text, show_absolute_syms, show_absolute_relocs, - show_reloc_info); + show_reloc_info, fgkaslr_opt);else process_32 (fp, use_real_mode, as_text, show_absolute_syms, show_absolute_relocs, - show_reloc_info); + show_reloc_info, fgkaslr_opt);fclose (fp);return 0 ; }
Signature · 1 line
2.31.1
PATCH v6 10/22 x86/boot/compressed: Avoid duplicate malloc() implementations
In reply to Alexander LobakinFrom: Kees Cook <[email protected]>
The preboot malloc() (and free()) implementation in include/linux/decompress/mm.h (which is also included by the static decompressors) is static. This is fine when the only thing interested in using malloc() is the decompression code, but the x86 preboot environment uses malloc() in a couple places, leading to a potential collision when the static copies of the available memory region ("malloc_ptr") gets reset to the global "free_mem_ptr" value. As it happens, the existing usage pattern happened to be safe because each user did 1 malloc() and 1 free() before returning and were not nested:
extract_kernel() (misc.c)
choose_random_location() (kaslr.c) mem_avoid_init() handle_mem_options() malloc() ... free() ... parse_elf() (misc.c) malloc() ... free()
Adding FGKASLR, however, will insert additional malloc() calls local to fgkaslr.c in the middle of parse_elf()'s malloc()/free() pair:
parse_elf() (misc.c)
malloc()
if (...) {
layout_randomized_image(output, &ehdr, phdrs);
malloc() <- boom
...
else
layout_image(output, &ehdr, phdrs);
free()To avoid collisions, there must be a single implementation of malloc(). Adjust include/linux/decompress/mm.h so that visibility can be controlled, provide prototypes in misc.h, and implement the functions in misc.c. This also results in a small size savings:
$ size vmlinux.before vmlinux.after text data bss dec hex filename 8842314 468 178320 9021102 89a6ae vmlinux.before 8842240 468 178320 9021028 89a664 vmlinux.after
--- arch/x86/boot/compressed/kaslr.c | 4 ---- arch/x86/boot/compressed/misc.c | 3 +++ arch/x86/boot/compressed/misc.h | 2 ++ include/linux/decompress/mm.h | 12 ++++++++++-- 4 files changed, 15 insertions(+), 6 deletions(-)
arch/x86/boot/compressed/kaslr.c +0 -4
@@ -32,10 +32,6 @@#include <generated/utsrelease.h> #include <asm/efi.h> -/* Macros used by the included decompressor code below. */ -#define STATIC -#include <linux/decompress/mm.h> -#define _SETUP#include <asm/setup.h> /* For COMMAND_LINE_SIZE */ #undef _SETUP
arch/x86/boot/compressed/misc.c +3 -0
@@ -28,6 +28,9 @@/* Macros used by the included decompressor code below. */ #define STATIC static +/* Define an externally visible malloc()/free(). */ +#define MALLOC_VISIBLE +#include <linux/decompress/mm.h> /* * Provide definitions of memzero and memmove as some of the decompressors will
arch/x86/boot/compressed/misc.h +2 -0
@@ -44,6 +44,8 @@ extern char _head[], _end[];/* misc.c */ extern memptr free_mem_ptr;extern memptr free_mem_end_ptr; +extern void *malloc (int size); +extern void free (void *where);extern struct boot_params *boot_params;void __putstr (const char *s);void __puthex (unsigned long value);
include/linux/decompress/mm.h +10 -2
@@ -25,13 +25,21 @@#define STATIC_RW_DATA static#endif +/* +* When an architecture needs to share the malloc()/free() implementation +* between compilation units, it needs to have non-local visibility. +*/ +#ifndef MALLOC_VISIBLE +#define MALLOC_VISIBLE static +#endif +/* A trivial malloc implementation, adapted from * malloc by Hannu Savolainen 1993 and Matthias Urlichs 1994 */ STATIC_RW_DATA unsigned long malloc_ptr; STATIC_RW_DATA int malloc_count; -static void *malloc (int size) +MALLOC_VISIBLE void *malloc(int size) {void *p;
@@ -52,7 +60,7 @@ static void *malloc(int size)return p; } -static void free (void *where) +MALLOC_VISIBLE void free(void *where) { malloc_count--;if (!malloc_count)
Signature · 1 line
2.31.1
PATCH v6 11/22 x86: Add support for function granular KASLR
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
This commit contains the changes required to re-layout the kernel text sections generated by -ffunction-sections shortly after decompression. Documentation of the feature is also added.
After decompression, the decompressed image's elf headers are parsed. In order to manually update certain data structures that are built with relative offsets during the kernel build process, certain symbols are not stripped by objdump and their location is retained in the elf symbol tables. These addresses are saved.
If the image was built with -ffunction-sections, there will be ELF section headers present which contain information about the address range of each section. Anything that is not broken out into function sections (i.e. is consolidated into .text) is left in it's original location, but any other executable section which begins with ".text." is located and shuffled randomly within the remaining text segment address range.
After the sections have been copied to their new locations, but before relocations have been applied, the kallsyms tables must be updated to reflect the new symbol locations. Because it is expected that these tables will be sorted by address, the kallsyms tables will need to be sorted after the update.
When applying relocations, the address of the relocation needs to be adjusted by the offset from the original location of the section that was randomized to it's new location. In addition, if a value at that relocation was a location in the text segment that was randomized, it's value will be adjusted to a new location.
After relocations have been applied, the exception table must be updated with new symbol locations, and then re-sorted by the new address. The orc table will have been updated as part of applying relocations, but since it is expected to be sorted by address, it will need to be resorted.
--- arch/x86/boot/compressed/Makefile | 2 + arch/x86/boot/compressed/fgkaslr.c | 905 +++++++++++++++++++++++ arch/x86/boot/compressed/misc.c | 154 +++- arch/x86/boot/compressed/misc.h | 28 + arch/x86/boot/compressed/utils.c | 13 + arch/x86/boot/compressed/vmlinux.symbols | 19 + arch/x86/include/asm/boot.h | 13 +- arch/x86/kernel/vmlinux.lds.S | 2 + include/uapi/linux/elf.h | 1 + 9 files changed, 1110 insertions(+), 27 deletions(-) create mode 100644 arch/x86/boot/compressed/fgkaslr.c create mode 100644 arch/x86/boot/compressed/utils.c create mode 100644 arch/x86/boot/compressed/vmlinux.symbols
arch/x86/boot/compressed/Makefile +2 -0
@@ -89,6 +89,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o +vmlinux-objs-$(CONFIG_FG_KASLR) += $(obj)/fgkaslr.o $(obj)/utils.o ifdef CONFIG_X86_64 vmlinux-objs-y += $(obj)/ident_map_64.o vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
@@ -108,6 +109,7 @@ $(obj)/vmlinux: $(vmlinux-objs-y) $(efi-obj-y) FORCE OBJCOPYFLAGS_vmlinux.bin := -R .comment -S ifdef CONFIG_FG_KASLR +OBJCOPYFLAGS += --keep-symbols=$(srctree)/$(src)/vmlinux.symbols RELOCS_ARGS += --fg-kaslr endif
arch/x86/boot/compressed/fgkaslr.c new file +905 -0
905 lines not on this page · raw message
arch/x86/boot/compressed/misc.c +128 -26
217 lines not on this page · raw message
arch/x86/boot/compressed/misc.h +28 -0
@@ -81,6 +81,34 @@ struct mem_vector {u64 size; }; +#ifdef CONFIG_X86_64 +#define Elf_Ehdr Elf64_Ehdr +#define Elf_Phdr Elf64_Phdr +#define Elf_Shdr Elf64_Shdr +#else +#define Elf_Ehdr Elf32_Ehdr +#define Elf_Phdr Elf32_Phdr +#define Elf_Shdr Elf32_Shdr +#endif + +#ifdef CONFIG_FG_KASLR +void layout_randomized_image (void *output,Elf_Ehdr *ehdr,Elf_Phdr *phdrs); +void pre_relocations_cleanup (unsigned long map); +void post_relocations_cleanup (unsigned long map); +Elf_Shdr *adjust_address (long *address); +void adjust_relative_offset (long pc,long *value,Elf_Shdr *section); +bool is_percpu_addr (long pc,long offset); +#else +static inline void layout_randomized_image (void *output,Elf_Ehdr *ehdr, +Elf_Phdr *phdrs) { } +static inline void pre_relocations_cleanup (unsigned long map) { } +static inline void post_relocations_cleanup (unsigned long map) { } +static inline Elf_Shdr *adjust_address (long *address) {return NULL ; } +static inline void adjust_relative_offset (long pc,long *value, +Elf_Shdr *section) { } +static inline bool is_percpu_addr (long pc,long offset) {return true; } +#endif +#ifdef CONFIG_RANDOMIZE_BASE /* kaslr.c */ void choose_random_location(unsigned long input,
arch/x86/boot/compressed/utils.c new file +13 -0
@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* This contains various libraries that are needed for FG-KASLR. +* +* Copyright (C) 2020-2021, Intel Corporation. +* Author: Kristen Carlson Accardi <[email protected]> +*/ + +#define _LINUX_KPROBES_H +#define NOKPROBE_SYMBOL (fname) + +#include "../../../../lib/sort.c" +#include "../../../../lib/bsearch.c"
arch/x86/boot/compressed/vmlinux.symbols new file +19 -0
@@ -0,0 +1,19 @@ +kallsyms_offsets +kallsyms_addresses +kallsyms_num_syms +kallsyms_relative_base +kallsyms_names +kallsyms_token_table +kallsyms_token_index +kallsyms_markers +__start___ex_table +__stop___ex_table +__altinstr_replacement +__altinstr_replacement_end +_sinittext +_einittext +_stext +_etext +__start_orc_unwind_ip +__stop_orc_unwind_ip +__start_orc_unwind
arch/x86/include/asm/boot.h +12 -1
@@ -24,7 +24,18 @@# error "Invalid value for CONFIG_PHYSICAL_ALIGN"#endif -#if defined(CONFIG_KERNEL_BZIP2 ) +#ifdef CONFIG_FG_KASLR +/* +* We need extra boot heap when using fgkaslr because we make a copy +* of the original decompressed kernel to avoid issues with writing +* over ourselves when shuffling the sections. We also need extra +* space for resorting kallsyms after shuffling. This value could +* be decreased if free() would release memory properly, or if we +* could avoid the kernel copy. It would need to be increased if we +* find additional tables that need to be resorted. +*/ +# define BOOT_HEAP_SIZE 0x4800000 +#elif defined(CONFIG_KERNEL_BZIP2 )# define BOOT_HEAP_SIZE 0x400000#elif defined(CONFIG_KERNEL_ZSTD )/*
arch/x86/kernel/vmlinux.lds.S +2 -0
@@ -290,7 +290,9 @@ SECTIONS * get the address and the length of them to patch the kernel safely. */ .altinstr_replacement : AT(ADDR(.altinstr_replacement) - LOAD_OFFSET) { + __altinstr_replacement = .; *(.altinstr_replacement) + __altinstr_replacement_end = .; } /*
include/uapi/linux/elf.h +1 -0
@@ -299,6 +299,7 @@ typedef struct elf64_phdr {#define SHN_LIVEPATCH 0xff20#define SHN_ABS 0xfff1#define SHN_COMMON 0xfff2 +#define SHN_XINDEX 0xffff#define SHN_HIRESERVE 0xffff typedef struct elf32_shdr {
Signature · 1 line
2.31.1
PATCH v6 12/22 linkage: add macros for putting ASM functions into own sections
In reply to Alexander LobakinWith ClangLTO or -ffunction-sections (DCE, FG-KASLR), compiler places C functions into separate sections by default. However, this doesn't happen with ASM functions which are still being placed into .text.
Introduce a pack of macros which generate a new unique section for the describing function named in the same fashion (.text.<func_name>).
This will be needed to make input .text section empty to harden the kernel even more.
--- include/linux/linkage.h | 76 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+)
include/linux/linkage.h +76 -0
@@ -355,4 +355,80 @@#endif /* __ASSEMBLY__ */ +/* +* Allow ASM symbols to have their own unique sections if they are being +* generated by the compiler for C functions (DCE, FG-KASLR, LTO). +*/ +#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION ) && !defined(MODULE )) || \ + (defined(CONFIG_FG_KASLR ) && !defined(MODULE )) || \ + (defined(CONFIG_MODULE_FG_KASLR ) && defined(MODULE )) || \ + (defined(CONFIG_LTO_CLANG )) + +#define ASM_PUSH_SECTION (name) \ + ".pushsection .text." #name + +#define SYM_TEXT_SECTION (name) \ + .pushsection .text.##name + +#else /* just .text */ + +#define ASM_PUSH_SECTION (name) \ + ".pushsection .text" + +#define SYM_TEXT_SECTION (name) \ + .pushsection .text + +#endif /* just .text */ + +#ifdef __ASSEMBLY__ + +#define SYM_TEXT_END_SECTION \ + .popsection + +#define SYM_FUNC_START_LOCAL_ALIAS_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_FUNC_START_LOCAL_ALIAS(name) + +#define SYM_FUNC_START_LOCAL_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_FUNC_START_LOCAL(name) + +#define SYM_FUNC_START_NOALIGN_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_FUNC_START_NOALIGN(name) + +#define SYM_FUNC_START_WEAK_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_FUNC_START_WEAK(name) + +#define SYM_FUNC_START_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_FUNC_START(name) + +#define SYM_CODE_START_LOCAL_NOALIGN_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_CODE_START_LOCAL_NOALIGN(name) + +#define SYM_CODE_START_NOALIGN_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_CODE_START_NOALIGN(name) + +#define SYM_CODE_START_SECTION (name) \ + SYM_TEXT_SECTION(name) ASM_NL \ + SYM_CODE_START(name) + +#define SYM_FUNC_END_ALIAS_SECTION (name) \ + SYM_FUNC_END_ALIAS(name) ASM_NL \ + SYM_TEXT_END_SECTION + +#define SYM_FUNC_END_SECTION (name) \ + SYM_FUNC_END(name) ASM_NL \ + SYM_TEXT_END_SECTION + +#define SYM_CODE_END_SECTION (name) \ + SYM_CODE_END(name) ASM_NL \ + SYM_TEXT_END_SECTION + +#endif /* __ASSEMBLY__ */ +#endif /* _LINUX_LINKAGE_H */
Signature · 1 line
2.31.1
PATCH v6 13/22 x86: conditionally place regular ASM functions into separate sections
In reply to Alexander LobakinUse the newly introduces macros to create unique separate sections for (almost) every "regular" ASM function (i.e. for those which aren't explicitly put into a specific one).
There should be no leftovers as input .text will be size-asserted in the LD script generated for FG-KASLR.
--- arch/x86/crypto/aegis128-aesni-asm.S | 36 +++--- arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 12 +- arch/x86/crypto/aesni-intel_asm.S | 116 +++++++++--------- arch/x86/crypto/aesni-intel_avx-x86_64.S | 32 ++--- arch/x86/crypto/blake2s-core.S | 8 +- arch/x86/crypto/blowfish-x86_64-asm_64.S | 16 +-- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 28 ++--- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 28 ++--- arch/x86/crypto/camellia-x86_64-asm_64.S | 16 +-- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 24 ++-- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 20 +-- arch/x86/crypto/chacha-avx2-x86_64.S | 12 +- arch/x86/crypto/chacha-avx512vl-x86_64.S | 12 +- arch/x86/crypto/chacha-ssse3-x86_64.S | 16 +-- arch/x86/crypto/crc32-pclmul_asm.S | 4 +- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 4 +- arch/x86/crypto/crct10dif-pcl-asm_64.S | 4 +- arch/x86/crypto/des3_ede-asm_64.S | 8 +- arch/x86/crypto/ghash-clmulni-intel_asm.S | 12 +- arch/x86/crypto/nh-avx2-x86_64.S | 4 +- arch/x86/crypto/nh-sse2-x86_64.S | 4 +- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 8 +- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 +-- arch/x86/crypto/serpent-avx2-asm_64.S | 20 +-- arch/x86/crypto/serpent-sse2-i586-asm_32.S | 8 +- arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 8 +- arch/x86/crypto/sha1_avx2_x86_64_asm.S | 4 +- arch/x86/crypto/sha1_ni_asm.S | 4 +- arch/x86/crypto/sha1_ssse3_asm.S | 4 +- arch/x86/crypto/sha256-avx-asm.S | 4 +- arch/x86/crypto/sha256-avx2-asm.S | 4 +- arch/x86/crypto/sha256-ssse3-asm.S | 4 +- arch/x86/crypto/sha256_ni_asm.S | 4 +- arch/x86/crypto/sha512-avx-asm.S | 4 +- arch/x86/crypto/sha512-avx2-asm.S | 4 +- arch/x86/crypto/sha512-ssse3-asm.S | 4 +- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 20 +-- arch/x86/crypto/twofish-i586-asm_32.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64.S | 8 +- arch/x86/entry/entry_32.S | 24 ++-- arch/x86/entry/entry_64.S | 18 +-- arch/x86/entry/thunk_32.S | 4 +- arch/x86/entry/thunk_64.S | 8 +- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 2 +- arch/x86/kernel/acpi/wakeup_32.S | 9 +- arch/x86/kernel/acpi/wakeup_64.S | 10 +- arch/x86/kernel/ftrace_32.S | 19 +-- arch/x86/kernel/ftrace_64.S | 28 ++--- arch/x86/kernel/irqflags.S | 4 +- arch/x86/kernel/kprobes/core.c | 3 +- arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/relocate_kernel_32.S | 2 + arch/x86/kernel/relocate_kernel_64.S | 2 + arch/x86/kvm/emulate.c | 2 +- arch/x86/kvm/vmx/vmenter.S | 8 +- arch/x86/lib/clear_page_64.S | 12 +- arch/x86/lib/cmpxchg16b_emu.S | 4 +- arch/x86/lib/copy_mc_64.S | 8 +- arch/x86/lib/copy_page_64.S | 7 +- arch/x86/lib/copy_user_64.S | 18 +-- arch/x86/lib/csum-copy_64.S | 4 +- arch/x86/lib/error-inject.c | 3 +- arch/x86/lib/getuser.S | 37 +++--- arch/x86/lib/hweight.S | 9 +- arch/x86/lib/iomap_copy_64.S | 4 +- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 12 +- arch/x86/lib/msr-reg.S | 8 +- arch/x86/lib/putuser.S | 18 +-- arch/x86/mm/mem_encrypt_boot.S | 8 +- arch/x86/platform/efi/efi_stub_64.S | 4 +- arch/x86/platform/efi/efi_thunk_64.S | 4 +- arch/x86/power/hibernate_asm_32.S | 14 ++- arch/x86/power/hibernate_asm_64.S | 14 ++- arch/x86/xen/xen-asm.S | 49 ++++---- arch/x86/xen/xen-head.S | 10 +- 78 files changed, 482 insertions(+), 480 deletions(-)
arch/x86/crypto/aegis128-aesni-asm.S +18 -18
@@ -71,7 +71,7 @@ * %r8 * %r9 */ -SYM_FUNC_START_LOCAL(__load_partial) +SYM_FUNC_START_LOCAL_SECTION(__load_partial) xor %r9d, %r9d pxor MSG, MSG
@@ -123,7 +123,7 @@ SYM_FUNC_START_LOCAL(__load_partial) .Lld_partial_8: ret -SYM_FUNC_END(__load_partial) +SYM_FUNC_END_SECTION(__load_partial) /* * __store_partial: internal ABI
@@ -137,7 +137,7 @@ SYM_FUNC_END(__load_partial) * %r9 * %r10 */ -SYM_FUNC_START_LOCAL(__store_partial) +SYM_FUNC_START_LOCAL_SECTION(__store_partial) mov LEN, %r8 mov DST, %r9
@@ -181,12 +181,12 @@ SYM_FUNC_START_LOCAL(__store_partial) .Lst_partial_1: ret -SYM_FUNC_END(__store_partial) +SYM_FUNC_END_SECTION(__store_partial) /* * void crypto_aegis128_aesni_init(void *state, const void *key, const void *iv); */ -SYM_FUNC_START(crypto_aegis128_aesni_init) +SYM_FUNC_START_SECTION(crypto_aegis128_aesni_init) FRAME_BEGIN /* load IV: */
@@ -226,13 +226,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_init) FRAME_END ret -SYM_FUNC_END(crypto_aegis128_aesni_init) +SYM_FUNC_END_SECTION(crypto_aegis128_aesni_init) /* * void crypto_aegis128_aesni_ad(void *state, unsigned int length, * const void *data); */ -SYM_FUNC_START(crypto_aegis128_aesni_ad) +SYM_FUNC_START_SECTION(crypto_aegis128_aesni_ad) FRAME_BEGIN cmp $0x10, LEN
@@ -378,7 +378,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad) .Lad_out: FRAME_END ret -SYM_FUNC_END(crypto_aegis128_aesni_ad) +SYM_FUNC_END_SECTION(crypto_aegis128_aesni_ad) .macro encrypt_block a s0 s1 s2 s3 s4 i movdq\a (\i * 0x10)(SRC), MSG
@@ -402,7 +402,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_ad) * void crypto_aegis128_aesni_enc(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_enc) +SYM_FUNC_START_SECTION(crypto_aegis128_aesni_enc) FRAME_BEGIN cmp $0x10, LEN
@@ -493,13 +493,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc) .Lenc_out: FRAME_END ret -SYM_FUNC_END(crypto_aegis128_aesni_enc) +SYM_FUNC_END_SECTION(crypto_aegis128_aesni_enc) /* * void crypto_aegis128_aesni_enc_tail(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_enc_tail) +SYM_FUNC_START_SECTION(crypto_aegis128_aesni_enc_tail) FRAME_BEGIN /* load the state: */
@@ -533,7 +533,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc_tail) FRAME_END ret -SYM_FUNC_END(crypto_aegis128_aesni_enc_tail) +SYM_FUNC_END_SECTION(crypto_aegis128_aesni_enc_tail) .macro decrypt_block a s0 s1 s2 s3 s4 i movdq\a (\i * 0x10)(SRC), MSG
@@ -556,7 +556,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_enc_tail) * void crypto_aegis128_aesni_dec(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_dec) +SYM_FUNC_START_SECTION(crypto_aegis128_aesni_dec) FRAME_BEGIN cmp $0x10, LEN
@@ -647,13 +647,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec) .Ldec_out: FRAME_END ret -SYM_FUNC_END(crypto_aegis128_aesni_dec) +SYM_FUNC_END_SECTION(crypto_aegis128_aesni_dec) /* * void crypto_aegis128_aesni_dec_tail(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_dec_tail) +SYM_FUNC_START_SECTION(crypto_aegis128_aesni_dec_tail) FRAME_BEGIN /* load the state: */
@@ -697,13 +697,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec_tail) FRAME_END ret -SYM_FUNC_END(crypto_aegis128_aesni_dec_tail) +SYM_FUNC_END_SECTION(crypto_aegis128_aesni_dec_tail) /* * void crypto_aegis128_aesni_final(void *state, void *tag_xor, * u64 assoclen, u64 cryptlen); */ -SYM_FUNC_START(crypto_aegis128_aesni_final) +SYM_FUNC_START_SECTION(crypto_aegis128_aesni_final) FRAME_BEGIN /* load the state: */
@@ -744,4 +744,4 @@ SYM_FUNC_START(crypto_aegis128_aesni_final) FRAME_END ret -SYM_FUNC_END(crypto_aegis128_aesni_final) +SYM_FUNC_END_SECTION(crypto_aegis128_aesni_final)
arch/x86/crypto/aes_ctrby8_avx-x86_64.S +6 -6
@@ -535,11 +535,11 @@ ddq_add_8: * aes_ctr_enc_128_avx_by8(void *in, void *iv, void *keys, void *out, * unsigned int num_bytes) */ -SYM_FUNC_START(aes_ctr_enc_128_avx_by8) +SYM_FUNC_START_SECTION(aes_ctr_enc_128_avx_by8) /* call the aes main loop */ do_aes_ctrmain KEY_128 -SYM_FUNC_END(aes_ctr_enc_128_avx_by8) +SYM_FUNC_END_SECTION(aes_ctr_enc_128_avx_by8) /* * routine to do AES192 CTR enc/decrypt "by8"
@@ -548,11 +548,11 @@ SYM_FUNC_END(aes_ctr_enc_128_avx_by8) * aes_ctr_enc_192_avx_by8(void *in, void *iv, void *keys, void *out, * unsigned int num_bytes) */ -SYM_FUNC_START(aes_ctr_enc_192_avx_by8) +SYM_FUNC_START_SECTION(aes_ctr_enc_192_avx_by8) /* call the aes main loop */ do_aes_ctrmain KEY_192 -SYM_FUNC_END(aes_ctr_enc_192_avx_by8) +SYM_FUNC_END_SECTION(aes_ctr_enc_192_avx_by8) /* * routine to do AES256 CTR enc/decrypt "by8"
@@ -561,8 +561,8 @@ SYM_FUNC_END(aes_ctr_enc_192_avx_by8) * aes_ctr_enc_256_avx_by8(void *in, void *iv, void *keys, void *out, * unsigned int num_bytes) */ -SYM_FUNC_START(aes_ctr_enc_256_avx_by8) +SYM_FUNC_START_SECTION(aes_ctr_enc_256_avx_by8) /* call the aes main loop */ do_aes_ctrmain KEY_256 -SYM_FUNC_END(aes_ctr_enc_256_avx_by8) +SYM_FUNC_END_SECTION(aes_ctr_enc_256_avx_by8)
arch/x86/crypto/aesni-intel_asm.S +56 -60
409 lines not on this page · raw message
arch/x86/crypto/aesni-intel_avx-x86_64.S +16 -16
@@ -1763,12 +1763,12 @@ _initial_blocks_done\@: # const u8 *aad, /* Additional Authentication Data (AAD)*/ # u64 aad_len) /* Length of AAD in bytes. With RFC4106 this is going to be 8 or 12 Bytes */ ############################################################# -SYM_FUNC_START(aesni_gcm_init_avx_gen2) +SYM_FUNC_START_SECTION(aesni_gcm_init_avx_gen2) FUNC_SAVE INIT GHASH_MUL_AVX, PRECOMPUTE_AVX FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_init_avx_gen2) +SYM_FUNC_END_SECTION(aesni_gcm_init_avx_gen2) ############################################################################### #void aesni_gcm_enc_update_avx_gen2(
@@ -1778,7 +1778,7 @@ SYM_FUNC_END(aesni_gcm_init_avx_gen2) # const u8 *in, /* Plaintext input */ # u64 plaintext_len) /* Length of data in Bytes for encryption. */ ############################################################################### -SYM_FUNC_START(aesni_gcm_enc_update_avx_gen2) +SYM_FUNC_START_SECTION(aesni_gcm_enc_update_avx_gen2) FUNC_SAVE mov keysize, %eax cmp $32, %eax
@@ -1797,7 +1797,7 @@ key_256_enc_update: GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GHASH_LAST_8_AVX, GHASH_MUL_AVX, ENC, 13 FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_enc_update_avx_gen2) +SYM_FUNC_END_SECTION(aesni_gcm_enc_update_avx_gen2) ############################################################################### #void aesni_gcm_dec_update_avx_gen2(
@@ -1807,7 +1807,7 @@ SYM_FUNC_END(aesni_gcm_enc_update_avx_gen2) # const u8 *in, /* Ciphertext input */ # u64 plaintext_len) /* Length of data in Bytes for encryption. */ ############################################################################### -SYM_FUNC_START(aesni_gcm_dec_update_avx_gen2) +SYM_FUNC_START_SECTION(aesni_gcm_dec_update_avx_gen2) FUNC_SAVE mov keysize,%eax cmp $32, %eax
@@ -1826,7 +1826,7 @@ key_256_dec_update: GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GHASH_LAST_8_AVX, GHASH_MUL_AVX, DEC, 13 FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_dec_update_avx_gen2) +SYM_FUNC_END_SECTION(aesni_gcm_dec_update_avx_gen2) ############################################################################### #void aesni_gcm_finalize_avx_gen2(
@@ -1836,7 +1836,7 @@ SYM_FUNC_END(aesni_gcm_dec_update_avx_gen2) # u64 auth_tag_len)# /* Authenticated Tag Length in bytes. # Valid values are 16 (most likely), 12 or 8. */ ############################################################################### -SYM_FUNC_START(aesni_gcm_finalize_avx_gen2) +SYM_FUNC_START_SECTION(aesni_gcm_finalize_avx_gen2) FUNC_SAVE mov keysize,%eax cmp $32, %eax
@@ -1855,7 +1855,7 @@ key_256_finalize: GCM_COMPLETE GHASH_MUL_AVX, 13, arg3, arg4 FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_finalize_avx_gen2) +SYM_FUNC_END_SECTION(aesni_gcm_finalize_avx_gen2) ############################################################################### # GHASH_MUL MACRO to implement: Data*HashKey mod (128,127,126,121,0)
@@ -2731,12 +2731,12 @@ _initial_blocks_done\@: # const u8 *aad, /* Additional Authentication Data (AAD)*/ # u64 aad_len) /* Length of AAD in bytes. With RFC4106 this is going to be 8 or 12 Bytes */ ############################################################# -SYM_FUNC_START(aesni_gcm_init_avx_gen4) +SYM_FUNC_START_SECTION(aesni_gcm_init_avx_gen4) FUNC_SAVE INIT GHASH_MUL_AVX2, PRECOMPUTE_AVX2 FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_init_avx_gen4) +SYM_FUNC_END_SECTION(aesni_gcm_init_avx_gen4) ############################################################################### #void aesni_gcm_enc_avx_gen4(
@@ -2746,7 +2746,7 @@ SYM_FUNC_END(aesni_gcm_init_avx_gen4) # const u8 *in, /* Plaintext input */ # u64 plaintext_len) /* Length of data in Bytes for encryption. */ ############################################################################### -SYM_FUNC_START(aesni_gcm_enc_update_avx_gen4) +SYM_FUNC_START_SECTION(aesni_gcm_enc_update_avx_gen4) FUNC_SAVE mov keysize,%eax cmp $32, %eax
@@ -2765,7 +2765,7 @@ key_256_enc_update4: GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, ENC, 13 FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_enc_update_avx_gen4) +SYM_FUNC_END_SECTION(aesni_gcm_enc_update_avx_gen4) ############################################################################### #void aesni_gcm_dec_update_avx_gen4(
@@ -2775,7 +2775,7 @@ SYM_FUNC_END(aesni_gcm_enc_update_avx_gen4) # const u8 *in, /* Ciphertext input */ # u64 plaintext_len) /* Length of data in Bytes for encryption. */ ############################################################################### -SYM_FUNC_START(aesni_gcm_dec_update_avx_gen4) +SYM_FUNC_START_SECTION(aesni_gcm_dec_update_avx_gen4) FUNC_SAVE mov keysize,%eax cmp $32, %eax
@@ -2794,7 +2794,7 @@ key_256_dec_update4: GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, DEC, 13 FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_dec_update_avx_gen4) +SYM_FUNC_END_SECTION(aesni_gcm_dec_update_avx_gen4) ############################################################################### #void aesni_gcm_finalize_avx_gen4(
@@ -2804,7 +2804,7 @@ SYM_FUNC_END(aesni_gcm_dec_update_avx_gen4) # u64 auth_tag_len)# /* Authenticated Tag Length in bytes. # Valid values are 16 (most likely), 12 or 8. */ ############################################################################### -SYM_FUNC_START(aesni_gcm_finalize_avx_gen4) +SYM_FUNC_START_SECTION(aesni_gcm_finalize_avx_gen4) FUNC_SAVE mov keysize,%eax cmp $32, %eax
@@ -2823,4 +2823,4 @@ key_256_finalize4: GCM_COMPLETE GHASH_MUL_AVX2, 13, arg3, arg4 FUNC_RESTORE ret -SYM_FUNC_END(aesni_gcm_finalize_avx_gen4) +SYM_FUNC_END_SECTION(aesni_gcm_finalize_avx_gen4)
arch/x86/crypto/blake2s-core.S +4 -4
@@ -46,7 +46,7 @@ SIGMA2: #endif /* CONFIG_AS_AVX512 */ .text -SYM_FUNC_START(blake2s_compress_ssse3) +SYM_FUNC_START_SECTION(blake2s_compress_ssse3) testq %rdx,%rdx je .Lendofloop movdqu (%rdi),%xmm0
@@ -172,10 +172,10 @@ SYM_FUNC_START(blake2s_compress_ssse3) movdqu %xmm14,0x20(%rdi) .Lendofloop: ret -SYM_FUNC_END(blake2s_compress_ssse3) +SYM_FUNC_END_SECTION(blake2s_compress_ssse3) #ifdef CONFIG_AS_AVX512 -SYM_FUNC_START(blake2s_compress_avx512) +SYM_FUNC_START_SECTION(blake2s_compress_avx512) vmovdqu (%rdi),%xmm0 vmovdqu 0x10(%rdi),%xmm1 vmovdqu 0x20(%rdi),%xmm4
@@ -252,5 +252,5 @@ SYM_FUNC_START(blake2s_compress_avx512) vmovdqu %xmm4,0x20(%rdi) vzeroupper retq -SYM_FUNC_END(blake2s_compress_avx512) +SYM_FUNC_END_SECTION(blake2s_compress_avx512) #endif /* CONFIG_AS_AVX512 */
arch/x86/crypto/blowfish-x86_64-asm_64.S +8 -8
@@ -103,7 +103,7 @@ bswapq RX0; \ xorq RX0, (RIO); -SYM_FUNC_START(__blowfish_enc_blk) +SYM_FUNC_START_SECTION(__blowfish_enc_blk) /* input: * %rdi: ctx * %rsi: dst
@@ -139,9 +139,9 @@ SYM_FUNC_START(__blowfish_enc_blk) .L__enc_xor: xor_block(); ret; -SYM_FUNC_END(__blowfish_enc_blk) +SYM_FUNC_END_SECTION(__blowfish_enc_blk) -SYM_FUNC_START(blowfish_dec_blk) +SYM_FUNC_START_SECTION(blowfish_dec_blk) /* input: * %rdi: ctx * %rsi: dst
@@ -171,7 +171,7 @@ SYM_FUNC_START(blowfish_dec_blk) movq %r11, %r12; ret; -SYM_FUNC_END(blowfish_dec_blk) +SYM_FUNC_END_SECTION(blowfish_dec_blk) /********************************************************************** 4-way blowfish, four blocks parallel
@@ -283,7 +283,7 @@ SYM_FUNC_END(blowfish_dec_blk) bswapq RX3; \ xorq RX3, 24(RIO); -SYM_FUNC_START(__blowfish_enc_blk_4way) +SYM_FUNC_START_SECTION(__blowfish_enc_blk_4way) /* input: * %rdi: ctx * %rsi: dst
@@ -330,9 +330,9 @@ SYM_FUNC_START(__blowfish_enc_blk_4way) popq %rbx; popq %r12; ret; -SYM_FUNC_END(__blowfish_enc_blk_4way) +SYM_FUNC_END_SECTION(__blowfish_enc_blk_4way) -SYM_FUNC_START(blowfish_dec_blk_4way) +SYM_FUNC_START_SECTION(blowfish_dec_blk_4way) /* input: * %rdi: ctx * %rsi: dst
@@ -365,4 +365,4 @@ SYM_FUNC_START(blowfish_dec_blk_4way) popq %r12; ret; -SYM_FUNC_END(blowfish_dec_blk_4way) +SYM_FUNC_END_SECTION(blowfish_dec_blk_4way)
arch/x86/crypto/camellia-aesni-avx-asm_64.S +14 -14
@@ -187,21 +187,21 @@ * Size optimization... with inlined roundsm16, binary would be over 5 times * larger and would only be 0.5% faster (on sandy-bridge). */ +SYM_FUNC_START_LOCAL_SECTION(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) .align 8 -SYM_FUNC_START_LOCAL(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) roundsm16(%xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm15, %rcx, (%r9)); ret; -SYM_FUNC_END(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) +SYM_FUNC_END_SECTION(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) +SYM_FUNC_START_LOCAL_SECTION(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .align 8 -SYM_FUNC_START_LOCAL(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) roundsm16(%xmm4, %xmm5, %xmm6, %xmm7, %xmm0, %xmm1, %xmm2, %xmm3, %xmm12, %xmm13, %xmm14, %xmm15, %xmm8, %xmm9, %xmm10, %xmm11, %rax, (%r9)); ret; -SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) +SYM_FUNC_END_SECTION(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) /* * IN/OUT:
@@ -712,8 +712,8 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .text +SYM_FUNC_START_LOCAL_SECTION(__camellia_enc_blk16) .align 8 -SYM_FUNC_START_LOCAL(__camellia_enc_blk16) /* input: * %rdi: ctx, CTX * %rax: temporary storage, 256 bytes
@@ -797,10 +797,10 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk16) %xmm15, %rax, %rcx, 24); jmp .Lenc_done; -SYM_FUNC_END(__camellia_enc_blk16) +SYM_FUNC_END_SECTION(__camellia_enc_blk16) +SYM_FUNC_START_LOCAL_SECTION(__camellia_dec_blk16) .align 8 -SYM_FUNC_START_LOCAL(__camellia_dec_blk16) /* input: * %rdi: ctx, CTX * %rax: temporary storage, 256 bytes
@@ -882,9 +882,9 @@ SYM_FUNC_START_LOCAL(__camellia_dec_blk16) ((key_table + (24) * 8) + 4)(CTX)); jmp .Ldec_max24; -SYM_FUNC_END(__camellia_dec_blk16) +SYM_FUNC_END_SECTION(__camellia_dec_blk16) -SYM_FUNC_START(camellia_ecb_enc_16way) +SYM_FUNC_START_SECTION(camellia_ecb_enc_16way) /* input: * %rdi: ctx, CTX * %rsi: dst (16 blocks)
@@ -907,9 +907,9 @@ SYM_FUNC_START(camellia_ecb_enc_16way) FRAME_END ret; -SYM_FUNC_END(camellia_ecb_enc_16way) +SYM_FUNC_END_SECTION(camellia_ecb_enc_16way) -SYM_FUNC_START(camellia_ecb_dec_16way) +SYM_FUNC_START_SECTION(camellia_ecb_dec_16way) /* input: * %rdi: ctx, CTX * %rsi: dst (16 blocks)
@@ -937,9 +937,9 @@ SYM_FUNC_START(camellia_ecb_dec_16way) FRAME_END ret; -SYM_FUNC_END(camellia_ecb_dec_16way) +SYM_FUNC_END_SECTION(camellia_ecb_dec_16way) -SYM_FUNC_START(camellia_cbc_dec_16way) +SYM_FUNC_START_SECTION(camellia_cbc_dec_16way) /* input: * %rdi: ctx, CTX * %rsi: dst (16 blocks)
@@ -988,4 +988,4 @@ SYM_FUNC_START(camellia_cbc_dec_16way) FRAME_END ret; -SYM_FUNC_END(camellia_cbc_dec_16way) +SYM_FUNC_END_SECTION(camellia_cbc_dec_16way)
arch/x86/crypto/camellia-aesni-avx2-asm_64.S +14 -14
@@ -221,21 +221,21 @@ * Size optimization... with inlined roundsm32 binary would be over 5 times * larger and would only marginally faster. */ +SYM_FUNC_START_LOCAL_SECTION(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) .align 8 -SYM_FUNC_START_LOCAL(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) roundsm32(%ymm0, %ymm1, %ymm2, %ymm3, %ymm4, %ymm5, %ymm6, %ymm7, %ymm8, %ymm9, %ymm10, %ymm11, %ymm12, %ymm13, %ymm14, %ymm15, %rcx, (%r9)); ret; -SYM_FUNC_END(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) +SYM_FUNC_END_SECTION(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) +SYM_FUNC_START_LOCAL_SECTION(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .align 8 -SYM_FUNC_START_LOCAL(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) roundsm32(%ymm4, %ymm5, %ymm6, %ymm7, %ymm0, %ymm1, %ymm2, %ymm3, %ymm12, %ymm13, %ymm14, %ymm15, %ymm8, %ymm9, %ymm10, %ymm11, %rax, (%r9)); ret; -SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) +SYM_FUNC_END_SECTION(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) /* * IN/OUT:
@@ -748,8 +748,8 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) .text +SYM_FUNC_START_LOCAL_SECTION(__camellia_enc_blk32) .align 8 -SYM_FUNC_START_LOCAL(__camellia_enc_blk32) /* input: * %rdi: ctx, CTX * %rax: temporary storage, 512 bytes
@@ -833,10 +833,10 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk32) %ymm15, %rax, %rcx, 24); jmp .Lenc_done; -SYM_FUNC_END(__camellia_enc_blk32) +SYM_FUNC_END_SECTION(__camellia_enc_blk32) +SYM_FUNC_START_LOCAL_SECTION(__camellia_dec_blk32) .align 8 -SYM_FUNC_START_LOCAL(__camellia_dec_blk32) /* input: * %rdi: ctx, CTX * %rax: temporary storage, 512 bytes
@@ -918,9 +918,9 @@ SYM_FUNC_START_LOCAL(__camellia_dec_blk32) ((key_table + (24) * 8) + 4)(CTX)); jmp .Ldec_max24; -SYM_FUNC_END(__camellia_dec_blk32) +SYM_FUNC_END_SECTION(__camellia_dec_blk32) -SYM_FUNC_START(camellia_ecb_enc_32way) +SYM_FUNC_START_SECTION(camellia_ecb_enc_32way) /* input: * %rdi: ctx, CTX * %rsi: dst (32 blocks)
@@ -947,9 +947,9 @@ SYM_FUNC_START(camellia_ecb_enc_32way) FRAME_END ret; -SYM_FUNC_END(camellia_ecb_enc_32way) +SYM_FUNC_END_SECTION(camellia_ecb_enc_32way) -SYM_FUNC_START(camellia_ecb_dec_32way) +SYM_FUNC_START_SECTION(camellia_ecb_dec_32way) /* input: * %rdi: ctx, CTX * %rsi: dst (32 blocks)
@@ -981,9 +981,9 @@ SYM_FUNC_START(camellia_ecb_dec_32way) FRAME_END ret; -SYM_FUNC_END(camellia_ecb_dec_32way) +SYM_FUNC_END_SECTION(camellia_ecb_dec_32way) -SYM_FUNC_START(camellia_cbc_dec_32way) +SYM_FUNC_START_SECTION(camellia_cbc_dec_32way) /* input: * %rdi: ctx, CTX * %rsi: dst (32 blocks)
@@ -1048,4 +1048,4 @@ SYM_FUNC_START(camellia_cbc_dec_32way) addq $(16 * 32), %rsp; FRAME_END ret; -SYM_FUNC_END(camellia_cbc_dec_32way) +SYM_FUNC_END_SECTION(camellia_cbc_dec_32way)
arch/x86/crypto/camellia-x86_64-asm_64.S +8 -8
@@ -175,7 +175,7 @@ bswapq RAB0; \ movq RAB0, 4*2(RIO); -SYM_FUNC_START(__camellia_enc_blk) +SYM_FUNC_START_SECTION(__camellia_enc_blk) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -220,9 +220,9 @@ SYM_FUNC_START(__camellia_enc_blk) movq RR12, %r12; ret; -SYM_FUNC_END(__camellia_enc_blk) +SYM_FUNC_END_SECTION(__camellia_enc_blk) -SYM_FUNC_START(camellia_dec_blk) +SYM_FUNC_START_SECTION(camellia_dec_blk) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -258,7 +258,7 @@ SYM_FUNC_START(camellia_dec_blk) movq RR12, %r12; ret; -SYM_FUNC_END(camellia_dec_blk) +SYM_FUNC_END_SECTION(camellia_dec_blk) /********************************************************************** 2-way camellia
@@ -409,7 +409,7 @@ SYM_FUNC_END(camellia_dec_blk) bswapq RAB1; \ movq RAB1, 12*2(RIO); -SYM_FUNC_START(__camellia_enc_blk_2way) +SYM_FUNC_START_SECTION(__camellia_enc_blk_2way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -456,9 +456,9 @@ SYM_FUNC_START(__camellia_enc_blk_2way) movq RR12, %r12; popq %rbx; ret; -SYM_FUNC_END(__camellia_enc_blk_2way) +SYM_FUNC_END_SECTION(__camellia_enc_blk_2way) -SYM_FUNC_START(camellia_dec_blk_2way) +SYM_FUNC_START_SECTION(camellia_dec_blk_2way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -496,4 +496,4 @@ SYM_FUNC_START(camellia_dec_blk_2way) movq RR12, %r12; movq RXOR, %rbx; ret; -SYM_FUNC_END(camellia_dec_blk_2way) +SYM_FUNC_END_SECTION(camellia_dec_blk_2way)
arch/x86/crypto/cast5-avx-x86_64-asm_64.S +12 -12
@@ -208,8 +208,8 @@ .text +SYM_FUNC_START_LOCAL_SECTION(__cast5_enc_blk16) .align 16 -SYM_FUNC_START_LOCAL(__cast5_enc_blk16) /* input: * %rdi: ctx * RL1: blocks 1 and 2
@@ -280,10 +280,10 @@ SYM_FUNC_START_LOCAL(__cast5_enc_blk16) outunpack_blocks(RR4, RL4, RTMP, RX, RKM); ret; -SYM_FUNC_END(__cast5_enc_blk16) +SYM_FUNC_END_SECTION(__cast5_enc_blk16) +SYM_FUNC_START_LOCAL_SECTION(__cast5_dec_blk16) .align 16 -SYM_FUNC_START_LOCAL(__cast5_dec_blk16) /* input: * %rdi: ctx * RL1: encrypted blocks 1 and 2
@@ -357,9 +357,9 @@ SYM_FUNC_START_LOCAL(__cast5_dec_blk16) .L__skip_dec: vpsrldq $4, RKR, RKR; jmp .L__dec_tail; -SYM_FUNC_END(__cast5_dec_blk16) +SYM_FUNC_END_SECTION(__cast5_dec_blk16) -SYM_FUNC_START(cast5_ecb_enc_16way) +SYM_FUNC_START_SECTION(cast5_ecb_enc_16way) /* input: * %rdi: ctx * %rsi: dst
@@ -394,9 +394,9 @@ SYM_FUNC_START(cast5_ecb_enc_16way) popq %r15; FRAME_END ret; -SYM_FUNC_END(cast5_ecb_enc_16way) +SYM_FUNC_END_SECTION(cast5_ecb_enc_16way) -SYM_FUNC_START(cast5_ecb_dec_16way) +SYM_FUNC_START_SECTION(cast5_ecb_dec_16way) /* input: * %rdi: ctx * %rsi: dst
@@ -432,9 +432,9 @@ SYM_FUNC_START(cast5_ecb_dec_16way) popq %r15; FRAME_END ret; -SYM_FUNC_END(cast5_ecb_dec_16way) +SYM_FUNC_END_SECTION(cast5_ecb_dec_16way) -SYM_FUNC_START(cast5_cbc_dec_16way) +SYM_FUNC_START_SECTION(cast5_cbc_dec_16way) /* input: * %rdi: ctx * %rsi: dst
@@ -484,9 +484,9 @@ SYM_FUNC_START(cast5_cbc_dec_16way) popq %r12; FRAME_END ret; -SYM_FUNC_END(cast5_cbc_dec_16way) +SYM_FUNC_END_SECTION(cast5_cbc_dec_16way) -SYM_FUNC_START(cast5_ctr_16way) +SYM_FUNC_START_SECTION(cast5_ctr_16way) /* input: * %rdi: ctx * %rsi: dst
@@ -560,4 +560,4 @@ SYM_FUNC_START(cast5_ctr_16way) popq %r12; FRAME_END ret; -SYM_FUNC_END(cast5_ctr_16way) +SYM_FUNC_END_SECTION(cast5_ctr_16way)
arch/x86/crypto/cast6-avx-x86_64-asm_64.S +10 -10
@@ -244,8 +244,8 @@ .text +SYM_FUNC_START_LOCAL_SECTION(__cast6_enc_blk8) .align 8 -SYM_FUNC_START_LOCAL(__cast6_enc_blk8) /* input: * %rdi: ctx * RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: blocks
@@ -290,10 +290,10 @@ SYM_FUNC_START_LOCAL(__cast6_enc_blk8) outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); ret; -SYM_FUNC_END(__cast6_enc_blk8) +SYM_FUNC_END_SECTION(__cast6_enc_blk8) +SYM_FUNC_START_LOCAL_SECTION(__cast6_dec_blk8) .align 8 -SYM_FUNC_START_LOCAL(__cast6_dec_blk8) /* input: * %rdi: ctx * RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: encrypted blocks
@@ -337,9 +337,9 @@ SYM_FUNC_START_LOCAL(__cast6_dec_blk8) outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); ret; -SYM_FUNC_END(__cast6_dec_blk8) +SYM_FUNC_END_SECTION(__cast6_dec_blk8) -SYM_FUNC_START(cast6_ecb_enc_8way) +SYM_FUNC_START_SECTION(cast6_ecb_enc_8way) /* input: * %rdi: ctx * %rsi: dst
@@ -360,9 +360,9 @@ SYM_FUNC_START(cast6_ecb_enc_8way) popq %r15; FRAME_END ret; -SYM_FUNC_END(cast6_ecb_enc_8way) +SYM_FUNC_END_SECTION(cast6_ecb_enc_8way) -SYM_FUNC_START(cast6_ecb_dec_8way) +SYM_FUNC_START_SECTION(cast6_ecb_dec_8way) /* input: * %rdi: ctx * %rsi: dst
@@ -383,9 +383,9 @@ SYM_FUNC_START(cast6_ecb_dec_8way) popq %r15; FRAME_END ret; -SYM_FUNC_END(cast6_ecb_dec_8way) +SYM_FUNC_END_SECTION(cast6_ecb_dec_8way) -SYM_FUNC_START(cast6_cbc_dec_8way) +SYM_FUNC_START_SECTION(cast6_cbc_dec_8way) /* input: * %rdi: ctx * %rsi: dst
@@ -409,4 +409,4 @@ SYM_FUNC_START(cast6_cbc_dec_8way) popq %r12; FRAME_END ret; -SYM_FUNC_END(cast6_cbc_dec_8way) +SYM_FUNC_END_SECTION(cast6_cbc_dec_8way)
arch/x86/crypto/chacha-avx2-x86_64.S +6 -6
@@ -34,7 +34,7 @@ CTR4BL: .octa 0x00000000000000000000000000000002 .text -SYM_FUNC_START(chacha_2block_xor_avx2) +SYM_FUNC_START_SECTION(chacha_2block_xor_avx2) # %rdi: Input state matrix, s # %rsi: up to 2 data blocks output, o # %rdx: up to 2 data blocks input, i
@@ -224,9 +224,9 @@ SYM_FUNC_START(chacha_2block_xor_avx2) lea -8(%r10),%rsp jmp .Ldone2 -SYM_FUNC_END(chacha_2block_xor_avx2) +SYM_FUNC_END_SECTION(chacha_2block_xor_avx2) -SYM_FUNC_START(chacha_4block_xor_avx2) +SYM_FUNC_START_SECTION(chacha_4block_xor_avx2) # %rdi: Input state matrix, s # %rsi: up to 4 data blocks output, o # %rdx: up to 4 data blocks input, i
@@ -529,9 +529,9 @@ SYM_FUNC_START(chacha_4block_xor_avx2) lea -8(%r10),%rsp jmp .Ldone4 -SYM_FUNC_END(chacha_4block_xor_avx2) +SYM_FUNC_END_SECTION(chacha_4block_xor_avx2) -SYM_FUNC_START(chacha_8block_xor_avx2) +SYM_FUNC_START_SECTION(chacha_8block_xor_avx2) # %rdi: Input state matrix, s # %rsi: up to 8 data blocks output, o # %rdx: up to 8 data blocks input, i
@@ -1018,4 +1018,4 @@ SYM_FUNC_START(chacha_8block_xor_avx2) jmp .Ldone8 -SYM_FUNC_END(chacha_8block_xor_avx2) +SYM_FUNC_END_SECTION(chacha_8block_xor_avx2)
arch/x86/crypto/chacha-avx512vl-x86_64.S +6 -6
@@ -24,7 +24,7 @@ CTR8BL: .octa 0x00000003000000020000000100000000 .text -SYM_FUNC_START(chacha_2block_xor_avx512vl) +SYM_FUNC_START_SECTION(chacha_2block_xor_avx512vl) # %rdi: Input state matrix, s # %rsi: up to 2 data blocks output, o # %rdx: up to 2 data blocks input, i
@@ -187,9 +187,9 @@ SYM_FUNC_START(chacha_2block_xor_avx512vl) jmp .Ldone2 -SYM_FUNC_END(chacha_2block_xor_avx512vl) +SYM_FUNC_END_SECTION(chacha_2block_xor_avx512vl) -SYM_FUNC_START(chacha_4block_xor_avx512vl) +SYM_FUNC_START_SECTION(chacha_4block_xor_avx512vl) # %rdi: Input state matrix, s # %rsi: up to 4 data blocks output, o # %rdx: up to 4 data blocks input, i
@@ -453,9 +453,9 @@ SYM_FUNC_START(chacha_4block_xor_avx512vl) jmp .Ldone4 -SYM_FUNC_END(chacha_4block_xor_avx512vl) +SYM_FUNC_END_SECTION(chacha_4block_xor_avx512vl) -SYM_FUNC_START(chacha_8block_xor_avx512vl) +SYM_FUNC_START_SECTION(chacha_8block_xor_avx512vl) # %rdi: Input state matrix, s # %rsi: up to 8 data blocks output, o # %rdx: up to 8 data blocks input, i
@@ -833,4 +833,4 @@ SYM_FUNC_START(chacha_8block_xor_avx512vl) jmp .Ldone8 -SYM_FUNC_END(chacha_8block_xor_avx512vl) +SYM_FUNC_END_SECTION(chacha_8block_xor_avx512vl)
arch/x86/crypto/chacha-ssse3-x86_64.S +8 -8
@@ -33,7 +33,7 @@ CTRINC: .octa 0x00000003000000020000000100000000 * * Clobbers: %r8d, %xmm4-%xmm7 */ -SYM_FUNC_START_LOCAL(chacha_permute) +SYM_FUNC_START_LOCAL_SECTION(chacha_permute) movdqa ROT8(%rip),%xmm4 movdqa ROT16(%rip),%xmm5
@@ -109,9 +109,9 @@ SYM_FUNC_START_LOCAL(chacha_permute) jnz .Ldoubleround ret -SYM_FUNC_END(chacha_permute) +SYM_FUNC_END_SECTION(chacha_permute) -SYM_FUNC_START(chacha_block_xor_ssse3) +SYM_FUNC_START_SECTION(chacha_block_xor_ssse3) # %rdi: Input state matrix, s # %rsi: up to 1 data block output, o # %rdx: up to 1 data block input, i
@@ -197,9 +197,9 @@ SYM_FUNC_START(chacha_block_xor_ssse3) lea -8(%r10),%rsp jmp .Ldone -SYM_FUNC_END(chacha_block_xor_ssse3) +SYM_FUNC_END_SECTION(chacha_block_xor_ssse3) -SYM_FUNC_START(hchacha_block_ssse3) +SYM_FUNC_START_SECTION(hchacha_block_ssse3) # %rdi: Input state matrix, s # %rsi: output (8 32-bit words) # %edx: nrounds
@@ -218,9 +218,9 @@ SYM_FUNC_START(hchacha_block_ssse3) FRAME_END ret -SYM_FUNC_END(hchacha_block_ssse3) +SYM_FUNC_END_SECTION(hchacha_block_ssse3) -SYM_FUNC_START(chacha_4block_xor_ssse3) +SYM_FUNC_START_SECTION(chacha_4block_xor_ssse3) # %rdi: Input state matrix, s # %rsi: up to 4 data blocks output, o # %rdx: up to 4 data blocks input, i
@@ -788,4 +788,4 @@ SYM_FUNC_START(chacha_4block_xor_ssse3) jmp .Ldone4 -SYM_FUNC_END(chacha_4block_xor_ssse3) +SYM_FUNC_END_SECTION(chacha_4block_xor_ssse3)
arch/x86/crypto/crc32-pclmul_asm.S +2 -2
@@ -102,7 +102,7 @@ * size_t len, uint crc32) */ -SYM_FUNC_START(crc32_pclmul_le_16) /* buffer and buffer size are 16 bytes aligned */ +SYM_FUNC_START_SECTION(crc32_pclmul_le_16) /* buffer and buffer size are 16 bytes aligned */ movdqa (BUF), %xmm1 movdqa 0x10(BUF), %xmm2 movdqa 0x20(BUF), %xmm3
@@ -237,4 +237,4 @@ fold_64: pextrd $0x01, %xmm1, %eax ret -SYM_FUNC_END(crc32_pclmul_le_16) +SYM_FUNC_END_SECTION(crc32_pclmul_le_16)
arch/x86/crypto/crc32c-pcl-intel-asm_64.S +2 -2
@@ -73,7 +73,7 @@ # unsigned int crc_pcl(u8 *buffer, int len, unsigned int crc_init); .text -SYM_FUNC_START(crc_pcl) +SYM_FUNC_START_SECTION(crc_pcl) #define bufp rdi #define bufp_dw %edi #define bufp_w %di
@@ -307,7 +307,7 @@ do_return: popq %rdi popq %rbx ret -SYM_FUNC_END(crc_pcl) +SYM_FUNC_END_SECTION(crc_pcl) .section .rodata, "a", @progbits ################################################################
arch/x86/crypto/crct10dif-pcl-asm_64.S +2 -2
@@ -94,8 +94,8 @@ # # Assumes len >= 16. # +SYM_FUNC_START_SECTION(crc_t10dif_pcl) .align 16 -SYM_FUNC_START(crc_t10dif_pcl) movdqa .Lbswap_mask(%rip), BSWAP_MASK
@@ -280,7 +280,7 @@ SYM_FUNC_START(crc_t10dif_pcl) jge .Lfold_16_bytes_loop # 32 <= len <= 255 add $16, len jmp .Lhandle_partial_segment # 17 <= len <= 31 -SYM_FUNC_END(crc_t10dif_pcl) +SYM_FUNC_END_SECTION(crc_t10dif_pcl) .section .rodata, "a", @progbits .align 16
arch/x86/crypto/des3_ede-asm_64.S +4 -4
@@ -162,7 +162,7 @@ movl left##d, (io); \ movl right##d, 4(io); -SYM_FUNC_START(des3_ede_x86_64_crypt_blk) +SYM_FUNC_START_SECTION(des3_ede_x86_64_crypt_blk) /* input: * %rdi: round keys, CTX * %rsi: dst
@@ -244,7 +244,7 @@ SYM_FUNC_START(des3_ede_x86_64_crypt_blk) popq %rbx; ret; -SYM_FUNC_END(des3_ede_x86_64_crypt_blk) +SYM_FUNC_END_SECTION(des3_ede_x86_64_crypt_blk) /*********************************************************************** * 3-way 3DES
@@ -418,7 +418,7 @@ SYM_FUNC_END(des3_ede_x86_64_crypt_blk) #define __movq(src, dst) \ movq src, dst; -SYM_FUNC_START(des3_ede_x86_64_crypt_blk_3way) +SYM_FUNC_START_SECTION(des3_ede_x86_64_crypt_blk_3way) /* input: * %rdi: ctx, round keys * %rsi: dst (3 blocks)
@@ -529,7 +529,7 @@ SYM_FUNC_START(des3_ede_x86_64_crypt_blk_3way) popq %rbx; ret; -SYM_FUNC_END(des3_ede_x86_64_crypt_blk_3way) +SYM_FUNC_END_SECTION(des3_ede_x86_64_crypt_blk_3way) .section .rodata, "a", @progbits .align 16
arch/x86/crypto/ghash-clmulni-intel_asm.S +6 -6
@@ -43,7 +43,7 @@ * T2 * T3 */ -SYM_FUNC_START_LOCAL(__clmul_gf128mul_ble) +SYM_FUNC_START_LOCAL_SECTION(__clmul_gf128mul_ble) movaps DATA, T1 pshufd $0b01001110, DATA, T2 pshufd $0b01001110, SHASH, T3
@@ -86,10 +86,10 @@ SYM_FUNC_START_LOCAL(__clmul_gf128mul_ble) pxor T2, T1 pxor T1, DATA ret -SYM_FUNC_END(__clmul_gf128mul_ble) +SYM_FUNC_END_SECTION(__clmul_gf128mul_ble) /* void clmul_ghash_mul(char *dst, const u128 *shash) */ -SYM_FUNC_START(clmul_ghash_mul) +SYM_FUNC_START_SECTION(clmul_ghash_mul) FRAME_BEGIN movups (%rdi), DATA movups (%rsi), SHASH
@@ -100,13 +100,13 @@ SYM_FUNC_START(clmul_ghash_mul) movups DATA, (%rdi) FRAME_END ret -SYM_FUNC_END(clmul_ghash_mul) +SYM_FUNC_END_SECTION(clmul_ghash_mul) /* * void clmul_ghash_update(char *dst, const char *src, unsigned int srclen, * const u128 *shash); */ -SYM_FUNC_START(clmul_ghash_update) +SYM_FUNC_START_SECTION(clmul_ghash_update) FRAME_BEGIN cmp $16, %rdx jb .Lupdate_just_ret # check length
@@ -129,4 +129,4 @@ SYM_FUNC_START(clmul_ghash_update) .Lupdate_just_ret: FRAME_END ret -SYM_FUNC_END(clmul_ghash_update) +SYM_FUNC_END_SECTION(clmul_ghash_update)
arch/x86/crypto/nh-avx2-x86_64.S +2 -2
@@ -69,7 +69,7 @@ * * It's guaranteed that message_len % 16 == 0. */ -SYM_FUNC_START(nh_avx2) +SYM_FUNC_START_SECTION(nh_avx2) vmovdqu 0x00(KEY), K0 vmovdqu 0x10(KEY), K1
@@ -154,4 +154,4 @@ SYM_FUNC_START(nh_avx2) vpaddq T4, T0, T0 vmovdqu T0, (HASH) ret -SYM_FUNC_END(nh_avx2) +SYM_FUNC_END_SECTION(nh_avx2)
arch/x86/crypto/nh-sse2-x86_64.S +2 -2
@@ -71,7 +71,7 @@ * * It's guaranteed that message_len % 16 == 0. */ -SYM_FUNC_START(nh_sse2) +SYM_FUNC_START_SECTION(nh_sse2) movdqu 0x00(KEY), K0 movdqu 0x10(KEY), K1
@@ -120,4 +120,4 @@ SYM_FUNC_START(nh_sse2) movdqu T0, 0x00(HASH) movdqu T1, 0x10(HASH) ret -SYM_FUNC_END(nh_sse2) +SYM_FUNC_END_SECTION(nh_sse2)
arch/x86/crypto/poly1305-x86_64-cryptogams.pl +6 -2
@@ -108,8 +108,8 @@ if (!$kernel) { sub declare_function() { my ($name, $align, $nargs) = @_; if($kernel) { + $code .= "SYM_FUNC_START_SECTION($name)\n"; $code .= ".align $align\n"; - $code .= "SYM_FUNC_START($name)\n"; $code .= ".L$name:\n"; } else { $code .= ".globl $name\n";
@@ -122,7 +122,7 @@ sub declare_function() { sub end_function() { my ($name) = @_; if($kernel) { - $code .= "SYM_FUNC_END($name)\n"; + $code .= "SYM_FUNC_END_SECTION($name)\n"; } else { $code .= ".size $name,.-$name\n"; }
@@ -421,6 +421,7 @@ my ($H0,$H1,$H2,$H3,$H4, $T0,$T1,$T2,$T3,$T4, $D0,$D1,$D2,$D3,$D4, $MASK) = map("%xmm$_",(0..15)); $code.=<<___; +SYM_TEXT_SECTION(__poly1305_block) .type __poly1305_block,\@abi-omnipotent .align 32 __poly1305_block:
@@ -431,7 +432,9 @@ $code.=<<___; pop $ctx ret .size __poly1305_block,.-__poly1305_block +SYM_TEXT_END_SECTION +SYM_TEXT_SECTION(__poly1305_init_avx) .type __poly1305_init_avx,\@abi-omnipotent .align 32 __poly1305_init_avx:
@@ -596,6 +599,7 @@ __poly1305_init_avx: pop %rbp ret .size __poly1305_init_avx,.-__poly1305_init_avx +SYM_TEXT_END_SECTION ___ &declare_function("poly1305_blocks_avx", 32, 4);
arch/x86/crypto/serpent-avx-x86_64-asm_64.S +10 -10
@@ -550,8 +550,8 @@ #define write_blocks(x0, x1, x2, x3, t0, t1, t2) \ transpose_4x4(x0, x1, x2, x3, t0, t1, t2) +SYM_FUNC_START_LOCAL_SECTION(__serpent_enc_blk8_avx) .align 8 -SYM_FUNC_START_LOCAL(__serpent_enc_blk8_avx) /* input: * %rdi: ctx, CTX * RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: blocks
@@ -602,10 +602,10 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk8_avx) write_blocks(RA2, RB2, RC2, RD2, RK0, RK1, RK2); ret; -SYM_FUNC_END(__serpent_enc_blk8_avx) +SYM_FUNC_END_SECTION(__serpent_enc_blk8_avx) +SYM_FUNC_START_LOCAL_SECTION(__serpent_dec_blk8_avx) .align 8 -SYM_FUNC_START_LOCAL(__serpent_dec_blk8_avx) /* input: * %rdi: ctx, CTX * RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: encrypted blocks
@@ -656,9 +656,9 @@ SYM_FUNC_START_LOCAL(__serpent_dec_blk8_avx) write_blocks(RC2, RD2, RB2, RE2, RK0, RK1, RK2); ret; -SYM_FUNC_END(__serpent_dec_blk8_avx) +SYM_FUNC_END_SECTION(__serpent_dec_blk8_avx) -SYM_FUNC_START(serpent_ecb_enc_8way_avx) +SYM_FUNC_START_SECTION(serpent_ecb_enc_8way_avx) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -674,9 +674,9 @@ SYM_FUNC_START(serpent_ecb_enc_8way_avx) FRAME_END ret; -SYM_FUNC_END(serpent_ecb_enc_8way_avx) +SYM_FUNC_END_SECTION(serpent_ecb_enc_8way_avx) -SYM_FUNC_START(serpent_ecb_dec_8way_avx) +SYM_FUNC_START_SECTION(serpent_ecb_dec_8way_avx) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -692,9 +692,9 @@ SYM_FUNC_START(serpent_ecb_dec_8way_avx) FRAME_END ret; -SYM_FUNC_END(serpent_ecb_dec_8way_avx) +SYM_FUNC_END_SECTION(serpent_ecb_dec_8way_avx) -SYM_FUNC_START(serpent_cbc_dec_8way_avx) +SYM_FUNC_START_SECTION(serpent_cbc_dec_8way_avx) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -710,4 +710,4 @@ SYM_FUNC_START(serpent_cbc_dec_8way_avx) FRAME_END ret; -SYM_FUNC_END(serpent_cbc_dec_8way_avx) +SYM_FUNC_END_SECTION(serpent_cbc_dec_8way_avx)
arch/x86/crypto/serpent-avx2-asm_64.S +10 -10
@@ -550,8 +550,8 @@ #define write_blocks(x0, x1, x2, x3, t0, t1, t2) \ transpose_4x4(x0, x1, x2, x3, t0, t1, t2) +SYM_FUNC_START_LOCAL_SECTION(__serpent_enc_blk16) .align 8 -SYM_FUNC_START_LOCAL(__serpent_enc_blk16) /* input: * %rdi: ctx, CTX * RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: plaintext
@@ -602,10 +602,10 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk16) write_blocks(RA2, RB2, RC2, RD2, RK0, RK1, RK2); ret; -SYM_FUNC_END(__serpent_enc_blk16) +SYM_FUNC_END_SECTION(__serpent_enc_blk16) +SYM_FUNC_START_LOCAL_SECTION(__serpent_dec_blk16) .align 8 -SYM_FUNC_START_LOCAL(__serpent_dec_blk16) /* input: * %rdi: ctx, CTX * RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: ciphertext
@@ -656,9 +656,9 @@ SYM_FUNC_START_LOCAL(__serpent_dec_blk16) write_blocks(RC2, RD2, RB2, RE2, RK0, RK1, RK2); ret; -SYM_FUNC_END(__serpent_dec_blk16) +SYM_FUNC_END_SECTION(__serpent_dec_blk16) -SYM_FUNC_START(serpent_ecb_enc_16way) +SYM_FUNC_START_SECTION(serpent_ecb_enc_16way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -678,9 +678,9 @@ SYM_FUNC_START(serpent_ecb_enc_16way) FRAME_END ret; -SYM_FUNC_END(serpent_ecb_enc_16way) +SYM_FUNC_END_SECTION(serpent_ecb_enc_16way) -SYM_FUNC_START(serpent_ecb_dec_16way) +SYM_FUNC_START_SECTION(serpent_ecb_dec_16way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -700,9 +700,9 @@ SYM_FUNC_START(serpent_ecb_dec_16way) FRAME_END ret; -SYM_FUNC_END(serpent_ecb_dec_16way) +SYM_FUNC_END_SECTION(serpent_ecb_dec_16way) -SYM_FUNC_START(serpent_cbc_dec_16way) +SYM_FUNC_START_SECTION(serpent_cbc_dec_16way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -723,4 +723,4 @@ SYM_FUNC_START(serpent_cbc_dec_16way) FRAME_END ret; -SYM_FUNC_END(serpent_cbc_dec_16way) +SYM_FUNC_END_SECTION(serpent_cbc_dec_16way)
arch/x86/crypto/serpent-sse2-i586-asm_32.S +4 -4
@@ -497,7 +497,7 @@ pxor t0, x3; \ movdqu x3, (3*4*4)(out); -SYM_FUNC_START(__serpent_enc_blk_4way) +SYM_FUNC_START_SECTION(__serpent_enc_blk_4way) /* input: * arg_ctx(%esp): ctx, CTX * arg_dst(%esp): dst
@@ -559,9 +559,9 @@ SYM_FUNC_START(__serpent_enc_blk_4way) xor_blocks(%eax, RA, RB, RC, RD, RT0, RT1, RE); ret; -SYM_FUNC_END(__serpent_enc_blk_4way) +SYM_FUNC_END_SECTION(__serpent_enc_blk_4way) -SYM_FUNC_START(serpent_dec_blk_4way) +SYM_FUNC_START_SECTION(serpent_dec_blk_4way) /* input: * arg_ctx(%esp): ctx, CTX * arg_dst(%esp): dst
@@ -613,4 +613,4 @@ SYM_FUNC_START(serpent_dec_blk_4way) write_blocks(%eax, RC, RD, RB, RE, RT0, RT1, RA); ret; -SYM_FUNC_END(serpent_dec_blk_4way) +SYM_FUNC_END_SECTION(serpent_dec_blk_4way)
arch/x86/crypto/serpent-sse2-x86_64-asm_64.S +4 -4
@@ -619,7 +619,7 @@ pxor t0, x3; \ movdqu x3, (3*4*4)(out); -SYM_FUNC_START(__serpent_enc_blk_8way) +SYM_FUNC_START_SECTION(__serpent_enc_blk_8way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -682,9 +682,9 @@ SYM_FUNC_START(__serpent_enc_blk_8way) xor_blocks(%rax, RA2, RB2, RC2, RD2, RK0, RK1, RK2); ret; -SYM_FUNC_END(__serpent_enc_blk_8way) +SYM_FUNC_END_SECTION(__serpent_enc_blk_8way) -SYM_FUNC_START(serpent_dec_blk_8way) +SYM_FUNC_START_SECTION(serpent_dec_blk_8way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -736,4 +736,4 @@ SYM_FUNC_START(serpent_dec_blk_8way) write_blocks(%rax, RC2, RD2, RB2, RE2, RK0, RK1, RK2); ret; -SYM_FUNC_END(serpent_dec_blk_8way) +SYM_FUNC_END_SECTION(serpent_dec_blk_8way)
arch/x86/crypto/sha1_avx2_x86_64_asm.S +2 -2
@@ -634,7 +634,7 @@ _loop3: * param: function's name */ .macro SHA1_VECTOR_ASM name - SYM_FUNC_START(\name) + SYM_FUNC_START_SECTION(\name) push %rbx push %r12
@@ -676,7 +676,7 @@ _loop3: ret - SYM_FUNC_END(\name) + SYM_FUNC_END_SECTION(\name) .endm .section .rodata
arch/x86/crypto/sha1_ni_asm.S +2 -2
@@ -92,8 +92,8 @@ * numBlocks: Number of blocks to process */ .text +SYM_FUNC_START_SECTION(sha1_ni_transform) .align 32 -SYM_FUNC_START(sha1_ni_transform) push %rbp mov %rsp, %rbp sub $FRAME_SIZE, %rsp
@@ -291,7 +291,7 @@ SYM_FUNC_START(sha1_ni_transform) pop %rbp ret -SYM_FUNC_END(sha1_ni_transform) +SYM_FUNC_END_SECTION(sha1_ni_transform) .section .rodata.cst16.PSHUFFLE_BYTE_FLIP_MASK, "aM", @progbits, 16 .align 16
arch/x86/crypto/sha1_ssse3_asm.S +2 -2
@@ -67,7 +67,7 @@ * param: function's name */ .macro SHA1_VECTOR_ASM name - SYM_FUNC_START(\name) + SYM_FUNC_START_SECTION(\name) push %rbx push %r12
@@ -101,7 +101,7 @@ pop %rbx ret - SYM_FUNC_END(\name) + SYM_FUNC_END_SECTION(\name) .endm /*
arch/x86/crypto/sha256-avx-asm.S +2 -2
@@ -346,7 +346,7 @@ a = TMP_ ## arg 3 : Num blocks ######################################################################## .text -SYM_FUNC_START(sha256_transform_avx) +SYM_FUNC_START_SECTION(sha256_transform_avx) .align 32 pushq %rbx pushq %r12
@@ -459,7 +459,7 @@ done_hash: popq %r12 popq %rbx ret -SYM_FUNC_END(sha256_transform_avx) +SYM_FUNC_END_SECTION(sha256_transform_avx) .section .rodata.cst256.K256, "aM", @progbits, 256 .align 64
arch/x86/crypto/sha256-avx2-asm.S +2 -2
@@ -523,7 +523,7 @@ STACK_SIZE = _CTX + _CTX_SIZE ## arg 3 : Num blocks ######################################################################## .text -SYM_FUNC_START(sha256_transform_rorx) +SYM_FUNC_START_SECTION(sha256_transform_rorx) .align 32 pushq %rbx pushq %r12
@@ -711,7 +711,7 @@ done_hash: popq %r12 popq %rbx ret -SYM_FUNC_END(sha256_transform_rorx) +SYM_FUNC_END_SECTION(sha256_transform_rorx) .section .rodata.cst512.K256, "aM", @progbits, 512 .align 64
arch/x86/crypto/sha256-ssse3-asm.S +2 -2
@@ -355,7 +355,7 @@ a = TMP_ ## arg 3 : Num blocks ######################################################################## .text -SYM_FUNC_START(sha256_transform_ssse3) +SYM_FUNC_START_SECTION(sha256_transform_ssse3) .align 32 pushq %rbx pushq %r12
@@ -473,7 +473,7 @@ done_hash: popq %rbx ret -SYM_FUNC_END(sha256_transform_ssse3) +SYM_FUNC_END_SECTION(sha256_transform_ssse3) .section .rodata.cst256.K256, "aM", @progbits, 256 .align 64
arch/x86/crypto/sha256_ni_asm.S +2 -2
@@ -96,8 +96,8 @@ */ .text +SYM_FUNC_START_SECTION(sha256_ni_transform) .align 32 -SYM_FUNC_START(sha256_ni_transform) shl $6, NUM_BLKS /* convert to bytes */ jz .Ldone_hash
@@ -327,7 +327,7 @@ SYM_FUNC_START(sha256_ni_transform) .Ldone_hash: ret -SYM_FUNC_END(sha256_ni_transform) +SYM_FUNC_END_SECTION(sha256_ni_transform) .section .rodata.cst256.K256, "aM", @progbits, 256 .align 64
arch/x86/crypto/sha512-avx-asm.S +2 -2
@@ -273,7 +273,7 @@ frame_size = frame_WK + WK_SIZE # of SHA512 message blocks. # "blocks" is the message length in SHA512 blocks ######################################################################## -SYM_FUNC_START(sha512_transform_avx) +SYM_FUNC_START_SECTION(sha512_transform_avx) test msglen, msglen je nowork
@@ -362,7 +362,7 @@ updateblock: nowork: ret -SYM_FUNC_END(sha512_transform_avx) +SYM_FUNC_END_SECTION(sha512_transform_avx) ######################################################################## ### Binary Data
arch/x86/crypto/sha512-avx2-asm.S +2 -2
@@ -565,7 +565,7 @@ frame_size = frame_CTX + CTX_SIZE # of SHA512 message blocks. # "blocks" is the message length in SHA512 blocks ######################################################################## -SYM_FUNC_START(sha512_transform_rorx) +SYM_FUNC_START_SECTION(sha512_transform_rorx) # Save GPRs push %rbx push %r12
@@ -680,7 +680,7 @@ done_hash: pop %rbx ret -SYM_FUNC_END(sha512_transform_rorx) +SYM_FUNC_END_SECTION(sha512_transform_rorx) ######################################################################## ### Binary Data
arch/x86/crypto/sha512-ssse3-asm.S +2 -2
@@ -274,7 +274,7 @@ frame_size = frame_WK + WK_SIZE # of SHA512 message blocks. # "blocks" is the message length in SHA512 blocks. ######################################################################## -SYM_FUNC_START(sha512_transform_ssse3) +SYM_FUNC_START_SECTION(sha512_transform_ssse3) test msglen, msglen je nowork
@@ -364,7 +364,7 @@ updateblock: nowork: ret -SYM_FUNC_END(sha512_transform_ssse3) +SYM_FUNC_END_SECTION(sha512_transform_ssse3) ######################################################################## ### Binary Data
arch/x86/crypto/twofish-avx-x86_64-asm_64.S +10 -10
@@ -228,8 +228,8 @@ vpxor x2, wkey, x2; \ vpxor x3, wkey, x3; +SYM_FUNC_START_LOCAL_SECTION(__twofish_enc_blk8) .align 8 -SYM_FUNC_START_LOCAL(__twofish_enc_blk8) /* input: * %rdi: ctx, CTX * RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: blocks
@@ -268,10 +268,10 @@ SYM_FUNC_START_LOCAL(__twofish_enc_blk8) outunpack_blocks(RC2, RD2, RA2, RB2, RK1, RX0, RY0, RK2); ret; -SYM_FUNC_END(__twofish_enc_blk8) +SYM_FUNC_END_SECTION(__twofish_enc_blk8) +SYM_FUNC_START_LOCAL_SECTION(__twofish_dec_blk8) .align 8 -SYM_FUNC_START_LOCAL(__twofish_dec_blk8) /* input: * %rdi: ctx, CTX * RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2: encrypted blocks
@@ -308,9 +308,9 @@ SYM_FUNC_START_LOCAL(__twofish_dec_blk8) outunpack_blocks(RA2, RB2, RC2, RD2, RK1, RX0, RY0, RK2); ret; -SYM_FUNC_END(__twofish_dec_blk8) +SYM_FUNC_END_SECTION(__twofish_dec_blk8) -SYM_FUNC_START(twofish_ecb_enc_8way) +SYM_FUNC_START_SECTION(twofish_ecb_enc_8way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -328,9 +328,9 @@ SYM_FUNC_START(twofish_ecb_enc_8way) FRAME_END ret; -SYM_FUNC_END(twofish_ecb_enc_8way) +SYM_FUNC_END_SECTION(twofish_ecb_enc_8way) -SYM_FUNC_START(twofish_ecb_dec_8way) +SYM_FUNC_START_SECTION(twofish_ecb_dec_8way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -348,9 +348,9 @@ SYM_FUNC_START(twofish_ecb_dec_8way) FRAME_END ret; -SYM_FUNC_END(twofish_ecb_dec_8way) +SYM_FUNC_END_SECTION(twofish_ecb_dec_8way) -SYM_FUNC_START(twofish_cbc_dec_8way) +SYM_FUNC_START_SECTION(twofish_cbc_dec_8way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -373,4 +373,4 @@ SYM_FUNC_START(twofish_cbc_dec_8way) FRAME_END ret; -SYM_FUNC_END(twofish_cbc_dec_8way) +SYM_FUNC_END_SECTION(twofish_cbc_dec_8way)
arch/x86/crypto/twofish-i586-asm_32.S +4 -4
@@ -207,7 +207,7 @@ xor %esi, d ## D;\ ror $1, d ## D; -SYM_FUNC_START(twofish_enc_blk) +SYM_FUNC_START_SECTION(twofish_enc_blk) push %ebp /* save registers according to calling convention*/ push %ebx push %esi
@@ -261,9 +261,9 @@ SYM_FUNC_START(twofish_enc_blk) pop %ebp mov $1, %eax ret -SYM_FUNC_END(twofish_enc_blk) +SYM_FUNC_END_SECTION(twofish_enc_blk) -SYM_FUNC_START(twofish_dec_blk) +SYM_FUNC_START_SECTION(twofish_dec_blk) push %ebp /* save registers according to calling convention*/ push %ebx push %esi
@@ -318,4 +318,4 @@ SYM_FUNC_START(twofish_dec_blk) pop %ebp mov $1, %eax ret -SYM_FUNC_END(twofish_dec_blk) +SYM_FUNC_END_SECTION(twofish_dec_blk)
arch/x86/crypto/twofish-x86_64-asm_64-3way.S +4 -4
@@ -220,7 +220,7 @@ rorq $32, RAB2; \ outunpack3(mov, RIO, 2, RAB, 2); -SYM_FUNC_START(__twofish_enc_blk_3way) +SYM_FUNC_START_SECTION(__twofish_enc_blk_3way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -267,9 +267,9 @@ SYM_FUNC_START(__twofish_enc_blk_3way) popq %r12; popq %r13; ret; -SYM_FUNC_END(__twofish_enc_blk_3way) +SYM_FUNC_END_SECTION(__twofish_enc_blk_3way) -SYM_FUNC_START(twofish_dec_blk_3way) +SYM_FUNC_START_SECTION(twofish_dec_blk_3way) /* input: * %rdi: ctx, CTX * %rsi: dst
@@ -302,4 +302,4 @@ SYM_FUNC_START(twofish_dec_blk_3way) popq %r12; popq %r13; ret; -SYM_FUNC_END(twofish_dec_blk_3way) +SYM_FUNC_END_SECTION(twofish_dec_blk_3way)
arch/x86/crypto/twofish-x86_64-asm_64.S +4 -4
@@ -202,7 +202,7 @@ xor %r8d, d ## D;\ ror $1, d ## D; -SYM_FUNC_START(twofish_enc_blk) +SYM_FUNC_START_SECTION(twofish_enc_blk) pushq R1 /* %rdi contains the ctx address */
@@ -253,9 +253,9 @@ SYM_FUNC_START(twofish_enc_blk) popq R1 movl $1,%eax ret -SYM_FUNC_END(twofish_enc_blk) +SYM_FUNC_END_SECTION(twofish_enc_blk) -SYM_FUNC_START(twofish_dec_blk) +SYM_FUNC_START_SECTION(twofish_dec_blk) pushq R1 /* %rdi contains the ctx address */
@@ -305,4 +305,4 @@ SYM_FUNC_START(twofish_dec_blk) popq R1 movl $1,%eax ret -SYM_FUNC_END(twofish_dec_blk) +SYM_FUNC_END_SECTION(twofish_dec_blk)
arch/x86/entry/entry_32.S +8 -16
@@ -675,8 +675,7 @@ __irqentry_text_end: * %eax: prev task * %edx: next task */ -.pushsection .text, "ax" -SYM_CODE_START(__switch_to_asm) +SYM_CODE_START_SECTION(__switch_to_asm) /* * Save callee-saved registers * This must match the order in struct inactive_task_frame
@@ -721,8 +720,7 @@ SYM_CODE_START(__switch_to_asm) popl %ebp jmp __switch_to -SYM_CODE_END(__switch_to_asm) -.popsection +SYM_CODE_END_SECTION(__switch_to_asm) /* * The unwinder expects the last frame on the stack to always be at the same
@@ -731,8 +729,7 @@ SYM_CODE_END(__switch_to_asm) * asmlinkage function so its argument has to be pushed on the stack. This * wrapper creates a proper "end of stack" frame header before the call. */ -.pushsection .text, "ax" -SYM_FUNC_START(schedule_tail_wrapper) +SYM_FUNC_START_SECTION(schedule_tail_wrapper) FRAME_BEGIN pushl %eax
@@ -741,8 +738,7 @@ SYM_FUNC_START(schedule_tail_wrapper) FRAME_END ret -SYM_FUNC_END(schedule_tail_wrapper) -.popsection +SYM_FUNC_END_SECTION(schedule_tail_wrapper) /* * A newly forked process directly context switches into this address.
@@ -751,8 +747,7 @@ SYM_FUNC_END(schedule_tail_wrapper) * ebx: kernel thread func (NULL for user thread) * edi: kernel thread arg */ -.pushsection .text, "ax" -SYM_CODE_START(ret_from_fork) +SYM_CODE_START_SECTION(ret_from_fork) call schedule_tail_wrapper testl %ebx, %ebx
@@ -774,8 +769,7 @@ SYM_CODE_START(ret_from_fork) */ movl $0, PT_EAX(%esp) jmp 2b -SYM_CODE_END(ret_from_fork) -.popsection +SYM_CODE_END_SECTION(ret_from_fork) SYM_ENTRY(__begin_SYSENTER_singlestep_region, SYM_L_GLOBAL, SYM_A_NONE) /*
@@ -1247,8 +1241,7 @@ SYM_CODE_START(asm_exc_nmi) #endif SYM_CODE_END(asm_exc_nmi) -.pushsection .text, "ax" -SYM_CODE_START(rewind_stack_do_exit) +SYM_CODE_START_SECTION(rewind_stack_do_exit) /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp
@@ -1257,5 +1250,4 @@ SYM_CODE_START(rewind_stack_do_exit) call do_exit 1: jmp 1b -SYM_CODE_END(rewind_stack_do_exit) -.popsection +SYM_CODE_END_SECTION(rewind_stack_do_exit)
arch/x86/entry/entry_64.S +6 -12
@@ -221,8 +221,7 @@ SYM_CODE_END(entry_SYSCALL_64) * %rdi: prev task * %rsi: next task */ -.pushsection .text, "ax" -SYM_FUNC_START(__switch_to_asm) +SYM_FUNC_START_SECTION(__switch_to_asm) /* * Save callee-saved registers * This must match the order in inactive_task_frame
@@ -263,8 +262,7 @@ SYM_FUNC_START(__switch_to_asm) popq %rbp jmp __switch_to -SYM_FUNC_END(__switch_to_asm) -.popsection +SYM_FUNC_END_SECTION(__switch_to_asm) /* * A newly forked process directly context switches into this address.
@@ -273,8 +271,7 @@ SYM_FUNC_END(__switch_to_asm) * rbx: kernel thread func (NULL for user thread) * r12: kernel thread arg */ -.pushsection .text, "ax" -SYM_CODE_START(ret_from_fork) +SYM_CODE_START_SECTION(ret_from_fork) UNWIND_HINT_EMPTY movq %rax, %rdi call schedule_tail /* rdi: 'prev' task parameter */
@@ -300,8 +297,7 @@ SYM_CODE_START(ret_from_fork) */ movq $0, RAX(%rsp) jmp 2b -SYM_CODE_END(ret_from_fork) -.popsection +SYM_CODE_END_SECTION(ret_from_fork) .macro DEBUG_ENTRY_ASSERT_IRQS_OFF #ifdef CONFIG_DEBUG_ENTRY
@@ -1428,8 +1424,7 @@ SYM_CODE_START(ignore_sysret) SYM_CODE_END(ignore_sysret) #endif -.pushsection .text, "ax" -SYM_CODE_START(rewind_stack_do_exit) +SYM_CODE_START_SECTION(rewind_stack_do_exit) UNWIND_HINT_FUNC /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp
@@ -1439,5 +1434,4 @@ SYM_CODE_START(rewind_stack_do_exit) UNWIND_HINT_REGS call do_exit -SYM_CODE_END(rewind_stack_do_exit) -.popsection +SYM_CODE_END_SECTION(rewind_stack_do_exit)
arch/x86/entry/thunk_32.S +2 -2
@@ -10,7 +10,7 @@ /* put return address in eax (arg1) */ .macro THUNK name, func, put_ret_addr_in_eax=0 -SYM_CODE_START_NOALIGN(\name) +SYM_CODE_START_NOALIGN_SECTION(\name) pushl %eax pushl %ecx pushl %edx
@@ -26,7 +26,7 @@ SYM_CODE_START_NOALIGN(\name) popl %eax ret _ASM_NOKPROBE(\name) -SYM_CODE_END(\name) +SYM_CODE_END_SECTION(\name) .endm #ifdef CONFIG_PREEMPTION
arch/x86/entry/thunk_64.S +4 -4
@@ -11,7 +11,7 @@ /* rdi: arg1 ... normal C conventions. rax is saved/restored. */ .macro THUNK name, func -SYM_FUNC_START_NOALIGN(\name) +SYM_FUNC_START_NOALIGN_SECTION(\name) pushq %rbp movq %rsp, %rbp
@@ -27,7 +27,7 @@ SYM_FUNC_START_NOALIGN(\name) call \func jmp __thunk_restore -SYM_FUNC_END(\name) +SYM_FUNC_END_SECTION(\name) _ASM_NOKPROBE(\name) .endm
@@ -39,7 +39,7 @@ SYM_FUNC_END(\name) #endif #ifdef CONFIG_PREEMPTION -SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore) +SYM_CODE_START_LOCAL_NOALIGN_SECTION(__thunk_restore) popq %r11 popq %r10 popq %r9
@@ -52,5 +52,5 @@ SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore) popq %rbp ret _ASM_NOKPROBE(__thunk_restore) -SYM_CODE_END(__thunk_restore) +SYM_CODE_END_SECTION(__thunk_restore) #endif
arch/x86/include/asm/paravirt.h +1 -1
@@ -656,7 +656,7 @@ bool __raw_callee_save___native_vcpu_is_preempted(long cpu);#define PV_CALLEE_SAVE_REGS_THUNK (func) \ extern typeof(func) __raw_callee_save_##func; \ \ - asm(".pushsection .text;" \ + asm(ASM_PUSH_SECTION(__raw_callee_save_##func) ";" \ ".globl " PV_THUNK_NAME(func) ";" \ ".type " PV_THUNK_NAME(func) ", @function;" \ PV_THUNK_NAME(func) ":" \
arch/x86/include/asm/qspinlock_paravirt.h +1 -1
@@ -34,7 +34,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_spin_unlock_slowpath);* rsi = lockval (second argument) * rdx = internal variable (set to 0) */ -asm (".pushsection .text;" +asm (ASM_PUSH_SECTION(__raw_callee_save___pv_queued_spin_unlock) ";"".globl " PV_UNLOCK ";" ".type " PV_UNLOCK ", @function;" ".align 4,0x90;"
arch/x86/kernel/acpi/wakeup_32.S +4 -5
@@ -7,9 +7,9 @@ # Copyright 2003, 2008 Pavel Machek <[email protected] .code32 +SYM_CODE_START_SECTION(wakeup_pmode_return) ALIGN -SYM_CODE_START(wakeup_pmode_return) movw $__KERNEL_DS, %ax movw %ax, %ss movw %ax, %fs
@@ -42,9 +42,9 @@ SYM_CODE_END(wakeup_pmode_return) bogus_magic: jmp bogus_magic +SYM_TEXT_END_SECTION - - +SYM_TEXT_SECTION(do_suspend_lowlevel) save_registers: sidt saved_idt sldt saved_ldt
@@ -87,7 +87,7 @@ ret_point: call restore_registers call restore_processor_state ret -SYM_CODE_END(do_suspend_lowlevel) +SYM_CODE_END_SECTION(do_suspend_lowlevel) .data ALIGN
@@ -98,4 +98,3 @@ saved_eip: .long 0 saved_idt: .long 0,0 saved_ldt: .long 0 saved_tss: .long 0 -
arch/x86/kernel/acpi/wakeup_64.S +5 -5
@@ -16,7 +16,7 @@ /* * Hooray, we are in Long 64-bit mode (but still running in low memory) */ -SYM_FUNC_START(wakeup_long64) +SYM_FUNC_START_SECTION(wakeup_long64) movq saved_magic, %rax movq $0x123456789abcdef0, %rdx cmpq %rdx, %rax
@@ -28,7 +28,7 @@ SYM_FUNC_START(wakeup_long64) jmp 1b 2: movw $__KERNEL_DS, %ax - movw %ax, %ss + movw %ax, %ss movw %ax, %ds movw %ax, %es movw %ax, %fs
@@ -43,9 +43,9 @@ SYM_FUNC_START(wakeup_long64) movq saved_rip, %rax ANNOTATE_RETPOLINE_SAFE jmp *%rax -SYM_FUNC_END(wakeup_long64) +SYM_FUNC_END_SECTION(wakeup_long64) -SYM_FUNC_START(do_suspend_lowlevel) +SYM_FUNC_START_SECTION(do_suspend_lowlevel) FRAME_BEGIN subq $8, %rsp xorl %eax, %eax
@@ -128,7 +128,7 @@ SYM_FUNC_START(do_suspend_lowlevel) addq $8, %rsp FRAME_END jmp restore_processor_state -SYM_FUNC_END(do_suspend_lowlevel) +SYM_FUNC_END_SECTION(do_suspend_lowlevel) STACK_FRAME_NON_STANDARD do_suspend_lowlevel .data
arch/x86/kernel/ftrace_32.S +10 -9
@@ -18,13 +18,12 @@ # define MCOUNT_FRAME 0 /* using frame = false */ #endif -SYM_FUNC_START(__fentry__) +SYM_FUNC_START_SECTION(__fentry__) ret -SYM_FUNC_END(__fentry__) +SYM_FUNC_END_SECTION(__fentry__) EXPORT_SYMBOL(__fentry__) -SYM_CODE_START(ftrace_caller) - +SYM_CODE_START_SECTION(ftrace_caller) #ifdef CONFIG_FRAME_POINTER /* * Frame pointers are of ip followed by bp.
@@ -85,9 +84,9 @@ ftrace_graph_call: /* This is weak to keep gas from relaxing the jumps */ SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK) ret -SYM_CODE_END(ftrace_caller) +SYM_CODE_END_SECTION(ftrace_caller) -SYM_CODE_START(ftrace_regs_caller) +SYM_CODE_START_SECTION(ftrace_regs_caller) /* * We're here from an mcount/fentry CALL, and the stack frame looks like: *
@@ -161,10 +160,10 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) popl %eax jmp .Lftrace_ret -SYM_CODE_END(ftrace_regs_caller) +SYM_CODE_END_SECTION(ftrace_regs_caller) #ifdef CONFIG_FUNCTION_GRAPH_TRACER -SYM_CODE_START(ftrace_graph_caller) +SYM_CODE_START_SECTION(ftrace_graph_caller) pushl %eax pushl %ecx pushl %edx
@@ -178,8 +177,9 @@ SYM_CODE_START(ftrace_graph_caller) popl %ecx popl %eax ret -SYM_CODE_END(ftrace_graph_caller) +SYM_CODE_END_SECTION(ftrace_graph_caller) +SYM_TEXT_SECTION(return_to_handler) .globl return_to_handler return_to_handler: pushl %eax
@@ -190,4 +190,5 @@ return_to_handler: popl %edx popl %eax JMP_NOSPEC ecx +SYM_TEXT_END_SECTION #endif
arch/x86/kernel/ftrace_64.S +14 -14
@@ -131,12 +131,12 @@ #ifdef CONFIG_DYNAMIC_FTRACE -SYM_FUNC_START(__fentry__) +SYM_FUNC_START_SECTION(__fentry__) retq -SYM_FUNC_END(__fentry__) +SYM_FUNC_END_SECTION(__fentry__) EXPORT_SYMBOL(__fentry__) -SYM_FUNC_START(ftrace_caller) +SYM_FUNC_START_SECTION(ftrace_caller) /* save_mcount_regs fills in first two parameters */ save_mcount_regs
@@ -171,9 +171,9 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL) jmp ftrace_epilogue -SYM_FUNC_END(ftrace_caller); +SYM_FUNC_END_SECTION(ftrace_caller) -SYM_FUNC_START(ftrace_epilogue) +SYM_FUNC_START_SECTION(ftrace_epilogue) #ifdef CONFIG_FUNCTION_GRAPH_TRACER SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL) jmp ftrace_stub
@@ -186,9 +186,9 @@ SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL) SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK) UNWIND_HINT_FUNC retq -SYM_FUNC_END(ftrace_epilogue) +SYM_FUNC_END_SECTION(ftrace_epilogue) -SYM_FUNC_START(ftrace_regs_caller) +SYM_FUNC_START_SECTION(ftrace_regs_caller) /* Save the current flags before any operations that can change them */ pushfq
@@ -280,12 +280,12 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) UNWIND_HINT_FUNC jmp ftrace_epilogue -SYM_FUNC_END(ftrace_regs_caller) +SYM_FUNC_END_SECTION(ftrace_regs_caller) #else /* ! CONFIG_DYNAMIC_FTRACE */ -SYM_FUNC_START(__fentry__) +SYM_FUNC_START_SECTION(__fentry__) cmpq $ftrace_stub, ftrace_trace_function jnz trace
@@ -316,12 +316,12 @@ trace: restore_mcount_regs jmp fgraph_trace -SYM_FUNC_END(__fentry__) +SYM_FUNC_END_SECTION(__fentry__) EXPORT_SYMBOL(__fentry__) #endif /* CONFIG_DYNAMIC_FTRACE */ #ifdef CONFIG_FUNCTION_GRAPH_TRACER -SYM_FUNC_START(ftrace_graph_caller) +SYM_FUNC_START_SECTION(ftrace_graph_caller) /* Saves rbp into %rdx and fills first parameter */ save_mcount_regs
@@ -332,9 +332,9 @@ SYM_FUNC_START(ftrace_graph_caller) restore_mcount_regs retq -SYM_FUNC_END(ftrace_graph_caller) +SYM_FUNC_END_SECTION(ftrace_graph_caller) -SYM_FUNC_START(return_to_handler) +SYM_FUNC_START_SECTION(return_to_handler) subq $24, %rsp /* Save the return values */
@@ -349,5 +349,5 @@ SYM_FUNC_START(return_to_handler) movq (%rsp), %rax addq $24, %rsp JMP_NOSPEC rdi -SYM_FUNC_END(return_to_handler) +SYM_FUNC_END_SECTION(return_to_handler) #endif
arch/x86/kernel/irqflags.S +2 -2
@@ -7,9 +7,9 @@ /* * unsigned long native_save_fl(void) */ -SYM_FUNC_START(native_save_fl) +SYM_FUNC_START_SECTION(native_save_fl) pushf pop %_ASM_AX ret -SYM_FUNC_END(native_save_fl) +SYM_FUNC_END_SECTION(native_save_fl) EXPORT_SYMBOL(native_save_fl)
arch/x86/kernel/kprobes/core.c +2 -1
@@ -1018,7 +1018,7 @@ NOKPROBE_SYMBOL(kprobe_int3_handler);* calls trampoline_handler() runs, which calls the kretprobe's handler. */ asm( -".text\n" + ASM_PUSH_SECTION(kretprobe_trampoline) "\n"".global kretprobe_trampoline\n" ".type kretprobe_trampoline, @function\n" "kretprobe_trampoline:\n"
@@ -1046,6 +1046,7 @@ asm(#endif " ret\n" ".size kretprobe_trampoline, .-kretprobe_trampoline\n" +".popsection\n" );NOKPROBE_SYMBOL (kretprobe_trampoline);STACK_FRAME_NON_STANDARD (kretprobe_trampoline);
arch/x86/kernel/kvm.c +1 -1
@@ -911,7 +911,7 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);* restoring to/from the stack. */ asm( -".pushsection .text;" +ASM_PUSH_SECTION(__raw_callee_save___kvm_vcpu_is_preempted) ";"".global __raw_callee_save___kvm_vcpu_is_preempted;" ".type __raw_callee_save___kvm_vcpu_is_preempted, @function;" "__raw_callee_save___kvm_vcpu_is_preempted:"
arch/x86/kernel/relocate_kernel_32.S +2 -0
@@ -35,6 +35,7 @@ #define CP_PA_BACKUP_PAGES_MAP DATA(0x1c) .text +SYM_TEXT_SECTION(kexec_control_code) SYM_CODE_START_NOALIGN(relocate_kernel) /* Save the CPU context, used for jumping back */
@@ -276,3 +277,4 @@ SYM_CODE_END(swap_pages) .globl kexec_control_code_size .set kexec_control_code_size, . - relocate_kernel +SYM_TEXT_END_SECTION
arch/x86/kernel/relocate_kernel_64.S +2 -0
@@ -38,6 +38,7 @@ #define CP_PA_BACKUP_PAGES_MAP DATA(0x30) .text +SYM_TEXT_SECTION(kexec_control_code) .align PAGE_SIZE .code64 SYM_CODE_START_NOALIGN(relocate_kernel)
@@ -293,3 +294,4 @@ SYM_CODE_END(swap_pages) .globl kexec_control_code_size .set kexec_control_code_size, . - relocate_kernel +SYM_TEXT_END_SECTION
arch/x86/kvm/emulate.c +1 -1
@@ -323,7 +323,7 @@ static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop);#define FOP_START (op) \ extern void em_##op(struct fastop *fake); \ - asm(".pushsection .text, \"ax\" \n\t" \ + asm(ASM_PUSH_SECTION(em_##op) ", \"ax\" \n\t" \ ".global em_" #op " \n\t" \ ".align " __stringify(FASTOP_SIZE) " \n\t" \ "em_" #op ":\n\t"
arch/x86/kvm/vmx/vmenter.S +4 -4
@@ -246,7 +246,7 @@ SYM_FUNC_END(__vmx_vcpu_run) * Save and restore volatile registers across a call to vmread_error(). Note, * all parameters are passed on the stack. */ -SYM_FUNC_START(vmread_error_trampoline) +SYM_FUNC_START_SECTION(vmread_error_trampoline) push %_ASM_BP mov %_ASM_SP, %_ASM_BP
@@ -294,9 +294,9 @@ SYM_FUNC_START(vmread_error_trampoline) pop %_ASM_BP ret -SYM_FUNC_END(vmread_error_trampoline) +SYM_FUNC_END_SECTION(vmread_error_trampoline) -SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff) +SYM_FUNC_START_SECTION(vmx_do_interrupt_nmi_irqoff) /* * Unconditionally create a stack frame, getting the correct RSP on the * stack (for x86-64) would take two instructions anyways, and RBP can
@@ -327,4 +327,4 @@ SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff) mov %_ASM_BP, %_ASM_SP pop %_ASM_BP ret -SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff) +SYM_FUNC_END_SECTION(vmx_do_interrupt_nmi_irqoff)
arch/x86/lib/clear_page_64.S +6 -6
@@ -13,15 +13,15 @@ * Zero a page. * %rdi - page */ -SYM_FUNC_START(clear_page_rep) +SYM_FUNC_START_SECTION(clear_page_rep) movl $4096/8,%ecx xorl %eax,%eax rep stosq ret -SYM_FUNC_END(clear_page_rep) +SYM_FUNC_END_SECTION(clear_page_rep) EXPORT_SYMBOL_GPL(clear_page_rep) -SYM_FUNC_START(clear_page_orig) +SYM_FUNC_START_SECTION(clear_page_orig) xorl %eax,%eax movl $4096/64,%ecx .p2align 4
@@ -40,13 +40,13 @@ SYM_FUNC_START(clear_page_orig) jnz .Lloop nop ret -SYM_FUNC_END(clear_page_orig) +SYM_FUNC_END_SECTION(clear_page_orig) EXPORT_SYMBOL_GPL(clear_page_orig) -SYM_FUNC_START(clear_page_erms) +SYM_FUNC_START_SECTION(clear_page_erms) movl $4096,%ecx xorl %eax,%eax rep stosb ret -SYM_FUNC_END(clear_page_erms) +SYM_FUNC_END_SECTION(clear_page_erms) EXPORT_SYMBOL_GPL(clear_page_erms)
arch/x86/lib/cmpxchg16b_emu.S +2 -2
@@ -13,7 +13,7 @@ * %rcx : high 64 bits of new value * %al : Operation successful */ -SYM_FUNC_START(this_cpu_cmpxchg16b_emu) +SYM_FUNC_START_SECTION(this_cpu_cmpxchg16b_emu) # # Emulate 'cmpxchg16b %gs:(%rsi)' except we return the result in %al not
@@ -44,4 +44,4 @@ SYM_FUNC_START(this_cpu_cmpxchg16b_emu) xor %al,%al ret -SYM_FUNC_END(this_cpu_cmpxchg16b_emu) +SYM_FUNC_END_SECTION(this_cpu_cmpxchg16b_emu)
arch/x86/lib/copy_mc_64.S +4 -4
@@ -17,7 +17,7 @@ * boundary. The non-fragile version is equivalent to memcpy() * regardless of CPU machine-check-recovery capability. */ -SYM_FUNC_START(copy_mc_fragile) +SYM_FUNC_START_SECTION(copy_mc_fragile) cmpl $8, %edx /* Less than 8 bytes? Go to byte copy loop */ jb .L_no_whole_words
@@ -78,7 +78,7 @@ SYM_FUNC_START(copy_mc_fragile) xorl %eax, %eax .L_done: ret -SYM_FUNC_END(copy_mc_fragile) +SYM_FUNC_END_SECTION(copy_mc_fragile) .section .fixup, "ax" /*
@@ -125,7 +125,7 @@ SYM_FUNC_END(copy_mc_fragile) * machine check recovery support this version should be no slower than * standard memcpy. */ -SYM_FUNC_START(copy_mc_enhanced_fast_string) +SYM_FUNC_START_SECTION(copy_mc_enhanced_fast_string) movq %rdi, %rax movq %rdx, %rcx .L_copy:
@@ -133,7 +133,7 @@ SYM_FUNC_START(copy_mc_enhanced_fast_string) /* Copy successful. Return zero */ xorl %eax, %eax ret -SYM_FUNC_END(copy_mc_enhanced_fast_string) +SYM_FUNC_END_SECTION(copy_mc_enhanced_fast_string) .section .fixup, "ax" .E_copy:
arch/x86/lib/copy_page_64.S +5 -2
@@ -12,6 +12,8 @@ * copy unless the CPU indicates X86_FEATURE_REP_GOOD. Could vary the * prefetch distance based on SMP/UP. */ + +SYM_TEXT_SECTION(copy_page) ALIGN SYM_FUNC_START(copy_page) ALTERNATIVE "jmp copy_page_regs", "", X86_FEATURE_REP_GOOD
@@ -19,9 +21,10 @@ SYM_FUNC_START(copy_page) rep movsq ret SYM_FUNC_END(copy_page) +SYM_TEXT_END_SECTION EXPORT_SYMBOL(copy_page) -SYM_FUNC_START_LOCAL(copy_page_regs) +SYM_FUNC_START_LOCAL_SECTION(copy_page_regs) subq $2*8, %rsp movq %rbx, (%rsp) movq %r12, 1*8(%rsp)
@@ -86,4 +89,4 @@ SYM_FUNC_START_LOCAL(copy_page_regs) movq 1*8(%rsp), %r12 addq $2*8, %rsp ret -SYM_FUNC_END(copy_page_regs) +SYM_FUNC_END_SECTION(copy_page_regs)
arch/x86/lib/copy_user_64.S +10 -8
@@ -54,7 +54,7 @@ * Output: * eax uncopied bytes or 0 if successful. */ -SYM_FUNC_START(copy_user_generic_unrolled) +SYM_FUNC_START_SECTION(copy_user_generic_unrolled) ASM_STAC cmpl $8,%edx jb 20f /* less then 8 bytes, go to byte copy loop */
@@ -137,7 +137,7 @@ SYM_FUNC_START(copy_user_generic_unrolled) _ASM_EXTABLE_CPY(19b, 40b) _ASM_EXTABLE_CPY(21b, 50b) _ASM_EXTABLE_CPY(22b, 50b) -SYM_FUNC_END(copy_user_generic_unrolled) +SYM_FUNC_END_SECTION(copy_user_generic_unrolled) EXPORT_SYMBOL(copy_user_generic_unrolled) /* Some CPUs run faster using the string copy instructions.
@@ -158,7 +158,7 @@ EXPORT_SYMBOL(copy_user_generic_unrolled) * Output: * eax uncopied bytes or 0 if successful. */ -SYM_FUNC_START(copy_user_generic_string) +SYM_FUNC_START_SECTION(copy_user_generic_string) ASM_STAC cmpl $8,%edx jb 2f /* less than 8 bytes, go to byte copy loop */
@@ -183,7 +183,7 @@ SYM_FUNC_START(copy_user_generic_string) _ASM_EXTABLE_CPY(1b, 11b) _ASM_EXTABLE_CPY(3b, 12b) -SYM_FUNC_END(copy_user_generic_string) +SYM_FUNC_END_SECTION(copy_user_generic_string) EXPORT_SYMBOL(copy_user_generic_string) /*
@@ -198,7 +198,7 @@ EXPORT_SYMBOL(copy_user_generic_string) * Output: * eax uncopied bytes or 0 if successful. */ -SYM_FUNC_START(copy_user_enhanced_fast_string) +SYM_FUNC_START_SECTION(copy_user_enhanced_fast_string) ASM_STAC cmpl $64,%edx jb .L_copy_short_string /* less then 64 bytes, avoid the costly 'rep' */
@@ -215,7 +215,7 @@ SYM_FUNC_START(copy_user_enhanced_fast_string) .previous _ASM_EXTABLE_CPY(1b, 12b) -SYM_FUNC_END(copy_user_enhanced_fast_string) +SYM_FUNC_END_SECTION(copy_user_enhanced_fast_string) EXPORT_SYMBOL(copy_user_enhanced_fast_string) /*
@@ -232,6 +232,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) * Output: * eax uncopied bytes or 0 if successful. */ +SYM_TEXT_SECTION(copy_user_handle_tail) SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */
@@ -254,6 +255,7 @@ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail) +SYM_TEXT_END_SECTION /* * copy_user_nocache - Uncached memory copy with exception handling
@@ -264,7 +266,7 @@ SYM_CODE_END(.Lcopy_user_handle_tail) * - Require 8-byte alignment when size is 8 bytes or larger. * - Require 4-byte alignment when size is 4 bytes. */ -SYM_FUNC_START(__copy_user_nocache) +SYM_FUNC_START_SECTION(__copy_user_nocache) ASM_STAC /* If size is less than 8 bytes, go to 4-byte copy */
@@ -403,5 +405,5 @@ SYM_FUNC_START(__copy_user_nocache) _ASM_EXTABLE_CPY(31b, .L_fixup_4b_copy) _ASM_EXTABLE_CPY(40b, .L_fixup_1b_copy) _ASM_EXTABLE_CPY(41b, .L_fixup_1b_copy) -SYM_FUNC_END(__copy_user_nocache) +SYM_FUNC_END_SECTION(__copy_user_nocache) EXPORT_SYMBOL(__copy_user_nocache)
arch/x86/lib/csum-copy_64.S +2 -2
@@ -36,7 +36,7 @@ _ASM_EXTABLE_UA(20b, .Lfault) .endm -SYM_FUNC_START(csum_partial_copy_generic) +SYM_FUNC_START_SECTION(csum_partial_copy_generic) subq $5*8, %rsp movq %rbx, 0*8(%rsp) movq %r12, 1*8(%rsp)
@@ -253,4 +253,4 @@ SYM_FUNC_START(csum_partial_copy_generic) .Lfault: xorl %eax, %eax jmp .Lout -SYM_FUNC_END(csum_partial_copy_generic) +SYM_FUNC_END_SECTION(csum_partial_copy_generic)
arch/x86/lib/error-inject.c +2 -1
@@ -6,12 +6,13 @@ asmlinkage void just_return_func(void); asm( -".text\n" + ASM_PUSH_SECTION(just_return_func) "\n"".type just_return_func, @function\n" ".globl just_return_func\n" "just_return_func:\n" " ret\n" ".size just_return_func, .-just_return_func\n" +".popsection\n" );void override_function_with_return (struct pt_regs *regs)
arch/x86/lib/getuser.S +20 -17
@@ -47,7 +47,7 @@ #endif .text -SYM_FUNC_START(__get_user_1) +SYM_FUNC_START_SECTION(__get_user_1) LOAD_TASK_SIZE_MINUS_N(0) cmp %_ASM_DX,%_ASM_AX jae bad_get_user
@@ -58,10 +58,10 @@ SYM_FUNC_START(__get_user_1) xor %eax,%eax ASM_CLAC ret -SYM_FUNC_END(__get_user_1) +SYM_FUNC_END_SECTION(__get_user_1) EXPORT_SYMBOL(__get_user_1) -SYM_FUNC_START(__get_user_2) +SYM_FUNC_START_SECTION(__get_user_2) LOAD_TASK_SIZE_MINUS_N(1) cmp %_ASM_DX,%_ASM_AX jae bad_get_user
@@ -72,10 +72,10 @@ SYM_FUNC_START(__get_user_2) xor %eax,%eax ASM_CLAC ret -SYM_FUNC_END(__get_user_2) +SYM_FUNC_END_SECTION(__get_user_2) EXPORT_SYMBOL(__get_user_2) -SYM_FUNC_START(__get_user_4) +SYM_FUNC_START_SECTION(__get_user_4) LOAD_TASK_SIZE_MINUS_N(3) cmp %_ASM_DX,%_ASM_AX jae bad_get_user
@@ -86,10 +86,10 @@ SYM_FUNC_START(__get_user_4) xor %eax,%eax ASM_CLAC ret -SYM_FUNC_END(__get_user_4) +SYM_FUNC_END_SECTION(__get_user_4) EXPORT_SYMBOL(__get_user_4) -SYM_FUNC_START(__get_user_8) +SYM_FUNC_START_SECTION(__get_user_8) #ifdef CONFIG_X86_64 LOAD_TASK_SIZE_MINUS_N(7) cmp %_ASM_DX,%_ASM_AX
@@ -114,41 +114,41 @@ SYM_FUNC_START(__get_user_8) ASM_CLAC ret #endif -SYM_FUNC_END(__get_user_8) +SYM_FUNC_END_SECTION(__get_user_8) EXPORT_SYMBOL(__get_user_8) /* .. and the same for __get_user, just without the range checks */ -SYM_FUNC_START(__get_user_nocheck_1) +SYM_FUNC_START_SECTION(__get_user_nocheck_1) ASM_STAC ASM_BARRIER_NOSPEC 6: movzbl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC ret -SYM_FUNC_END(__get_user_nocheck_1) +SYM_FUNC_END_SECTION(__get_user_nocheck_1) EXPORT_SYMBOL(__get_user_nocheck_1) -SYM_FUNC_START(__get_user_nocheck_2) +SYM_FUNC_START_SECTION(__get_user_nocheck_2) ASM_STAC ASM_BARRIER_NOSPEC 7: movzwl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC ret -SYM_FUNC_END(__get_user_nocheck_2) +SYM_FUNC_END_SECTION(__get_user_nocheck_2) EXPORT_SYMBOL(__get_user_nocheck_2) -SYM_FUNC_START(__get_user_nocheck_4) +SYM_FUNC_START_SECTION(__get_user_nocheck_4) ASM_STAC ASM_BARRIER_NOSPEC 8: movl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC ret -SYM_FUNC_END(__get_user_nocheck_4) +SYM_FUNC_END_SECTION(__get_user_nocheck_4) EXPORT_SYMBOL(__get_user_nocheck_4) -SYM_FUNC_START(__get_user_nocheck_8) +SYM_FUNC_START_SECTION(__get_user_nocheck_8) ASM_STAC ASM_BARRIER_NOSPEC #ifdef CONFIG_X86_64
@@ -160,10 +160,10 @@ SYM_FUNC_START(__get_user_nocheck_8) xor %eax,%eax ASM_CLAC ret -SYM_FUNC_END(__get_user_nocheck_8) +SYM_FUNC_END_SECTION(__get_user_nocheck_8) EXPORT_SYMBOL(__get_user_nocheck_8) - +SYM_TEXT_SECTION(bad_get_user) SYM_CODE_START_LOCAL(.Lbad_get_user_clac) ASM_CLAC bad_get_user:
@@ -171,8 +171,10 @@ bad_get_user: mov $(-EFAULT),%_ASM_AX ret SYM_CODE_END(.Lbad_get_user_clac) +SYM_TEXT_END_SECTION #ifdef CONFIG_X86_32 +SYM_TEXT_SECTION(bad_get_user_8) SYM_CODE_START_LOCAL(.Lbad_get_user_8_clac) ASM_CLAC bad_get_user_8:
@@ -181,6 +183,7 @@ bad_get_user_8: mov $(-EFAULT),%_ASM_AX ret SYM_CODE_END(.Lbad_get_user_8_clac) +SYM_TEXT_END_SECTION #endif /* get_user */
arch/x86/lib/hweight.S +4 -5
@@ -8,8 +8,7 @@ * unsigned int __sw_hweight32(unsigned int w) * %rdi: w */ -SYM_FUNC_START(__sw_hweight32) - +SYM_FUNC_START_SECTION(__sw_hweight32) #ifdef CONFIG_X86_64 movl %edi, %eax # w #endif
@@ -33,10 +32,10 @@ SYM_FUNC_START(__sw_hweight32) shrl $24, %eax # w = w_tmp >> 24 __ASM_SIZE(pop,) %__ASM_REG(dx) ret -SYM_FUNC_END(__sw_hweight32) +SYM_FUNC_END_SECTION(__sw_hweight32) EXPORT_SYMBOL(__sw_hweight32) -SYM_FUNC_START(__sw_hweight64) +SYM_FUNC_START_SECTION(__sw_hweight64) #ifdef CONFIG_X86_64 pushq %rdi pushq %rdx
@@ -79,5 +78,5 @@ SYM_FUNC_START(__sw_hweight64) popl %ecx ret #endif -SYM_FUNC_END(__sw_hweight64) +SYM_FUNC_END_SECTION(__sw_hweight64) EXPORT_SYMBOL(__sw_hweight64)
arch/x86/lib/iomap_copy_64.S +2 -2
@@ -8,8 +8,8 @@ /* * override generic version in lib/iomap_copy.c */ -SYM_FUNC_START(__iowrite32_copy) +SYM_FUNC_START_SECTION(__iowrite32_copy) movl %edx,%ecx rep movsd ret -SYM_FUNC_END(__iowrite32_copy) +SYM_FUNC_END_SECTION(__iowrite32_copy)
arch/x86/lib/memmove_64.S +2 -2
@@ -24,7 +24,7 @@ * Output: * rax: dest */ -SYM_FUNC_START_WEAK(memmove) +SYM_FUNC_START_WEAK_SECTION(memmove) SYM_FUNC_START(__memmove) mov %rdi, %rax
@@ -207,6 +207,6 @@ SYM_FUNC_START(__memmove) 13: retq SYM_FUNC_END(__memmove) -SYM_FUNC_END_ALIAS(memmove) +SYM_FUNC_END_ALIAS_SECTION(memmove) EXPORT_SYMBOL(__memmove) EXPORT_SYMBOL(memmove)
arch/x86/lib/memset_64.S +6 -6
@@ -17,7 +17,7 @@ * * rax original destination */ -SYM_FUNC_START_WEAK(memset) +SYM_FUNC_START_WEAK_SECTION(memset) SYM_FUNC_START(__memset) /* * Some CPUs support enhanced REP MOVSB/STOSB feature. It is recommended
@@ -42,7 +42,7 @@ SYM_FUNC_START(__memset) movq %r9,%rax ret SYM_FUNC_END(__memset) -SYM_FUNC_END_ALIAS(memset) +SYM_FUNC_END_ALIAS_SECTION(memset) EXPORT_SYMBOL(memset) EXPORT_SYMBOL(__memset)
@@ -57,16 +57,16 @@ EXPORT_SYMBOL(__memset) * * rax original destination */ -SYM_FUNC_START_LOCAL(memset_erms) +SYM_FUNC_START_LOCAL_SECTION(memset_erms) movq %rdi,%r9 movb %sil,%al movq %rdx,%rcx rep stosb movq %r9,%rax ret -SYM_FUNC_END(memset_erms) +SYM_FUNC_END_SECTION(memset_erms) -SYM_FUNC_START_LOCAL(memset_orig) +SYM_FUNC_START_LOCAL_SECTION(memset_orig) movq %rdi,%r10 /* expand byte value */
@@ -137,4 +137,4 @@ SYM_FUNC_START_LOCAL(memset_orig) subq %r8,%rdx jmp .Lafter_bad_alignment .Lfinal: -SYM_FUNC_END(memset_orig) +SYM_FUNC_END_SECTION(memset_orig)
arch/x86/lib/msr-reg.S +4 -4
@@ -12,7 +12,7 @@ * */ .macro op_safe_regs op -SYM_FUNC_START(\op\()_safe_regs) +SYM_FUNC_START_SECTION(\op\()_safe_regs) pushq %rbx pushq %r12 movq %rdi, %r10 /* Save pointer */
@@ -41,13 +41,13 @@ SYM_FUNC_START(\op\()_safe_regs) jmp 2b _ASM_EXTABLE(1b, 3b) -SYM_FUNC_END(\op\()_safe_regs) +SYM_FUNC_END_SECTION(\op\()_safe_regs) .endm #else /* X86_32 */ .macro op_safe_regs op -SYM_FUNC_START(\op\()_safe_regs) +SYM_FUNC_START_SECTION(\op\()_safe_regs) pushl %ebx pushl %ebp pushl %esi
@@ -83,7 +83,7 @@ SYM_FUNC_START(\op\()_safe_regs) jmp 2b _ASM_EXTABLE(1b, 3b) -SYM_FUNC_END(\op\()_safe_regs) +SYM_FUNC_END_SECTION(\op\()_safe_regs) .endm #endif
arch/x86/lib/putuser.S +10 -8
@@ -43,7 +43,7 @@ #endif .text -SYM_FUNC_START(__put_user_1) +SYM_FUNC_START_SECTION(__put_user_1) LOAD_TASK_SIZE_MINUS_N(0) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user
@@ -53,11 +53,11 @@ SYM_INNER_LABEL(__put_user_nocheck_1, SYM_L_GLOBAL) xor %ecx,%ecx ASM_CLAC ret -SYM_FUNC_END(__put_user_1) +SYM_FUNC_END_SECTION(__put_user_1) EXPORT_SYMBOL(__put_user_1) EXPORT_SYMBOL(__put_user_nocheck_1) -SYM_FUNC_START(__put_user_2) +SYM_FUNC_START_SECTION(__put_user_2) LOAD_TASK_SIZE_MINUS_N(1) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user
@@ -67,11 +67,11 @@ SYM_INNER_LABEL(__put_user_nocheck_2, SYM_L_GLOBAL) xor %ecx,%ecx ASM_CLAC ret -SYM_FUNC_END(__put_user_2) +SYM_FUNC_END_SECTION(__put_user_2) EXPORT_SYMBOL(__put_user_2) EXPORT_SYMBOL(__put_user_nocheck_2) -SYM_FUNC_START(__put_user_4) +SYM_FUNC_START_SECTION(__put_user_4) LOAD_TASK_SIZE_MINUS_N(3) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user
@@ -81,11 +81,11 @@ SYM_INNER_LABEL(__put_user_nocheck_4, SYM_L_GLOBAL) xor %ecx,%ecx ASM_CLAC ret -SYM_FUNC_END(__put_user_4) +SYM_FUNC_END_SECTION(__put_user_4) EXPORT_SYMBOL(__put_user_4) EXPORT_SYMBOL(__put_user_nocheck_4) -SYM_FUNC_START(__put_user_8) +SYM_FUNC_START_SECTION(__put_user_8) LOAD_TASK_SIZE_MINUS_N(7) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user
@@ -98,16 +98,18 @@ SYM_INNER_LABEL(__put_user_nocheck_8, SYM_L_GLOBAL) xor %ecx,%ecx ASM_CLAC RET -SYM_FUNC_END(__put_user_8) +SYM_FUNC_END_SECTION(__put_user_8) EXPORT_SYMBOL(__put_user_8) EXPORT_SYMBOL(__put_user_nocheck_8) +SYM_TEXT_SECTION(bad_put_user) SYM_CODE_START_LOCAL(.Lbad_put_user_clac) ASM_CLAC .Lbad_put_user: movl $-EFAULT,%ecx RET SYM_CODE_END(.Lbad_put_user_clac) +SYM_TEXT_END_SECTION _ASM_EXTABLE_UA(1b, .Lbad_put_user_clac) _ASM_EXTABLE_UA(2b, .Lbad_put_user_clac)
arch/x86/mm/mem_encrypt_boot.S +4 -4
@@ -16,7 +16,7 @@ .text .code64 -SYM_FUNC_START(sme_encrypt_execute) +SYM_FUNC_START_SECTION(sme_encrypt_execute) /* * Entry parameters:
@@ -66,9 +66,9 @@ SYM_FUNC_START(sme_encrypt_execute) pop %rbp ret -SYM_FUNC_END(sme_encrypt_execute) +SYM_FUNC_END_SECTION(sme_encrypt_execute) -SYM_FUNC_START(__enc_copy) +SYM_FUNC_START_SECTION(__enc_copy) /* * Routine used to encrypt memory in place. * This routine must be run outside of the kernel proper since
@@ -153,4 +153,4 @@ SYM_FUNC_START(__enc_copy) ret .L__enc_copy_end: -SYM_FUNC_END(__enc_copy) +SYM_FUNC_END_SECTION(__enc_copy)
arch/x86/platform/efi/efi_stub_64.S +2 -2
@@ -10,7 +10,7 @@ #include <linux/linkage.h> #include <asm/nospec-branch.h> -SYM_FUNC_START(__efi_call) +SYM_FUNC_START_SECTION(__efi_call) pushq %rbp movq %rsp, %rbp and $~0xf, %rsp
@@ -24,4 +24,4 @@ SYM_FUNC_START(__efi_call) CALL_NOSPEC rdi leave ret -SYM_FUNC_END(__efi_call) +SYM_FUNC_END_SECTION(__efi_call)
arch/x86/platform/efi/efi_thunk_64.S +2 -2
@@ -25,7 +25,7 @@ .text .code64 -SYM_CODE_START(__efi64_thunk) +SYM_CODE_START_SECTION(__efi64_thunk) push %rbp push %rbx
@@ -69,7 +69,7 @@ SYM_CODE_START(__efi64_thunk) 2: pushl $__KERNEL_CS pushl %ebp lret -SYM_CODE_END(__efi64_thunk) +SYM_CODE_END_SECTION(__efi64_thunk) .bss .balign 8
arch/x86/power/hibernate_asm_32.S +8 -6
@@ -16,7 +16,7 @@ .text -SYM_FUNC_START(swsusp_arch_suspend) +SYM_FUNC_START_SECTION(swsusp_arch_suspend) movl %esp, saved_context_esp movl %ebx, saved_context_ebx movl %ebp, saved_context_ebp
@@ -33,9 +33,9 @@ SYM_FUNC_START(swsusp_arch_suspend) call swsusp_save FRAME_END ret -SYM_FUNC_END(swsusp_arch_suspend) +SYM_FUNC_END_SECTION(swsusp_arch_suspend) -SYM_CODE_START(restore_image) +SYM_CODE_START_SECTION(restore_image) /* prepare to jump to the image kernel */ movl restore_jump_address, %ebx movl restore_cr3, %ebp
@@ -45,10 +45,10 @@ SYM_CODE_START(restore_image) /* jump to relocated restore code */ movl relocated_restore_code, %eax jmpl *%eax -SYM_CODE_END(restore_image) +SYM_CODE_END_SECTION(restore_image) /* code below has been relocated to a safe page */ -SYM_CODE_START(core_restore_code) +SYM_CODE_START_SECTION(core_restore_code) movl temp_pgt, %eax movl %eax, %cr3
@@ -78,8 +78,9 @@ copy_loop: done: jmpl *%ebx -SYM_CODE_END(core_restore_code) +SYM_CODE_END_SECTION(core_restore_code) +SYM_TEXT_SECTION(restore_registers) /* code below belongs to the image kernel */ .align PAGE_SIZE SYM_FUNC_START(restore_registers)
@@ -110,3 +111,4 @@ SYM_FUNC_START(restore_registers) ret SYM_FUNC_END(restore_registers) +SYM_TEXT_END_SECTION
arch/x86/power/hibernate_asm_64.S +8 -6
@@ -23,6 +23,7 @@ #include <asm/frame.h> #include <asm/nospec-branch.h> +SYM_TEXT_SECTION(restore_registers) /* code below belongs to the image kernel */ .align PAGE_SIZE SYM_FUNC_START(restore_registers)
@@ -68,8 +69,9 @@ SYM_FUNC_START(restore_registers) ret SYM_FUNC_END(restore_registers) +SYM_TEXT_END_SECTION -SYM_FUNC_START(swsusp_arch_suspend) +SYM_FUNC_START_SECTION(swsusp_arch_suspend) movq $saved_context, %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax)
@@ -97,9 +99,9 @@ SYM_FUNC_START(swsusp_arch_suspend) call swsusp_save FRAME_END ret -SYM_FUNC_END(swsusp_arch_suspend) +SYM_FUNC_END_SECTION(swsusp_arch_suspend) -SYM_FUNC_START(restore_image) +SYM_FUNC_START_SECTION(restore_image) /* prepare to jump to the image kernel */
Lines of this file are not shown. View the raw message.
arch/x86/xen/xen-asm.S +24 -25
49 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.31.1
Part of the diff is not shown here. View the raw message.
PATCH v6 14/22 FG-KASLR: use a scripted approach to handle .text.* sections
In reply to Alexander LobakinInstead of relying on the linker and his heuristics about where to place (orphan) .text.* section, use a script to read vmlinux.o and generate a new .tmp_vmlinux.lds which will contain an entry for each of them. It relies on a magic marker inside the preprocessed vmlinux.lds (which is harmless in case FG-KASLR is disabled) and injects a list of input text sections there.
As a bonus, this approach allows us to configure FG-KASLR in terms of number of functions per each section. The zero value means one section per each functions, it is the strongest choice, but the resulting vmlinux also has the biggest size here, as well as the total number of sections and the boottime delay (which is still barely noticeable). The values of 4-8 are still strong enough and allows to save some space, and so on.
We also keep tracking the maximal alignment we found while traversing through the readelf output and the number of times we spotted it. It's actual only for values >= 64 and is required to reserve some space between the last .text.* section and the _etext marker.
The reason is that e.g. x86 has at least 3 ASM sections (4 with ClangCFI) aligned to 4096, and when mixing them with the small sections, we could go past the _etext and render the kernel unbootable. This reserved space ensures this won't happen.
--- arch/x86/kernel/vmlinux.lds.S | 4 +- include/asm-generic/vmlinux.lds.h | 6 ++ init/Kconfig | 14 +++ scripts/generate_text_sections.pl | 144 ++++++++++++++++++++++++++++++ scripts/link-vmlinux.sh | 25 +++++- 5 files changed, 191 insertions(+), 2 deletions(-) create mode 100755 scripts/generate_text_sections.pl
arch/x86/kernel/vmlinux.lds.S +3 -1
@@ -147,9 +147,11 @@ SECTIONS #endif } :text =0xcccc + TEXT_FG_KASLR + /* End of text section, which should occupy whole number of pages */ - _etext = .; . = ALIGN(PAGE_SIZE); + _etext = .; X86_ALIGN_RODATA_BEGIN RO_DATA(PAGE_SIZE)
include/asm-generic/vmlinux.lds.h +6 -0
@@ -127,6 +127,12 @@#define TEXT_MAIN .text#endif +/* +* Used by scripts/generate_text_sections.pl to inject text sections, +* harmless if FG-KASLR is disabled. +*/ +#define TEXT_FG_KASLR __fg_kaslr_magic = .; +/* * GCC 4.5 and later have a 32 bytes section alignment for structures. * Except GCC 4.9, that feels the need to align on 64 bytes.
init/Kconfig +14 -0
@@ -2033,6 +2033,20 @@ config FG_KASLR If unsure, say N. +config FG_KASLR_SHIFT + int "FG-KASLR granularity (number of functions per section shift)" + depends on FG_KASLR + range 0 16 + default 0 + help + This sets the number of functions that will be put in each section + as a power of two. + Decreasing the value increases the randomization, but also increases + the size of the final kernel/vmlinux due to the amount of sections. + 0 means that a separate section will be created for each function. + 16 almost disables the randomization, leaving only the manual + separation. + endmenu # General setup source "arch/Kconfig"
scripts/generate_text_sections.pl new file +144 -0
144 lines not on this page · raw message
scripts/link-vmlinux.sh +24 -1
@@ -66,6 +66,18 @@ gen_symversions() done } +# If CONFIG_FG_KASLR is selected, generate a linker script which will +# declare all custom text sections for future boottime shuffling +gen_text_sections() +{ + info GEN .tmp_vmlinux.lds + + ${PERL} ${srctree}/scripts/generate_text_sections.pl \ + -s "${CONFIG_FG_KASLR_SHIFT}" vmlinux.o \ + < "${objtree}/${KBUILD_LDS}" \ + > .tmp_vmlinux.lds +} + # Link of vmlinux.o used for section mismatch analysis # ${1} output file modpost_link()
@@ -155,12 +167,19 @@ vmlinux_link() local ld local ldflags local ldlibs + local lds info LD ${output} # skip output file argument shift + if [ -n "${CONFIG_FG_KASLR}" ]; then + lds=".tmp_vmlinux.lds" + else + lds="${objtree}/${KBUILD_LDS}" + fi + if [ -n "${CONFIG_LTO_CLANG}" ]; then # Use vmlinux.o instead of performing the slow LTO link again. objs=vmlinux.o
@@ -182,7 +201,7 @@ vmlinux_link() ldlibs= fi - ldflags="${ldflags} ${wl}--script=${objtree}/${KBUILD_LDS}" + ldflags="${ldflags} ${wl}--script=${lds}" # The kallsyms linking does not need debug symbols included. if [ "$output" != "${output#.tmp_vmlinux.kallsyms}" ] ; then
@@ -351,6 +370,10 @@ info GEN modules.builtin tr '\0' '\n' < modules.builtin.modinfo | sed -n 's/^[[:alnum:]:_]*\.file=//p' | tr ' ' '\n' | uniq | sed -e 's:^:kernel/:' -e 's/$/.ko/' > modules.builtin +if [ -n "${CONFIG_FG_KASLR}" ]; then + gen_text_sections +fi + btf_vmlinux_bin_o="" if [ -n "${CONFIG_DEBUG_INFO_BTF}" ]; then btf_vmlinux_bin_o=.btf.vmlinux.bin.o
Signature · 1 line
2.31.1
PATCH v6 15/22 kallsyms: Hide layout
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
This patch makes /proc/kallsyms display in a random order, rather than sorted by address in order to hide the newly randomized address layout.
--- kernel/kallsyms.c | 158 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 157 insertions(+), 1 deletion(-)
kernel/kallsyms.c +157 -1
@@ -563,6 +563,12 @@ struct kallsym_iter {int show_value; }; +struct kallsyms_shuffled_iter { +struct kallsym_iter iter; +loff_t total_syms; +loff_t shuffled_index[]; +}; +int __weak arch_get_kallsym (unsigned int symnum,unsigned long *value,char *type,char *name) {
@@ -810,7 +816,7 @@ bool kallsyms_show_value(const struct cred *cred) } } -static int kallsyms_open (struct inode *inode,struct file *file) +static int __kallsyms_open (struct inode *inode,struct file *file) {/* * We keep iterator in m->private, since normal case is to
@@ -831,6 +837,156 @@ static int kallsyms_open(struct inode *inode, struct file *file)return 0 ; } +/* +* When function granular kaslr is enabled, we need to print out the symbols +* at random so we don't reveal the new layout. +*/ +#ifdef CONFIG_FG_KASLR +static int update_random_pos (struct kallsyms_shuffled_iter *s_iter, +loff_t pos,loff_t *new_pos) +{ +loff_t new; + +if (pos >= s_iter->total_syms) +return 0 ; + + new = s_iter->shuffled_index[pos]; + +/* +* normally this would be done as part of update_iter, however, +* we want to avoid triggering this in the event that new is +* zero since we don't want to blow away our pos end indicators. +*/ +if (new ==0 ) { + s_iter->iter.name[0 ] ='\0' ; + s_iter->iter.nameoff =get_symbol_offset (new); + s_iter->iter.pos = new; + } + + *new_pos = new; +return 1 ; +} + +static void *shuffled_start (struct seq_file *m,loff_t *pos) +{ +struct kallsyms_shuffled_iter *s_iter = m->private; +loff_t new_pos; + +if (!update_random_pos (s_iter, *pos, &new_pos)) +return NULL ; + +return s_start (m, &new_pos); +} + +static void *shuffled_next (struct seq_file *m,void *p,loff_t *pos) +{ +struct kallsyms_shuffled_iter *s_iter = m->private; +loff_t new_pos; + + (*pos)++; + +if (!update_random_pos (s_iter, *pos, &new_pos)) +return NULL ; + +if (!update_iter (m->private, new_pos)) +return NULL ; + +return p; +} + +/* +* shuffle_index_list() +* Use a Fisher Yates algorithm to shuffle a list of text sections. +*/ +static void shuffle_index_list (loff_t *indexes,loff_t size) +{ +u32 i, j; + +for (i = size -1 ; i >0 ; i--) { +/* pick a random index from 0 to i */ + j =get_random_u32 () % (i +1 ); + +swap (indexes[i], indexes[j]); + } +} + +static const struct seq_operations kallsyms_shuffled_op = { + .start = shuffled_start, + .next = shuffled_next, + .stop = s_stop, + .show = s_show +}; + +static int kallsyms_random_open (struct inode *inode,struct file *file) +{ +loff_t pos; +struct kallsyms_shuffled_iter *shuffled_iter; +struct kallsym_iter iter; +bool show_value; + +/* +* If privileged, go ahead and use the normal algorithm for +* displaying symbols +*/ + show_value =kallsyms_show_value (file->f_cred); +if (show_value) +return __kallsyms_open (inode, file); + +/* +* we need to figure out how many extra symbols there are +* to print out past kallsyms_num_syms +*/ + pos = kallsyms_num_syms; +reset_iter (&iter,0 ); +do { +if (!update_iter (&iter, pos)) +break ; + pos++; + }while (1 ); + +/* +* add storage space for an array of loff_t equal to the size +* of the total number of symbols we need to print +*/ + shuffled_iter =__seq_open_private (file, &kallsyms_shuffled_op, +sizeof (*shuffled_iter) + + (sizeof (pos) * pos)); +if (!shuffled_iter) +return -ENOMEM ; + +reset_iter (&shuffled_iter->iter,0 ); + shuffled_iter->iter.show_value = show_value; + shuffled_iter->total_syms = pos; + +/* +* the existing update_iter algorithm requires that we +* are either moving along increasing pos sequentially, +* or that these values are correct. Since these values +* were discovered above, initialize our new iter so we +* can use update_iter non-sequentially. +*/ + shuffled_iter->iter.pos_arch_end = iter.pos_arch_end; + shuffled_iter->iter.pos_mod_end = iter.pos_mod_end; + shuffled_iter->iter.pos_ftrace_mod_end = iter.pos_ftrace_mod_end; + +/* +* initialize the array with all possible pos values, then +* shuffle the array so that the values will display in a random +* order. +*/ +for (pos =0 ; pos < shuffled_iter->total_syms; pos++) + shuffled_iter->shuffled_index[pos] = pos; + +shuffle_index_list (shuffled_iter->shuffled_index, shuffled_iter->total_syms); + +return 0 ; +} + +#define kallsyms_open kallsyms_random_open +#else +#define kallsyms_open __kallsyms_open +#endif /* !CONFIG_FG_KASLR */ +#ifdef CONFIG_KGDB_KDB const char *kdb_walk_kallsyms (loff_t *pos) {
Signature · 1 line
2.31.1
PATCH v6 16/22 livepatch: only match unique symbols when using fgkaslr
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
If any type of function granular randomization is enabled, the sympos algorithm will fail, as it will be impossible to resolve symbols when there are duplicates using the previous symbol position.
Override the value of sympos to always be zero if fgkaslr is enabled for either the core kernel or modules, forcing the algorithm
to require that only unique symbols are allowed to be patched.
--- kernel/livepatch/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
kernel/livepatch/core.c +11 -0
@@ -169,6 +169,17 @@ static int klp_find_object_symbol(const char *objname, const char *name,else kallsyms_on_each_symbol (klp_find_callback, &args); +/* +* If any type of function granular randomization is enabled, it +* will be impossible to resolve symbols when there are duplicates +* using the previous symbol position (i.e. sympos != 0). Override +* the value of sympos to always be zero in this case. This will +* force the algorithm to require that only unique symbols are +* allowed to be patched. +*/ +if (IS_ENABLED (CONFIG_FG_KASLR )) + sympos =0 ; +/* * Ensure an address was found. If sympos is 0, ensure symbol is unique; * otherwise ensure the symbol position count matches sympos.
Signature · 1 line
2.31.1
Miroslav Benes
maintainerRe: [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr
Hi,
Quoted 25 lines · Alexander Lobakin
From: Kristen Carlson Accardi <[email protected]>
If any type of function granular randomization is enabled, the sympos algorithm will fail, as it will be impossible to resolve symbols when there are duplicates using the previous symbol position.
Override the value of sympos to always be zero if fgkaslr is enabled for either the core kernel or modules, forcing the algorithm
to require that only unique symbols are allowed to be patched.Signed-off-by Kristen Carlson Accardi <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>--- kernel/livepatch/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/livepatch/core.c +1
-0@@ -169,6 +169,17 @@ static int klp_find_object_symbol(const char *objname, const char *name,else kallsyms_on_each_symbol (klp_find_callback, &args); +/*
kernel/livepatch/core.c static int klp_find_object_symbol(const char *objname, const char *name, · line 182
Quoted 10 lines
kernel/livepatch/core.c +10
-0+* If any type of function granular randomization is enabled, it +* will be impossible to resolve symbols when there are duplicates +* using the previous symbol position (i.e. sympos != 0). Override +* the value of sympos to always be zero in this case. This will +* force the algorithm to require that only unique symbols are +* allowed to be patched. +*/ +if (IS_ENABLED (CONFIG_FG_KASLR )) + sympos =0 ; +
I ran the live patching tests and no problem occurred, which is great. We do not have a test for old_sympos, which makes the testing less telling, but at least nothing blows up with the section randomization itself.
However, I want to reiterate what I wrote for the same patch in v5 series.
The above hunk should work, but I wonder if we should make it more explicit. With the change the user will get the error with "unresolvable ambiguity for symbol..." if they specify sympos and the symbol is not unique. It could confuse them.
So, how about it making it something like
if (IS_ENABLED(CONFIG_FG_KASLR) || IS_ENABLED(CONFIG_MODULE_FG_KASLR))
if (sympos) {
pr_err("fgkaslr is enabled, specifying sympos for symbol '%s' in object '%s' does not work.\n",
name, objname);
*addr = 0;
return -EINVAL;
}? (there could be goto to the error out at the end of the function to save copy-pasting).
In that case, if sympos is not specified, the user will get the message which matches the reality. If the user specifies it, they will get the error in case of fgkaslr (no matter if the symbol is found or not).
What do you think?
Miroslav
Re: [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr
From: Miroslav Benes <[email protected]>
Date: Thu, 9 Sep 2021 13:53:35 +0200 (CEST)
Quoted 1 line
Hi,
Hi!
Quoted 26 lines · Alexander Lobakin
Quoted 25 lines
From: Kristen Carlson Accardi <[email protected]>
If any type of function granular randomization is enabled, the sympos algorithm will fail, as it will be impossible to resolve symbols when there are duplicates using the previous symbol position.
Override the value of sympos to always be zero if fgkaslr is enabled for either the core kernel or modules, forcing the algorithm
to require that only unique symbols are allowed to be patched.Signed-off-by Kristen Carlson Accardi <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>--- kernel/livepatch/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/livepatch/core.c +0
-0@@ -169,6 +169,17 @@ static int klp_find_object_symbol(const char *objname, const char *name,else kallsyms_on_each_symbol (klp_find_callback, &args);
Quoted 15 lines
kernel/livepatch/core.c static int klp_find_object_symbol(const char *objname, const char *name, · line 182
Quoted 11 lines
kernel/livepatch/core.c +11
-0+/* +* If any type of function granular randomization is enabled, it +* will be impossible to resolve symbols when there are duplicates +* using the previous symbol position (i.e. sympos != 0). Override +* the value of sympos to always be zero in this case. This will +* force the algorithm to require that only unique symbols are +* allowed to be patched. +*/ +if (IS_ENABLED (CONFIG_FG_KASLR )) + sympos =0 ; +I ran the live patching tests and no problem occurred, which is great. We do not have a test for
old_sympos, which makes the testing less telling, but at least nothing blows up with the section randomization itself.
Great, thanks!
Quoted 18 lines
However, I want to reiterate what I wrote for the same patch in v5 series.
The above hunk should work, but I wonder if we should make it more explicit. With the change the user will get the error with "unresolvable ambiguity for symbol..." if they specify sympos and the symbol is not unique. It could confuse them.
So, how about it making it something like
if (
IS_ENABLED(CONFIG_FG_KASLR)||IS_ENABLED(CONFIG_MODULE_FG_KASLR))if (sympos) { pr_err("fgkaslr is enabled, specifying sympos for symbol '%s' in object '%s' does not work.\n", name, objname); *addr = 0; return -EINVAL; }
Quoted 6 lines
? (there could be goto to the error out at the end of the function to save copy-pasting).
In that case, if sympos is not specified, the user will get the message which matches the reality. If the user specifies it, they will get the error in case of fgkaslr (no matter if the symbol is found or not).
Not familiar with livepatching unfortunately, hope Kristen and/or Kees will comment on this. Looks fine for me anyways.
Quoted 3 lines
What do you think?
Miroslav
Thanks,
Al
PATCH v6 17/22 x86/boot: allow FG-KASLR to be selected
In reply to Alexander LobakinNow that we have full support of FG-KASLR from both kernel core and x86 code, allow FG-KASLR to be enabled for x86_64 if the "regular" KASLR is also turned on.
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -74,6 +74,7 @@ config X86 select ARCH_HAS_EARLY_DEBUG if KGDB select ARCH_HAS_ELF_RANDOMIZE select ARCH_HAS_FAST_MULTIPLIER + select ARCH_HAS_FG_KASLR if X86_64 && RANDOMIZE_BASE select ARCH_HAS_FILTER_PGPROT select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL
Signature · 1 line
2.31.1
PATCH v6 18/22 arm64/crypto: conditionally place ASM functions into separate sections
In reply to Alexander LobakinThe resulting LD script generated by FG-KASLR script contains a size assertion for the input .text function. In case if it's not empty, the build will stop plug a potentional layout leakage.
As FG-KASLR for modules tends to be arch-independent, we should take care of the modular ASM code of every architecture to not break the build.
This is the ARM64 part.
--- arch/arm64/crypto/aes-ce-ccm-core.S | 16 +++++------ arch/arm64/crypto/aes-ce-core.S | 16 +++++------ arch/arm64/crypto/aes-ce.S | 4 +-- arch/arm64/crypto/aes-cipher-core.S | 8 +++--- arch/arm64/crypto/aes-modes.S | 16 +++++------ arch/arm64/crypto/aes-neon.S | 4 +-- arch/arm64/crypto/aes-neonbs-core.S | 38 +++++++++++++-------------- arch/arm64/crypto/chacha-neon-core.S | 18 ++++++------- arch/arm64/crypto/crct10dif-ce-core.S | 14 +++++----- arch/arm64/crypto/ghash-ce-core.S | 24 ++++++++--------- arch/arm64/crypto/nh-neon-core.S | 4 +-- arch/arm64/crypto/poly1305-armv8.pl | 17 ++++++++++++ arch/arm64/crypto/sha1-ce-core.S | 4 +-- arch/arm64/crypto/sha2-ce-core.S | 4 +-- arch/arm64/crypto/sha3-ce-core.S | 4 +-- arch/arm64/crypto/sha512-armv8.pl | 11 ++++++++ arch/arm64/crypto/sha512-ce-core.S | 4 +-- arch/arm64/crypto/sm3-ce-core.S | 4 +-- arch/arm64/crypto/sm4-ce-core.S | 4 +-- 19 files changed, 121 insertions(+), 93 deletions(-)
arch/arm64/crypto/aes-ce-ccm-core.S +8 -8
@@ -15,7 +15,7 @@ * void ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes, * u32 *macp, u8 const rk[], u32 rounds); */ -SYM_FUNC_START(ce_aes_ccm_auth_data) +SYM_FUNC_START_SECTION(ce_aes_ccm_auth_data) ldr w8, [x3] /* leftover from prev round? */ ld1 {v0.16b}, [x0] /* load mac */ cbz w8, 1f
@@ -81,13 +81,13 @@ SYM_FUNC_START(ce_aes_ccm_auth_data) st1 {v0.16b}, [x0] 10: str w8, [x3] ret -SYM_FUNC_END(ce_aes_ccm_auth_data) +SYM_FUNC_END_SECTION(ce_aes_ccm_auth_data) /* * void ce_aes_ccm_final(u8 mac[], u8 const ctr[], u8 const rk[], * u32 rounds); */ -SYM_FUNC_START(ce_aes_ccm_final) +SYM_FUNC_START_SECTION(ce_aes_ccm_final) ld1 {v3.4s}, [x2], #16 /* load first round key */ ld1 {v0.16b}, [x0] /* load mac */ cmp w3, #12 /* which key size? */
@@ -121,7 +121,7 @@ SYM_FUNC_START(ce_aes_ccm_final) eor v0.16b, v0.16b, v1.16b /* en-/decrypt the mac */ st1 {v0.16b}, [x0] /* store result */ ret -SYM_FUNC_END(ce_aes_ccm_final) +SYM_FUNC_END_SECTION(ce_aes_ccm_final) .macro aes_ccm_do_crypt,enc ldr x8, [x6, #8] /* load lower ctr */
@@ -212,10 +212,10 @@ CPU_LE( rev x8, x8 ) * u8 const rk[], u32 rounds, u8 mac[], * u8 ctr[]); */ -SYM_FUNC_START(ce_aes_ccm_encrypt) +SYM_FUNC_START_SECTION(ce_aes_ccm_encrypt) aes_ccm_do_crypt 1 -SYM_FUNC_END(ce_aes_ccm_encrypt) +SYM_FUNC_END_SECTION(ce_aes_ccm_encrypt) -SYM_FUNC_START(ce_aes_ccm_decrypt) +SYM_FUNC_START_SECTION(ce_aes_ccm_decrypt) aes_ccm_do_crypt 0 -SYM_FUNC_END(ce_aes_ccm_decrypt) +SYM_FUNC_END_SECTION(ce_aes_ccm_decrypt)
arch/arm64/crypto/aes-ce-core.S +8 -8
@@ -8,7 +8,7 @@ .arch armv8-a+crypto -SYM_FUNC_START(__aes_ce_encrypt) +SYM_FUNC_START_SECTION(__aes_ce_encrypt) sub w3, w3, #2 ld1 {v0.16b}, [x2] ld1 {v1.4s}, [x0], #16
@@ -34,9 +34,9 @@ SYM_FUNC_START(__aes_ce_encrypt) eor v0.16b, v0.16b, v3.16b st1 {v0.16b}, [x1] ret -SYM_FUNC_END(__aes_ce_encrypt) +SYM_FUNC_END_SECTION(__aes_ce_encrypt) -SYM_FUNC_START(__aes_ce_decrypt) +SYM_FUNC_START_SECTION(__aes_ce_decrypt) sub w3, w3, #2 ld1 {v0.16b}, [x2] ld1 {v1.4s}, [x0], #16
@@ -62,23 +62,23 @@ SYM_FUNC_START(__aes_ce_decrypt) eor v0.16b, v0.16b, v3.16b st1 {v0.16b}, [x1] ret -SYM_FUNC_END(__aes_ce_decrypt) +SYM_FUNC_END_SECTION(__aes_ce_decrypt) /* * __aes_ce_sub() - use the aese instruction to perform the AES sbox * substitution on each byte in 'input' */ -SYM_FUNC_START(__aes_ce_sub) +SYM_FUNC_START_SECTION(__aes_ce_sub) dup v1.4s, w0 movi v0.16b, #0 aese v0.16b, v1.16b umov w0, v0.s[0] ret -SYM_FUNC_END(__aes_ce_sub) +SYM_FUNC_END_SECTION(__aes_ce_sub) -SYM_FUNC_START(__aes_ce_invert) +SYM_FUNC_START_SECTION(__aes_ce_invert) ld1 {v0.4s}, [x1] aesimc v1.16b, v0.16b st1 {v1.4s}, [x0] ret -SYM_FUNC_END(__aes_ce_invert) +SYM_FUNC_END_SECTION(__aes_ce_invert)
arch/arm64/crypto/aes-ce.S +2 -2
@@ -9,8 +9,8 @@ #include <linux/linkage.h> #include <asm/assembler.h> -#define AES_FUNC_START(func) SYM_FUNC_START(ce_ ## func) -#define AES_FUNC_END(func) SYM_FUNC_END(ce_ ## func) +#define AES_FUNC_START(func) SYM_FUNC_START_SECTION(ce_ ## func) +#define AES_FUNC_END(func) SYM_FUNC_END_SECTION(ce_ ## func) .arch armv8-a+crypto
arch/arm64/crypto/aes-cipher-core.S +4 -4
@@ -122,11 +122,11 @@ CPU_BE( rev w7, w7 ) ret .endm -SYM_FUNC_START(__aes_arm64_encrypt) +SYM_FUNC_START_SECTION(__aes_arm64_encrypt) do_crypt fround, crypto_ft_tab, crypto_ft_tab + 1, 2 -SYM_FUNC_END(__aes_arm64_encrypt) +SYM_FUNC_END_SECTION(__aes_arm64_encrypt) +SYM_FUNC_START_SECTION(__aes_arm64_decrypt) .align 5 -SYM_FUNC_START(__aes_arm64_decrypt) do_crypt iround, crypto_it_tab, crypto_aes_inv_sbox, 0 -SYM_FUNC_END(__aes_arm64_decrypt) +SYM_FUNC_END_SECTION(__aes_arm64_decrypt)
arch/arm64/crypto/aes-modes.S +8 -8
@@ -22,26 +22,26 @@ #define ST5(x...) x #endif -SYM_FUNC_START_LOCAL(aes_encrypt_block4x) +SYM_FUNC_START_LOCAL_SECTION(aes_encrypt_block4x) encrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 ret -SYM_FUNC_END(aes_encrypt_block4x) +SYM_FUNC_END_SECTION(aes_encrypt_block4x) -SYM_FUNC_START_LOCAL(aes_decrypt_block4x) +SYM_FUNC_START_LOCAL_SECTION(aes_decrypt_block4x) decrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 ret -SYM_FUNC_END(aes_decrypt_block4x) +SYM_FUNC_END_SECTION(aes_decrypt_block4x) #if MAX_STRIDE == 5 -SYM_FUNC_START_LOCAL(aes_encrypt_block5x) +SYM_FUNC_START_LOCAL_SECTION(aes_encrypt_block5x) encrypt_block5x v0, v1, v2, v3, v4, w3, x2, x8, w7 ret -SYM_FUNC_END(aes_encrypt_block5x) +SYM_FUNC_END_SECTION(aes_encrypt_block5x) -SYM_FUNC_START_LOCAL(aes_decrypt_block5x) +SYM_FUNC_START_LOCAL_SECTION(aes_decrypt_block5x) decrypt_block5x v0, v1, v2, v3, v4, w3, x2, x8, w7 ret -SYM_FUNC_END(aes_decrypt_block5x) +SYM_FUNC_END_SECTION(aes_decrypt_block5x) #endif /*
arch/arm64/crypto/aes-neon.S +2 -2
@@ -8,8 +8,8 @@ #include <linux/linkage.h> #include <asm/assembler.h> -#define AES_FUNC_START(func) SYM_FUNC_START(neon_ ## func) -#define AES_FUNC_END(func) SYM_FUNC_END(neon_ ## func) +#define AES_FUNC_START(func) SYM_FUNC_START_SECTION(neon_ ## func) +#define AES_FUNC_END(func) SYM_FUNC_END_SECTION(neon_ ## func) xtsmask .req v7 cbciv .req v7
arch/arm64/crypto/aes-neonbs-core.S +19 -19
@@ -380,7 +380,7 @@ ISRM0: .octa 0x0306090c00070a0d01040b0e0205080f /* * void aesbs_convert_key(u8 out[], u32 const rk[], int rounds) */ -SYM_FUNC_START(aesbs_convert_key) +SYM_FUNC_START_SECTION(aesbs_convert_key) ld1 {v7.4s}, [x1], #16 // load round 0 key ld1 {v17.4s}, [x1], #16 // load round 1 key
@@ -425,10 +425,10 @@ SYM_FUNC_START(aesbs_convert_key) eor v17.16b, v17.16b, v7.16b str q17, [x0] ret -SYM_FUNC_END(aesbs_convert_key) +SYM_FUNC_END_SECTION(aesbs_convert_key) +SYM_FUNC_START_LOCAL_SECTION(aesbs_encrypt8) .align 4 -SYM_FUNC_START_LOCAL(aesbs_encrypt8) ldr q9, [bskey], #16 // round 0 key ldr q8, M0SR ldr q24, SR
@@ -488,10 +488,10 @@ SYM_FUNC_START_LOCAL(aesbs_encrypt8) eor v2.16b, v2.16b, v12.16b eor v5.16b, v5.16b, v12.16b ret -SYM_FUNC_END(aesbs_encrypt8) +SYM_FUNC_END_SECTION(aesbs_encrypt8) +SYM_FUNC_START_LOCAL_SECTION(aesbs_decrypt8) .align 4 -SYM_FUNC_START_LOCAL(aesbs_decrypt8) lsl x9, rounds, #7 add bskey, bskey, x9
@@ -553,7 +553,7 @@ SYM_FUNC_START_LOCAL(aesbs_decrypt8) eor v3.16b, v3.16b, v12.16b eor v5.16b, v5.16b, v12.16b ret -SYM_FUNC_END(aesbs_decrypt8) +SYM_FUNC_END_SECTION(aesbs_decrypt8) /* * aesbs_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
@@ -619,13 +619,13 @@ SYM_FUNC_END(aesbs_decrypt8) ret .endm +SYM_FUNC_START_SECTION(aesbs_ecb_encrypt) .align 4 -SYM_FUNC_START(aesbs_ecb_encrypt) __ecb_crypt aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5 -SYM_FUNC_END(aesbs_ecb_encrypt) +SYM_FUNC_END_SECTION(aesbs_ecb_encrypt) +SYM_FUNC_START_SECTION(aesbs_ecb_decrypt) .align 4 -SYM_FUNC_START(aesbs_ecb_decrypt) __ecb_crypt aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5 SYM_FUNC_END(aesbs_ecb_decrypt)
@@ -633,8 +633,8 @@ SYM_FUNC_END(aesbs_ecb_decrypt) * aesbs_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, * int blocks, u8 iv[]) */ +SYM_FUNC_START_SECTION(aesbs_cbc_decrypt) .align 4 -SYM_FUNC_START(aesbs_cbc_decrypt) frame_push 6 mov x19, x0
@@ -718,7 +718,7 @@ SYM_FUNC_START(aesbs_cbc_decrypt) 2: frame_pop ret -SYM_FUNC_END(aesbs_cbc_decrypt) +SYM_FUNC_END_SECTION(aesbs_cbc_decrypt) .macro next_tweak, out, in, const, tmp sshr \tmp\().2d, \in\().2d, #63
@@ -734,7 +734,7 @@ SYM_FUNC_END(aesbs_cbc_decrypt) * aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, * int blocks, u8 iv[]) */ -SYM_FUNC_START_LOCAL(__xts_crypt8) +SYM_FUNC_START_LOCAL_SECTION(__xts_crypt8) mov x6, #1 lsl x6, x6, x23 subs w23, w23, #8
@@ -787,7 +787,7 @@ SYM_FUNC_START_LOCAL(__xts_crypt8) 0: mov bskey, x21 mov rounds, x22 br x16 -SYM_FUNC_END(__xts_crypt8) +SYM_FUNC_END_SECTION(__xts_crypt8) .macro __xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7 frame_push 6, 64
@@ -851,13 +851,13 @@ SYM_FUNC_END(__xts_crypt8) ret .endm -SYM_FUNC_START(aesbs_xts_encrypt) +SYM_FUNC_START_SECTION(aesbs_xts_encrypt) __xts_crypt aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5 -SYM_FUNC_END(aesbs_xts_encrypt) +SYM_FUNC_END_SECTION(aesbs_xts_encrypt) -SYM_FUNC_START(aesbs_xts_decrypt) +SYM_FUNC_START_SECTION(aesbs_xts_decrypt) __xts_crypt aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5 -SYM_FUNC_END(aesbs_xts_decrypt) +SYM_FUNC_END_SECTION(aesbs_xts_decrypt) .macro next_ctr, v mov \v\().d[1], x8
@@ -871,7 +871,7 @@ SYM_FUNC_END(aesbs_xts_decrypt) * aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[], * int rounds, int blocks, u8 iv[], u8 final[]) */ -SYM_FUNC_START(aesbs_ctr_encrypt) +SYM_FUNC_START_SECTION(aesbs_ctr_encrypt) frame_push 8 mov x19, x0
@@ -998,4 +998,4 @@ CPU_LE( rev x8, x8 ) 7: cbz x25, 8b st1 {v5.16b}, [x25] b 8b -SYM_FUNC_END(aesbs_ctr_encrypt) +SYM_FUNC_END_SECTION(aesbs_ctr_encrypt)
arch/arm64/crypto/chacha-neon-core.S +9 -9
@@ -23,7 +23,6 @@ #include <asm/cache.h> .text - .align 6 /* * chacha_permute - permute one block
@@ -36,7 +35,8 @@ * * Clobbers: w3, x10, v4, v12 */ -SYM_FUNC_START_LOCAL(chacha_permute) +SYM_FUNC_START_LOCAL_SECTION(chacha_permute) + .align 6 adr_l x10, ROT8 ld1 {v12.4s}, [x10]
@@ -104,9 +104,9 @@ SYM_FUNC_START_LOCAL(chacha_permute) b.ne .Ldoubleround ret -SYM_FUNC_END(chacha_permute) +SYM_FUNC_END_SECTION(chacha_permute) -SYM_FUNC_START(chacha_block_xor_neon) +SYM_FUNC_START_SECTION(chacha_block_xor_neon) // x0: Input state matrix, s // x1: 1 data block output, o // x2: 1 data block input, i
@@ -143,9 +143,9 @@ SYM_FUNC_START(chacha_block_xor_neon) ldp x29, x30, [sp], #16 ret -SYM_FUNC_END(chacha_block_xor_neon) +SYM_FUNC_END_SECTION(chacha_block_xor_neon) -SYM_FUNC_START(hchacha_block_neon) +SYM_FUNC_START_SECTION(hchacha_block_neon) // x0: Input state matrix, s // x1: output (8 32-bit words) // w2: nrounds
@@ -163,7 +163,7 @@ SYM_FUNC_START(hchacha_block_neon) ldp x29, x30, [sp], #16 ret -SYM_FUNC_END(hchacha_block_neon) +SYM_FUNC_END_SECTION(hchacha_block_neon) a0 .req w12 a1 .req w13
@@ -182,8 +182,8 @@ SYM_FUNC_END(hchacha_block_neon) a14 .req w27 a15 .req w28 +SYM_FUNC_START_SECTION(chacha_4block_xor_neon) .align 6 -SYM_FUNC_START(chacha_4block_xor_neon) frame_push 10 // x0: Input state matrix, s
@@ -790,7 +790,7 @@ CPU_BE( rev a15, a15 ) st1 {v28.16b-v31.16b}, [x7] // overlapping stores 3: st1 {v24.16b-v27.16b}, [x1] b .Lout -SYM_FUNC_END(chacha_4block_xor_neon) +SYM_FUNC_END_SECTION(chacha_4block_xor_neon) .section ".rodata", "a", %progbits .align L1_CACHE_SHIFT
arch/arm64/crypto/crct10dif-ce-core.S +7 -7
@@ -131,7 +131,7 @@ tbl bd4.16b, {\bd\().16b}, perm4.16b .endm -SYM_FUNC_START_LOCAL(__pmull_p8_core) +SYM_FUNC_START_LOCAL_SECTION(__pmull_p8_core) .L__pmull_p8_core: ext t4.8b, ad.8b, ad.8b, #1 // A1 ext t5.8b, ad.8b, ad.8b, #2 // A2
@@ -194,7 +194,7 @@ SYM_FUNC_START_LOCAL(__pmull_p8_core) eor t4.16b, t4.16b, t5.16b eor t6.16b, t6.16b, t3.16b ret -SYM_FUNC_END(__pmull_p8_core) +SYM_FUNC_END_SECTION(__pmull_p8_core) .macro __pmull_p8, rq, ad, bd, i .ifnc \bd, fold_consts
@@ -465,21 +465,21 @@ CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 ) // // Assumes len >= 16. // -SYM_FUNC_START(crc_t10dif_pmull_p8) +SYM_FUNC_START_SECTION(crc_t10dif_pmull_p8) stp x29, x30, [sp, #-16]! mov x29, sp crc_t10dif_pmull p8 -SYM_FUNC_END(crc_t10dif_pmull_p8) +SYM_FUNC_END_SECTION(crc_t10dif_pmull_p8) - .align 5 // // u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len); // // Assumes len >= 16. // -SYM_FUNC_START(crc_t10dif_pmull_p64) +SYM_FUNC_START_SECTION(crc_t10dif_pmull_p64) + .align 5 crc_t10dif_pmull p64 -SYM_FUNC_END(crc_t10dif_pmull_p64) +SYM_FUNC_END_SECTION(crc_t10dif_pmull_p64) .section ".rodata", "a" .align 4
arch/arm64/crypto/ghash-ce-core.S +12 -12
@@ -350,13 +350,13 @@ CPU_LE( rev64 T1.16b, T1.16b ) * void pmull_ghash_update(int blocks, u64 dg[], const char *src, * struct ghash_key const *k, const char *head) */ -SYM_FUNC_START(pmull_ghash_update_p64) +SYM_FUNC_START_SECTION(pmull_ghash_update_p64) __pmull_ghash p64 -SYM_FUNC_END(pmull_ghash_update_p64) +SYM_FUNC_END_SECTION(pmull_ghash_update_p64) -SYM_FUNC_START(pmull_ghash_update_p8) +SYM_FUNC_START_SECTION(pmull_ghash_update_p8) __pmull_ghash p8 -SYM_FUNC_END(pmull_ghash_update_p8) +SYM_FUNC_END_SECTION(pmull_ghash_update_p8) KS0 .req v8 KS1 .req v9
@@ -602,20 +602,20 @@ CPU_LE( rev w8, w8 ) * struct ghash_key const *k, u64 dg[], u8 ctr[], * int rounds, u8 tag) */ -SYM_FUNC_START(pmull_gcm_encrypt) +SYM_FUNC_START_SECTION(pmull_gcm_encrypt) pmull_gcm_do_crypt 1 -SYM_FUNC_END(pmull_gcm_encrypt) +SYM_FUNC_END_SECTION(pmull_gcm_encrypt) /* * void pmull_gcm_decrypt(int blocks, u8 dst[], const u8 src[], * struct ghash_key const *k, u64 dg[], u8 ctr[], * int rounds, u8 tag) */ -SYM_FUNC_START(pmull_gcm_decrypt) +SYM_FUNC_START_SECTION(pmull_gcm_decrypt) pmull_gcm_do_crypt 0 -SYM_FUNC_END(pmull_gcm_decrypt) +SYM_FUNC_END_SECTION(pmull_gcm_decrypt) -SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x) +SYM_FUNC_START_LOCAL_SECTION(pmull_gcm_ghash_4x) movi MASK.16b, #0xe1 shl MASK.2d, MASK.2d, #57
@@ -696,9 +696,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x) eor XL.16b, XL.16b, T2.16b ret -SYM_FUNC_END(pmull_gcm_ghash_4x) +SYM_FUNC_END_SECTION(pmull_gcm_ghash_4x) -SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x) +SYM_FUNC_START_LOCAL_SECTION(pmull_gcm_enc_4x) ld1 {KS0.16b}, [x5] // load upper counter sub w10, w8, #4 sub w11, w8, #3
@@ -761,7 +761,7 @@ SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x) eor INP3.16b, INP3.16b, KS3.16b ret -SYM_FUNC_END(pmull_gcm_enc_4x) +SYM_FUNC_END_SECTION(pmull_gcm_enc_4x) .section ".rodata", "a" .align 6
arch/arm64/crypto/nh-neon-core.S +2 -2
@@ -62,7 +62,7 @@ * * It's guaranteed that message_len % 16 == 0. */ -SYM_FUNC_START(nh_neon) +SYM_FUNC_START_SECTION(nh_neon) ld1 {K0.4s,K1.4s}, [KEY], #32 movi PASS0_SUMS.2d, #0
@@ -100,4 +100,4 @@ SYM_FUNC_START(nh_neon) addp T1.2d, PASS2_SUMS.2d, PASS3_SUMS.2d st1 {T0.16b,T1.16b}, [HASH] ret -SYM_FUNC_END(nh_neon) +SYM_FUNC_END_SECTION(nh_neon)
arch/arm64/crypto/poly1305-armv8.pl +17 -0
@@ -48,8 +48,12 @@ my ($h0,$h1,$h2,$r0,$r1,$s1,$t0,$t1,$d0,$d1,$d2) = map("x$_",(4..14)); $code.=<<___; #ifndef __KERNEL__ +# define SYM_TEXT_SECTION() +# define SYM_TEXT_END_SECTION # include "arm_arch.h" .extern OPENSSL_armcap_P +#else +# include <linux/linkage.h> #endif .text
@@ -58,6 +62,7 @@ $code.=<<___; .globl poly1305_blocks .globl poly1305_emit +SYM_TEXT_SECTION(poly1305_init) .globl poly1305_init .type poly1305_init,%function .align 5
@@ -107,7 +112,9 @@ poly1305_init: .Lno_key: ret .size poly1305_init,.-poly1305_init +SYM_TEXT_END_SECTION +SYM_TEXT_SECTION(poly1305_blocks) .type poly1305_blocks,%function .align 5 poly1305_blocks:
@@ -198,7 +205,9 @@ poly1305_blocks: .Lno_data: ret .size poly1305_blocks,.-poly1305_blocks +SYM_TEXT_END_SECTION +SYM_TEXT_SECTION(poly1305_emit) .type poly1305_emit,%function .align 5 poly1305_emit:
@@ -258,6 +267,7 @@ poly1305_emit: ret .size poly1305_emit,.-poly1305_emit +SYM_TEXT_END_SECTION ___ my ($R0,$R1,$S1,$R2,$S2,$R3,$S3,$R4,$S4) = map("v$_.4s",(0..8)); my ($IN01_0,$IN01_1,$IN01_2,$IN01_3,$IN01_4) = map("v$_.2s",(9..13));
@@ -270,6 +280,7 @@ my ($in2,$zeros)=("x16","x17"); my $is_base2_26 = $zeros; # borrow $code.=<<___; +SYM_TEXT_SECTION(poly1305_mult) .type poly1305_mult,%function .align 5 poly1305_mult:
@@ -306,7 +317,9 @@ poly1305_mult: ret .size poly1305_mult,.-poly1305_mult +SYM_TEXT_END_SECTION +SYM_TEXT_SECTION(poly1305_splat) .type poly1305_splat,%function .align 4 poly1305_splat:
@@ -333,7 +346,9 @@ poly1305_splat: ret .size poly1305_splat,.-poly1305_splat +SYM_TEXT_END_SECTION +SYM_TEXT_SECTION(poly1305_blocks_neon) #ifdef __KERNEL__ .globl poly1305_blocks_neon #endif
@@ -888,6 +903,8 @@ poly1305_blocks_neon: .align 5 .Lzeros: .long 0,0,0,0,0,0,0,0 +SYM_TEXT_END_SECTION + .asciz "Poly1305 for ARMv8, CRYPTOGAMS by \@dot-asm" .align 2 #if !defined(__KERNEL__) && !defined(_WIN64)
arch/arm64/crypto/sha1-ce-core.S +2 -2
@@ -65,7 +65,7 @@ * int sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src, * int blocks) */ -SYM_FUNC_START(sha1_ce_transform) +SYM_FUNC_START_SECTION(sha1_ce_transform) /* load round constants */ loadrc k0.4s, 0x5a827999, w6 loadrc k1.4s, 0x6ed9eba1, w6
@@ -147,4 +147,4 @@ CPU_LE( rev32 v11.16b, v11.16b ) str dgb, [x0, #16] mov w0, w2 ret -SYM_FUNC_END(sha1_ce_transform) +SYM_FUNC_END_SECTION(sha1_ce_transform)
arch/arm64/crypto/sha2-ce-core.S +2 -2
@@ -75,7 +75,7 @@ * int blocks) */ .text -SYM_FUNC_START(sha2_ce_transform) +SYM_FUNC_START_SECTION(sha2_ce_transform) /* load round constants */ adr_l x8, .Lsha2_rcon ld1 { v0.4s- v3.4s}, [x8], #64
@@ -154,4 +154,4 @@ CPU_LE( rev32 v19.16b, v19.16b ) 3: st1 {dgav.4s, dgbv.4s}, [x0] mov w0, w2 ret -SYM_FUNC_END(sha2_ce_transform) +SYM_FUNC_END_SECTION(sha2_ce_transform)
arch/arm64/crypto/sha3-ce-core.S +2 -2
@@ -40,7 +40,7 @@ * int sha3_ce_transform(u64 *st, const u8 *data, int blocks, int dg_size) */ .text -SYM_FUNC_START(sha3_ce_transform) +SYM_FUNC_START_SECTION(sha3_ce_transform) /* load state */ add x8, x0, #32 ld1 { v0.1d- v3.1d}, [x0]
@@ -197,7 +197,7 @@ SYM_FUNC_START(sha3_ce_transform) st1 {v24.1d}, [x0] mov w0, w2 ret -SYM_FUNC_END(sha3_ce_transform) +SYM_FUNC_END_SECTION(sha3_ce_transform) .section ".rodata", "a" .align 8
arch/arm64/crypto/sha512-armv8.pl +11 -0
@@ -195,11 +195,16 @@ ___ $code.=<<___; #ifndef __KERNEL__ # include "arm_arch.h" +# define SYM_TEXT_SECTION() +# define SYM_TEXT_END_SECTION +#else +# include <linux/linkage.h> #endif .text .extern OPENSSL_armcap_P +SYM_TEXT_SECTION($func) .globl $func .type $func,%function .align 6
@@ -285,7 +290,9 @@ $code.=<<___; ldp x29,x30,[sp],#128 ret .size $func,.-$func +SYM_TEXT_END_SECTION +SYM_TEXT_SECTION(K$BITS) .align 6 .type .LK$BITS,%object .LK$BITS:
@@ -354,6 +361,8 @@ $code.=<<___ if ($SZ==4); ___ $code.=<<___; .size .LK$BITS,.-.LK$BITS +SYM_TEXT_END_SECTION + #ifndef __KERNEL__ .align 3 .LOPENSSL_armcap_P:
@@ -637,6 +646,7 @@ sub body_00_15 () { } $code.=<<___; +SYM_TEXT_SECTION(sha256_block_neon) #ifdef __KERNEL__ .globl sha256_block_neon #endif
@@ -736,6 +746,7 @@ $code.=<<___; add sp,sp,#16*4+16 ret .size sha256_block_neon,.-sha256_block_neon +SYM_TEXT_END_SECTION ___ }
arch/arm64/crypto/sha512-ce-core.S +2 -2
@@ -106,7 +106,7 @@ * int blocks) */ .text -SYM_FUNC_START(sha512_ce_transform) +SYM_FUNC_START_SECTION(sha512_ce_transform) /* load state */ ld1 {v8.2d-v11.2d}, [x0]
@@ -203,4 +203,4 @@ CPU_LE( rev64 v19.16b, v19.16b ) 3: st1 {v8.2d-v11.2d}, [x0] mov w0, w2 ret -SYM_FUNC_END(sha512_ce_transform) +SYM_FUNC_END_SECTION(sha512_ce_transform)
arch/arm64/crypto/sm3-ce-core.S +2 -2
@@ -73,7 +73,7 @@ * int blocks) */ .text -SYM_FUNC_START(sm3_ce_transform) +SYM_FUNC_START_SECTION(sm3_ce_transform) /* load state */ ld1 {v8.4s-v9.4s}, [x0] rev64 v8.4s, v8.4s
@@ -131,7 +131,7 @@ CPU_LE( rev32 v3.16b, v3.16b ) ext v9.16b, v9.16b, v9.16b, #8 st1 {v8.4s-v9.4s}, [x0] ret -SYM_FUNC_END(sm3_ce_transform) +SYM_FUNC_END_SECTION(sm3_ce_transform) .section ".rodata", "a" .align 3
arch/arm64/crypto/sm4-ce-core.S +2 -2
@@ -15,7 +15,7 @@ * void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in); */ .text -SYM_FUNC_START(sm4_ce_do_crypt) +SYM_FUNC_START_SECTION(sm4_ce_do_crypt) ld1 {v8.4s}, [x2] ld1 {v0.4s-v3.4s}, [x0], #64 CPU_LE( rev32 v8.16b, v8.16b )
@@ -33,4 +33,4 @@ CPU_LE( rev32 v8.16b, v8.16b ) CPU_LE( rev32 v8.16b, v8.16b ) st1 {v8.4s}, [x1] ret -SYM_FUNC_END(sm4_ce_do_crypt) +SYM_FUNC_END_SECTION(sm4_ce_do_crypt)
Signature · 1 line
2.31.1
PATCH v6 19/22 module: Reorder functions
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
Introduce a new config option to allow modules to be re-ordered by function. This option can be enabled independently of the kernel text KASLR or FG_KASLR settings so that it can be used by architectures that do not support either of these features. This option will be selected by default if CONFIG_FG_KASLR is selected.
If a module has functions split out into separate text sections (i.e. compiled with the -ffunction-sections flag), reorder the functions to provide some code diversification to modules.
--- Makefile | 4 ++ init/Kconfig | 12 ++++++ kernel/kallsyms.c | 4 +- kernel/livepatch/core.c | 2 +- kernel/module.c | 91 ++++++++++++++++++++++++++++++++++++++++- 5 files changed, 108 insertions(+), 5 deletions(-)
Makefile +4 -0
@@ -921,6 +921,10 @@ endif # ClangLTO implies -ffunction-sections -fdata-sections, no need # to specify them manually and trigger a pointless full rebuild ifndef CONFIG_LTO_CLANG +ifdef CONFIG_MODULE_FG_KASLR +KBUILD_CFLAGS_MODULE += -ffunction-sections +endif + ifneq ($(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION)$(CONFIG_FG_KASLR),) KBUILD_CFLAGS_KERNEL += -ffunction-sections endif
init/Kconfig +12 -0
@@ -2330,6 +2330,18 @@ config UNUSED_KSYMS_WHITELIST one per line. The path can be absolute, or relative to the kernel source tree. +config MODULE_FG_KASLR + bool "Module Function Granular Layout Randomization" + default FG_KASLR + depends on BROKEN + help + This option randomizes the module text section by reordering the text + section by function at module load time. In order to use this + feature, the module must have been compiled with the + -ffunction-sections compiler flag. + + If unsure, say N. + endif # MODULES config MODULES_TREE_LOOKUP
kernel/kallsyms.c +2 -2
@@ -841,7 +841,7 @@ static int __kallsyms_open(struct inode *inode, struct file *file)* When function granular kaslr is enabled, we need to print out the symbols * at random so we don't reveal the new layout. */ -#ifdef CONFIG_FG_KASLR +#if defined(CONFIG_FG_KASLR ) || defined(CONFIG_MODULE_FG_KASLR )static int update_random_pos (struct kallsyms_shuffled_iter *s_iter,loff_t pos,loff_t *new_pos) {
@@ -985,7 +985,7 @@ static int kallsyms_random_open(struct inode *inode, struct file *file)#define kallsyms_open kallsyms_random_open#else #define kallsyms_open __kallsyms_open -#endif /* !CONFIG_FG_KASLR */ +#endif /* !CONFIG_FG_KASLR && !CONFIG_MODULE_FG_KASLR */ #ifdef CONFIG_KGDB_KDB const char *kdb_walk_kallsyms (loff_t *pos)
kernel/livepatch/core.c +1 -1
@@ -177,7 +177,7 @@ static int klp_find_object_symbol(const char *objname, const char *name,* force the algorithm to require that only unique symbols are * allowed to be patched. */ -if (IS_ENABLED (CONFIG_FG_KASLR )) +if (IS_ENABLED (CONFIG_FG_KASLR ) ||IS_ENABLED (CONFIG_MODULE_FG_KASLR )) sympos =0 ;/*
kernel/module.c +89 -2
121 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v6 20/22 module: use a scripted approach for FG-KASLR
In reply to Alexander LobakinUse the same methods and scripts to generate an LD script for every module containing all the output text sections.
The only difference there is that we don't need to reserve any space as the memory for every section is being allocated dynamically.
--- include/asm-generic/vmlinux.lds.h | 12 ++++++++++++ init/Kconfig | 15 ++++++++++++++- scripts/Makefile.modfinal | 19 ++++++++++++++++--- scripts/generate_text_sections.pl | 7 ++++++- scripts/module.lds.S | 14 +++++++++++++- 5 files changed, 61 insertions(+), 6 deletions(-)
include/asm-generic/vmlinux.lds.h +12 -0
@@ -127,6 +127,18 @@#define TEXT_MAIN .text#endif +/* +* Same for modules. However, LD_DEAD_CODE_DATA_ELIMINATION doesn't touch +* them, so no need to check for it here. +*/ +#if defined(CONFIG_LTO_CLANG ) && !defined(CONFIG_MODULE_FG_KASLR ) +#define TEXT_MAIN_MODULE .text .text.[0-9a-zA-Z_]* +#elif defined(CONFIG_MODULE_FG_KASLR ) +#define TEXT_MAIN_MODULE .text.__unused__ +#else +#define TEXT_MAIN_MODULE .text +#endif +/* * Used by scripts/generate_text_sections.pl to inject text sections, * harmless if FG-KASLR is disabled.
init/Kconfig +14 -1
@@ -2333,7 +2333,6 @@ config UNUSED_KSYMS_WHITELIST config MODULE_FG_KASLR bool "Module Function Granular Layout Randomization" default FG_KASLR - depends on BROKEN help This option randomizes the module text section by reordering the text section by function at module load time. In order to use this
@@ -2342,6 +2341,20 @@ config MODULE_FG_KASLR If unsure, say N. +config MODULE_FG_KASLR_SHIFT + int "Module FG-KASLR granularity (functions per section shift)" + depends on MODULE_FG_KASLR + range 0 16 + default 0 + help + This sets the number of functions that will be put in each section + as a power of two. + Decreasing the value increases the randomization, but also increases + the size of the final kernel module due to the amount of sections. + 0 means that a separate section will be created for each function. + 16 almost disables the randomization, leaving only the manual + separation. + endif # MODULES config MODULES_TREE_LOOKUP
scripts/Makefile.modfinal +16 -3
@@ -28,13 +28,24 @@ quiet_cmd_cc_o_c = CC [M] $@ %.mod.o: %.mod.c FORCE $(call if_changed_dep,cc_o_c) +ifdef CONFIG_MODULE_FG_KASLR +quiet_cmd_gen_modules_lds = GEN [M] $@ + cmd_gen_modules_lds = \ + $(PERL) $(srctree)/scripts/generate_text_sections.pl \ + -s $(CONFIG_MODULE_FG_KASLR_SHIFT) $(filter %.o, $^) \ + < $(filter %.lds, $^) > $@ + +%.lds: %$(mod-prelink-ext).o scripts/module.lds FORCE + $(call if_changed,gen_modules_lds) +endif + ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink) quiet_cmd_ld_ko_o = LD [M] $@ cmd_ld_ko_o += \ $(LD) -r $(KBUILD_LDFLAGS) \ $(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) \ - -T scripts/module.lds -o $@ $(filter %.o, $^); \ + -T $(filter %.lds, $^) -o $@ $(filter %.o, $^); \ $(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true) quiet_cmd_btf_ko = BTF [M] $@
@@ -55,13 +66,15 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \ # Re-generate module BTFs if either module's .ko or vmlinux changed -$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE +$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o +$(modules): %.ko: $(if $(CONFIG_MODULE_FG_KASLR),%.lds,scripts/module.lds) +$(modules): %.ko: $(if $(KBUILD_BUILTIN),vmlinux) FORCE +$(call if_changed_except,ld_ko_o,vmlinux) ifdef CONFIG_DEBUG_INFO_BTF_MODULES +$(if $(newer-prereqs),$(call cmd,btf_ko)) endif -targets += $(modules) $(modules:.ko=.mod.o) +targets += $(modules) $(modules:.ko=.mod.o) $(if $(CONFIG_MODULE_FG_KASLR),$(modules:.ko=.lds)) # Add FORCE to the prequisites of a target to force it to be always rebuilt. # ---------------------------------------------------------------------------
scripts/generate_text_sections.pl +6 -1
@@ -44,6 +44,7 @@ my $readelf = $ENV{'READELF'} || die "$0: ERROR: READELF not set?"; ## text sections array my @sections = (); +my $vmlinux = 0; ## max alignment found to reserve some space my $max_align = 64;
@@ -64,6 +65,10 @@ sub read_sections { next; } + if ($name eq ".sched.text") { + $vmlinux = 1; + } + if (!($name =~ /^\.text\.[0-9a-zA-Z_]*((\.constprop|\.isra|\.part)\.[0-9])*(|\.[0-9cfi]*)$/)) { next; }
@@ -120,7 +125,7 @@ sub print_reserve { ## we reserve some space for them to not overlap _etext while shuffling ## sections - if (!$count) { + if (!$vmlinux or !$count) { return; }
scripts/module.lds.S +13 -1
@@ -3,6 +3,11 @@ * Archs are free to supply their own linker scripts. ld will * combine them automatically. */ + +#include <asm-generic/vmlinux.lds.h> + +#undef SANITIZER_DISCARDS + #ifdef CONFIG_CFI_CLANG # include <asm/page.h> # define ALIGN_CFI ALIGN(PAGE_SIZE)
@@ -57,9 +62,16 @@ SECTIONS { */ .text : ALIGN_CFI { *(.text.__cfi_check) - *(.text .text.[0-9a-zA-Z_]* .text..L.cfi*) + *(TEXT_MAIN_MODULE) + *(.text..L.cfi.jumptable .text..L.cfi.jumptable.*) + } +#elif defined(CONFIG_MODULE_FG_KASLR) + .text : { + *(TEXT_MAIN_MODULE) } #endif + + TEXT_FG_KASLR } /* bring in arch-specific sections */
Signature · 1 line
2.31.1
PATCH v6 21/22 Documentation: add a documentation for FG-KASLR
In reply to Alexander LobakinFrom: Kristen Carlson Accardi <[email protected]>
Describe the main principles behind the FG-KASLR hardening feature in a new doc section.
--- .../admin-guide/kernel-parameters.txt | 6 + Documentation/security/fgkaslr.rst | 172 ++++++++++++++++++ Documentation/security/index.rst | 1 + 3 files changed, 179 insertions(+) create mode 100644 Documentation/security/fgkaslr.rst
Documentation/admin-guide/kernel-parameters.txt +6 -0
@@ -2211,6 +2211,12 @@ kernel and module base offset ASLR (Address Space Layout Randomization). + nofgkaslr [KNL] + When CONFIG_FG_KASLR is set, this parameter + disables kernel function granular ASLR + (Address Space Layout Randomization). + See Documentation/security/fgkaslr.rst. + kasan_multi_shot [KNL] Enforce KASAN (Kernel Address Sanitizer) to print report on every invalid memory access. Without this
Documentation/security/fgkaslr.rst new file +172 -0
172 lines not on this page · raw message
Documentation/security/index.rst +1 -0
@@ -7,6 +7,7 @@ Security Documentation credentials IMA-templates + fgkaslr keys/index lsm lsm-development
Signature · 1 line
2.31.1
PATCH v6 22/22 maintainers: add MAINTAINERS entry for FG-KASLR
In reply to Alexander LobakinAdd an entry for FG-KASLR containing the maintainers, reviewers, public mailing lists, files and so on.
--- MAINTAINERS | 12 ++++++++++++ 1 file changed, 12 insertions(+)
MAINTAINERS +12 -0
@@ -7637,6 +7637,18 @@ L: [email protected] S: Maintained F: drivers/platform/x86/fujitsu-tablet.c +FUNCTION-GRAINED KASLR (FG-KASLR) +M: Alexander Lobakin <[email protected]> +R: Kristen Carlson Accardi <[email protected]> +R: Kees Cook <[email protected]> +L: [email protected] +S: Supported +F: Documentation/security/fgkaslr.rst +F: arch/x86/boot/compressed/fgkaslr.c +F: arch/x86/boot/compressed/utils.c +F: arch/x86/boot/compressed/vmlinux.symbols +F: scripts/generate_text_sections.pl + FUSE: FILESYSTEM IN USERSPACE M: Miklos Szeredi <[email protected]> L: [email protected]
Signature · 1 line
2.31.1
Quoted 2 lines · Alexander Lobakin
This is a massive rework and a respin of Kristen Accardi's marvellous FG-KASLR series (v5).
Thanks for working on this! I know Marios has been looking at some of this as well. I think he tracked down a kretprobes bug and has a fixed prepared.
Quoted 7 lines
The major differences since v5 [0]:
* You can now tune the number of functions per each section to achieve the preferable vmlinux size or protection level. Default is still as one section per each function.
This can be handy for storage-constrained systems. 4-8 fps are still strong, but reduce the size of the final vmlinu{x,z} significantly;
Interesting, but I'm not sure what the size issue is. v5's on-disk image size issues were related to the large relocation table that was used during decompress and layout, but would get discarded. The final in-core image size was roughly the same size as a non-FGKASLR kernel (since functions were already aligned even without -ffunction-sections). How does the functions-per-section knob change image size?
Quoted 5 lines
* I don't use orphan sections anymore. It's not reliable at all / may differ from linker to linker, and also conflicts with
CONFIG_LD_ORPHAN_WARNwhich is great for catching random bugs ->
* All the .text.* sections are now being described explicitly in the linker script. A Perl script is used to take the original LDS, the
Quoted 8 lines
original object file, read a list of input sections from it and generate the resulting LDS.
This costs a bit of linking time as LD tends to think hard when processing scripts > 1 Mb. It adds about 40-60 seconds to the whole linking process (BTF step, 2-3 kallsyms steps and the final step), but "better safe than sorry".
In addition, that approach allows to reserve some space at the end and add some link assertions ->
Yeah, this "hope that orphan handling does it right" bugged me too, but my attempts to solve it looked much like yours: creating a linker file that named all the sections. I found this to be prohibitively expensive at link time (and that seems backed by your own measurements of an extra minute or so at link time). If that's still the result of using a generated linker file, we just need to depend on orphan handling. LD_ORPHAN_WARN will still exist for non-FGKASLR builds, so the benefits will continue to exist -- I think the correct solution is to have the linker grow a "pass through" special target like "DISCARD", which just maps given input section patterns into same-named output sections.
Quoted 4 lines
* Input .text section now must be empty, otherwise the linkage will be stopped. This is implemented by the size assertion in the resulting LD script and is designed to plug the potentional layout leakage. This also means that ->
I worry this will create unexpected problems for named sections that weren't originally being randomized with the v5 FGKASLR.
Quoted 6 lines
* "Regular" ASM functions are now being placed into unique separate functions the same way compiler does this for C functions. This is achieved by introducing and using several new macros which take the symbol name as a base for its new section name. This gives a better opportunity to both DCE and FG-KASLR, as ASM code now can also be randomized or garbage-collected;
This is interesting! I think it'd be a good evolutionary step on top of "basic FGKASLR".
Quoted 3 lines
* It's now fully compatible with ClangLTO, ClangCFI,
CONFIG_LD_ORPHAN_WARNand some more stuff landed since the last revision was published;
FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it work with ClangLTO and ClangCFI.
Quoted 2 lines
* Includes several fixes: relocations inside .altinstr_replacement code and minor issues found and/or suggested by LKP robot.
Excellent!
Quoted 4 lines
The series was compile-time and runtime tested on the following setups with no issues:
-x86_64, GCC 11, Binutils 2.35;
-x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).
Great, this is a good start. One place we saw problems in the past was with i386 build gotchas, so that'll need testing too.
Quoted 2 lines
The first 4 patches are from the linux-kbuild tree and included to avoid merge conflicts and non-intuitive resolving of them.
Sounds good. It might be easier to base the series on linux-next, so a smaller series. Though given the merge window just opened, it might make more sense for a v7 to be based on v5.15-rc2 in three weeks.
Quoted 6 lines
The series is also available here: [1]
[0] https://lore.kernel.org/kernel-hardening/[email protected]
[1] https://github.com/alobakin/linux/pull/3The original v5 cover letter:
More notes below...
Quoted 278 lines
Function Granular Kernel Address Space Layout Randomization (fgkaslr)
---------------------------------------------------------------------This patch set is an implementation of finer grained kernel address space randomization. It rearranges your kernel code at load time
on a per-function level granularity, with only around a second added to boot time.Changes in v5:
--------------
* fixed a bug in the code which increases boot heap size forCONFIG_FG_KASLRwhich prevented the boot heap from being increased forCONFIG_FG_KASLRwhen using bzip2 compression. Thanks to Andy Lavr for finding the problem and identifying the solution.
* changed the adjustment of theorc_unwind_iptable at boot time to disregard relocs associated with this table, and instead inspect the entries separately. Relocs are not able to be used since they are no longer correct once the table is resorted at buildtime.
* changed howorc_unwind_ipaddresses in randomized sections are identified to include the byte immediately after the end of the section.
* updated module code to use kvmalloc/kvfree based on suggestions from Evgenii Shatokhin <[email protected]>.
* changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to match the nokaslr option. fgkaslr="X" can be added at a later date if it is needed.
* Added a patch to force livepatch to require symbols to be unique if using while fgkaslr either for core or modules.Changes in v4:
-------------
* dropped the patch to split out change to STATIC definition inx86/boot/compressed/misc.cand replaced with a patch authored by Kees Cook to avoid the duplicate malloc definitions
* Added a section toDocumentation/admin-guide/kernel-parameters.txtto document the fgkaslr boot option.
* redesigned the patch to hide the new layout when reading/proc/kallsyms. The previous implementation utilized a dynamically allocated linked list to display the kernel and module symbols in alphabetical order. The new implementation uses a randomly shuffled index array to display the kernel and module symbols in a random order.Changes in v3:
-------------
*Makefilechanges to accommodateCONFIG_LD_DEAD_CODE_DATA_ELIMINATION
* removal of extraneousALIGN_PAGEfrom_etextchanges
* changed variable names in x86/tools/relocs to be less confusing
* split out change to STATIC definition inx86/boot/compressed/misc.c
* Updates to Documentation to make it more clear what is preserved in .text
* much more detailed commit message for function granular KASLR patch
* minor tweaks and changes that make for more readable code
* this cover letter updated slightly to add additional detailsChanges in v2:
--------------
* Fix to address i386 build failure
* Allow module reordering patch to be configured separately so that arm (or other non-x86_64 arches) can take advantage of module function reordering. This support has not be tested by me, but smoke tested by Ard Biesheuvel <[email protected]> on arm.
* Fix build issue when building on arm as reported by
Ard Biesheuvel <[email protected]>Patches to objtool are included because they are dependencies for this patchset, however they have been submitted by their maintainer separately.
Background
----------
KASLR was merged into the kernel with the objective of increasing the difficulty of code reuse attacks. Code reuse attacks reused existing code snippets to get around existing memory protections. They exploit software bugs which expose addresses of useful code snippets to control the flow of execution for their own nefarious purposes. KASLR moves the entire kernel code text as a unit at boot time in order to make addresses less predictable. The order of the code within the segment is unchanged - only the base address is shifted. There are a few shortcomings to this algorithm.1. Low Entropy - there are only so many locations the kernel can fit in. This means an attacker could guess without too much trouble.
2. Knowledge of a single address can reveal the offset of the base address, exposing all other locations for a published/known kernel image.
3. Info leaks abound.Finer grained ASLR has been proposed as a way to make ASLR more resistant to info leaks. It is not a new concept at all, and there are many variations possible. Function reordering is an implementation of finer grained ASLR which randomizes the layout of an address space on a function level granularity. We use the term "fgkaslr" in this document to refer to the technique of function reordering when used with KASLR, as well as finer grained KASLR in general.
Proposed Improvement
--------------------
This patch set proposes adding function reordering on top of the existing KASLR base address randomization. The over-arching objective is incremental improvement over what we already have. It is designed to work in combination with the existing solution. The implementation is really pretty simple, and there are 2 main area where changes occur:* Build time
GCC has had an option to place functions into individual .text sections for many years now. This option can be used to implement function reordering at load time. The final compiled vmlinux retains all the section headers, which can be used to help find the address ranges of each function. Using this information and an expanded table of relocation addresses, individual text sections can be suffled immediately after decompression. Some data tables inside the kernel that have assumptions about order require re-sorting after being updated when applying relocations. In order to modify these tables, a few key symbols are excluded from the objcopy symbol stripping process for use after shuffling the text segments.
Some highlights from the build time changes to look for:
The top level kernel
Makefilewas modified to add the gcc flag if it is supported. Currently, I am applying this flag to everything it is possible to randomize. Anything that is written in C and not present in a special input section is randomized. The final binary segment 0 retains a consolidated .text section, as well as all the individual .text.* sections. Future work could turn off this flags for selected files or even entire subsystems, although obviously at the cost of security.The relocs tool is updated to add relative relocations. This information previously wasn't included because it wasn't necessary when moving the entire .text segment as a unit.
A new file was created to contain a list of symbols that objcopy should keep. We use those symbols at load time as described below.
* Load time
The boot kernel was modified to parse the vmlinux elf file after decompression to check for our interesting symbols that we kept, and to look for any .text.* sections to randomize. The consolidated .text section is skipped and not moved. The sections are shuffled randomly, and copied into memory following the .text section in a new random order. The existing code which updated relocation addresses was modified to account for not just a fixed delta from the load address, but the offset that the function section was moved to. This requires inspection of each address to see if it was impacted by a randomization. We use a bsearch to make this less horrible on performance. Any tables that need to be modified with new addresses or resorted are updated using the symbol addresses parsed from the elf symbol table.
In order to hide our new layout, symbols reported through
/proc/kallsymswill be displayed in a random order.Security Considerations
-----------------------
The objective of this patch set is to improve a technology that is already merged into the kernel (KASLR). This code will not prevent all attacks, but should instead be considered as one of several tools that can be used. In particular, this code is meant to make KASLR more effective in the presence of info leaks.How much entropy we are adding to the existing entropy of standard KASLR will depend on a few variables. Firstly and most obviously, the number of functions that are randomized matters. This implementation keeps the existing .text section for code that cannot be randomized - for example, because it was assembly code. The less sections to randomize, the less entropy. In addition, due to alignment (16 bytes for
x86_64), the number of bits in a address that the attacker needs to guess is reduced, as the lower bits are identical.Performance Impact
------------------
There are two areas where function reordering can impact performance: boot time latency, and run time performance.* Boot time latency
This implementation of finer grained KASLR impacts the boot time of the kernel in several places. It requires additional parsing of the kernel ELF file to obtain the section headers of the sections to be randomized. It calls the random number generator for each section to be randomized to determine that section's new memory location. It copies the decompressed kernel into a new area of memory to avoid corruption when laying out the newly randomized sections. It increases the number of relocations the kernel has to perform at boot time vs. standard KASLR, and it also requires a lookup on each address that needs to be relocated to see if it was in a randomized section and needs to be adjusted by a new offset. Finally, it re-sorts a few data tables that are required to be sorted by address.Booting a test VM on a modern, well appointed system showed an increase in latency of approximately 1 second.
* Run time
The performance impact at run-time of function reordering varies by workload. Using kcbench, a kernel compilation benchmark, the performance of a kernel build with finer grained KASLR was about 1% slower than a kernel with standard KASLR. Analysis with perf showed a slightly higher percentage of L1-icache-load-misses. Other workloads were examined as well, with varied results. Some workloads performed significantly worse under FGKASLR, while others stayed the same or were mysteriously better. In general, it will depend on the code flow whether or not finer grained KASLR will impact your workload, and how the underlying code was designed. Because the layout changes per boot, each time a system is rebooted the performance of a workload may change.Future work could identify hot areas that may not be randomized and either leave them in the .text section or group them together into a single section that may be randomized. If grouping things together helps, one other thing to consider is that if we could identify text blobs that should be grouped together to benefit a particular code flow, it could be interesting to explore whether this security feature could be also be used as a performance feature if you are interested in optimizing your kernel layout for a particular workload at boot time. Optimizing function layout for a particular workload has been researched and proven effective - for more information read the Facebook paper "Optimizing Function Placement for Large-Scale Data-Center Applications" (see references section below).
Image Size
----------
Adding additional section headers as a result of compiling with -ffunction-sections will increase the size of the vmlinux ELF file. With a standard distro config, the resulting vmlinux was increased by about 3%. The compressed image is also increased due to the header files, as well as the extra relocations that must be added. You can expect fgkaslr to increase the size of the compressed image by about 15%.Memory Usage
------------
fgkaslr increases the amount of heap that is required at boot time, although this extra memory is released when the kernel has finished decompression. As a result, it may not be appropriate to use this feature on systems without much memory.Building
--------
To enable fine grained KASLR, you need to have the following config options set (including all the ones you would use to build normal KASLR)
CONFIG_FG_KASLR=yIn addition, fgkaslr is only supported for the
X86_64architecture.Modules
-------
Modules are randomized similarly to the rest of the kernel by shuffling the sections at load time prior to moving them into memory. The module must also have been build with the -ffunction-sections compiler option.Although fgkaslr for the kernel is only supported for the
X86_64architecture, it is possible to use fgkaslr with modules on other architectures. To enable this feature, select
CONFIG_MODULE_FG_KASLR=yThis option is selected automatically for
X86_64whenCONFIG_FG_KASLRis set.Disabling
---------
Disabling normal KASLR using the nokaslr command line option also disables fgkaslr. It is also possible to disable fgkaslr separately by booting with nofgkaslr on the commandline.References
----------
There are a lot of academic papers which explore finer grained ASLR. This paper in particular contributed the most to my implementation design as well as my overall understanding of the problem space:Selfrando: Securing the Tor Browser against De-anonymization Exploits, M. Conti, S. Crane, T. Frassetto, et al.
For more information on how function layout impacts performance, see:
Optimizing Function Placement for Large-Scale Data-Center Applications, G. Ottoni, B. Maher
Alexander Lobakin (7):
linkage: add macros for putting ASM functions into own sections
x86: conditionally place regular ASM functions into separate sections
FG-KASLR: use a scripted approach to handle .text.* sections x86/boot: allow FG-KASLR to be selected
arm64/crypto: conditionally place ASM functions into separate sections
module: use a scripted approach for FG-KASLR
maintainers: addMAINTAINERSentry for FG-KASLR
Quoted 3 lines
Kees Cook (2):
x86/boot: Allow a "silent" kaslr random byte fetch x86/boot/compressed: Avoid duplicatemalloc()implementations
These two can get landed right away -- they're standalone fixes that can safely go in -tip.
Quoted 3 lines
Kristen Carlson Accardi (9):
x86:tools/relocs: Support >64K section headers
Same for this.
Quoted 8 lines
x86:
Makefile: Add build and config option forCONFIG_FG_KASLRMake sure ORC lookup covers the entire_etext-_stextx86/tools: Add relative relocs for randomized functions
x86: Add support for function granular KASLR
kallsyms: Hide layout
livepatch: only match unique symbols when using fgkaslr
module: Reorder functions
Documentation: add a documentation for FG-KASLR
I suspect it'll still be easier to review this series as a rebase v5 followed by the evolutionary improvements, since the "basic FGKASLR" has been reviewed in the past, and is fairly noninvasive. The changes for ASM, new .text rules, etc, make a lot more changes that I think would be nice to have separate so reasonable a/b testing can be done.
I'll try to go through the individual patches soon, though I'm currently pretty swamped. :)
I'm looking forward to having this feature finally landed; it's a nice complement to future eXecute-Only memory work too.
-Kees
Quoted 152 lines
Masahiro Yamada (3):
kbuild: mergevmlinux_link()between the ordinary link and Clang LTO
kbuild: do not remove 'linux' link inscripts/link-vmlinux.sh
kbuild: mergevmlinux_link()between ARCH=um and other architecturesSami Tolvanen (1):
kbuild: FixTRIM_UNUSED_KSYMSwithLTO_CLANG.../admin-guide/kernel-parameters.txt | 6 + Documentation/security/fgkaslr.rst | 172 ++++ Documentation/security/index.rst | 1 + MAINTAINERS | 12 + Makefile | 17 +- arch/Kconfig | 3 + arch/arm64/crypto/aes-ce-ccm-core.S | 16 +- arch/arm64/crypto/aes-ce-core.S | 16 +- arch/arm64/crypto/aes-ce.S | 4 +- arch/arm64/crypto/aes-cipher-core.S | 8 +- arch/arm64/crypto/aes-modes.S | 16 +- arch/arm64/crypto/aes-neon.S | 4 +- arch/arm64/crypto/aes-neonbs-core.S | 38 +- arch/arm64/crypto/chacha-neon-core.S | 18 +- arch/arm64/crypto/crct10dif-ce-core.S | 14 +- arch/arm64/crypto/ghash-ce-core.S | 24 +- arch/arm64/crypto/nh-neon-core.S | 4 +- arch/arm64/crypto/poly1305-armv8.pl | 17 + arch/arm64/crypto/sha1-ce-core.S | 4 +- arch/arm64/crypto/sha2-ce-core.S | 4 +- arch/arm64/crypto/sha3-ce-core.S | 4 +- arch/arm64/crypto/sha512-armv8.pl | 11 + arch/arm64/crypto/sha512-ce-core.S | 4 +- arch/arm64/crypto/sm3-ce-core.S | 4 +- arch/arm64/crypto/sm4-ce-core.S | 4 +- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 9 +- arch/x86/boot/compressed/fgkaslr.c | 905 ++++++++++++++++++ arch/x86/boot/compressed/kaslr.c | 4 - arch/x86/boot/compressed/misc.c | 157 ++- arch/x86/boot/compressed/misc.h | 30 + arch/x86/boot/compressed/utils.c | 13 + arch/x86/boot/compressed/vmlinux.symbols | 19 + arch/x86/crypto/aegis128-aesni-asm.S | 36 +- arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 12 +- arch/x86/crypto/aesni-intel_asm.S | 116 ++- arch/x86/crypto/aesni-intel_avx-x86_64.S | 32 +- arch/x86/crypto/blake2s-core.S | 8 +- arch/x86/crypto/blowfish-x86_64-asm_64.S | 16 +- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 28 +- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 28 +- arch/x86/crypto/camellia-x86_64-asm_64.S | 16 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 24 +- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/chacha-avx2-x86_64.S | 12 +- arch/x86/crypto/chacha-avx512vl-x86_64.S | 12 +- arch/x86/crypto/chacha-ssse3-x86_64.S | 16 +- arch/x86/crypto/crc32-pclmul_asm.S | 4 +- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 4 +- arch/x86/crypto/crct10dif-pcl-asm_64.S | 4 +- arch/x86/crypto/des3_ede-asm_64.S | 8 +- arch/x86/crypto/ghash-clmulni-intel_asm.S | 12 +- arch/x86/crypto/nh-avx2-x86_64.S | 4 +- arch/x86/crypto/nh-sse2-x86_64.S | 4 +- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 8 +- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/serpent-avx2-asm_64.S | 20 +- arch/x86/crypto/serpent-sse2-i586-asm_32.S | 8 +- arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 8 +- arch/x86/crypto/sha1_avx2_x86_64_asm.S | 4 +- arch/x86/crypto/sha1_ni_asm.S | 4 +- arch/x86/crypto/sha1_ssse3_asm.S | 4 +- arch/x86/crypto/sha256-avx-asm.S | 4 +- arch/x86/crypto/sha256-avx2-asm.S | 4 +- arch/x86/crypto/sha256-ssse3-asm.S | 4 +- arch/x86/crypto/sha256_ni_asm.S | 4 +- arch/x86/crypto/sha512-avx-asm.S | 4 +- arch/x86/crypto/sha512-avx2-asm.S | 4 +- arch/x86/crypto/sha512-ssse3-asm.S | 4 +- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/twofish-i586-asm_32.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64.S | 8 +- arch/x86/entry/entry_32.S | 24 +- arch/x86/entry/entry_64.S | 18 +- arch/x86/entry/thunk_32.S | 4 +- arch/x86/entry/thunk_64.S | 8 +- arch/x86/include/asm/boot.h | 13 +- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 2 +- arch/x86/kernel/acpi/wakeup_32.S | 9 +- arch/x86/kernel/acpi/wakeup_64.S | 10 +- arch/x86/kernel/ftrace_32.S | 19 +- arch/x86/kernel/ftrace_64.S | 28 +- arch/x86/kernel/irqflags.S | 4 +- arch/x86/kernel/kprobes/core.c | 3 +- arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/relocate_kernel_32.S | 2 + arch/x86/kernel/relocate_kernel_64.S | 2 + arch/x86/kernel/vmlinux.lds.S | 6 +- arch/x86/kvm/emulate.c | 2 +- arch/x86/kvm/vmx/vmenter.S | 8 +- arch/x86/lib/clear_page_64.S | 12 +- arch/x86/lib/cmpxchg16b_emu.S | 4 +- arch/x86/lib/copy_mc_64.S | 8 +- arch/x86/lib/copy_page_64.S | 7 +- arch/x86/lib/copy_user_64.S | 18 +- arch/x86/lib/csum-copy_64.S | 4 +- arch/x86/lib/error-inject.c | 3 +- arch/x86/lib/getuser.S | 37 +- arch/x86/lib/hweight.S | 9 +- arch/x86/lib/iomap_copy_64.S | 4 +- arch/x86/lib/kaslr.c | 18 +- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 12 +- arch/x86/lib/msr-reg.S | 8 +- arch/x86/lib/putuser.S | 18 +- arch/x86/mm/mem_encrypt_boot.S | 8 +- arch/x86/platform/efi/efi_stub_64.S | 4 +- arch/x86/platform/efi/efi_thunk_64.S | 4 +- arch/x86/power/hibernate_asm_32.S | 14 +- arch/x86/power/hibernate_asm_64.S | 14 +- arch/x86/tools/relocs.c | 135 ++- arch/x86/tools/relocs.h | 4 +- arch/x86/tools/relocs_common.c | 15 +- arch/x86/xen/xen-asm.S | 49 +- arch/x86/xen/xen-head.S | 10 +- include/asm-generic/vmlinux.lds.h | 41 +- include/linux/decompress/mm.h | 12 +- include/linux/linkage.h | 76 ++ include/uapi/linux/elf.h | 1 + init/Kconfig | 51 + kernel/kallsyms.c | 158 ++- kernel/livepatch/core.c | 11 + kernel/module.c | 91 +- scripts/Makefile.build | 27 +- scripts/Makefile.lib | 7 + scripts/Makefile.modfinal | 36 +- scripts/Makefile.modpost | 22 +- scripts/gen_autoksyms.sh | 12 - scripts/generate_text_sections.pl | 149 +++ scripts/link-vmlinux.sh | 104 +- scripts/module.lds.S | 14 +- 133 files changed, 2771 insertions(+), 757 deletions(-) create mode 100644 Documentation/security/fgkaslr.rst create mode 100644 arch/x86/boot/compressed/fgkaslr.c create mode 100644 arch/x86/boot/compressed/utils.c create mode 100644 arch/x86/boot/compressed/vmlinux.symbols create mode 100755 scripts/generate_text_sections.plSignature · 1 line
2.31.1
Signature · 1 line
Kees Cook
From: Kees Cook <[email protected]>
Date: Tue, 31 Aug 2021 10:27:45 -0700
Quoted 7 lines · Alexander Lobakin
Quoted 2 lines
This is a massive rework and a respin of Kristen Accardi's marvellous FG-KASLR series (v5).
Thanks for working on this! I know Marios has been looking at some of this as well. I think he tracked down a kretprobes bug and has a fixed prepared.
I was waiting for the fix to be landed in our discussion, but it hasn't appeared there, so I queued the series without it. Will be glad to finally see the fix and include it in v7.
Quoted 8 lines
Quoted 7 lines
The major differences since v5 [0]:
* You can now tune the number of functions per each section to achieve the preferable vmlinux size or protection level. Default is still as one section per each function.
This can be handy for storage-constrained systems. 4-8 fps are still strong, but reduce the size of the final vmlinu{x,z} significantly;
Quoted 6 lines
Interesting, but I'm not sure what the size issue is. v5's on-disk image size issues were related to the large relocation table that was used during decompress and layout, but would get discarded. The final in-core image size was roughly the same size as a non-FGKASLR kernel (since functions were already aligned even without -ffunction-sections). How does the functions-per-section knob change image size?
Without FG-KASLR, we have only one .text section, and the total section number is relatively small.
With FG-KASLR enabled, we have 40K+ separate text sections (I have 40K on a setup with ClangLTO and ClangCFI and about 48K on a "regular" one) and each of them is described in the ELF header. Plus a separate .rela.text section for every single of them. That's the main reason of the size increases.
Quoted 14 lines
Quoted 5 lines
* I don't use orphan sections anymore. It's not reliable at all / may differ from linker to linker, and also conflicts with
CONFIG_LD_ORPHAN_WARNwhich is great for catching random bugs ->
* All the .text.* sections are now being described explicitly in the linker script. A Perl script is used to take the original LDS, theQuoted 8 lines
original object file, read a list of input sections from it and generate the resulting LDS.
This costs a bit of linking time as LD tends to think hard when processing scripts > 1 Mb. It adds about 40-60 seconds to the whole linking process (BTF step, 2-3 kallsyms steps and the final step), but "better safe than sorry".
In addition, that approach allows to reserve some space at the end and add some link assertions ->
Quoted 10 lines
Yeah, this "hope that orphan handling does it right" bugged me too, but my attempts to solve it looked much like yours: creating a linker file that named all the sections. I found this to be prohibitively expensive at link time (and that seems backed by your own measurements of an extra minute or so at link time). If that's still the result of using a generated linker file, we just need to depend on orphan handling.
LD_ORPHAN_WARNwill still exist for non-FGKASLR builds, so the benefits will continue to exist -- I think the correct solution is to have the linker grow a "pass through" special target like "DISCARD", which just maps given input section patterns into same-named output sections.
We still have LD_ORPHAN_WARN on non-FG-KASLR builds, but we also have a rather different set of sections with FG-KASLR enabled. For example, I noticed the appearing of .symtab_shndx section only in virtue of LD_ORPHAN_WARN. So it's kinda not the same.
I don't see a problem in this extra minute. FG-KASLR is all about security, and you often pay something for this. We already have a size increase, and a small delay while booting, and we can't get rid of them. With orphan sections you leave a space for potentional flaws of the code, linker and/or linker script, which is really unwanted in case of a security feature.
After all, ClangLTO increases the linking time at lot, and TRIM_UNUSED_KSYMS builds almost the entire kernel two times in a row, but nobody complains about this as there's nothing we can do with it and it's the price you pay for the optimizations, so again, I don't see a problem here.
I'll be glad to see approaches with no link time penalties and still without "grey zones" like orphans and stuff, but I could come only with this. This can be a room for future patches and optimizations.
Quoted 7 lines
Quoted 4 lines
* Input .text section now must be empty, otherwise the linkage will be stopped. This is implemented by the size assertion in the resulting LD script and is designed to plug the potentional layout leakage. This also means that ->
I worry this will create unexpected problems for named sections that weren't originally being randomized with the v5 FGKASLR.
1. Input .text just contained a bunch of ASM functions (described below), none of them required any kind of special handling.
2. This was tested a lot.
3. We have plenty of time to test on a wide variety of setups since we miss 5.15 window.
Quoted 9 lines
Quoted 6 lines
* "Regular" ASM functions are now being placed into unique separate functions the same way compiler does this for C functions. This is achieved by introducing and using several new macros which take the symbol name as a base for its new section name. This gives a better opportunity to both DCE and FG-KASLR, as ASM code now can also be randomized or garbage-collected;
This is interesting! I think it'd be a good evolutionary step on top of "basic FGKASLR".
I still don't get why you're trying to split this series into two. It's been almost a year since v5 was published, I doubt you can get "basic FG-KASLR" accepted quickly just because it was reviewed back then.
I prefer to provide a full picture of what I'm trying to bring, so the community could review it all and throw much more ideas and stuff.
Quoted 6 lines
Quoted 3 lines
* It's now fully compatible with ClangLTO, ClangCFI,
CONFIG_LD_ORPHAN_WARNand some more stuff landed since the last revision was published;FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it work with ClangLTO and ClangCFI.
Once again, repeating the thing I wrote earlier in our discussion:
ClangCFI, at least shadowed implementation, requires the first text section of the module to be page-aligned and contain __cfi_check() at the very beginning of this section. With FG-KASLR and without special handling, this section gets randomized along with the others, and ClangCFI either rejects almost all modules or panics the kernel.
Quoted 5 lines
Quoted 2 lines
* Includes several fixes: relocations inside .altinstr_replacement code and minor issues found and/or suggested by LKP robot.
Excellent!
Quoted 7 lines
Quoted 4 lines
The series was compile-time and runtime tested on the following setups with no issues:
-x86_64, GCC 11, Binutils 2.35;
-x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).Great, this is a good start. One place we saw problems in the past was with i386 build gotchas, so that'll need testing too.
For now, FG_KASLR for x86 depends on X86_64. We might relax this dependency later after enough testing or whatsoever (like it's been done for ClangLTO).
Quoted 6 lines
Quoted 2 lines
The first 4 patches are from the linux-kbuild tree and included to avoid merge conflicts and non-intuitive resolving of them.
Sounds good. It might be easier to base the series on linux-next, so a smaller series. Though given the merge window just opened, it might make more sense for a v7 to be based on v5.15-rc2 in three weeks.
I don't usually base any series on linux-next, because it contains all the changes from all "for-next" branches and repos, while the series finally gets accepted to the specific repo based on just v5.x-rc1 (sometimes on -rc2). This may bring additional apply/merge problems.
Quoted 291 lines
Quoted 6 lines
The series is also available here: [1]
[0] https://lore.kernel.org/kernel-hardening/[email protected]
[1] https://github.com/alobakin/linux/pull/3The original v5 cover letter:
More notes below...
Quoted 278 lines
Function Granular Kernel Address Space Layout Randomization (fgkaslr)
---------------------------------------------------------------------This patch set is an implementation of finer grained kernel address space randomization. It rearranges your kernel code at load time
on a per-function level granularity, with only around a second added to boot time.Changes in v5:
--------------
* fixed a bug in the code which increases boot heap size forCONFIG_FG_KASLRwhich prevented the boot heap from being increased forCONFIG_FG_KASLRwhen using bzip2 compression. Thanks to Andy Lavr for finding the problem and identifying the solution.
* changed the adjustment of theorc_unwind_iptable at boot time to disregard relocs associated with this table, and instead inspect the entries separately. Relocs are not able to be used since they are no longer correct once the table is resorted at buildtime.
* changed howorc_unwind_ipaddresses in randomized sections are identified to include the byte immediately after the end of the section.
* updated module code to use kvmalloc/kvfree based on suggestions from Evgenii Shatokhin <[email protected]>.
* changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to match the nokaslr option. fgkaslr="X" can be added at a later date if it is needed.
* Added a patch to force livepatch to require symbols to be unique if using while fgkaslr either for core or modules.Changes in v4:
-------------
* dropped the patch to split out change to STATIC definition inx86/boot/compressed/misc.cand replaced with a patch authored by Kees Cook to avoid the duplicate malloc definitions
* Added a section toDocumentation/admin-guide/kernel-parameters.txtto document the fgkaslr boot option.
* redesigned the patch to hide the new layout when reading/proc/kallsyms. The previous implementation utilized a dynamically allocated linked list to display the kernel and module symbols in alphabetical order. The new implementation uses a randomly shuffled index array to display the kernel and module symbols in a random order.Changes in v3:
-------------
*Makefilechanges to accommodateCONFIG_LD_DEAD_CODE_DATA_ELIMINATION
* removal of extraneousALIGN_PAGEfrom_etextchanges
* changed variable names in x86/tools/relocs to be less confusing
* split out change to STATIC definition inx86/boot/compressed/misc.c
* Updates to Documentation to make it more clear what is preserved in .text
* much more detailed commit message for function granular KASLR patch
* minor tweaks and changes that make for more readable code
* this cover letter updated slightly to add additional detailsChanges in v2:
--------------
* Fix to address i386 build failure
* Allow module reordering patch to be configured separately so that arm (or other non-x86_64 arches) can take advantage of module function reordering. This support has not be tested by me, but smoke tested by Ard Biesheuvel <[email protected]> on arm.
* Fix build issue when building on arm as reported by
Ard Biesheuvel <[email protected]>Patches to objtool are included because they are dependencies for this patchset, however they have been submitted by their maintainer separately.
Background
----------
KASLR was merged into the kernel with the objective of increasing the difficulty of code reuse attacks. Code reuse attacks reused existing code snippets to get around existing memory protections. They exploit software bugs which expose addresses of useful code snippets to control the flow of execution for their own nefarious purposes. KASLR moves the entire kernel code text as a unit at boot time in order to make addresses less predictable. The order of the code within the segment is unchanged - only the base address is shifted. There are a few shortcomings to this algorithm.1. Low Entropy - there are only so many locations the kernel can fit in. This means an attacker could guess without too much trouble.
2. Knowledge of a single address can reveal the offset of the base address, exposing all other locations for a published/known kernel image.
3. Info leaks abound.Finer grained ASLR has been proposed as a way to make ASLR more resistant to info leaks. It is not a new concept at all, and there are many variations possible. Function reordering is an implementation of finer grained ASLR which randomizes the layout of an address space on a function level granularity. We use the term "fgkaslr" in this document to refer to the technique of function reordering when used with KASLR, as well as finer grained KASLR in general.
Proposed Improvement
--------------------
This patch set proposes adding function reordering on top of the existing KASLR base address randomization. The over-arching objective is incremental improvement over what we already have. It is designed to work in combination with the existing solution. The implementation is really pretty simple, and there are 2 main area where changes occur:* Build time
GCC has had an option to place functions into individual .text sections for many years now. This option can be used to implement function reordering at load time. The final compiled vmlinux retains all the section headers, which can be used to help find the address ranges of each function. Using this information and an expanded table of relocation addresses, individual text sections can be suffled immediately after decompression. Some data tables inside the kernel that have assumptions about order require re-sorting after being updated when applying relocations. In order to modify these tables, a few key symbols are excluded from the objcopy symbol stripping process for use after shuffling the text segments.
Some highlights from the build time changes to look for:
The top level kernel
Makefilewas modified to add the gcc flag if it is supported. Currently, I am applying this flag to everything it is possible to randomize. Anything that is written in C and not present in a special input section is randomized. The final binary segment 0 retains a consolidated .text section, as well as all the individual .text.* sections. Future work could turn off this flags for selected files or even entire subsystems, although obviously at the cost of security.The relocs tool is updated to add relative relocations. This information previously wasn't included because it wasn't necessary when moving the entire .text segment as a unit.
A new file was created to contain a list of symbols that objcopy should keep. We use those symbols at load time as described below.
* Load time
The boot kernel was modified to parse the vmlinux elf file after decompression to check for our interesting symbols that we kept, and to look for any .text.* sections to randomize. The consolidated .text section is skipped and not moved. The sections are shuffled randomly, and copied into memory following the .text section in a new random order. The existing code which updated relocation addresses was modified to account for not just a fixed delta from the load address, but the offset that the function section was moved to. This requires inspection of each address to see if it was impacted by a randomization. We use a bsearch to make this less horrible on performance. Any tables that need to be modified with new addresses or resorted are updated using the symbol addresses parsed from the elf symbol table.
In order to hide our new layout, symbols reported through
/proc/kallsymswill be displayed in a random order.Security Considerations
-----------------------
The objective of this patch set is to improve a technology that is already merged into the kernel (KASLR). This code will not prevent all attacks, but should instead be considered as one of several tools that can be used. In particular, this code is meant to make KASLR more effective in the presence of info leaks.How much entropy we are adding to the existing entropy of standard KASLR will depend on a few variables. Firstly and most obviously, the number of functions that are randomized matters. This implementation keeps the existing .text section for code that cannot be randomized - for example, because it was assembly code. The less sections to randomize, the less entropy. In addition, due to alignment (16 bytes for
x86_64), the number of bits in a address that the attacker needs to guess is reduced, as the lower bits are identical.Performance Impact
------------------
There are two areas where function reordering can impact performance: boot time latency, and run time performance.* Boot time latency
This implementation of finer grained KASLR impacts the boot time of the kernel in several places. It requires additional parsing of the kernel ELF file to obtain the section headers of the sections to be randomized. It calls the random number generator for each section to be randomized to determine that section's new memory location. It copies the decompressed kernel into a new area of memory to avoid corruption when laying out the newly randomized sections. It increases the number of relocations the kernel has to perform at boot time vs. standard KASLR, and it also requires a lookup on each address that needs to be relocated to see if it was in a randomized section and needs to be adjusted by a new offset. Finally, it re-sorts a few data tables that are required to be sorted by address.Booting a test VM on a modern, well appointed system showed an increase in latency of approximately 1 second.
* Run time
The performance impact at run-time of function reordering varies by workload. Using kcbench, a kernel compilation benchmark, the performance of a kernel build with finer grained KASLR was about 1% slower than a kernel with standard KASLR. Analysis with perf showed a slightly higher percentage of L1-icache-load-misses. Other workloads were examined as well, with varied results. Some workloads performed significantly worse under FGKASLR, while others stayed the same or were mysteriously better. In general, it will depend on the code flow whether or not finer grained KASLR will impact your workload, and how the underlying code was designed. Because the layout changes per boot, each time a system is rebooted the performance of a workload may change.Future work could identify hot areas that may not be randomized and either leave them in the .text section or group them together into a single section that may be randomized. If grouping things together helps, one other thing to consider is that if we could identify text blobs that should be grouped together to benefit a particular code flow, it could be interesting to explore whether this security feature could be also be used as a performance feature if you are interested in optimizing your kernel layout for a particular workload at boot time. Optimizing function layout for a particular workload has been researched and proven effective - for more information read the Facebook paper "Optimizing Function Placement for Large-Scale Data-Center Applications" (see references section below).
Image Size
----------
Adding additional section headers as a result of compiling with -ffunction-sections will increase the size of the vmlinux ELF file. With a standard distro config, the resulting vmlinux was increased by about 3%. The compressed image is also increased due to the header files, as well as the extra relocations that must be added. You can expect fgkaslr to increase the size of the compressed image by about 15%.Memory Usage
------------
fgkaslr increases the amount of heap that is required at boot time, although this extra memory is released when the kernel has finished decompression. As a result, it may not be appropriate to use this feature on systems without much memory.Building
--------
To enable fine grained KASLR, you need to have the following config options set (including all the ones you would use to build normal KASLR)
CONFIG_FG_KASLR=yIn addition, fgkaslr is only supported for the
X86_64architecture.Modules
-------
Modules are randomized similarly to the rest of the kernel by shuffling the sections at load time prior to moving them into memory. The module must also have been build with the -ffunction-sections compiler option.Although fgkaslr for the kernel is only supported for the
X86_64architecture, it is possible to use fgkaslr with modules on other architectures. To enable this feature, select
CONFIG_MODULE_FG_KASLR=yThis option is selected automatically for
X86_64whenCONFIG_FG_KASLRis set.Disabling
---------
Disabling normal KASLR using the nokaslr command line option also disables fgkaslr. It is also possible to disable fgkaslr separately by booting with nofgkaslr on the commandline.References
----------
There are a lot of academic papers which explore finer grained ASLR. This paper in particular contributed the most to my implementation design as well as my overall understanding of the problem space:Selfrando: Securing the Tor Browser against De-anonymization Exploits, M. Conti, S. Crane, T. Frassetto, et al.
For more information on how function layout impacts performance, see:
Optimizing Function Placement for Large-Scale Data-Center Applications, G. Ottoni, B. Maher
Alexander Lobakin (7):
linkage: add macros for putting ASM functions into own sections
x86: conditionally place regular ASM functions into separate sections
FG-KASLR: use a scripted approach to handle .text.* sections x86/boot: allow FG-KASLR to be selected
arm64/crypto: conditionally place ASM functions into separate sections
module: use a scripted approach for FG-KASLR
maintainers: addMAINTAINERSentry for FG-KASLRQuoted 3 lines
Kees Cook (2):
x86/boot: Allow a "silent" kaslr random byte fetch x86/boot/compressed: Avoid duplicatemalloc()implementations
Quoted 8 lines
These two can get landed right away -- they're standalone fixes that can safely go in -tip.
Quoted 3 lines
Kristen Carlson Accardi (9):
x86:tools/relocs: Support >64K section headersSame for this.
They make little to no sense for non-FG-KASLR systems. And none of them are "pure" fixes.
The same could be said about e.g. ORC lookup patch, but again, it makes no sense right now.
Quoted 9 lines
Quoted 8 lines
x86:
Makefile: Add build and config option forCONFIG_FG_KASLRMake sure ORC lookup covers the entire_etext-_stextx86/tools: Add relative relocs for randomized functions
x86: Add support for function granular KASLR
kallsyms: Hide layout
livepatch: only match unique symbols when using fgkaslr
module: Reorder functions
Documentation: add a documentation for FG-KASLR
Quoted 5 lines
I suspect it'll still be easier to review this series as a rebase v5 followed by the evolutionary improvements, since the "basic FGKASLR" has been reviewed in the past, and is fairly noninvasive. The changes for ASM, new .text rules, etc, make a lot more changes that I think would be nice to have separate so reasonable a/b testing can be done.
I don't see a point in testing it two times instead of just one, as well as in delivering this feature in two halves. It sounds like "let's introduce ClangLTO, but firstly only for modules, as LTO for vmlinux requires changes in objtool code and a special handling for the initcalls".
The changes you mentioned only seem invasive, in fact, they can carry way less harm than the "basic FG-KASLR" itself.
Quoted 157 lines
I'll try to go through the individual patches soon, though I'm currently pretty swamped. :)
I'm looking forward to having this feature finally landed; it's a nice complement to future eXecute-Only memory work too.
-Kees
Quoted 149 lines
Masahiro Yamada (3):
kbuild: mergevmlinux_link()between the ordinary link and Clang LTO
kbuild: do not remove 'linux' link inscripts/link-vmlinux.sh
kbuild: mergevmlinux_link()between ARCH=um and other architecturesSami Tolvanen (1):
kbuild: FixTRIM_UNUSED_KSYMSwithLTO_CLANG.../admin-guide/kernel-parameters.txt | 6 + Documentation/security/fgkaslr.rst | 172 ++++ Documentation/security/index.rst | 1 + MAINTAINERS | 12 + Makefile | 17 +- arch/Kconfig | 3 + arch/arm64/crypto/aes-ce-ccm-core.S | 16 +- arch/arm64/crypto/aes-ce-core.S | 16 +- arch/arm64/crypto/aes-ce.S | 4 +- arch/arm64/crypto/aes-cipher-core.S | 8 +- arch/arm64/crypto/aes-modes.S | 16 +- arch/arm64/crypto/aes-neon.S | 4 +- arch/arm64/crypto/aes-neonbs-core.S | 38 +- arch/arm64/crypto/chacha-neon-core.S | 18 +- arch/arm64/crypto/crct10dif-ce-core.S | 14 +- arch/arm64/crypto/ghash-ce-core.S | 24 +- arch/arm64/crypto/nh-neon-core.S | 4 +- arch/arm64/crypto/poly1305-armv8.pl | 17 + arch/arm64/crypto/sha1-ce-core.S | 4 +- arch/arm64/crypto/sha2-ce-core.S | 4 +- arch/arm64/crypto/sha3-ce-core.S | 4 +- arch/arm64/crypto/sha512-armv8.pl | 11 + arch/arm64/crypto/sha512-ce-core.S | 4 +- arch/arm64/crypto/sm3-ce-core.S | 4 +- arch/arm64/crypto/sm4-ce-core.S | 4 +- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 9 +- arch/x86/boot/compressed/fgkaslr.c | 905 ++++++++++++++++++ arch/x86/boot/compressed/kaslr.c | 4 - arch/x86/boot/compressed/misc.c | 157 ++- arch/x86/boot/compressed/misc.h | 30 + arch/x86/boot/compressed/utils.c | 13 + arch/x86/boot/compressed/vmlinux.symbols | 19 + arch/x86/crypto/aegis128-aesni-asm.S | 36 +- arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 12 +- arch/x86/crypto/aesni-intel_asm.S | 116 ++- arch/x86/crypto/aesni-intel_avx-x86_64.S | 32 +- arch/x86/crypto/blake2s-core.S | 8 +- arch/x86/crypto/blowfish-x86_64-asm_64.S | 16 +- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 28 +- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 28 +- arch/x86/crypto/camellia-x86_64-asm_64.S | 16 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 24 +- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/chacha-avx2-x86_64.S | 12 +- arch/x86/crypto/chacha-avx512vl-x86_64.S | 12 +- arch/x86/crypto/chacha-ssse3-x86_64.S | 16 +- arch/x86/crypto/crc32-pclmul_asm.S | 4 +- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 4 +- arch/x86/crypto/crct10dif-pcl-asm_64.S | 4 +- arch/x86/crypto/des3_ede-asm_64.S | 8 +- arch/x86/crypto/ghash-clmulni-intel_asm.S | 12 +- arch/x86/crypto/nh-avx2-x86_64.S | 4 +- arch/x86/crypto/nh-sse2-x86_64.S | 4 +- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 8 +- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/serpent-avx2-asm_64.S | 20 +- arch/x86/crypto/serpent-sse2-i586-asm_32.S | 8 +- arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 8 +- arch/x86/crypto/sha1_avx2_x86_64_asm.S | 4 +- arch/x86/crypto/sha1_ni_asm.S | 4 +- arch/x86/crypto/sha1_ssse3_asm.S | 4 +- arch/x86/crypto/sha256-avx-asm.S | 4 +- arch/x86/crypto/sha256-avx2-asm.S | 4 +- arch/x86/crypto/sha256-ssse3-asm.S | 4 +- arch/x86/crypto/sha256_ni_asm.S | 4 +- arch/x86/crypto/sha512-avx-asm.S | 4 +- arch/x86/crypto/sha512-avx2-asm.S | 4 +- arch/x86/crypto/sha512-ssse3-asm.S | 4 +- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/twofish-i586-asm_32.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64.S | 8 +- arch/x86/entry/entry_32.S | 24 +- arch/x86/entry/entry_64.S | 18 +- arch/x86/entry/thunk_32.S | 4 +- arch/x86/entry/thunk_64.S | 8 +- arch/x86/include/asm/boot.h | 13 +- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 2 +- arch/x86/kernel/acpi/wakeup_32.S | 9 +- arch/x86/kernel/acpi/wakeup_64.S | 10 +- arch/x86/kernel/ftrace_32.S | 19 +- arch/x86/kernel/ftrace_64.S | 28 +- arch/x86/kernel/irqflags.S | 4 +- arch/x86/kernel/kprobes/core.c | 3 +- arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/relocate_kernel_32.S | 2 + arch/x86/kernel/relocate_kernel_64.S | 2 + arch/x86/kernel/vmlinux.lds.S | 6 +- arch/x86/kvm/emulate.c | 2 +- arch/x86/kvm/vmx/vmenter.S | 8 +- arch/x86/lib/clear_page_64.S | 12 +- arch/x86/lib/cmpxchg16b_emu.S | 4 +- arch/x86/lib/copy_mc_64.S | 8 +- arch/x86/lib/copy_page_64.S | 7 +- arch/x86/lib/copy_user_64.S | 18 +- arch/x86/lib/csum-copy_64.S | 4 +- arch/x86/lib/error-inject.c | 3 +- arch/x86/lib/getuser.S | 37 +- arch/x86/lib/hweight.S | 9 +- arch/x86/lib/iomap_copy_64.S | 4 +- arch/x86/lib/kaslr.c | 18 +- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 12 +- arch/x86/lib/msr-reg.S | 8 +- arch/x86/lib/putuser.S | 18 +- arch/x86/mm/mem_encrypt_boot.S | 8 +- arch/x86/platform/efi/efi_stub_64.S | 4 +- arch/x86/platform/efi/efi_thunk_64.S | 4 +- arch/x86/power/hibernate_asm_32.S | 14 +- arch/x86/power/hibernate_asm_64.S | 14 +- arch/x86/tools/relocs.c | 135 ++- arch/x86/tools/relocs.h | 4 +- arch/x86/tools/relocs_common.c | 15 +- arch/x86/xen/xen-asm.S | 49 +- arch/x86/xen/xen-head.S | 10 +- include/asm-generic/vmlinux.lds.h | 41 +- include/linux/decompress/mm.h | 12 +- include/linux/linkage.h | 76 ++ include/uapi/linux/elf.h | 1 + init/Kconfig | 51 + kernel/kallsyms.c | 158 ++- kernel/livepatch/core.c | 11 + kernel/module.c | 91 +- scripts/Makefile.build | 27 +- scripts/Makefile.lib | 7 + scripts/Makefile.modfinal | 36 +- scripts/Makefile.modpost | 22 +- scripts/gen_autoksyms.sh | 12 - scripts/generate_text_sections.pl | 149 +++ scripts/link-vmlinux.sh | 104 +- scripts/module.lds.S | 14 +- 133 files changed, 2771 insertions(+), 757 deletions(-) create mode 100644 Documentation/security/fgkaslr.rst create mode 100644 arch/x86/boot/compressed/fgkaslr.c create mode 100644 arch/x86/boot/compressed/utils.c create mode 100644 arch/x86/boot/compressed/vmlinux.symbols create mode 100755 scripts/generate_text_sections.pl
Quoted 6 lines
Quoted 3 lines
Signature · 1 line
2.31.1Signature · 1 line
Kees Cook
Thanks,
Al
Quoted 7 lines · Alexander Lobakin
Without FG-KASLR, we have only one .text section, and the total section number is relatively small.
With FG-KASLR enabled, we have 40K+ separate text sections (I have 40K on a setup with ClangLTO and ClangCFI and about 48K on a "regular" one) and each of them is described in the ELF header. Plus a separate .rela.text section for every single of them. That's the main reason of the size increases.
If you have the size comparisons handy, I'd love to see them. My memory from v5 was that none of that end up in-core. And in that case, why limit the entropy of the resulting layout?
Quoted 4 lines
We still have
LD_ORPHAN_WARNon non-FG-KASLR builds, but we also have a rather different set of sections with FG-KASLR enabled. For example, I noticed the appearing of .symtab_shndx section only in virtue ofLD_ORPHAN_WARN. So it's kinda not the same.
Agreed: I'd rather have LD_ORPHAN_WARN always enabled.
Quoted 1 line
I don't see a problem in this extra minute. FG-KASLR is all about
But not at this cost. Maybe the x86 maintainers will disagree, but I see this as a prohibitive cost to doing development work under FGKASLR, and if we expect this to become the default in distros, no one is going to be happy with that change. Link time dominates the partial rebuild time, so my opinion is that it should not be so inflated if not absolutely needed. Perhaps once the link time bugs in ld.bfd and ld.lld get fixed, but not now.
Quoted 3 lines
security, and you often pay something for this. We already have a size increase, and a small delay while booting, and we can't get rid of them. With orphan sections you leave a space for potentional
There's a difference between development time costs and run time costs. I don't think the LD_ORPHAN_WARN coverage is worth it in this case.
Either way, we need to fix the linker.
Quoted 7 lines
flaws of the code, linker and/or linker script, which is really unwanted in case of a security feature.
After all, ClangLTO increases the linking time at lot, andTRIM_UNUSED_KSYMSbuilds almost the entire kernel two times in a row, but nobody complains about this as there's nothing we can do with it and it's the price you pay for the optimizations, so again, I don't see a problem here.
I get what you mean with regard to getting the perfect situation, but the kernel went 29 years without LD_ORPHAN_WARN. :) Anyway, we'll see what other folks think, I guess.
Quoted 4 lines
I still don't get why you're trying to split this series into two. It's been almost a year since v5 was published, I doubt you can get "basic FG-KASLR" accepted quickly just because it was reviewed back then.
Well, because it was blocked then by a single bug, and everything else you've described are distinct improvements on v5, so to me it makes sense to have it separated into those phases. I don't mean split the series, I mean rearrange the series so that a rebased v5 is at the start, and the improvements follow.
Quoted 3 lines
I prefer to provide a full picture of what I'm trying to bring, so the community could review it all and throw much more ideas and stuff.
Understood. I am suggesting some ideas about how it might help with review. :)
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
* It's now fully compatible with ClangLTO, ClangCFI,
CONFIG_LD_ORPHAN_WARNand some more stuff landed since the last revision was published;FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it work with ClangLTO and ClangCFI.
Quoted 7 lines
Once again, repeating the thing I wrote earlier in our discussion:
ClangCFI, at least shadowed implementation, requires the first text section of the module to be page-aligned and contain__cfi_check()at the very beginning of this section. With FG-KASLR and without special handling, this section gets randomized along with the others, and ClangCFI either rejects almost all modules or panics the kernel.
Ah-ha, thanks. I must have missed your answer to this earlier. I had probably done my initial v5 testing without modules.
Quoted 6 lines
Quoted 2 lines
Great, this is a good start. One place we saw problems in the past was with i386 build gotchas, so that'll need testing too.
For now,
FG_KASLRfor x86 depends onX86_64. We might relax this dependency later after enough testing or whatsoever (like it's been done for ClangLTO).
Yes, but we've had a history of making big patches that do _intend_ to break the i386 build, but they do anyway. Hence my question.
Quoted 9 lines
Quoted 3 lines
Sounds good. It might be easier to base the series on linux-next, so a smaller series. Though given the merge window just opened, it might make more sense for a v7 to be based on v5.15-rc2 in three weeks.
I don't usually base any series on linux-next, because it contains all the changes from all "for-next" branches and repos, while the series finally gets accepted to the specific repo based on just v5.x-rc1 (sometimes on -rc2). This may bring additional apply/merge problems.
Understood. I just find it confusing to include patches on lkml that already exist in a -next branch. Perhaps base on kbuild -next?
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Kees Cook (2):
x86/boot: Allow a "silent" kaslr random byte fetch x86/boot/compressed: Avoid duplicatemalloc()implementationsThese two can get landed right away -- they're standalone fixes that can safely go in -tip.
Quoted 3 lines
Kristen Carlson Accardi (9):
x86:tools/relocs: Support >64K section headers
Quoted 6 lines
Quoted 1 line
Same for this.
They make little to no sense for non-FG-KASLR systems. And none of them are "pure" fixes.
The same could be said about e.g. ORC lookup patch, but again, it makes no sense right now.
*shrug* They're trivial changes that have been reviewed before, so it seems like we can avoid resending them every time.
Quoted 6 lines
Quoted 5 lines
I suspect it'll still be easier to review this series as a rebase v5 followed by the evolutionary improvements, since the "basic FGKASLR" has been reviewed in the past, and is fairly noninvasive. The changes for ASM, new .text rules, etc, make a lot more changes that I think would be nice to have separate so reasonable a/b testing can be done.
Quoted 7 lines
I don't see a point in testing it two times instead of just one, as well as in delivering this feature in two halves. It sounds like "let's introduce ClangLTO, but firstly only for modules, as LTO for vmlinux requires changes in objtool code and a special handling for the initcalls".
The changes you mentioned only seem invasive, in fact, they can carry way less harm than the "basic FG-KASLR" itself.
Mostly it's a question of building on prior testing (v5 worked), so that new changes can be debugged if they cause problems. Regardless, it's been so long, perhaps it won't matter to other reviewers and they'll want to just start over from scratch.
-Kees
Signature · 1 line
Kees Cook
Quoted 12 lines
From: Kees Cook <[email protected]>
Date: Wed, 1 Sep 2021 18:36:59 -0700Quoted 7 lines · Alexander Lobakin
Without FG-KASLR, we have only one .text section, and the total section number is relatively small.
With FG-KASLR enabled, we have 40K+ separate text sections (I have 40K on a setup with ClangLTO and ClangCFI and about 48K on a "regular" one) and each of them is described in the ELF header. Plus a separate .rela.text section for every single of them. That's the main reason of the size increases.
Quoted 3 lines
If you have the size comparisons handy, I'd love to see them. My memory from v5 was that none of that end up in-core. And in that case, why limit the entropy of the resulting layout?
My testing machine is down for now, but I could send a size comparison later. It's something about 10 Mb of uncompressed kernel between 1 and 4 fps or so.
Quoted 9 lines
Quoted 4 lines
We still have
LD_ORPHAN_WARNon non-FG-KASLR builds, but we also have a rather different set of sections with FG-KASLR enabled. For example, I noticed the appearing of .symtab_shndx section only in virtue ofLD_ORPHAN_WARN. So it's kinda not the same.Agreed: I'd rather have
LD_ORPHAN_WARNalways enabled.Quoted 1 line
I don't see a problem in this extra minute. FG-KASLR is all about
Quoted 7 lines
But not at this cost. Maybe the x86 maintainers will disagree, but I see this as a prohibitive cost to doing development work under FGKASLR, and if we expect this to become the default in distros, no one is going to be happy with that change. Link time dominates the partial rebuild time, so my opinion is that it should not be so inflated if not absolutely needed. Perhaps once the link time bugs in ld.bfd and ld.lld get fixed, but not now.
I don't think FG-KASLR will be enabled by default in distros. Apart from linking time, it also increases cache misses a lot, and when it comes to performance critical usecases like high-speed servers and datacenters, I don't believe their maintainers would consider FG-KASLR.
Speaking about distros, almost no build systems to my knowledge use partial building, so this is only a downside for developers.
Quoted 8 lines
Quoted 3 lines
security, and you often pay something for this. We already have a size increase, and a small delay while booting, and we can't get rid of them. With orphan sections you leave a space for potentional
There's a difference between development time costs and run time costs. I don't think the
LD_ORPHAN_WARNcoverage is worth it in this case.Either way, we need to fix the linker.
I agree on that, I was surprised both BFD and LLD choke on big LD scripts.
Quoted 11 lines
Quoted 7 lines
flaws of the code, linker and/or linker script, which is really unwanted in case of a security feature.
After all, ClangLTO increases the linking time at lot, andTRIM_UNUSED_KSYMSbuilds almost the entire kernel two times in a row, but nobody complains about this as there's nothing we can do with it and it's the price you pay for the optimizations, so again, I don't see a problem here.I get what you mean with regard to getting the perfect situation, but the kernel went 29 years without
LD_ORPHAN_WARN. :) Anyway, we'll see what other folks think, I guess.
Also agree, let's wait for more opinions on that, I'm open to everything.
Quoted 50 lines
Quoted 4 lines
I still don't get why you're trying to split this series into two. It's been almost a year since v5 was published, I doubt you can get "basic FG-KASLR" accepted quickly just because it was reviewed back then.
Well, because it was blocked then by a single bug, and everything else you've described are distinct improvements on v5, so to me it makes sense to have it separated into those phases. I don't mean split the series, I mean rearrange the series so that a rebased v5 is at the start, and the improvements follow.
Quoted 3 lines
I prefer to provide a full picture of what I'm trying to bring, so the community could review it all and throw much more ideas and stuff.
Understood. I am suggesting some ideas about how it might help with review. :)
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
* It's now fully compatible with ClangLTO, ClangCFI,
CONFIG_LD_ORPHAN_WARNand some more stuff landed since the last revision was published;FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it work with ClangLTO and ClangCFI.
Quoted 7 lines
Once again, repeating the thing I wrote earlier in our discussion:
ClangCFI, at least shadowed implementation, requires the first text section of the module to be page-aligned and contain__cfi_check()at the very beginning of this section. With FG-KASLR and without special handling, this section gets randomized along with the others, and ClangCFI either rejects almost all modules or panics the kernel.Ah-ha, thanks. I must have missed your answer to this earlier. I had probably done my initial v5 testing without modules.
Quoted 6 lines
Quoted 2 lines
Great, this is a good start. One place we saw problems in the past was with i386 build gotchas, so that'll need testing too.
For now,
FG_KASLRfor x86 depends onX86_64. We might relax this dependency later after enough testing or whatsoever (like it's been done for ClangLTO).Yes, but we've had a history of making big patches that do _intend_ to break the i386 build, but they do anyway. Hence my question.
Quoted 4 lines
Quoted 3 lines
Sounds good. It might be easier to base the series on linux-next, so a smaller series. Though given the merge window just opened, it might make more sense for a v7 to be based on v5.15-rc2 in three weeks.
Quoted 8 lines
Quoted 5 lines
I don't usually base any series on linux-next, because it contains all the changes from all "for-next" branches and repos, while the series finally gets accepted to the specific repo based on just v5.x-rc1 (sometimes on -rc2). This may bring additional apply/merge problems.
Understood. I just find it confusing to include patches on lkml that already exist in a -next branch. Perhaps base on kbuild -next?
That's not a problem anymore I believe, since it doesn't hit 5.15 window, so the rebased v7 will be on top of 5.15-rc1 which will already contain those Kbuild fixes.
Quoted 35 lines
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Kees Cook (2):
x86/boot: Allow a "silent" kaslr random byte fetch x86/boot/compressed: Avoid duplicatemalloc()implementationsThese two can get landed right away -- they're standalone fixes that can safely go in -tip.
Quoted 3 lines
Kristen Carlson Accardi (9):
x86:tools/relocs: Support >64K section headersQuoted 6 lines
Quoted 1 line
Same for this.
They make little to no sense for non-FG-KASLR systems. And none of them are "pure" fixes.
The same could be said about e.g. ORC lookup patch, but again, it makes no sense right now.*shrug* They're trivial changes that have been reviewed before, so it seems like we can avoid resending them every time.
Quoted 6 lines
Quoted 5 lines
I suspect it'll still be easier to review this series as a rebase v5 followed by the evolutionary improvements, since the "basic FGKASLR" has been reviewed in the past, and is fairly noninvasive. The changes for ASM, new .text rules, etc, make a lot more changes that I think would be nice to have separate so reasonable a/b testing can be done.
Quoted 7 lines
I don't see a point in testing it two times instead of just one, as well as in delivering this feature in two halves. It sounds like "let's introduce ClangLTO, but firstly only for modules, as LTO for vmlinux requires changes in objtool code and a special handling for the initcalls".
The changes you mentioned only seem invasive, in fact, they can carry way less harm than the "basic FG-KASLR" itself.
Quoted 9 lines
Mostly it's a question of building on prior testing (v5 worked), so that new changes can be debugged if they cause problems. Regardless, it's been so long, perhaps it won't matter to other reviewers and they'll want to just start over from scratch.
-Kees
Signature · 1 line
Kees Cook
Thanks,
Al
(dropping all external emails)
Hey, going external was pretty premature here. We do have a process of internal review required for any patches that touch x86 subsystem that requires getting signed off by x86 reviewers - with a major revision such as this you should have waited until one of the x86 reviewers gave you a reviewed by before going external. I appreciate that this is an own time kind of project for you, but the community just sees one intel and we all need to follow the same standards and practices. Please read the process.
https://intelpedia.intel.com/Intel_Patch_Review_Process#Review_Process_for_x86_submissions
-----Original Message-----
From: Lobakin, Alexandr <[email protected]>
Sent: Tuesday, August 31, 2021 7:41 AM
To: [email protected]
Cc: Accardi, Kristen C <[email protected]>; Kristen Carlson Accardi <[email protected]>; Kees Cook <[email protected]>; Masahiro Yamada <[email protected]>; H. Peter Anvin <[email protected]>; Jessica Yu <[email protected]>; Nathan Chancellor <[email protected]>; Nick Desaulniers <[email protected]>; Marios Pomonis <[email protected]>; Sami Tolvanen <[email protected]>; Luck, Tony <[email protected]>; Ard Biesheuvel <[email protected]>; Brandeburg, Jesse <[email protected]>; Czapnik, Lukasz <[email protected]>; Plantykow, Marta A <[email protected]>; Kubiak, Michal <[email protected]>; Swiatkowski, Michal <[email protected]>; Lobakin, Alexandr <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v6 kspp-next 00/22] Function Granular KASLR
This is a massive rework and a respin of Kristen Accardi's marvellous FG-KASLR series (v5).
The major differences since v5 [0]:
* You can now tune the number of functions per each section to achieve the preferable vmlinux size or protection level. Default is still as one section per each function.
This can be handy for storage-constrained systems. 4-8 fps are still strong, but reduce the size of the final vmlinu{x,z} significantly;
* I don't use orphan sections anymore. It's not reliable at all / may differ from linker to linker, and also conflicts with CONFIG_LD_ORPHAN_WARN which is great for catching random bugs ->
* All the .text.* sections are now being described explicitly in the linker script. A Perl script is used to take the original LDS, the original object file, read a list of input sections from it and generate the resulting LDS.
This costs a bit of linking time as LD tends to think hard when processing scripts > 1 Mb. It adds about 40-60 seconds to the whole linking process (BTF step, 2-3 kallsyms steps and the final step), but "better safe than sorry".
In addition, that approach allows to reserve some space at the end and add some link assertions ->
* Input .text section now must be empty, otherwise the linkage will be stopped. This is implemented by the size assertion in the resulting LD script and is designed to plug the potentional layout leakage. This also means that ->
* "Regular" ASM functions are now being placed into unique separate functions the same way compiler does this for C functions. This is achieved by introducing and using several new macros which take the symbol name as a base for its new section name. This gives a better opportunity to both DCE and FG-KASLR, as ASM code now can also be randomized or garbage-collected;
* It's now fully compatible with ClangLTO, ClangCFI, CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last revision was published;
* Includes several fixes: relocations inside .altinstr_replacement code and minor issues found and/or suggested by LKP robot.
The series was compile-time and runtime tested on the following setups with no issues:
- x86_64, GCC 11, Binutils 2.35;
- x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).
The first 4 patches are from the linux-kbuild tree and included to avoid merge conflicts and non-intuitive resolving of them.
The series is also available here: [1]
[0] https://lore.kernel.org/kernel-hardening/[email protected]
[1] https://github.com/alobakin/linux/pull/3
The original v5 cover letter:
Function Granular Kernel Address Space Layout Randomization (fgkaslr)
---------------------------------------------------------------------
This patch set is an implementation of finer grained kernel address space randomization. It rearranges your kernel code at load time on a per-function level granularity, with only around a second added to boot time.
Changes in v5:
--------------
* fixed a bug in the code which increases boot heap size for CONFIG_FG_KASLR which prevented the boot heap from being increased for CONFIG_FG_KASLR when using bzip2 compression. Thanks to Andy Lavr for finding the problem and identifying the solution.
* changed the adjustment of the orc_unwind_ip table at boot time to disregard relocs associated with this table, and instead inspect the entries separately. Relocs are not able to be used since they are no longer correct once the table is resorted at buildtime.
* changed how orc_unwind_ip addresses in randomized sections are identified to include the byte immediately after the end of the section.
* updated module code to use kvmalloc/kvfree based on suggestions from Evgenii Shatokhin <[email protected]>.
* changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to match the nokaslr option. fgkaslr="X" can be added at a later date if it is needed.
* Added a patch to force livepatch to require symbols to be unique if using while fgkaslr either for core or modules.
Changes in v4:
-------------
* dropped the patch to split out change to STATIC definition in x86/boot/compressed/misc.c and replaced with a patch authored by Kees Cook to avoid the duplicate malloc definitions
* Added a section to Documentation/admin-guide/kernel-parameters.txt to document the fgkaslr boot option.
* redesigned the patch to hide the new layout when reading /proc/kallsyms. The previous implementation utilized a dynamically allocated linked list to display the kernel and module symbols in alphabetical order. The new implementation uses a randomly shuffled index array to display the kernel and module symbols in a random order.
Changes in v3:
-------------
* Makefile changes to accommodate CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
* removal of extraneous ALIGN_PAGE from _etext changes
* changed variable names in x86/tools/relocs to be less confusing
* split out change to STATIC definition in x86/boot/compressed/misc.c
* Updates to Documentation to make it more clear what is preserved in .text
* much more detailed commit message for function granular KASLR patch
* minor tweaks and changes that make for more readable code
* this cover letter updated slightly to add additional details
Changes in v2:
--------------
* Fix to address i386 build failure
* Allow module reordering patch to be configured separately so that arm (or other non-x86_64 arches) can take advantage of module function reordering. This support has not be tested by me, but smoke tested by Ard Biesheuvel <[email protected]> on arm.
* Fix build issue when building on arm as reported by
Ard Biesheuvel <[email protected]>
Patches to objtool are included because they are dependencies for this patchset, however they have been submitted by their maintainer separately.
Background
----------
KASLR was merged into the kernel with the objective of increasing the difficulty of code reuse attacks. Code reuse attacks reused existing code snippets to get around existing memory protections. They exploit software bugs which expose addresses of useful code snippets to control the flow of execution for their own nefarious purposes. KASLR moves the entire kernel code text as a unit at boot time in order to make addresses less predictable. The order of the code within the segment is unchanged - only the base address is shifted. There are a few shortcomings to this algorithm.
1. Low Entropy - there are only so many locations the kernel can fit in. This means an attacker could guess without too much trouble.
2. Knowledge of a single address can reveal the offset of the base address, exposing all other locations for a published/known kernel image.
3. Info leaks abound.
Finer grained ASLR has been proposed as a way to make ASLR more resistant to info leaks. It is not a new concept at all, and there are many variations possible. Function reordering is an implementation of finer grained ASLR which randomizes the layout of an address space on a function level granularity. We use the term "fgkaslr" in this document to refer to the technique of function reordering when used with KASLR, as well as finer grained KASLR in general.
Proposed Improvement
--------------------
This patch set proposes adding function reordering on top of the existing KASLR base address randomization. The over-arching objective is incremental improvement over what we already have. It is designed to work in combination with the existing solution. The implementation is really pretty simple, and there are 2 main area where changes occur:
* Build time
GCC has had an option to place functions into individual .text sections for many years now. This option can be used to implement function reordering at load time. The final compiled vmlinux retains all the section headers, which can be used to help find the address ranges of each function. Using this information and an expanded table of relocation addresses, individual text sections can be suffled immediately after decompression. Some data tables inside the kernel that have assumptions about order require re-sorting after being updated when applying relocations. In order to modify these tables, a few key symbols are excluded from the objcopy symbol stripping process for use after shuffling the text segments.
Some highlights from the build time changes to look for:
The top level kernel Makefile was modified to add the gcc flag if it is supported. Currently, I am applying this flag to everything it is possible to randomize. Anything that is written in C and not present in a special input section is randomized. The final binary segment 0 retains a consolidated .text section, as well as all the individual .text.* sections. Future work could turn off this flags for selected files or even entire subsystems, although obviously at the cost of security.
The relocs tool is updated to add relative relocations. This information previously wasn't included because it wasn't necessary when moving the entire .text segment as a unit.
A new file was created to contain a list of symbols that objcopy should keep. We use those symbols at load time as described below.
* Load time
The boot kernel was modified to parse the vmlinux elf file after decompression to check for our interesting symbols that we kept, and to look for any .text.* sections to randomize. The consolidated .text section is skipped and not moved. The sections are shuffled randomly, and copied into memory following the .text section in a new random order. The existing code which updated relocation addresses was modified to account for not just a fixed delta from the load address, but the offset that the function section was moved to. This requires inspection of each address to see if it was impacted by a randomization. We use a bsearch to make this less horrible on performance. Any tables that need to be modified with new addresses or resorted are updated using the symbol addresses parsed from the elf symbol table.
In order to hide our new layout, symbols reported through /proc/kallsyms will be displayed in a random order.
Security Considerations
-----------------------
The objective of this patch set is to improve a technology that is already merged into the kernel (KASLR). This code will not prevent all attacks, but should instead be considered as one of several tools that can be used. In particular, this code is meant to make KASLR more effective in the presence of info leaks.
How much entropy we are adding to the existing entropy of standard KASLR will depend on a few variables. Firstly and most obviously, the number of functions that are randomized matters. This implementation keeps the existing .text section for code that cannot be randomized - for example, because it was assembly code. The less sections to randomize, the less entropy. In addition, due to alignment (16 bytes for x86_64), the number of bits in a address that the attacker needs to guess is reduced, as the lower bits are identical.
Performance Impact
------------------
There are two areas where function reordering can impact performance: boot time latency, and run time performance.
* Boot time latency
This implementation of finer grained KASLR impacts the boot time of the kernel in several places. It requires additional parsing of the kernel ELF file to obtain the section headers of the sections to be randomized. It calls the random number generator for each section to be randomized to determine that section's new memory location. It copies the decompressed kernel into a new area of memory to avoid corruption when laying out the newly randomized sections. It increases the number of relocations the kernel has to perform at boot time vs. standard KASLR, and it also requires a lookup on each address that needs to be relocated to see if it was in a randomized section and needs to be adjusted by a new offset. Finally, it re-sorts a few data tables that are required to be sorted by address.
Booting a test VM on a modern, well appointed system showed an increase in latency of approximately 1 second.
* Run time
The performance impact at run-time of function reordering varies by workload.
Using kcbench, a kernel compilation benchmark, the performance of a kernel build with finer grained KASLR was about 1% slower than a kernel with standard KASLR. Analysis with perf showed a slightly higher percentage of L1-icache-load-misses. Other workloads were examined as well, with varied results. Some workloads performed significantly worse under FGKASLR, while others stayed the same or were mysteriously better. In general, it will depend on the code flow whether or not finer grained KASLR will impact your workload, and how the underlying code was designed. Because the layout changes per boot, each time a system is rebooted the performance of a workload may change.
Future work could identify hot areas that may not be randomized and either leave them in the .text section or group them together into a single section that may be randomized. If grouping things together helps, one other thing to consider is that if we could identify text blobs that should be grouped together to benefit a particular code flow, it could be interesting to explore whether this security feature could be also be used as a performance feature if you are interested in optimizing your kernel layout for a particular workload at boot time. Optimizing function layout for a particular workload has been researched and proven effective - for more information read the Facebook paper "Optimizing Function Placement for Large-Scale Data-Center Applications" (see references section below).
Image Size
----------
Adding additional section headers as a result of compiling with -ffunction-sections will increase the size of the vmlinux ELF file.
With a standard distro config, the resulting vmlinux was increased by about 3%. The compressed image is also increased due to the header files, as well as the extra relocations that must be added. You can expect fgkaslr to increase the size of the compressed image by about 15%.
Memory Usage
------------
fgkaslr increases the amount of heap that is required at boot time, although this extra memory is released when the kernel has finished decompression. As a result, it may not be appropriate to use this feature on systems without much memory.
Building
--------
To enable fine grained KASLR, you need to have the following config options set (including all the ones you would use to build normal KASLR)
CONFIG_FG_KASLR=y
In addition, fgkaslr is only supported for the X86_64 architecture.
Modules
-------
Modules are randomized similarly to the rest of the kernel by shuffling the sections at load time prior to moving them into memory. The module must also have been build with the -ffunction-sections compiler option.
Although fgkaslr for the kernel is only supported for the X86_64 architecture, it is possible to use fgkaslr with modules on other architectures. To enable this feature, select
CONFIG_MODULE_FG_KASLR=y
This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.
Disabling
---------
Disabling normal KASLR using the nokaslr command line option also disables fgkaslr. It is also possible to disable fgkaslr separately by booting with nofgkaslr on the commandline.
References
----------
There are a lot of academic papers which explore finer grained ASLR.
This paper in particular contributed the most to my implementation design as well as my overall understanding of the problem space:
Selfrando: Securing the Tor Browser against De-anonymization Exploits, M. Conti, S. Crane, T. Frassetto, et al.
For more information on how function layout impacts performance, see:
Optimizing Function Placement for Large-Scale Data-Center Applications, G. Ottoni, B. Maher
Alexander Lobakin (7):
linkage: add macros for putting ASM functions into own sections
x86: conditionally place regular ASM functions into separate sections
FG-KASLR: use a scripted approach to handle .text.* sections x86/boot: allow FG-KASLR to be selected
arm64/crypto: conditionally place ASM functions into separate sections
module: use a scripted approach for FG-KASLR
maintainers: add MAINTAINERS entry for FG-KASLR
Kees Cook (2):
x86/boot: Allow a "silent" kaslr random byte fetch x86/boot/compressed: Avoid duplicate malloc() implementations
Kristen Carlson Accardi (9):
x86: tools/relocs: Support >64K section headers
x86: Makefile: Add build and config option for CONFIG_FG_KASLR Make sure ORC lookup covers the entire _etext - _stext x86/tools: Add relative relocs for randomized functions
x86: Add support for function granular KASLR
kallsyms: Hide layout
livepatch: only match unique symbols when using fgkaslr
module: Reorder functions
Documentation: add a documentation for FG-KASLR
Masahiro Yamada (3):
kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
kbuild: merge vmlinux_link() between ARCH=um and other architectures
Sami Tolvanen (1):
kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG
.../admin-guide/kernel-parameters.txt | 6 + Documentation/security/fgkaslr.rst | 172 ++++ Documentation/security/index.rst | 1 + MAINTAINERS | 12 + Makefile | 17 +- arch/Kconfig | 3 + arch/arm64/crypto/aes-ce-ccm-core.S | 16 +- arch/arm64/crypto/aes-ce-core.S | 16 +- arch/arm64/crypto/aes-ce.S | 4 +- arch/arm64/crypto/aes-cipher-core.S | 8 +- arch/arm64/crypto/aes-modes.S | 16 +- arch/arm64/crypto/aes-neon.S | 4 +- arch/arm64/crypto/aes-neonbs-core.S | 38 +- arch/arm64/crypto/chacha-neon-core.S | 18 +- arch/arm64/crypto/crct10dif-ce-core.S | 14 +- arch/arm64/crypto/ghash-ce-core.S | 24 +- arch/arm64/crypto/nh-neon-core.S | 4 +- arch/arm64/crypto/poly1305-armv8.pl | 17 + arch/arm64/crypto/sha1-ce-core.S | 4 +- arch/arm64/crypto/sha2-ce-core.S | 4 +- arch/arm64/crypto/sha3-ce-core.S | 4 +- arch/arm64/crypto/sha512-armv8.pl | 11 + arch/arm64/crypto/sha512-ce-core.S | 4 +- arch/arm64/crypto/sm3-ce-core.S | 4 +- arch/arm64/crypto/sm4-ce-core.S | 4 +- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 9 +- arch/x86/boot/compressed/fgkaslr.c | 905 ++++++++++++++++++ arch/x86/boot/compressed/kaslr.c | 4 - arch/x86/boot/compressed/misc.c | 157 ++- arch/x86/boot/compressed/misc.h | 30 + arch/x86/boot/compressed/utils.c | 13 + arch/x86/boot/compressed/vmlinux.symbols | 19 + arch/x86/crypto/aegis128-aesni-asm.S | 36 +- arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 12 +- arch/x86/crypto/aesni-intel_asm.S | 116 ++- arch/x86/crypto/aesni-intel_avx-x86_64.S | 32 +- arch/x86/crypto/blake2s-core.S | 8 +- arch/x86/crypto/blowfish-x86_64-asm_64.S | 16 +- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 28 +- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 28 +- arch/x86/crypto/camellia-x86_64-asm_64.S | 16 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 24 +- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/chacha-avx2-x86_64.S | 12 +- arch/x86/crypto/chacha-avx512vl-x86_64.S | 12 +- arch/x86/crypto/chacha-ssse3-x86_64.S | 16 +- arch/x86/crypto/crc32-pclmul_asm.S | 4 +- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 4 +- arch/x86/crypto/crct10dif-pcl-asm_64.S | 4 +- arch/x86/crypto/des3_ede-asm_64.S | 8 +- arch/x86/crypto/ghash-clmulni-intel_asm.S | 12 +- arch/x86/crypto/nh-avx2-x86_64.S | 4 +- arch/x86/crypto/nh-sse2-x86_64.S | 4 +- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 8 +- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/serpent-avx2-asm_64.S | 20 +- arch/x86/crypto/serpent-sse2-i586-asm_32.S | 8 +- arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 8 +- arch/x86/crypto/sha1_avx2_x86_64_asm.S | 4 +- arch/x86/crypto/sha1_ni_asm.S | 4 +- arch/x86/crypto/sha1_ssse3_asm.S | 4 +- arch/x86/crypto/sha256-avx-asm.S | 4 +- arch/x86/crypto/sha256-avx2-asm.S | 4 +- arch/x86/crypto/sha256-ssse3-asm.S | 4 +- arch/x86/crypto/sha256_ni_asm.S | 4 +- arch/x86/crypto/sha512-avx-asm.S | 4 +- arch/x86/crypto/sha512-avx2-asm.S | 4 +- arch/x86/crypto/sha512-ssse3-asm.S | 4 +- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 20 +- arch/x86/crypto/twofish-i586-asm_32.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 8 +- arch/x86/crypto/twofish-x86_64-asm_64.S | 8 +- arch/x86/entry/entry_32.S | 24 +- arch/x86/entry/entry_64.S | 18 +- arch/x86/entry/thunk_32.S | 4 +- arch/x86/entry/thunk_64.S | 8 +- arch/x86/include/asm/boot.h | 13 +- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 2 +- arch/x86/kernel/acpi/wakeup_32.S | 9 +- arch/x86/kernel/acpi/wakeup_64.S | 10 +- arch/x86/kernel/ftrace_32.S | 19 +- arch/x86/kernel/ftrace_64.S | 28 +- arch/x86/kernel/irqflags.S | 4 +- arch/x86/kernel/kprobes/core.c | 3 +- arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/relocate_kernel_32.S | 2 + arch/x86/kernel/relocate_kernel_64.S | 2 + arch/x86/kernel/vmlinux.lds.S | 6 +- arch/x86/kvm/emulate.c | 2 +- arch/x86/kvm/vmx/vmenter.S | 8 +- arch/x86/lib/clear_page_64.S | 12 +- arch/x86/lib/cmpxchg16b_emu.S | 4 +- arch/x86/lib/copy_mc_64.S | 8 +- arch/x86/lib/copy_page_64.S | 7 +- arch/x86/lib/copy_user_64.S | 18 +- arch/x86/lib/csum-copy_64.S | 4 +- arch/x86/lib/error-inject.c | 3 +- arch/x86/lib/getuser.S | 37 +- arch/x86/lib/hweight.S | 9 +- arch/x86/lib/iomap_copy_64.S | 4 +- arch/x86/lib/kaslr.c | 18 +- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 12 +- arch/x86/lib/msr-reg.S | 8 +- arch/x86/lib/putuser.S | 18 +- arch/x86/mm/mem_encrypt_boot.S | 8 +- arch/x86/platform/efi/efi_stub_64.S | 4 +- arch/x86/platform/efi/efi_thunk_64.S | 4 +- arch/x86/power/hibernate_asm_32.S | 14 +- arch/x86/power/hibernate_asm_64.S | 14 +- arch/x86/tools/relocs.c | 135 ++- arch/x86/tools/relocs.h | 4 +- arch/x86/tools/relocs_common.c | 15 +- arch/x86/xen/xen-asm.S | 49 +- arch/x86/xen/xen-head.S | 10 +- include/asm-generic/vmlinux.lds.h | 41 +- include/linux/decompress/mm.h | 12 +- include/linux/linkage.h | 76 ++ include/uapi/linux/elf.h | 1 + init/Kconfig | 51 + kernel/kallsyms.c | 158 ++- kernel/livepatch/core.c | 11 + kernel/module.c | 91 +- scripts/Makefile.build | 27 +- scripts/Makefile.lib | 7 + scripts/Makefile.modfinal | 36 +- scripts/Makefile.modpost | 22 +- scripts/gen_autoksyms.sh | 12 - scripts/generate_text_sections.pl | 149 +++ scripts/link-vmlinux.sh | 104 +- scripts/module.lds.S | 14 +- 133 files changed, 2771 insertions(+), 757 deletions(-) create mode 100644 Documentation/security/fgkaslr.rst create mode 100644 arch/x86/boot/compressed/fgkaslr.c create mode 100644 arch/x86/boot/compressed/utils.c create mode 100644 arch/x86/boot/compressed/vmlinux.symbols create mode 100755 scripts/generate_text_sections.pl
--
2.31.1