Compile-time stack metadata validation

This is v13 of the compile-time stack metadata validation patch set, along with proposed fixes for many of the warnings it found. It's based on the tip/master branch.

v12 can be found here:

https://lkml.kernel.org/r/[email protected]

For more information about the motivation behind this patch set, and more details about what it does, see the patch 4 changelog and tools/stacktool/Documentation/stack-validation.txt.

Patch 1 fixes the formatting for the "make -C tools" help message.

Patches 2-6 add stacktool and some related macros.

Patches 7-21 are some proposed fixes for several of the warnings reported by stacktool. They've been compile-tested and boot-tested in a VM, but I haven't attempted any meaningful testing for many of them.

Patches 22-23 are some additional warning fixes from Chris J Arges.

v13:
- LDFLAGS order fix from Chris J Arges
- new warning fix patches from Chris J Arges
- "--frame-pointer" -> "--check-frame-pointer"

v12:
- rename "stackvalidate" -> "stacktool"
- move from scripts/ to tools/:
- makefile rework
- make a copy of the x86 insn code (and warn if the code diverges)
- use tools/include/linux/list.h
- move warning macros to a new warn.h file
- change wording: "stack validation" -> "stack metadata validation"

v11:
- attempt to answer the "why" question better in the documentation and commit message
- s/FP_SAVE/FRAME_BEGIN/ in documentation

v10:
- add scripts/mod to directory ignores
- remove circular dependencies for ignored objects which are built before stackvalidate
- fix CONFIG_MODVERSIONS incompatibility

v9:
- rename FRAME/ENDFRAME -> FRAME_BEGIN/FRAME_END
- fix jump table issue for when the original instruction is a jump
- drop paravirt thunk alignment patch
- add maintainers to CC for proposed warning fixes

v8:
- add proposed fixes for warnings
- fix all memory leaks
- process ignores earlier and add more ignore checks
- always assume POPCNT alternative is enabled
- drop hweight inline asm fix
- drop __schedule() ignore patch
- change .Ltemp_\@ to .Lstackvalidate_ignore_\@ in asm macro
- fix CONFIG_* checks in asm macros
- add C versions of ignore macros and frame macros
- change ";" to "\n" in C macros
- add ifdef CONFIG_STACK_VALIDATION checks in C ignore macros
- use numbered label in C ignore macro
- add missing break in switch case statement in arch-x86.c

v7:
- sibling call support
- document proposed solution for inline asm() frame pointer issues
- say "kernel entry/exit" instead of "context switch"
- clarify the checking of switch statement jump tables
- discard __stackvalidate_ignore_* sections in linker script
- use .Ltemp_\@ to get a unique label instead of static 3-digit number
- change STACKVALIDATE_IGNORE_FUNC variable to a static
- move STACKVALIDATE_IGNORE_INSN to arch-specific .h file

v6:
- rename asmvalidate -> stackvalidate (again)
- gcc-generated object file support
- recursive branch state analysis
- external jump support
- fixup/exception table support
- jump label support
- switch statement jump table support
- added documentation
- detection of "noreturn" dead end functions
- added a Kbuild mechanism for skipping files and dirs
- moved frame pointer macros to arch/x86/include/asm/frame.h
- moved ignore macros to include/linux/stackvalidate.h

v5:
- stackvalidate -> asmvalidate
- frame pointers only required for non-leaf functions
- check for the use of the FP_SAVE/RESTORE macros instead of manually analyzing code to detect frame pointer usage
- additional checks to ensure each function doesn't leave its boundaries
- make the macros simpler and more flexible
- support for analyzing ALTERNATIVE macros
- simplified the arch interfaces in scripts/asmvalidate/arch.h
- fixed some asmvalidate warnings
- rebased onto latest tip asm cleanups
- many more small changes

v4:
- Changed the default to CONFIG_STACK_VALIDATION=n, until all the asm code can get cleaned up.
- Fixed a stackvalidate error path exit code issue found by Michal Marek.

v3:
- Added a patch to make the push/pop CFI macros arch-independent, as suggested by H. Peter Anvin

v2:
- Fixed memory leaks reported by Petr Mladek

Chris J Arges (2):
x86/uaccess: Add stack frame output operand in get_user inline asm x86/stacktool: Ignore head_$(BITS) files.

Josh Poimboeuf (21):
tools: Fix formatting of the "make -C tools" help message x86/asm: Frame pointer macro cleanup
x86/asm: Add C versions of frame pointer macros x86/stacktool: Compile-time stack metadata validation x86/stacktool: Add file and directory ignores x86/stacktool: Add ignore macros
x86/xen: Add stack frame dependency to hypercall inline asm calls x86/paravirt: Add stack frame dependency to PVOP inline asm calls x86/paravirt: Create a stack frame in PV_CALLEE_SAVE_REGS_THUNK x86/amd: Set ELF function type for vide()
x86/reboot: Add ljmp instructions to stacktool whitelist x86/xen: Add xen_cpuid() and xen_setup_gdt() to stacktool whitelists x86/asm/crypto: Create stack frames in aesni-intel_asm.S x86/asm/crypto: Move .Lbswap_mask data to .rodata section x86/asm/crypto: Move jump_table to .rodata section x86/asm/crypto: Create stack frames in clmul_ghash_mul/update() x86/asm/entry: Create stack frames in thunk functions x86/asm/acpi: Create a stack frame in do_suspend_lowlevel() x86/asm: Create stack frames in rwsem functions x86/asm/efi: Create a stack frame in efi_call() x86/asm/power: Create stack frames in hibernate_asm_64.S

 MAINTAINERS                                        |   7 +
 Makefile                                           |   5 +-
 arch/Kconfig                                       |   6 +
 arch/x86/Kconfig                                   |   1 +
 arch/x86/boot/Makefile                             |   3 +-
 arch/x86/boot/compressed/Makefile                  |   3 +-
 arch/x86/crypto/aesni-intel_asm.S                  |  21 +
 arch/x86/crypto/crc32c-pcl-intel-asm_64.S          |   8 +-
 arch/x86/crypto/ghash-clmulni-intel_asm.S          |   5 +
 arch/x86/entry/thunk_64.S                          |   4 +
 arch/x86/entry/vdso/Makefile                       |   5 +-
 arch/x86/include/asm/frame.h                       |  51 +-
 arch/x86/include/asm/paravirt.h                    |   9 +-
 arch/x86/include/asm/paravirt_types.h              |  18 +-
 arch/x86/include/asm/stacktool.h                   |  45 +
 arch/x86/include/asm/uaccess.h                     |   5 +-
 arch/x86/include/asm/xen/hypercall.h               |   5 +-
 arch/x86/kernel/Makefile                           |   1 +
 arch/x86/kernel/acpi/wakeup_64.S                   |   3 +
 arch/x86/kernel/cpu/amd.c                          |   5 +-
 arch/x86/kernel/reboot.c                           |   7 +-
 arch/x86/kernel/vmlinux.lds.S                      |   5 +-
 arch/x86/lib/rwsem.S                               |  11 +-
 arch/x86/platform/efi/efi_stub_64.S                |   3 +
 arch/x86/power/hibernate_asm_64.S                  |   7 +
 arch/x86/purgatory/Makefile                        |   2 +
 arch/x86/realmode/Makefile                         |   4 +-
 arch/x86/realmode/rm/Makefile                      |   3 +-
 arch/x86/xen/enlighten.c                           |   4 +-
 drivers/firmware/efi/libstub/Makefile              |   1 +
 include/linux/stacktool.h                          |  29 +
 lib/Kconfig.debug                                  |  12 +
 scripts/Makefile.build                             |  37 +-
 scripts/mod/Makefile                               |   2 +
 tools/Makefile                                     |  43 +-
 tools/stacktool/.gitignore                         |   1 +
 tools/stacktool/Build                              |   4 +
 tools/stacktool/Documentation/stack-validation.txt | 336 +++++++
 tools/stacktool/Makefile                           |  51 ++
 tools/stacktool/arch.h                             |  44 +
 tools/stacktool/arch/x86/Build                     |  12 +
 tools/stacktool/arch/x86/decode.c                  | 163 ++++
 .../stacktool/arch/x86/insn/gen-insn-attr-x86.awk  | 387 ++++++++
 tools/stacktool/arch/x86/insn/inat.c               |  97 ++
 tools/stacktool/arch/x86/insn/inat.h               | 221 +++++
 tools/stacktool/arch/x86/insn/inat_types.h         |  29 +
 tools/stacktool/arch/x86/insn/insn.c               | 594 +++++++++++++
 tools/stacktool/arch/x86/insn/insn.h               | 201 +++++
 tools/stacktool/arch/x86/insn/x86-opcode-map.txt   | 984 +++++++++++++++++++++
 tools/stacktool/elf.c                              | 403 +++++++++
 tools/stacktool/elf.h                              |  79 ++
 tools/stacktool/special.c                          | 199 +++++
 tools/stacktool/special.h                          |  42 +
 tools/stacktool/stacktool.c                        | 980 ++++++++++++++++++++
 tools/stacktool/warn.h                             |  60 ++
 55 files changed, 5196 insertions(+), 71 deletions(-)
 create mode 100644 arch/x86/include/asm/stacktool.h
 create mode 100644 include/linux/stacktool.h
 create mode 100644 tools/stacktool/.gitignore
 create mode 100644 tools/stacktool/Build
 create mode 100644 tools/stacktool/Documentation/stack-validation.txt
 create mode 100644 tools/stacktool/Makefile
 create mode 100644 tools/stacktool/arch.h
 create mode 100644 tools/stacktool/arch/x86/Build
 create mode 100644 tools/stacktool/arch/x86/decode.c
 create mode 100644 tools/stacktool/arch/x86/insn/gen-insn-attr-x86.awk
 create mode 100644 tools/stacktool/arch/x86/insn/inat.c
 create mode 100644 tools/stacktool/arch/x86/insn/inat.h
 create mode 100644 tools/stacktool/arch/x86/insn/inat_types.h
 create mode 100644 tools/stacktool/arch/x86/insn/insn.c
 create mode 100644 tools/stacktool/arch/x86/insn/insn.h
 create mode 100644 tools/stacktool/arch/x86/insn/x86-opcode-map.txt
 create mode 100644 tools/stacktool/elf.c
 create mode 100644 tools/stacktool/elf.h
 create mode 100644 tools/stacktool/special.c
 create mode 100644 tools/stacktool/special.h
 create mode 100644 tools/stacktool/stacktool.c
 create mode 100644 tools/stacktool/warn.h

2.4.3

Align the x86_energy_perf_policy line with the others and restore the original alphabetical sorting.

---
 tools/Makefile | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)

index d6f307d..c7400a6 100644
@@ -8,23 +8,23 @@ include scripts/Makefile.include
 help:
 	@echo 'Possible targets:'
 	@echo ''
-	@echo '  acpi       - ACPI tools'
-	@echo '  cgroup     - cgroup tools'
-	@echo '  cpupower   - a tool for all things x86 CPU power'
-	@echo '  firewire   - the userspace part of nosy, an IEEE-1394 traffic sniffer'
-	@echo '  hv         - tools used when in Hyper-V clients'
-	@echo '  iio        - IIO tools'
-	@echo '  lguest     - a minimal 32-bit x86 hypervisor'
-	@echo '  perf       - Linux performance measurement and analysis tool'
-	@echo '  selftests  - various kernel selftests'
-	@echo '  turbostat  - Intel CPU idle stats and freq reporting tool'
-	@echo '  usb        - USB testing tools'
-	@echo '  virtio     - vhost test module'
-	@echo '  net        - misc networking tools'
-	@echo '  vm         - misc vm tools'
+	@echo '  acpi                   - ACPI tools'
+	@echo '  cgroup                 - cgroup tools'
+	@echo '  cpupower               - a tool for all things x86 CPU power'
+	@echo '  firewire               - the userspace part of nosy, an IEEE-1394 traffic sniffer'
+	@echo '  freefall               - laptop accelerometer program for disk protection'
+	@echo '  hv                     - tools used when in Hyper-V clients'
+	@echo '  iio                    - IIO tools'
+	@echo '  lguest                 - a minimal 32-bit x86 hypervisor'
+	@echo '  net                    - misc networking tools'
+	@echo '  perf                   - Linux performance measurement and analysis tool'
+	@echo '  selftests              - various kernel selftests'
+	@echo '  tmon                   - thermal monitoring and tuning tool'
+	@echo '  turbostat              - Intel CPU idle stats and freq reporting tool'
+	@echo '  usb                    - USB testing tools'
+	@echo '  virtio                 - vhost test module'
+	@echo '  vm                     - misc vm tools'
 	@echo '  x86_energy_perf_policy - Intel energy policy tool'
-	@echo '  tmon       - thermal monitoring and tuning tool'
-	@echo '  freefall   - laptop accelerometer program for disk protection'
 	@echo ''
 	@echo 'You can do:'
 	@echo ' $$ make -C tools/ <tool>_install'

2.4.3

The asm macros for setting up and restoring the frame pointer aren't currently being used. However, they will be needed soon to help asm functions to comply with stacktool.

Rename FRAME/ENDFRAME to FRAME_BEGIN/FRAME_END for more symmetry. Also make the code more readable and improve the comments.

---
 arch/x86/include/asm/frame.h | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)

index 793179c..8a6cd26 100644
@@ -1,23 +1,26 @@
+#ifndef _ASM_X86_FRAME_H
+#define _ASM_X86_FRAME_H
+
 #ifdef __ASSEMBLY__
 
 #include <asm/asm.h>
 
-/* The annotation hides the frame from the unwinder and makes it look
-   like a ordinary ebp save/restore. This avoids some special cases for
-   frame pointer later */
+/*
+ * These are stack frame creation macros.  They should be used by every
+ * callable non-leaf asm function to make kernel stack traces more reliable.
+ */
+.macro FRAME_BEGIN
+#ifdef CONFIG_FRAME_POINTER
+	push %_ASM_BP
+	_ASM_MOV %_ASM_SP, %_ASM_BP
+#endif
+.endm
+
+.macro FRAME_END
 #ifdef CONFIG_FRAME_POINTER
-	.macro FRAME
-	__ASM_SIZE(push,)	%__ASM_REG(bp)
-	__ASM_SIZE(mov)		%__ASM_REG(sp), %__ASM_REG(bp)
-	.endm
-	.macro ENDFRAME
-	__ASM_SIZE(pop,)	%__ASM_REG(bp)
-	.endm
-#else
-	.macro FRAME
-	.endm
-	.macro ENDFRAME
-	.endm
+	pop %_ASM_BP
 #endif
+.endm
 
 #endif  /*  __ASSEMBLY__  */
+#endif /* _ASM_X86_FRAME_H */

2.4.3

Add C versions of the frame pointer macros which can be used to create a stack frame in inline assembly.

---
 arch/x86/include/asm/frame.h | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

index 8a6cd26..9a30ec7 100644
@@ -1,10 +1,10 @@
 #ifndef _ASM_X86_FRAME_H
 #define _ASM_X86_FRAME_H
 
-#ifdef __ASSEMBLY__
-
 #include <asm/asm.h>
 
+#ifdef __ASSEMBLY__
+
 /*
  * These are stack frame creation macros.  They should be used by every
  * callable non-leaf asm function to make kernel stack traces more reliable.

@@ -22,5 +22,21 @@
 #endif
 .endm
 
+#else /* !__ASSEMBLY__ */
+
+#ifdef CONFIG_FRAME_POINTER
+
+#define FRAME_BEGIN				\
+	"push %" _ASM_BP "\n"			\
+	_ASM_MOV "%" _ASM_SP ", %" _ASM_BP "\n"
+
+#define FRAME_END "pop %" _ASM_BP "\n"
+
+#else /* !CONFIG_FRAME_POINTER */
+
+#define FRAME_BEGIN ""
+#define FRAME_END ""
+
+#endif /* CONFIG_FRAME_POINTER */
 #endif  /*  __ASSEMBLY__  */
 #endif /* _ASM_X86_FRAME_H */

2.4.3

This adds a CONFIG_STACK_VALIDATION option which enables a host tool named stacktool which runs at compile time. It analyzes every .o file and ensures the validity of its stack metadata. It enforces a set of rules on asm code and C inline assembly code so that stack traces can be reliable.

For each function, it recursively follows all possible code paths and validates the correct frame pointer state at each instruction.

It also follows code paths involving special sections, like .altinstructions, __jump_table, and __ex_table, which can add alternative execution paths to a given instruction (or set of instructions). Similarly, it knows how to follow switch statements, for which gcc sometimes uses jump tables.

Here are some of the benefits of validating stack metadata:

a) More reliable stack traces for frame pointer enabled kernels

Frame pointers are used for debugging purposes. They allow runtime code and debug tools to be able to walk the stack to determine the chain of function call sites that led to the currently executing code.

For some architectures, frame pointers are enabled by CONFIG_FRAME_POINTER. For some other architectures they may be required by the ABI (sometimes referred to as "backchain pointers").

For C code, gcc automatically generates instructions for setting up frame pointers when the -fno-omit-frame-pointer option is used.

But for asm code, the frame setup instructions have to be written by hand, which most people don't do. So the end result is that CONFIG_FRAME_POINTER is honored for C code but not for most asm code.

For stack traces based on frame pointers to be reliable, all functions which call other functions must first create a stack frame and update the frame pointer. If a first function doesn't properly create a stack frame before calling a second function, the *caller* of the first function will be skipped on the stack trace.

For example, consider the following example backtrace with frame pointers enabled:

     [<ffffffff81812584>] dump_stack+0x4b/0x63
     [<ffffffff812d6dc2>] cmdline_proc_show+0x12/0x30
     [<ffffffff8127f568>] seq_read+0x108/0x3e0
     [<ffffffff812cce62>] proc_reg_read+0x42/0x70
     [<ffffffff81256197>] __vfs_read+0x37/0x100
     [<ffffffff81256b16>] vfs_read+0x86/0x130
     [<ffffffff81257898>] SyS_read+0x58/0xd0
     [<ffffffff8181c1f2>] entry_SYSCALL_64_fastpath+0x12/0x76

It correctly shows that the caller of cmdline_proc_show() is seq_read().

If we remove the frame pointer logic from cmdline_proc_show() by replacing the frame pointer related instructions with nops, here's what it looks like instead:

     [<ffffffff81812584>] dump_stack+0x4b/0x63
     [<ffffffff812d6dc2>] cmdline_proc_show+0x12/0x30
     [<ffffffff812cce62>] proc_reg_read+0x42/0x70
     [<ffffffff81256197>] __vfs_read+0x37/0x100
     [<ffffffff81256b16>] vfs_read+0x86/0x130
     [<ffffffff81257898>] SyS_read+0x58/0xd0
     [<ffffffff8181c1f2>] entry_SYSCALL_64_fastpath+0x12/0x76

Notice that cmdline_proc_show()'s caller, seq_read(), has been skipped. Instead the stack trace seems to show that cmdline_proc_show() was called by proc_reg_read().

The benefit of stacktool here is that because it ensures that *all* functions honor CONFIG_FRAME_POINTER, no functions will ever[*] be skipped on a stack trace.

[*] unless an interrupt or exception has occurred at the very
beginning of a function before the stack frame has been created, or at the very end of the function after the stack frame has been destroyed. This is an inherent limitation of frame pointers.

b) 100% reliable stack traces for DWARF enabled kernels

This is not yet implemented. For more details about what is planned, see tools/stacktool/Documentation/stack-validation.txt.

c) Higher live patching compatibility rate

This is not yet implemented. For more details about what is planned, see tools/stacktool/Documentation/stack-validation.txt.

To achieve the validation, stacktool enforces the following rules:

1. Each callable function must be annotated as such with the ELF function type. In asm code, this is typically done using the ENTRY/ENDPROC macros. If stacktool finds a return instruction outside of a function, it flags an error since that usually indicates callable code which should be annotated accordingly.

This rule is needed so that stacktool can properly identify each callable function in order to analyze its stack metadata.

2. Conversely, each section of code which is *not* callable should *not* be annotated as an ELF function. The ENDPROC macro shouldn't be used in this case.

This rule is needed so that stacktool can ignore non-callable code. Such code doesn't have to follow any of the other rules.

3. Each callable function which calls another function must have the correct frame pointer logic, if required by CONFIG_FRAME_POINTER or the architecture's back chain rules. This can by done in asm code with the FRAME_BEGIN/FRAME_END macros.

This rule ensures that frame pointer based stack traces will work as designed. If function A doesn't create a stack frame before calling function B, the _caller_ of function A will be skipped on the stack trace.

4. Dynamic jumps and jumps to undefined symbols are only allowed if:

a) the jump is part of a switch statement; or

b) the jump matches sibling call semantics and the frame pointer has the same value it had on function entry.

This rule is needed so that stacktool can reliably analyze all of a function's code paths. If a function jumps to code in another file, and it's not a sibling call, stacktool has no way to follow the jump because it only analyzes a single file at a time.

5. A callable function may not execute kernel entry/exit instructions. The only code which needs such instructions is kernel entry code, which shouldn't be be in callable functions anyway.

This rule is just a sanity check to ensure that callable functions return normally.

It currently only supports x86_64. I tried to make the code generic so that support for other architectures can hopefully be plugged in relatively easily.

As a first step, CONFIG_STACK_VALIDATION is disabled by default, and all reported non-compliances result in warnings. Once we get them all cleaned up, we can change the default to CONFIG_STACK_VALIDATION=y and change the warnings to errors to keep the stack metadata clean.

On my Lenovo laptop with a i7-4810MQ 4-core/8-thread CPU, building the kernel with it enabled adds about three seconds of total build time. It hasn't been optimized for performance yet, so there are probably some opportunities for better build performance.

---
 MAINTAINERS                                        |   7 +
 Makefile                                           |   5 +-
 arch/Kconfig                                       |   6 +
 arch/x86/Kconfig                                   |   1 +
 lib/Kconfig.debug                                  |  12 +
 scripts/Makefile.build                             |  37 +-
 tools/Makefile                                     |  11 +-
 tools/stacktool/.gitignore                         |   1 +
 tools/stacktool/Build                              |   4 +
 tools/stacktool/Documentation/stack-validation.txt | 336 +++++++
 tools/stacktool/Makefile                           |  51 ++
 tools/stacktool/arch.h                             |  44 +
 tools/stacktool/arch/x86/Build                     |  12 +
 tools/stacktool/arch/x86/decode.c                  | 163 ++++
 .../stacktool/arch/x86/insn/gen-insn-attr-x86.awk  | 387 ++++++++
 tools/stacktool/arch/x86/insn/inat.c               |  97 ++
 tools/stacktool/arch/x86/insn/inat.h               | 221 +++++
 tools/stacktool/arch/x86/insn/inat_types.h         |  29 +
 tools/stacktool/arch/x86/insn/insn.c               | 594 +++++++++++++
 tools/stacktool/arch/x86/insn/insn.h               | 201 +++++
 tools/stacktool/arch/x86/insn/x86-opcode-map.txt   | 984 +++++++++++++++++++++
 tools/stacktool/elf.c                              | 403 +++++++++
 tools/stacktool/elf.h                              |  79 ++
 tools/stacktool/special.c                          | 199 +++++
 tools/stacktool/special.h                          |  42 +
 tools/stacktool/stacktool.c                        | 980 ++++++++++++++++++++
 tools/stacktool/warn.h                             |  60 ++
 27 files changed, 4956 insertions(+), 10 deletions(-)
 create mode 100644 tools/stacktool/.gitignore
 create mode 100644 tools/stacktool/Build
 create mode 100644 tools/stacktool/Documentation/stack-validation.txt
 create mode 100644 tools/stacktool/Makefile
 create mode 100644 tools/stacktool/arch.h
 create mode 100644 tools/stacktool/arch/x86/Build
 create mode 100644 tools/stacktool/arch/x86/decode.c
 create mode 100644 tools/stacktool/arch/x86/insn/gen-insn-attr-x86.awk
 create mode 100644 tools/stacktool/arch/x86/insn/inat.c
 create mode 100644 tools/stacktool/arch/x86/insn/inat.h
 create mode 100644 tools/stacktool/arch/x86/insn/inat_types.h
 create mode 100644 tools/stacktool/arch/x86/insn/insn.c
 create mode 100644 tools/stacktool/arch/x86/insn/insn.h
 create mode 100644 tools/stacktool/arch/x86/insn/x86-opcode-map.txt
 create mode 100644 tools/stacktool/elf.c
 create mode 100644 tools/stacktool/elf.h
 create mode 100644 tools/stacktool/special.c
 create mode 100644 tools/stacktool/special.h
 create mode 100644 tools/stacktool/stacktool.c
 create mode 100644 tools/stacktool/warn.h

index ea3e3ae..a128801 100644
@@ -9879,6 +9879,13 @@ L:	[email protected]
 S:	Supported
 F:	Documentation/stable_kernel_rules.txt
 
+STACK METADATA VALIDATION
+M:	Josh Poimboeuf <[email protected]>
+S:	Supported
+F:	tools/stacktool/
+F:	include/linux/stacktool.h
+F:	arch/x86/include/asm/stacktool.h
+
 STAGING SUBSYSTEM
 M:	Greg Kroah-Hartman <[email protected]>
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git

index 84f4b31..0b9a1ca 100644
@@ -984,7 +984,10 @@ prepare0: archprepare FORCE
 	$(Q)$(MAKE) $(build)=.
 
 # All the preparing..
-prepare: prepare0
+prepare: prepare0 prepare-stacktool
+
+PHONY += prepare-stacktool
+prepare-stacktool: $(if $(CONFIG_STACK_VALIDATION), tools/stacktool FORCE)
 
 # Generate some files
 # ---------------------------------------------------------------------------

index 671810c..b20f472 100644
@@ -527,6 +527,12 @@ config HAVE_COPY_THREAD_TLS
 	  normal C parameter passing, rather than extracting the syscall
 	  argument from pt_regs.
 
+config HAVE_STACK_VALIDATION
+	bool
+	help
+	  Architecture supports the stacktool host tool, which adds
+	  compile-time stack metadata validation.
+
 #
 # ABI hall of shame
 #

index d8546d4..006b4af 100644
@@ -153,6 +153,7 @@ config X86
 	select VIRT_TO_BUS
 	select X86_DEV_DMA_OPS			if X86_64
 	select X86_FEATURE_NAMES		if PROC_FS
+	select HAVE_STACK_VALIDATION		if X86_64
 
 config INSTRUCTION_DECODER
 	def_bool y

index ab76b99..24d2e88 100644
@@ -332,6 +332,18 @@ config FRAME_POINTER
 	  larger and slower, but it gives very useful debugging information
 	  in case of kernel bugs. (precise oopses/stacktraces/warnings)
 
+config STACK_VALIDATION
+	bool "Enable compile-time stack metadata validation"
+	depends on HAVE_STACK_VALIDATION
+	default n
+	help
+	  Add compile-time checks to validate stack metadata, including frame
+	  pointers (if CONFIG_FRAME_POINTER is enabled).  This helps ensure
+	  that runtime stack traces are more reliable.
+
+	  For more information, see
+	  tools/stacktool/Documentation/stack-validation.txt.
+
 config DEBUG_FORCE_WEAK_PER_CPU
 	bool "Force weak per-cpu definitions"
 	depends on DEBUG_KERNEL

index 01df30a..3ffd994 100644
@@ -241,10 +241,30 @@ cmd_record_mcount =						\
 	fi;
 endif
 
+ifdef CONFIG_STACK_VALIDATION
+
+__stacktool_obj := $(objtree)/tools/stacktool/stacktool
+
+ifdef CONFIG_FRAME_POINTER
+stacktool_args += --check-frame-pointer
+endif
+
+# Set STACKTOOL_foo.o=n to skip stack metadata validation for a file.
+# Set STACKTOOL=n to skip stack metadata validation for a directory.
+stacktool_obj = $(if $(patsubst n%,, \
+	$(STACKTOOL_$(basetarget).o)$(STACKTOOL)y), \
+	$(__stacktool_obj))
+cmd_stacktool = $(if $(patsubst n%,, \
+	$(STACKTOOL_$(basetarget).o)$(STACKTOOL)y), \
+	$(__stacktool_obj) $(stacktool_args) "$(@)";)
+
+endif # CONFIG_STACK_VALIDATION
+
 define rule_cc_o_c
 	$(call echo-cmd,checksrc) $(cmd_checksrc)			  \
 	$(call echo-cmd,cc_o_c) $(cmd_cc_o_c);				  \
 	$(cmd_modversions)						  \
+	$(cmd_stacktool)						  \
 	$(call echo-cmd,record_mcount)					  \
 	$(cmd_record_mcount)						  \
 	scripts/basic/fixdep $(depfile) $@ '$(call make-cmd,cc_o_c)' >    \

@@ -253,14 +273,23 @@ define rule_cc_o_c
 	mv -f $(dot-target).tmp $(dot-target).cmd
 endef
 
+define rule_as_o_S
+	$(call echo-cmd,as_o_S) $(cmd_as_o_S);				  \
+	$(cmd_stacktool)						  \
+	scripts/basic/fixdep $(depfile) $@ '$(call make-cmd,as_o_S)' >    \
+	                                              $(dot-target).tmp;  \
+	rm -f $(depfile);						  \
+	mv -f $(dot-target).tmp $(dot-target).cmd
+endef
+
 # Built-in and composite module parts
-$(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
+$(obj)/%.o: $(src)/%.c $(recordmcount_source) $(stacktool_obj) FORCE
 	$(call cmd,force_checksrc)
 	$(call if_changed_rule,cc_o_c)
 
 # Single-part modules are special since we need to mark them in $(MODVERDIR)
 
-$(single-used-m): $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
+$(single-used-m): $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(stacktool_obj) FORCE
 	$(call cmd,force_checksrc)
 	$(call if_changed_rule,cc_o_c)
 	@{ echo $(@:.o=.ko); echo $@; } > $(MODVERDIR)/$(@F:.o=.mod)

@@ -290,8 +319,8 @@ $(obj)/%.s: $(src)/%.S FORCE
 quiet_cmd_as_o_S = AS $(quiet_modtag)  $@
 cmd_as_o_S       = $(CC) $(a_flags) -c -o $@ $<
 
-$(obj)/%.o: $(src)/%.S FORCE
-	$(call if_changed_dep,as_o_S)
+$(obj)/%.o: $(src)/%.S $(stacktool_obj) FORCE
+	$(call if_changed_rule,as_o_S)
 
 targets += $(real-objs-y) $(real-objs-m) $(lib-y)
 targets += $(extra-y) $(MAKECMDGOALS) $(always)

index c7400a6..39d5d60 100644
@@ -19,6 +19,7 @@ help:
 	@echo '  net                    - misc networking tools'
 	@echo '  perf                   - Linux performance measurement and analysis tool'
 	@echo '  selftests              - various kernel selftests'
+	@echo '  stacktool              - a stack metadata validation tool'
 	@echo '  tmon                   - thermal monitoring and tuning tool'
 	@echo '  turbostat              - Intel CPU idle stats and freq reporting tool'
 	@echo '  usb                    - USB testing tools'

@@ -48,7 +49,7 @@ acpi: FORCE
 cpupower: FORCE
 	$(call descend,power/$@)
 
-cgroup firewire hv guest usb virtio vm net iio: FORCE
+cgroup firewire hv guest usb virtio vm net iio stacktool: FORCE
 	$(call descend,$@)
 
 liblockdep: FORCE

@@ -83,7 +84,7 @@ acpi_install:
 cpupower_install:
 	$(call descend,power/$(@:_install=),install)
 
-cgroup_install firewire_install hv_install lguest_install perf_install usb_install virtio_install vm_install net_install:
+cgroup_install firewire_install hv_install lguest_install perf_install usb_install virtio_install vm_install net_install stacktool_install:
 	$(call descend,$(@:_install=),install)
 
 selftests_install:

@@ -101,7 +102,7 @@ freefall_install:
 install: acpi_install cgroup_install cpupower_install hv_install firewire_install lguest_install \
 		perf_install selftests_install turbostat_install usb_install \
 		virtio_install vm_install net_install x86_energy_perf_policy_install \
-		tmon freefall_install
+		tmon freefall_install stacktool_install
 
 acpi_clean:
 	$(call descend,power/acpi,clean)

@@ -109,7 +110,7 @@ acpi_clean:
 cpupower_clean:
 	$(call descend,power/cpupower,clean)
 
-cgroup_clean hv_clean firewire_clean lguest_clean usb_clean virtio_clean vm_clean net_clean iio_clean:
+cgroup_clean hv_clean firewire_clean lguest_clean usb_clean virtio_clean vm_clean net_clean iio_clean stacktool_clean:
 	$(call descend,$(@:_clean=),clean)
 
 liblockdep_clean:

@@ -136,6 +137,6 @@ freefall_clean:
 clean: acpi_clean cgroup_clean cpupower_clean hv_clean firewire_clean lguest_clean \
 		perf_clean selftests_clean turbostat_clean usb_clean virtio_clean \
 		vm_clean net_clean iio_clean x86_energy_perf_policy_clean tmon_clean \
-		freefall_clean
+		freefall_clean stacktool_clean
 
 .PHONY: FORCE

new file mode 100644
index 0000000..a4014ec
@@ -0,0 +1 @@
+arch/x86/insn/inat-tables.c

new file mode 100644
index 0000000..7688d3d
@@ -0,0 +1,4 @@
+stacktool-y += stacktool.o
+stacktool-y += elf.o
+stacktool-y += special.o
+stacktool-y += arch/$(ARCH)/

new file mode 100644
index 0000000..9b9777d
@@ -0,0 +1,12 @@
+stacktool-y += decode.o
+
+inat_tables_script = arch/x86/insn/gen-insn-attr-x86.awk
+inat_tables_maps = arch/x86/insn/x86-opcode-map.txt
+
+$(OUTPUT)arch/x86/insn/inat-tables.c: $(inat_tables_script) $(inat_tables_maps)
+	$(call rule_mkdir)
+	$(Q)$(call echo-cmd,gen)$(AWK) -f $(inat_tables_script) $(inat_tables_maps) > $@
+
+$(OUTPUT)arch/x86/decode.o: $(OUTPUT)arch/x86/insn/inat-tables.c
+
+CFLAGS_decode.o += -I$(OUTPUT)arch/x86/insn

new file mode 100644
index 0000000..cb3c20c
@@ -0,0 +1,29 @@
+#ifndef _ASM_X86_INAT_TYPES_H
+#define _ASM_X86_INAT_TYPES_H
+/*
+ * x86 instruction attributes
+ *
+ * Written by Masami Hiramatsu <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ */
+
+/* Instruction attributes */
+typedef unsigned int insn_attr_t;
+typedef unsigned char insn_byte_t;
+typedef signed int insn_value_t;
+
+#endif

2.4.3

Part of the diff is not shown here. View the raw message.

On 09/22/2015, 05:46 PM, Josh Poimboeuf wrote:
...

...

...

     __THROW __attribute_pure__ __nonnull ((1));

...

...

js
suse labs

I erased too much here. Index is a local var here:

        int index, ret;

js
suse labs

Thanks Jiri! Here's the patch with s/index/idx/.

---8<---

Subject: [PATCH v13.1 04/23] x86/stacktool: Compile-time stack metadata validation

This adds a CONFIG_STACK_VALIDATION option which enables a host tool named stacktool which runs at compile time. It analyzes every .o file and ensures the validity of its stack metadata. It enforces a set of rules on asm code and C inline assembly code so that stack traces can be reliable.

For each function, it recursively follows all possible code paths and validates the correct frame pointer state at each instruction.

It also follows code paths involving special sections, like .altinstructions, __jump_table, and __ex_table, which can add alternative execution paths to a given instruction (or set of instructions). Similarly, it knows how to follow switch statements, for which gcc sometimes uses jump tables.

Here are some of the benefits of validating stack metadata:

a) More reliable stack traces for frame pointer enabled kernels

Frame pointers are used for debugging purposes. They allow runtime code and debug tools to be able to walk the stack to determine the chain of function call sites that led to the currently executing code.

For some architectures, frame pointers are enabled by CONFIG_FRAME_POINTER. For some other architectures they may be required by the ABI (sometimes referred to as "backchain pointers").

For C code, gcc automatically generates instructions for setting up frame pointers when the -fno-omit-frame-pointer option is used.

But for asm code, the frame setup instructions have to be written by hand, which most people don't do. So the end result is that CONFIG_FRAME_POINTER is honored for C code but not for most asm code.

For stack traces based on frame pointers to be reliable, all functions which call other functions must first create a stack frame and update the frame pointer. If a first function doesn't properly create a stack frame before calling a second function, the *caller* of the first function will be skipped on the stack trace.

For example, consider the following example backtrace with frame pointers enabled:

     [<ffffffff81812584>] dump_stack+0x4b/0x63
     [<ffffffff812d6dc2>] cmdline_proc_show+0x12/0x30
     [<ffffffff8127f568>] seq_read+0x108/0x3e0
     [<ffffffff812cce62>] proc_reg_read+0x42/0x70
     [<ffffffff81256197>] __vfs_read+0x37/0x100
     [<ffffffff81256b16>] vfs_read+0x86/0x130
     [<ffffffff81257898>] SyS_read+0x58/0xd0
     [<ffffffff8181c1f2>] entry_SYSCALL_64_fastpath+0x12/0x76

It correctly shows that the caller of cmdline_proc_show() is seq_read().

If we remove the frame pointer logic from cmdline_proc_show() by replacing the frame pointer related instructions with nops, here's what it looks like instead:

     [<ffffffff81812584>] dump_stack+0x4b/0x63
     [<ffffffff812d6dc2>] cmdline_proc_show+0x12/0x30
     [<ffffffff812cce62>] proc_reg_read+0x42/0x70
     [<ffffffff81256197>] __vfs_read+0x37/0x100
     [<ffffffff81256b16>] vfs_read+0x86/0x130
     [<ffffffff81257898>] SyS_read+0x58/0xd0
     [<ffffffff8181c1f2>] entry_SYSCALL_64_fastpath+0x12/0x76

Notice that cmdline_proc_show()'s caller, seq_read(), has been skipped. Instead the stack trace seems to show that cmdline_proc_show() was called by proc_reg_read().

The benefit of stacktool here is that because it ensures that *all* functions honor CONFIG_FRAME_POINTER, no functions will ever[*] be skipped on a stack trace.

[*] unless an interrupt or exception has occurred at the very
beginning of a function before the stack frame has been created, or at the very end of the function after the stack frame has been destroyed. This is an inherent limitation of frame pointers.

b) 100% reliable stack traces for DWARF enabled kernels

This is not yet implemented. For more details about what is planned, see tools/stacktool/Documentation/stack-validation.txt.

c) Higher live patching compatibility rate

This is not yet implemented. For more details about what is planned, see tools/stacktool/Documentation/stack-validation.txt.

To achieve the validation, stacktool enforces the following rules:

1. Each callable function must be annotated as such with the ELF function type. In asm code, this is typically done using the ENTRY/ENDPROC macros. If stacktool finds a return instruction outside of a function, it flags an error since that usually indicates callable code which should be annotated accordingly.

This rule is needed so that stacktool can properly identify each callable function in order to analyze its stack metadata.

2. Conversely, each section of code which is *not* callable should *not* be annotated as an ELF function. The ENDPROC macro shouldn't be used in this case.

This rule is needed so that stacktool can ignore non-callable code. Such code doesn't have to follow any of the other rules.

3. Each callable function which calls another function must have the correct frame pointer logic, if required by CONFIG_FRAME_POINTER or the architecture's back chain rules. This can by done in asm code with the FRAME_BEGIN/FRAME_END macros.

This rule ensures that frame pointer based stack traces will work as designed. If function A doesn't create a stack frame before calling function B, the _caller_ of function A will be skipped on the stack trace.

4. Dynamic jumps and jumps to undefined symbols are only allowed if:

a) the jump is part of a switch statement; or

b) the jump matches sibling call semantics and the frame pointer has the same value it had on function entry.

This rule is needed so that stacktool can reliably analyze all of a function's code paths. If a function jumps to code in another file, and it's not a sibling call, stacktool has no way to follow the jump because it only analyzes a single file at a time.

5. A callable function may not execute kernel entry/exit instructions. The only code which needs such instructions is kernel entry code, which shouldn't be be in callable functions anyway.

This rule is just a sanity check to ensure that callable functions return normally.

It currently only supports x86_64. I tried to make the code generic so that support for other architectures can hopefully be plugged in relatively easily.

As a first step, CONFIG_STACK_VALIDATION is disabled by default, and all reported non-compliances result in warnings. Once we get them all cleaned up, we can change the default to CONFIG_STACK_VALIDATION=y and change the warnings to errors to keep the stack metadata clean.

On my Lenovo laptop with a i7-4810MQ 4-core/8-thread CPU, building the kernel with it enabled adds about three seconds of total build time. It hasn't been optimized for performance yet, so there are probably some opportunities for better build performance.

---
 MAINTAINERS                                        |   7 +
 Makefile                                           |   5 +-
 arch/Kconfig                                       |   6 +
 arch/x86/Kconfig                                   |   1 +
 lib/Kconfig.debug                                  |  12 +
 scripts/Makefile.build                             |  37 +-
 tools/Makefile                                     |  11 +-
 tools/stacktool/.gitignore                         |   1 +
 tools/stacktool/Build                              |   4 +
 tools/stacktool/Documentation/stack-validation.txt | 336 +++++++
 tools/stacktool/Makefile                           |  51 ++
 tools/stacktool/arch.h                             |  44 +
 tools/stacktool/arch/x86/Build                     |  12 +
 tools/stacktool/arch/x86/decode.c                  | 163 ++++
 .../stacktool/arch/x86/insn/gen-insn-attr-x86.awk  | 387 ++++++++
 tools/stacktool/arch/x86/insn/inat.c               |  97 ++
 tools/stacktool/arch/x86/insn/inat.h               | 221 +++++
 tools/stacktool/arch/x86/insn/inat_types.h         |  29 +
 tools/stacktool/arch/x86/insn/insn.c               | 594 +++++++++++++
 tools/stacktool/arch/x86/insn/insn.h               | 201 +++++
 tools/stacktool/arch/x86/insn/x86-opcode-map.txt   | 984 +++++++++++++++++++++
 tools/stacktool/elf.c                              | 403 +++++++++
 tools/stacktool/elf.h                              |  79 ++
 tools/stacktool/special.c                          | 199 +++++
 tools/stacktool/special.h                          |  42 +
 tools/stacktool/stacktool.c                        | 980 ++++++++++++++++++++
 tools/stacktool/warn.h                             |  60 ++
 27 files changed, 4956 insertions(+), 10 deletions(-)
 create mode 100644 tools/stacktool/.gitignore
 create mode 100644 tools/stacktool/Build
 create mode 100644 tools/stacktool/Documentation/stack-validation.txt
 create mode 100644 tools/stacktool/Makefile
 create mode 100644 tools/stacktool/arch.h
 create mode 100644 tools/stacktool/arch/x86/Build
 create mode 100644 tools/stacktool/arch/x86/decode.c
 create mode 100644 tools/stacktool/arch/x86/insn/gen-insn-attr-x86.awk
 create mode 100644 tools/stacktool/arch/x86/insn/inat.c
 create mode 100644 tools/stacktool/arch/x86/insn/inat.h
 create mode 100644 tools/stacktool/arch/x86/insn/inat_types.h
 create mode 100644 tools/stacktool/arch/x86/insn/insn.c
 create mode 100644 tools/stacktool/arch/x86/insn/insn.h
 create mode 100644 tools/stacktool/arch/x86/insn/x86-opcode-map.txt
 create mode 100644 tools/stacktool/elf.c
 create mode 100644 tools/stacktool/elf.h
 create mode 100644 tools/stacktool/special.c
 create mode 100644 tools/stacktool/special.h
 create mode 100644 tools/stacktool/stacktool.c
 create mode 100644 tools/stacktool/warn.h

index ea3e3ae..a128801 100644
@@ -9879,6 +9879,13 @@ L:	[email protected]
 S:	Supported
 F:	Documentation/stable_kernel_rules.txt
 
+STACK METADATA VALIDATION
+M:	Josh Poimboeuf <[email protected]>
+S:	Supported
+F:	tools/stacktool/
+F:	include/linux/stacktool.h
+F:	arch/x86/include/asm/stacktool.h
+
 STAGING SUBSYSTEM
 M:	Greg Kroah-Hartman <[email protected]>
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git

index 84f4b31..0b9a1ca 100644
@@ -984,7 +984,10 @@ prepare0: archprepare FORCE
 	$(Q)$(MAKE) $(build)=.
 
 # All the preparing..
-prepare: prepare0
+prepare: prepare0 prepare-stacktool
+
+PHONY += prepare-stacktool
+prepare-stacktool: $(if $(CONFIG_STACK_VALIDATION), tools/stacktool FORCE)
 
 # Generate some files
 # ---------------------------------------------------------------------------

index 671810c..b20f472 100644
@@ -527,6 +527,12 @@ config HAVE_COPY_THREAD_TLS
 	  normal C parameter passing, rather than extracting the syscall
 	  argument from pt_regs.
 
+config HAVE_STACK_VALIDATION
+	bool
+	help
+	  Architecture supports the stacktool host tool, which adds
+	  compile-time stack metadata validation.
+
 #
 # ABI hall of shame
 #

index d8546d4..006b4af 100644
@@ -153,6 +153,7 @@ config X86
 	select VIRT_TO_BUS
 	select X86_DEV_DMA_OPS			if X86_64
 	select X86_FEATURE_NAMES		if PROC_FS
+	select HAVE_STACK_VALIDATION		if X86_64
 
 config INSTRUCTION_DECODER
 	def_bool y

index ab76b99..24d2e88 100644
@@ -332,6 +332,18 @@ config FRAME_POINTER
 	  larger and slower, but it gives very useful debugging information
 	  in case of kernel bugs. (precise oopses/stacktraces/warnings)
 
+config STACK_VALIDATION
+	bool "Enable compile-time stack metadata validation"
+	depends on HAVE_STACK_VALIDATION
+	default n
+	help
+	  Add compile-time checks to validate stack metadata, including frame
+	  pointers (if CONFIG_FRAME_POINTER is enabled).  This helps ensure
+	  that runtime stack traces are more reliable.
+
+	  For more information, see
+	  tools/stacktool/Documentation/stack-validation.txt.
+
 config DEBUG_FORCE_WEAK_PER_CPU
 	bool "Force weak per-cpu definitions"
 	depends on DEBUG_KERNEL

index 01df30a..3ffd994 100644
@@ -241,10 +241,30 @@ cmd_record_mcount =						\
 	fi;
 endif
 
+ifdef CONFIG_STACK_VALIDATION
+
+__stacktool_obj := $(objtree)/tools/stacktool/stacktool
+
+ifdef CONFIG_FRAME_POINTER
+stacktool_args += --check-frame-pointer
+endif
+
+# Set STACKTOOL_foo.o=n to skip stack metadata validation for a file.
+# Set STACKTOOL=n to skip stack metadata validation for a directory.
+stacktool_obj = $(if $(patsubst n%,, \
+	$(STACKTOOL_$(basetarget).o)$(STACKTOOL)y), \
+	$(__stacktool_obj))
+cmd_stacktool = $(if $(patsubst n%,, \
+	$(STACKTOOL_$(basetarget).o)$(STACKTOOL)y), \
+	$(__stacktool_obj) $(stacktool_args) "$(@)";)
+
+endif # CONFIG_STACK_VALIDATION
+
 define rule_cc_o_c
 	$(call echo-cmd,checksrc) $(cmd_checksrc)			  \
 	$(call echo-cmd,cc_o_c) $(cmd_cc_o_c);				  \
 	$(cmd_modversions)						  \
+	$(cmd_stacktool)						  \
 	$(call echo-cmd,record_mcount)					  \
 	$(cmd_record_mcount)						  \
 	scripts/basic/fixdep $(depfile) $@ '$(call make-cmd,cc_o_c)' >    \

@@ -253,14 +273,23 @@ define rule_cc_o_c
 	mv -f $(dot-target).tmp $(dot-target).cmd
 endef
 
+define rule_as_o_S
+	$(call echo-cmd,as_o_S) $(cmd_as_o_S);				  \
+	$(cmd_stacktool)						  \
+	scripts/basic/fixdep $(depfile) $@ '$(call make-cmd,as_o_S)' >    \
+	                                              $(dot-target).tmp;  \
+	rm -f $(depfile);						  \
+	mv -f $(dot-target).tmp $(dot-target).cmd
+endef
+
 # Built-in and composite module parts
-$(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
+$(obj)/%.o: $(src)/%.c $(recordmcount_source) $(stacktool_obj) FORCE
 	$(call cmd,force_checksrc)
 	$(call if_changed_rule,cc_o_c)
 
 # Single-part modules are special since we need to mark them in $(MODVERDIR)
 
-$(single-used-m): $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
+$(single-used-m): $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(stacktool_obj) FORCE
 	$(call cmd,force_checksrc)
 	$(call if_changed_rule,cc_o_c)
 	@{ echo $(@:.o=.ko); echo $@; } > $(MODVERDIR)/$(@F:.o=.mod)

@@ -290,8 +319,8 @@ $(obj)/%.s: $(src)/%.S FORCE
 quiet_cmd_as_o_S = AS $(quiet_modtag)  $@
 cmd_as_o_S       = $(CC) $(a_flags) -c -o $@ $<
 
-$(obj)/%.o: $(src)/%.S FORCE
-	$(call if_changed_dep,as_o_S)
+$(obj)/%.o: $(src)/%.S $(stacktool_obj) FORCE
+	$(call if_changed_rule,as_o_S)
 
 targets += $(real-objs-y) $(real-objs-m) $(lib-y)
 targets += $(extra-y) $(MAKECMDGOALS) $(always)

index c7400a6..39d5d60 100644
@@ -19,6 +19,7 @@ help:
 	@echo '  net                    - misc networking tools'
 	@echo '  perf                   - Linux performance measurement and analysis tool'
 	@echo '  selftests              - various kernel selftests'
+	@echo '  stacktool              - a stack metadata validation tool'
 	@echo '  tmon                   - thermal monitoring and tuning tool'
 	@echo '  turbostat              - Intel CPU idle stats and freq reporting tool'
 	@echo '  usb                    - USB testing tools'

@@ -48,7 +49,7 @@ acpi: FORCE
 cpupower: FORCE
 	$(call descend,power/$@)
 
-cgroup firewire hv guest usb virtio vm net iio: FORCE
+cgroup firewire hv guest usb virtio vm net iio stacktool: FORCE
 	$(call descend,$@)
 
 liblockdep: FORCE

@@ -83,7 +84,7 @@ acpi_install:
 cpupower_install:
 	$(call descend,power/$(@:_install=),install)
 
-cgroup_install firewire_install hv_install lguest_install perf_install usb_install virtio_install vm_install net_install:
+cgroup_install firewire_install hv_install lguest_install perf_install usb_install virtio_install vm_install net_install stacktool_install:
 	$(call descend,$(@:_install=),install)
 
 selftests_install:

@@ -101,7 +102,7 @@ freefall_install:
 install: acpi_install cgroup_install cpupower_install hv_install firewire_install lguest_install \
 		perf_install selftests_install turbostat_install usb_install \
 		virtio_install vm_install net_install x86_energy_perf_policy_install \
-		tmon freefall_install
+		tmon freefall_install stacktool_install
 
 acpi_clean:
 	$(call descend,power/acpi,clean)

@@ -109,7 +110,7 @@ acpi_clean:
 cpupower_clean:
 	$(call descend,power/cpupower,clean)
 
-cgroup_clean hv_clean firewire_clean lguest_clean usb_clean virtio_clean vm_clean net_clean iio_clean:
+cgroup_clean hv_clean firewire_clean lguest_clean usb_clean virtio_clean vm_clean net_clean iio_clean stacktool_clean:
 	$(call descend,$(@:_clean=),clean)
 
 liblockdep_clean:

@@ -136,6 +137,6 @@ freefall_clean:
 clean: acpi_clean cgroup_clean cpupower_clean hv_clean firewire_clean lguest_clean \
 		perf_clean selftests_clean turbostat_clean usb_clean virtio_clean \
 		vm_clean net_clean iio_clean x86_energy_perf_policy_clean tmon_clean \
-		freefall_clean
+		freefall_clean stacktool_clean
 
 .PHONY: FORCE

new file mode 100644
index 0000000..a4014ec
@@ -0,0 +1 @@
+arch/x86/insn/inat-tables.c

new file mode 100644
index 0000000..7688d3d
@@ -0,0 +1,4 @@
+stacktool-y += stacktool.o
+stacktool-y += elf.o
+stacktool-y += special.o
+stacktool-y += arch/$(ARCH)/

new file mode 100644
index 0000000..9b9777d
@@ -0,0 +1,12 @@
+stacktool-y += decode.o
+
+inat_tables_script = arch/x86/insn/gen-insn-attr-x86.awk
+inat_tables_maps = arch/x86/insn/x86-opcode-map.txt
+
+$(OUTPUT)arch/x86/insn/inat-tables.c: $(inat_tables_script) $(inat_tables_maps)
+	$(call rule_mkdir)
+	$(Q)$(call echo-cmd,gen)$(AWK) -f $(inat_tables_script) $(inat_tables_maps) > $@
+
+$(OUTPUT)arch/x86/decode.o: $(OUTPUT)arch/x86/insn/inat-tables.c
+
+CFLAGS_decode.o += -I$(OUTPUT)arch/x86/insn

new file mode 100644
index 0000000..cb3c20c
@@ -0,0 +1,29 @@
+#ifndef _ASM_X86_INAT_TYPES_H
+#define _ASM_X86_INAT_TYPES_H
+/*
+ * x86 instruction attributes
+ *
+ * Written by Masami Hiramatsu <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ */
+
+/* Instruction attributes */
+typedef unsigned int insn_attr_t;
+typedef unsigned char insn_byte_t;
+typedef signed int insn_value_t;
+
+#endif

2.4.3

Part of the diff is not shown here. View the raw message.

<snip>

Why do these files essentially need to be duplicated? Would it be possible to point Make at the files that are already in ../../arch/x86/*?

--chris j arges

It's just a warning, not an error, right? Just want to make sure it didn't fail the build. The files are in sync on tip/master.

They're duplicated because the "tools" dir is supposed to be source independent from the rest of the kernel tree.

Josh

Correct just a warning. And once I rebased correctly things are in sync. Thanks and sorry for the noise.

--chris

Tell stacktool to skip validation of the following code:

- boot image
- vdso image
- kexec purgatory
- realmode
- efi libstub
- scripts/mod

They all run outside the kernel's normal mode of operation and they don't affect runtime kernel stack traces, so they're free to skirt the stacktool rules.

---
 arch/x86/boot/Makefile                | 3 ++-
 arch/x86/boot/compressed/Makefile     | 3 ++-
 arch/x86/entry/vdso/Makefile          | 5 ++++-
 arch/x86/purgatory/Makefile           | 2 ++
 arch/x86/realmode/Makefile            | 4 +++-
 arch/x86/realmode/rm/Makefile         | 3 ++-
 drivers/firmware/efi/libstub/Makefile | 1 +
 scripts/mod/Makefile                  | 2 ++
 8 files changed, 18 insertions(+), 5 deletions(-)

index 0d553e5..fe96b2f 100644
@@ -14,7 +14,8 @@
 # Set it to -DSVGA_MODE=NORMAL_VGA if you just want the EGA/VGA mode.
 # The number is the same as you would ordinarily press at bootup.
 
-KASAN_SANITIZE := n
+KASAN_SANITIZE	:= n
+STACKTOOL	:= n
 
 SVGA_MODE	:= -DSVGA_MODE=NORMAL_VGA
 

index 0a291cd..8cea814 100644
@@ -16,7 +16,8 @@
 #	(see scripts/Makefile.lib size_append)
 #	compressed vmlinux.bin.all + u32 size of vmlinux.bin.all
 
-KASAN_SANITIZE := n
+KASAN_SANITIZE	:= n
+STACKTOOL	:= n
 
 targets := vmlinux vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma \
 	vmlinux.bin.xz vmlinux.bin.lzo vmlinux.bin.lz4

index a3d0767..8095694 100644
@@ -3,7 +3,9 @@
 #
 
 KBUILD_CFLAGS += $(DISABLE_LTO)
-KASAN_SANITIZE := n
+
+KASAN_SANITIZE	:= n
+STACKTOOL	:= n
 
 VDSO64-$(CONFIG_X86_64)		:= y
 VDSOX32-$(CONFIG_X86_X32_ABI)	:= y

@@ -15,6 +17,7 @@ vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o
 
 # files to link into kernel
 obj-y				+= vma.o
+STACKTOOL_vma.o			:= y
 
 # vDSO images to build
 vdso_img-$(VDSO64-y)		+= 64

index 2c835e3..30d7d58 100644
@@ -1,3 +1,5 @@
+STACKTOOL := n
+
 purgatory-y := purgatory.o stack.o setup-x86_$(BITS).o sha256.o entry64.o string.o
 
 targets += $(purgatory-y)

index e02c2c6..0c24689 100644
@@ -6,7 +6,9 @@
 # for more details.
 #
 #
-KASAN_SANITIZE := n
+KASAN_SANITIZE	:= n
+STACKTOOL	:= n
+
 subdir- := rm
 
 obj-y += init.o

index 2730d77..1da2e5b 100644
@@ -6,7 +6,8 @@
 # for more details.
 #
 #
-KASAN_SANITIZE := n
+KASAN_SANITIZE	:= n
+STACKTOOL	:= n
 
 always := realmode.bin realmode.relocs
 

index 816dbe9..a612cdb 100644
@@ -20,6 +20,7 @@ KBUILD_CFLAGS			:= $(cflags-y) \
 
 GCOV_PROFILE			:= n
 KASAN_SANITIZE			:= n
+STACKTOOL			:= n
 
 lib-y				:= efi-stub-helper.o
 lib-$(CONFIG_EFI_ARMSTUB)	+= arm-stub.o fdt.o

index c11212f..496184d 100644
@@ -1,3 +1,5 @@
+STACKTOOL	:= n
+
 hostprogs-y	:= modpost mk_elfconfig
 always		:= $(hostprogs-y) empty.o
 

2.4.3

Add new stacktool ignore macros: STACKTOOL_IGNORE_INSN and STACKTOOL_IGNORE_FUNC. These can be used to tell stacktool to skip validation of an instruction or a function, respectively.

---
 arch/x86/include/asm/stacktool.h | 45 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/vmlinux.lds.S    |  5 ++++-
 include/linux/stacktool.h        | 29 ++++++++++++++++++++++++++
 3 files changed, 78 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/stacktool.h
 create mode 100644 include/linux/stacktool.h

new file mode 100644
index 0000000..250a37e
@@ -0,0 +1,45 @@
+#ifndef _ASM_X86_STACKTOOL_H
+#define _ASM_X86_STACKTOOL_H
+
+#include <asm/asm.h>
+
+#ifdef __ASSEMBLY__
+
+/*
+ * This asm macro tells stacktool to ignore the instruction immediately after
+ * the macro when doing stack metadata validation.  It should only be used in
+ * special cases where you're 100% sure it won't affect the reliability of
+ * frame pointers and kernel stack traces.
+ *
+ * For more information, see tools/stacktool/Documentation/stack-validation.txt.
+ */
+.macro STACKTOOL_IGNORE_INSN
+#ifdef CONFIG_STACK_VALIDATION
+	.Lstacktool_ignore_\@:
+	.pushsection __stacktool_ignore_insn, "a"
+	_ASM_ALIGN
+	.long .Lstacktool_ignore_\@ - .
+	.popsection
+#endif
+.endm
+
+#else /* !__ASSEMBLY__ */
+
+#ifdef CONFIG_STACK_VALIDATION
+
+#define STACKTOOL_IGNORE_INSN				\
+	"1:\n"							\
+	".pushsection __stacktool_ignore_insn, \"a\"\n"	\
+	_ASM_ALIGN "\n"						\
+	".long 1b - .\n"					\
+	".popsection\n"
+
+#else /* !CONFIG_STACK_VALIDATION */
+
+#define STACKTOOL_IGNORE_INSN ""
+
+#endif /* CONFIG_STACK_VALIDATION */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_X86_STACKTOOL_H */

index 74e4bf1..8512b2d 100644
@@ -332,7 +332,10 @@ SECTIONS
 
 	/* Sections to be discarded */
 	DISCARDS
-	/DISCARD/ : { *(.eh_frame) }
+	/DISCARD/ : {
+		*(.eh_frame)
+		*(__stacktool_ignore_*)
+	}
 }
 
 

new file mode 100644
index 0000000..c1e151b
@@ -0,0 +1,29 @@
+#ifndef _LINUX_STACKTOOL_H
+#define _LINUX_STACKTOOL_H
+
+#include <asm/stacktool.h>
+
+#ifndef __ASSEMBLY__
+
+#ifdef CONFIG_STACK_VALIDATION
+/*
+ * This C macro tells stacktool to ignore the function when doing stack
+ * metadata validation.  It should only be used in special cases where you're
+ * 100% sure it won't affect the reliability of frame pointers and kernel stack
+ * traces.
+ *
+ * For more information, see tools/stacktool/Documentation/stack-validation.txt.
+ */
+#define STACKTOOL_IGNORE_FUNC(_func) \
+	static void __used __section(__stacktool_ignore_func) \
+		*__stacktool_ignore_func_##_func = _func
+
+#else /* !CONFIG_STACK_VALIDATION */
+
+#define STACKTOOL_IGNORE_FUNC(_func)
+
+#endif /* CONFIG_STACK_VALIDATION */
+
+#endif /* !__ASSEMBLY__ */
+
+#endif /* _LINUX_STACKTOOL_H */

2.4.3

If a hypercall is inlined at the beginning of a function, gcc can insert the call instruction before setting up a stack frame, which breaks frame pointer convention if CONFIG_FRAME_POINTER is enabled and can result in a bad stack trace.

Force a stack frame to be created if CONFIG_FRAME_POINTER is enabled by listing the stack pointer as an output operand for the hypercall inline asm statements.

---
 arch/x86/include/asm/xen/hypercall.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index 83aea80..9b7dc62 100644
@@ -110,9 +110,10 @@ extern struct { char _entry[32]; } hypercall_page[];
 	register unsigned long __arg2 asm(__HYPERCALL_ARG2REG) = __arg2; \
 	register unsigned long __arg3 asm(__HYPERCALL_ARG3REG) = __arg3; \
 	register unsigned long __arg4 asm(__HYPERCALL_ARG4REG) = __arg4; \
-	register unsigned long __arg5 asm(__HYPERCALL_ARG5REG) = __arg5;
+	register unsigned long __arg5 asm(__HYPERCALL_ARG5REG) = __arg5; \
+	register void *__sp asm(_ASM_SP);
 
-#define __HYPERCALL_0PARAM	"=r" (__res)
+#define __HYPERCALL_0PARAM	"=r" (__res), "+r" (__sp)
 #define __HYPERCALL_1PARAM	__HYPERCALL_0PARAM, "+r" (__arg1)
 #define __HYPERCALL_2PARAM	__HYPERCALL_1PARAM, "+r" (__arg2)
 #define __HYPERCALL_3PARAM	__HYPERCALL_2PARAM, "+r" (__arg3)

2.4.3

If a PVOP call macro is inlined at the beginning of a function, gcc can insert the call instruction before setting up a stack frame, which breaks frame pointer convention if CONFIG_FRAME_POINTER is enabled and can result in a bad stack trace.

Force a stack frame to be created if CONFIG_FRAME_POINTER is enabled by listing the stack pointer as an output operand for the PVOP inline asm statements.

---
 arch/x86/include/asm/paravirt_types.h | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

index 31247b5..1885961 100644
@@ -496,8 +496,9 @@ int paravirt_disable_iospace(void);
  * makes sure the incoming and outgoing types are always correct.
  */
 #ifdef CONFIG_X86_32
-#define PVOP_VCALL_ARGS				\
-	unsigned long __eax = __eax, __edx = __edx, __ecx = __ecx
+#define PVOP_VCALL_ARGS							\
+	unsigned long __eax = __eax, __edx = __edx, __ecx = __ecx;	\
+	register void *__sp asm("esp")
 #define PVOP_CALL_ARGS			PVOP_VCALL_ARGS
 
 #define PVOP_CALL_ARG1(x)		"a" ((unsigned long)(x))

@@ -515,9 +516,10 @@ int paravirt_disable_iospace(void);
 #define VEXTRA_CLOBBERS
 #else  /* CONFIG_X86_64 */
 /* [re]ax isn't an arg, but the return val */
-#define PVOP_VCALL_ARGS					\
-	unsigned long __edi = __edi, __esi = __esi,	\
-		__edx = __edx, __ecx = __ecx, __eax = __eax
+#define PVOP_VCALL_ARGS						\
+	unsigned long __edi = __edi, __esi = __esi,		\
+		__edx = __edx, __ecx = __ecx, __eax = __eax;	\
+	register void *__sp asm("rsp")
 #define PVOP_CALL_ARGS		PVOP_VCALL_ARGS
 
 #define PVOP_CALL_ARG1(x)		"D" ((unsigned long)(x))

@@ -556,7 +558,7 @@ int paravirt_disable_iospace(void);
 			asm volatile(pre				\
 				     paravirt_alt(PARAVIRT_CALL)	\
 				     post				\
-				     : call_clbr			\
+				     : "+r" (__sp), call_clbr		\
 				     : paravirt_type(op),		\
 				       paravirt_clobber(clbr),		\
 				       ##__VA_ARGS__			\

@@ -566,7 +568,7 @@ int paravirt_disable_iospace(void);
 			asm volatile(pre				\
 				     paravirt_alt(PARAVIRT_CALL)	\
 				     post				\
-				     : call_clbr			\
+				     : "+r" (__sp), call_clbr		\
 				     : paravirt_type(op),		\
 				       paravirt_clobber(clbr),		\
 				       ##__VA_ARGS__			\

@@ -593,7 +595,7 @@ int paravirt_disable_iospace(void);
 		asm volatile(pre					\
 			     paravirt_alt(PARAVIRT_CALL)		\
 			     post					\
-			     : call_clbr				\
+			     : "+r" (__sp), call_clbr			\
 			     : paravirt_type(op),			\
 			       paravirt_clobber(clbr),			\
 			       ##__VA_ARGS__				\

2.4.3

A function created with the PV_CALLEE_SAVE_REGS_THUNK macro doesn't set up a new stack frame before the call instruction, which breaks frame pointer convention if CONFIG_FRAME_POINTER is enabled and can result in a bad stack trace. Also, the thunk functions aren't annotated as ELF callable functions.

Create a stack frame when CONFIG_FRAME_POINTER is enabled and add the ELF function type.

---
 arch/x86/include/asm/paravirt.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

index 10d0596..23a382b 100644
@@ -13,6 +13,7 @@
 #include <linux/bug.h>
 #include <linux/types.h>
 #include <linux/cpumask.h>
+#include <asm/frame.h>
 
 static inline int paravirt_enabled(void)
 {

@@ -776,15 +777,19 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
  * call. The return value in rax/eax will not be saved, even for void
  * functions.
  */
+#define PV_THUNK_NAME(func) "__raw_callee_save_" #func
 #define PV_CALLEE_SAVE_REGS_THUNK(func)					\
 	extern typeof(func) __raw_callee_save_##func;			\
 									\
 	asm(".pushsection .text;"					\
-	    ".globl __raw_callee_save_" #func " ; "			\
-	    "__raw_callee_save_" #func ": "				\
+	    ".globl " PV_THUNK_NAME(func) ";"				\
+	    ".type " PV_THUNK_NAME(func) ", @function;"			\
+	    PV_THUNK_NAME(func) ":"					\
+	    FRAME_BEGIN							\
 	    PV_SAVE_ALL_CALLER_REGS					\
 	    "call " #func ";"						\
 	    PV_RESTORE_ALL_CALLER_REGS					\
+	    FRAME_END							\
 	    "ret;"							\
 	    ".popsection")
 

2.4.3

vide() is a callable function, but is missing the ELF function type, which confuses tools like stacktool.

Properly annotate it to be a callable function. The generated code is unchanged.

---
 arch/x86/kernel/cpu/amd.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

index 4a70fc6..c010de5 100644
@@ -75,7 +75,10 @@ static inline int wrmsrl_amd_safe(unsigned msr, unsigned long long val)
  */
 
 extern __visible void vide(void);
-__asm__(".globl vide\n\t.align 4\nvide: ret");
+__asm__(".globl vide\n"
+	".type vide, @function\n"
+	".align 4\n"
+	"vide: ret\n");
 
 static void init_amd_k5(struct cpuinfo_x86 *c)
 {

2.4.3

stacktool reports a false positive warning for the ljmp instruction in machine_real_restart(). Normally, ljmp isn't allowed in a function, but this is a special case where it's jumping into real mode.

Add the jumps to a whitelist which tells stacktool to ignore them.

---
 arch/x86/kernel/reboot.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

index 02693dd..1ea1c5e 100644
@@ -9,6 +9,7 @@
 #include <linux/sched.h>
 #include <linux/tboot.h>
 #include <linux/delay.h>
+#include <linux/stacktool.h>
 #include <acpi/reboot.h>
 #include <asm/io.h>
 #include <asm/apic.h>

@@ -97,11 +98,13 @@ void __noreturn machine_real_restart(unsigned int type)
 
 	/* Jump to the identity-mapped low memory code */
 #ifdef CONFIG_X86_32
-	asm volatile("jmpl *%0" : :
+	asm volatile(STACKTOOL_IGNORE_INSN
+		     "jmpl *%0;" : :
 		     "rm" (real_mode_header->machine_real_restart_asm),
 		     "a" (type));
 #else
-	asm volatile("ljmpl *%0" : :
+	asm volatile(STACKTOOL_IGNORE_INSN
+		     "ljmpl *%0" : :
 		     "m" (real_mode_header->machine_real_restart_asm),
 		     "D" (type));
 #endif

2.4.3

stacktool reports the following false positive warnings:

stacktool: arch/x86/xen/enlighten.o: xen_cpuid()+0x41: can't find jump dest instruction at .text+0x108
stacktool: arch/x86/xen/enlighten.o: xen_setup_gdt.constprop.23()+0x2e: kernel entry/exit from callable instruction

The first warning is due to xen_cpuid()'s use of XEN_EMULATE_PREFIX to insert some fake instructions which stacktool doesn't know how to decode.

The second warning is due to xen_setup_gdt()'s use of an lret instruction, which stacktool normally doesn't allow in callable functions. But this seems to be a valid use of the instruction.

Add both functions to the stacktool whitelist.

---
 arch/x86/xen/enlighten.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index 30d12af..1e0fe3a 100644
@@ -32,6 +32,7 @@
 #include <linux/gfp.h>
 #include <linux/memblock.h>
 #include <linux/edd.h>
+#include <linux/stacktool.h>
 
 #include <xen/xen.h>
 #include <xen/events.h>

@@ -346,8 +347,8 @@ static void xen_cpuid(unsigned int *ax, unsigned int *bx,
 	*cx &= maskecx;
 	*cx |= setecx;
 	*dx &= maskedx;
-
 }
+STACKTOOL_IGNORE_FUNC(xen_cpuid);
 
 static bool __init xen_check_mwait(void)
 {

@@ -1451,6 +1452,7 @@ static void __ref xen_setup_gdt(int cpu)
 	pv_cpu_ops.write_gdt_entry = xen_write_gdt_entry;
 	pv_cpu_ops.load_gdt = xen_load_gdt;
 }
+STACKTOOL_IGNORE_FUNC(xen_setup_gdt);
 
 #ifdef CONFIG_XEN_PVH
 /*

2.4.3

aesni-intel_asm.S has several callable non-leaf functions which don't honor CONFIG_FRAME_POINTER, which can result in bad stack traces.

Create stack frames for them when CONFIG_FRAME_POINTER is enabled.

---
 arch/x86/crypto/aesni-intel_asm.S | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

index 6bd2c6c..289ef12 100644
@@ -31,6 +31,7 @@
 
 #include <linux/linkage.h>
 #include <asm/inst.h>
+#include <asm/frame.h>
 
 /*
  * The following macros are used to move an (un)aligned 16 byte value to/from

@@ -1800,6 +1801,7 @@ ENDPROC(_key_expansion_256b)
  *                   unsigned int key_len)
  */
 ENTRY(aesni_set_key)
+	FRAME_BEGIN
 #ifndef __x86_64__
 	pushl KEYP
 	movl 8(%esp), KEYP		# ctx

@@ -1905,6 +1907,7 @@ ENTRY(aesni_set_key)
 #ifndef __x86_64__
 	popl KEYP
 #endif
+	FRAME_END
 	ret
 ENDPROC(aesni_set_key)
 

@@ -1912,6 +1915,7 @@ ENDPROC(aesni_set_key)
  * void aesni_enc(struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src)
  */
 ENTRY(aesni_enc)
+	FRAME_BEGIN
 #ifndef __x86_64__
 	pushl KEYP
 	pushl KLEN

@@ -1927,6 +1931,7 @@ ENTRY(aesni_enc)
 	popl KLEN
 	popl KEYP
 #endif
+	FRAME_END
 	ret
 ENDPROC(aesni_enc)
 

@@ -2101,6 +2106,7 @@ ENDPROC(_aesni_enc4)
  * void aesni_dec (struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src)
  */
 ENTRY(aesni_dec)
+	FRAME_BEGIN
 #ifndef __x86_64__
 	pushl KEYP
 	pushl KLEN

@@ -2117,6 +2123,7 @@ ENTRY(aesni_dec)
 	popl KLEN
 	popl KEYP
 #endif
+	FRAME_END
 	ret
 ENDPROC(aesni_dec)
 

@@ -2292,6 +2299,7 @@ ENDPROC(_aesni_dec4)
  *		      size_t len)
  */
 ENTRY(aesni_ecb_enc)
+	FRAME_BEGIN
 #ifndef __x86_64__
 	pushl LEN
 	pushl KEYP

@@ -2342,6 +2350,7 @@ ENTRY(aesni_ecb_enc)
 	popl KEYP
 	popl LEN
 #endif
+	FRAME_END
 	ret
 ENDPROC(aesni_ecb_enc)
 

@@ -2350,6 +2359,7 @@ ENDPROC(aesni_ecb_enc)
  *		      size_t len);
  */
 ENTRY(aesni_ecb_dec)
+	FRAME_BEGIN
 #ifndef __x86_64__
 	pushl LEN
 	pushl KEYP

@@ -2401,6 +2411,7 @@ ENTRY(aesni_ecb_dec)
 	popl KEYP
 	popl LEN
 #endif
+	FRAME_END
 	ret
 ENDPROC(aesni_ecb_dec)
 

@@ -2409,6 +2420,7 @@ ENDPROC(aesni_ecb_dec)
  *		      size_t len, u8 *iv)
  */
 ENTRY(aesni_cbc_enc)
+	FRAME_BEGIN
 #ifndef __x86_64__
 	pushl IVP
 	pushl LEN

@@ -2443,6 +2455,7 @@ ENTRY(aesni_cbc_enc)
 	popl LEN
 	popl IVP
 #endif
+	FRAME_END
 	ret
 ENDPROC(aesni_cbc_enc)
 

@@ -2451,6 +2464,7 @@ ENDPROC(aesni_cbc_enc)
  *		      size_t len, u8 *iv)
  */
 ENTRY(aesni_cbc_dec)
+	FRAME_BEGIN
 #ifndef __x86_64__
 	pushl IVP
 	pushl LEN

@@ -2534,6 +2548,7 @@ ENTRY(aesni_cbc_dec)
 	popl LEN
 	popl IVP
 #endif
+	FRAME_END
 	ret
 ENDPROC(aesni_cbc_dec)
 

@@ -2598,6 +2613,7 @@ ENDPROC(_aesni_inc)
  *		      size_t len, u8 *iv)
  */
 ENTRY(aesni_ctr_enc)
+	FRAME_BEGIN
 	cmp $16, LEN
 	jb .Lctr_enc_just_ret
 	mov 480(KEYP), KLEN

@@ -2651,6 +2667,7 @@ ENTRY(aesni_ctr_enc)
 .Lctr_enc_ret:
 	movups IV, (IVP)
 .Lctr_enc_just_ret:
+	FRAME_END
 	ret
 ENDPROC(aesni_ctr_enc)
 

@@ -2677,6 +2694,7 @@ ENDPROC(aesni_ctr_enc)
  *			 bool enc, u8 *iv)
  */
 ENTRY(aesni_xts_crypt8)
+	FRAME_BEGIN
 	cmpb $0, %cl
 	movl $0, %ecx
 	movl $240, %r10d

@@ -2777,6 +2795,7 @@ ENTRY(aesni_xts_crypt8)
 	pxor INC, STATE4
 	movdqu STATE4, 0x70(OUTP)
 
+	FRAME_END
 	ret
 ENDPROC(aesni_xts_crypt8)
 

2.4.3

This will break 32 bit builds using the aesni-intel.ko module. You need to adjust the esp-based offsets for the non-x86_64 case, as FRAME_BEGIN may do another push.

How about adding a FRAME_OFFSET() macro to <asm/frame.h> to wrap the offsets?:

#ifdef CONFIG_FRAME_POINTER

# define FRAME_OFFSET(x)	((x) + (BITS_PER_LONG / 8))
#else
# define FRAME_OFFSET(x)	(x)
#endif

And using them like this:

	movl FRAME_OFFSET(8)(%esp), KEYP		# ctx

Ah, right. The 32-bit ABI passes arguments on the stack instead of via registers.

For now, I'm inclined to just make FRAME_BEGIN and FRAME_END do nothing on 32-bit. We're only doing stack validation on x86_64 and I don't know if anybody cares about frame pointers on 32-bit at this point.

Josh

Well, we had issues in the past, especially in that very module, but only on 32 bit systems. So it would be nice to get frame pointers right for 32 bit, too.

Regards,
Mathias

Ok, I'll make your suggested changes for this file with FRAME_OFFSET.

(But note that stacktool doesn't currently support 32 bit so it won't detect any frame pointer issues in 32 bit code.)

Josh

stacktool reports the following warning:

stacktool: arch/x86/crypto/aesni-intel_asm.o: _aesni_inc_init(): can't find starting instruction

stacktool gets confused when it tries to disassemble the following data in the .text section:

.Lbswap_mask:

          .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0

Move it to .rodata which is a more appropriate section for read-only data.

---
 arch/x86/crypto/aesni-intel_asm.S | 2 ++
 1 file changed, 2 insertions(+)

index 289ef12..18f0ced 100644
@@ -2553,9 +2553,11 @@ ENTRY(aesni_cbc_dec)
 ENDPROC(aesni_cbc_dec)
 
 #ifdef __x86_64__
+.pushsection .rodata
 .align 16
 .Lbswap_mask:
 	.byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
+.popsection
 
 /*
  * _aesni_inc_init:	internal ABI

2.4.3

stacktool reports the following warning:

stacktool: arch/x86/crypto/crc32c-pcl-intel-asm_64.o: crc_pcl()+0x11dd: can't decode instruction

It gets confused when trying to decode jump_table data. Move jump_table to the .rodata section which is a more appropriate home for read-only data.

---
 arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

index 225be06..dc05f01 100644
@@ -170,8 +170,8 @@ continue_block:
 	## branch into array
 	lea	jump_table(%rip), bufp
 	movzxw  (bufp, %rax, 2), len
-	offset=crc_array-jump_table
-	lea     offset(bufp, len, 1), bufp
+	lea	crc_array(%rip), bufp
+	lea     (bufp, len, 1), bufp
 	jmp     *bufp
 
 	################################################################

@@ -310,7 +310,9 @@ do_return:
 	popq    %rdi
 	popq    %rbx
         ret
+ENDPROC(crc_pcl)
 
+.section	.rodata, "a", %progbits
         ################################################################
         ## jump table        Table is 129 entries x 2 bytes each
         ################################################################

@@ -324,13 +326,11 @@ JMPTBL_ENTRY %i
 	i=i+1
 .endr
 
-ENDPROC(crc_pcl)
 
 	################################################################
 	## PCLMULQDQ tables
 	## Table is 128 entries x 2 words (8 bytes) each
 	################################################################
-.section	.rotata, "a", %progbits
 .align 8
 K_table:
 	.long 0x493c7d27, 0x00000001

2.4.3

clmul_ghash_mul() and clmul_ghash_update() are callable non-leaf functions which don't honor CONFIG_FRAME_POINTER, which can result in bad stack traces.

Create stack frames for them when CONFIG_FRAME_POINTER is enabled.

---
 arch/x86/crypto/ghash-clmulni-intel_asm.S | 5 +++++
 1 file changed, 5 insertions(+)

index 5d1e007..eed55c8 100644
@@ -18,6 +18,7 @@
 
 #include <linux/linkage.h>
 #include <asm/inst.h>
+#include <asm/frame.h>
 
 .data
 

@@ -94,6 +95,7 @@ ENDPROC(__clmul_gf128mul_ble)
 
 /* void clmul_ghash_mul(char *dst, const u128 *shash) */
 ENTRY(clmul_ghash_mul)
+	FRAME_BEGIN
 	movups (%rdi), DATA
 	movups (%rsi), SHASH
 	movaps .Lbswap_mask, BSWAP

@@ -101,6 +103,7 @@ ENTRY(clmul_ghash_mul)
 	call __clmul_gf128mul_ble
 	PSHUFB_XMM BSWAP DATA
 	movups DATA, (%rdi)
+	FRAME_END
 	ret
 ENDPROC(clmul_ghash_mul)
 

@@ -109,6 +112,7 @@ ENDPROC(clmul_ghash_mul)
  *			   const u128 *shash);
  */
 ENTRY(clmul_ghash_update)
+	FRAME_BEGIN
 	cmp $16, %rdx
 	jb .Lupdate_just_ret	# check length
 	movaps .Lbswap_mask, BSWAP

@@ -128,5 +132,6 @@ ENTRY(clmul_ghash_update)
 	PSHUFB_XMM BSWAP DATA
 	movups DATA, (%rdi)
 .Lupdate_just_ret:
+	FRAME_END
 	ret
 ENDPROC(clmul_ghash_update)

2.4.3

Thunk functions are callable non-leaf functions that don't honor CONFIG_FRAME_POINTER, which can result in bad stack traces. Also they aren't annotated as ELF callable functions which can confuse tooling.

Create stack frames for them when CONFIG_FRAME_POINTER is enabled and add the ELF function type.

---
 arch/x86/entry/thunk_64.S | 4 ++++
 1 file changed, 4 insertions(+)

index efb2b93..98df1fa 100644
@@ -8,11 +8,14 @@
 #include <linux/linkage.h>
 #include "calling.h"
 #include <asm/asm.h>
+#include <asm/frame.h>
 
 	/* rdi:	arg1 ... normal C conventions. rax is saved/restored. */
 	.macro THUNK name, func, put_ret_addr_in_rdi=0
 	.globl \name
+	.type \name, @function
 \name:
+	FRAME_BEGIN
 
 	/* this one pushes 9 elems, the next one would be %rIP */
 	pushq %rdi

@@ -62,6 +65,7 @@ restore:
 	popq %rdx
 	popq %rsi
 	popq %rdi
+	FRAME_END
 	ret
 	_ASM_NOKPROBE(restore)
 #endif

2.4.3

do_suspend_lowlevel() is a callable non-leaf function which doesn't honor CONFIG_FRAME_POINTER, which can result in bad stack traces.

Create a stack frame for it when CONFIG_FRAME_POINTER is enabled.

---
 arch/x86/kernel/acpi/wakeup_64.S | 3 +++
 1 file changed, 3 insertions(+)

index 8c35df4..169963f 100644
@@ -5,6 +5,7 @@
 #include <asm/page_types.h>
 #include <asm/msr.h>
 #include <asm/asm-offsets.h>
+#include <asm/frame.h>
 
 # Copyright 2003 Pavel Machek <[email protected]>, distribute under GPLv2
 

@@ -39,6 +40,7 @@ bogus_64_magic:
 	jmp	bogus_64_magic
 
 ENTRY(do_suspend_lowlevel)
+	FRAME_BEGIN
 	subq	$8, %rsp
 	xorl	%eax, %eax
 	call	save_processor_state

@@ -109,6 +111,7 @@ ENTRY(do_suspend_lowlevel)
 
 	xorl	%eax, %eax
 	addq	$8, %rsp
+	FRAME_END
 	jmp	restore_processor_state
 ENDPROC(do_suspend_lowlevel)
 

2.4.3

rwsem.S has several callable non-leaf functions which don't honor CONFIG_FRAME_POINTER, which can result in bad stack traces.

Create stack frames for them when CONFIG_FRAME_POINTER is enabled.

---
 arch/x86/lib/rwsem.S | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

index 40027db..be110ef 100644
@@ -15,6 +15,7 @@
 
 #include <linux/linkage.h>
 #include <asm/alternative-asm.h>
+#include <asm/frame.h>
 
 #define __ASM_HALF_REG(reg)	__ASM_SEL(reg, e##reg)
 #define __ASM_HALF_SIZE(inst)	__ASM_SEL(inst##w, inst##l)

@@ -84,24 +85,29 @@
 
 /* Fix up special calling conventions */
 ENTRY(call_rwsem_down_read_failed)
+	FRAME_BEGIN
 	save_common_regs
 	__ASM_SIZE(push,) %__ASM_REG(dx)
 	movq %rax,%rdi
 	call rwsem_down_read_failed
 	__ASM_SIZE(pop,) %__ASM_REG(dx)
 	restore_common_regs
+	FRAME_END
 	ret
 ENDPROC(call_rwsem_down_read_failed)
 
 ENTRY(call_rwsem_down_write_failed)
+	FRAME_BEGIN
 	save_common_regs
 	movq %rax,%rdi
 	call rwsem_down_write_failed
 	restore_common_regs
+	FRAME_END
 	ret
 ENDPROC(call_rwsem_down_write_failed)
 
 ENTRY(call_rwsem_wake)
+	FRAME_BEGIN
 	/* do nothing if still outstanding active readers */
 	__ASM_HALF_SIZE(dec) %__ASM_HALF_REG(dx)
 	jnz 1f

@@ -109,15 +115,18 @@ ENTRY(call_rwsem_wake)
 	movq %rax,%rdi
 	call rwsem_wake
 	restore_common_regs
-1:	ret
+1:	FRAME_END
+	ret
 ENDPROC(call_rwsem_wake)
 
 ENTRY(call_rwsem_downgrade_wake)
+	FRAME_BEGIN
 	save_common_regs
 	__ASM_SIZE(push,) %__ASM_REG(dx)
 	movq %rax,%rdi
 	call rwsem_downgrade_wake
 	__ASM_SIZE(pop,) %__ASM_REG(dx)
 	restore_common_regs
+	FRAME_END
 	ret
 ENDPROC(call_rwsem_downgrade_wake)

2.4.3

efi_call() is a callable non-leaf function which doesn't honor CONFIG_FRAME_POINTER, which can result in bad stack traces.

Create a stack frame for it when CONFIG_FRAME_POINTER is enabled.

---
 arch/x86/platform/efi/efi_stub_64.S | 3 +++
 1 file changed, 3 insertions(+)

index 86d0f9e..0df2dcc 100644
@@ -11,6 +11,7 @@
 #include <asm/msr.h>
 #include <asm/processor-flags.h>
 #include <asm/page_types.h>
+#include <asm/frame.h>
 
 #define SAVE_XMM			\
 	mov %rsp, %rax;			\

@@ -74,6 +75,7 @@
 	.endm
 
 ENTRY(efi_call)
+	FRAME_BEGIN
 	SAVE_XMM
 	mov (%rsp), %rax
 	mov 8(%rax), %rax

@@ -88,6 +90,7 @@ ENTRY(efi_call)
 	RESTORE_PGT
 	addq $48, %rsp
 	RESTORE_XMM
+	FRAME_END
 	ret
 ENDPROC(efi_call)
 

2.4.3

swsusp_arch_suspend() and restore_registers() are callable non-leaf functions which don't honor CONFIG_FRAME_POINTER, which can result in bad stack traces. Also they aren't annotated as ELF callable functions which can confuse tooling.

Create a stack frame for them when CONFIG_FRAME_POINTER is enabled and give them proper ELF function annotations.

---
 arch/x86/power/hibernate_asm_64.S | 7 +++++++
 1 file changed, 7 insertions(+)

index e2386cb..4400a43 100644
@@ -21,8 +21,10 @@
 #include <asm/page_types.h>
 #include <asm/asm-offsets.h>
 #include <asm/processor-flags.h>
+#include <asm/frame.h>
 
 ENTRY(swsusp_arch_suspend)
+	FRAME_BEGIN
 	movq	$saved_context, %rax
 	movq	%rsp, pt_regs_sp(%rax)
 	movq	%rbp, pt_regs_bp(%rax)

@@ -50,7 +52,9 @@ ENTRY(swsusp_arch_suspend)
 	movq	%rax, restore_cr3(%rip)
 
 	call swsusp_save
+	FRAME_END
 	ret
+ENDPROC(swsusp_arch_suspend)
 
 ENTRY(restore_image)
 	/* switch to temporary page tables */

@@ -107,6 +111,7 @@ ENTRY(core_restore_code)
 	 */
 
 ENTRY(restore_registers)
+	FRAME_BEGIN
 	/* go back to the original page tables */
 	movq    %rbx, %cr3
 

@@ -147,4 +152,6 @@ ENTRY(restore_registers)
 	/* tell the hibernation core that we've just restored the memory */
 	movq	%rax, in_suspend(%rip)
 
+	FRAME_END
 	ret
+ENDPROC(restore_registers)

2.4.3

From: Chris J Arges <[email protected]>

Numerous 'call without frame pointer save/setup' warnings are introduced by stacktool because of functions using the get_user macro. Bad stack traces could occur due to lack of or misplacement of stack frame setup code.

This patch forces a stack frame to be created before the inline asm code if CONFIG_FRAME_POINTER is enabled by listing the stack pointer as an output operand for the get_user inline assembly statement.

---
 arch/x86/include/asm/uaccess.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index a8df874..55b8db5 100644
@@ -176,10 +176,11 @@ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL))
 ({									\
 	int __ret_gu;							\
 	register __inttype(*(ptr)) __val_gu asm("%"_ASM_DX);		\
+	register void *__sp asm(_ASM_SP);				\
 	__chk_user_ptr(ptr);						\
 	might_fault();							\
-	asm volatile("call __get_user_%P3"				\
-		     : "=a" (__ret_gu), "=r" (__val_gu)			\
+	asm volatile("call __get_user_%P4"				\
+		     : "=a" (__ret_gu), "=r" (__val_gu), "+r" (__sp)	\
 		     : "0" (ptr), "i" (sizeof(*(ptr))));		\
 	(x) = (__force __typeof__(*(ptr))) __val_gu;			\
 	__ret_gu;							\

2.4.3

From: Chris J Arges <[email protected]>

Tell stacktool to skip validation of head_64, head_32 as they don't affect runtime kernel stack traces.

---
 arch/x86/kernel/Makefile | 1 +
 1 file changed, 1 insertion(+)

index b1b78ff..f331fef 100644
@@ -19,6 +19,7 @@ endif
 KASAN_SANITIZE_head$(BITS).o := n
 KASAN_SANITIZE_dumpstack.o := n
 KASAN_SANITIZE_dumpstack_$(BITS).o := n
+STACKTOOL_head_$(BITS).o := n
 
 CFLAGS_irq.o := -I$(src)/../include/asm/trace
 

2.4.3

Hi Ingo,

Do you have any more objections to these patches? Would you be willing to apply them?

Thanks!
Josh

So I still don't like the tool namespace you picked: Git-alike generic naming plus subcommands work so much better that I'm not sure why we are even having that
discussion: if you name your tool 'stacktool' and alias everything you have today to under 'stacktool run ...' and add 'stacktool help' as a second, obvious subcommand then you'll have your current syntax and a lot more future flexibility and ability to branch off various functionality a'la Git, perf or kvmtool ...

It's an 100% equivalent superset of your current stacktool's parameter structure, so none of your prior objections apply.

Thanks,

Ingo

Because subcommands are useful in _some_ cases, but they aren't a panacea that should be blindly applied everywhere.

Sure, subcommands work great for monolithic framework tools like git, perf, yum, docker, etc. But stacktool is not (and never will be) a monolithic framework type of tool.

The suggestion to put 100% of the functionality under 'stacktool run [options]' is certainly possible. But 'run' is so broad. What else could the tool ever do but 'run'?

Josh

Hi Ingo,

Do you still have objections to my proposed command-line interface for stacktool?

As I described below, I still don't think subcommands are a good fit for stacktool. However, if you strongly disagree, I can change it.

The patch set is now quite mature at v14. Further, it's holding up work on several other items:

- more frame pointer fixes
- CFI generation and validation
- livepatch consistency model
- DWARF unwinder

So I'm wondering if it would be possible for you to merge stacktool in time for the next merge window (4.5).

Thanks!
Josh