x86/entry/64: interrupt entry size reduction
Subsystems (2)
v3 cover x86/entry/64: interrupt entry size reduction
Here is another re-spin of the interrupt entry size reduction patchset, which applies on top of tip/pti and tries to implement what Linus suggested a few days ago.[*]
Patch 1/5 provides the most significant cuttings (-3k) and gets us below the text size of entry_64.o (by about 2k) compared to before I started meddling with it. This patch is the most evolved of the set, yet saw the helper function being renamed since v2.
The other five patches still need more discussion on whether each additional step and the code complexity it adds is really worth it. And stringent review, of course.
Overall, these patches provide for a sizeable cutting of up to 4.5k compared to tip/pti:
text data bss dec hex filename
19500 0 0 19500 4c2c entry_64.o-orig (before 2e3f0098bc45) 21088 0 0 21088 5260 entry_64.o-pti (as of ced5d0bf603f) 18006 0 0 18006 4656 entry_64.o (patch 1/6) 16623 0 0 16623 40ef entry_64.o (patch 6/6)
Changes for v3:
- patch 1:
- rename interrupt_helper to interrupt_entry (suggested by Josh Poimboeuf)
- do not add cld to interrupt_entry yet
- patch 2 (follow-up changes only)
- patch 3:
- move cld from interrupt macro to interrupt_entry (suggestion by Linus Torvalds to make this more explicit)
- do not add cld to switch_to_thread_stack wrapper function, but keep cld where it is entry_INT80_compat.
- patch 4: (follow-up changes only)
- patch 5 (NEW):
- move ASM_CLAC to interrupt helper (suggested by Brian Gerst)
- patch 6 (was patch 5/5 in v2):
- update commentary (suggested by Josh Poimboeuf)
- remove unused UNWIND hint from entry_64_compat.S (suggested by Josh Poimboeuf)
Changes for v2:
- patch 1 (unchanged)
- patch 2:
- use leaq 8() instead of mov+addq in ENTER_IRQ_STACK (suggested by Brian Gerst)
- add UNWIND_HINT_REGS indirect=1 after call to interrupt_helper; update size information accordingly
- patch 3:
- add cld to switch_to_thread_stack wrapper function
- improve commit message
- update size information
- patch 4:
- add UNWIND_HINT_REGS indirect=1 to the now-removed interrupt macro
- patch 5 (NEW):
- open-code the DO_SWITCH_TO_THREAD_STACK macro (suggested by Brian Gerst)
- improve (?) UNWIND hints
[*] http://lkml.kernel.org/r/CA+55aFwLTF3EtaQ4OpDv2UM41J=EU7gfemv=eVq+uQi31-usSg@mail.gmail.com
Dominik Brodowski (6):
x86/entry/64: move PUSH_AND_CLEAR_REGS from interrupt macro to helper
function
x86/entry/64: move ENTER_IRQ_STACK from interrupt macro to
interrupt_entry
x86/entry/64: move switch_to_thread_stack to interrupt_entry x86/entry/64: remove interrupt macro
x86/entry/64: move ASM_CLAC to interrupt_entry x86/entry/64: open-code switch_to_thread_stack
arch/x86/entry/entry_64.S | 132 ++++++++++++++++++++++++--------------- arch/x86/entry/entry_64_compat.S | 17 ++++- 2 files changed, 98 insertions(+), 51 deletions(-)
Signature · 1 line
2.16.2
RFC PATCH v3 1/6 x86/entry/64: move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
The PUSH_AND_CLEAR_REGS macro is able to insert the GP registers "above" the original return address. This allows us to move a sizeable part of the interrupt entry macro to an interrupt entry helper function:
text data bss dec hex filename
21088 0 0 21088 5260 entry_64.o-orig 18006 0 0 18006 4656 entry_64.o
--- arch/x86/entry/entry_64.S | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +9 -2
@@ -525,6 +525,14 @@ END(irq_entries_start) * * Entry runs with interrupts off. */ +ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + + PUSH_AND_CLEAR_REGS save_ret=1 + ENCODE_FRAME_POINTER 8 + + ret +END(interrupt_entry) /* 0(%rsp): ~(interrupt number) */ .macro interrupt func
@@ -536,8 +544,7 @@ END(irq_entries_start) call switch_to_thread_stack 1: - PUSH_AND_CLEAR_REGS - ENCODE_FRAME_POINTER + call interrupt_entry testb $3, CS(%rsp) jz 1f
Signature · 1 line
2.16.2
Re: [RFC PATCH v3 1/6] x86/entry/64: move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
arch/x86/entry/entry_64.S END(irq_entries_start) · line 535
Quoted 8 lines · Dominik Brodowski
arch/x86/entry/entry_64.S +8
-0+ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + + PUSH_AND_CLEAR_REGS save_ret=1 + ENCODE_FRAME_POINTER 8 + + ret +END(interrupt_entry)
There's nothing wrong with this patch, but it does expose what a nasty hack our "ENCODE_FRAME_POINTER" thing is.
It generates (when there is an offset, like this):
leaq \ptregs_offset(%rsp), %rbp
orq $0x1, %rbp
and I would _really_ hope that the stack pointer is always aligned on interrupt entry, so I don't see why it's not just
leaq 1+\ptregs_offset(%rsp), %rbp
instead.
I dunno. Let's ask Josh what the reason for the separate "or" was.
But this is an independent issue of this patch, really.
Linus
PATCH 1/1 x86/entry/64: Simplify ENCODE_FRAME_POINTER
Quoted 22 lines · Linus Torvalds
Review
Quoted 8 lines · Dominik Brodowski
(quoted diff) +8
-0+ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + + PUSH_AND_CLEAR_REGS save_ret=1 + ENCODE_FRAME_POINTER 8 + + ret +END(interrupt_entry)There's nothing wrong with this patch, but it does expose what a nasty hack our "
ENCODE_FRAME_POINTER" thing is.It generates (when there is an offset, like this):
leaq \ptregs_offset(%rsp), %rbp
orq $0x1, %rbpand I would _really_ hope that the stack pointer is always aligned on interrupt entry, so I don't see why it's not just
Quoted 7 lines
leaq 1+\ptregs_offset(%rsp), %rbpinstead.
I dunno. Let's ask Josh what the reason for the separate "or" was.
But this is an independent issue of this patch, really.
Indeed, how about this?
----
From: Josh Poimboeuf <[email protected]>
Subject: [PATCH] x86/entry/64: Simplify ENCODE_FRAME_POINTER
On 64-bit, the stack pointer is always aligned on interrupt, so instead of setting the LSB of the pt_regs address, we can just add 1 to it.
--- arch/x86/entry/calling.h | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
arch/x86/entry/calling.h +1 -6
@@ -172,12 +172,7 @@ For 32-bit we have the following conventions - kernel is built with*/ .macro ENCODE_FRAME_POINTER ptregs_offset=0#ifdef CONFIG_FRAME_POINTER - .if \ptregs_offset - leaq \ptregs_offset(%rsp), %rbp - .else - mov %rsp, %rbp - .endif - orq $0x1, %rbp + leaq 1+\ptregs_offset(%rsp), %rbp #endif .endm
Signature · 1 line
2.14.3
tip-bot for Josh Poimboeuf
merge bot[tip:x86/pti] x86/entry/64: Simplify ENCODE_FRAME_POINTER
x86/entry/64: Simplify ENCODE_FRAME_POINTER
Commit-ID: fa509cf0f52628bb646f85fa12b16231060053e6 Gitweb: https://git.kernel.org/tip/fa509cf0f52628bb646f85fa12b16231060053e6 Author: Josh Poimboeuf <[email protected]> AuthorDate: Tue, 20 Feb 2018 20:42:14 -0600 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 10:13:59 +0100
x86/entry/64: Simplify ENCODE_FRAME_POINTER
On 64-bit, the stack pointer is always aligned on interrupt, so instead of setting the LSB of the pt_regs address, we can just add 1 to it.
--- arch/x86/entry/calling.h | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
arch/x86/entry/calling.h +1 -6
@@ -181,12 +181,7 @@ For 32-bit we have the following conventions - kernel is built with*/ .macro ENCODE_FRAME_POINTER ptregs_offset=0#ifdef CONFIG_FRAME_POINTER - .if \ptregs_offset - leaq \ptregs_offset(%rsp), %rbp - .else - mov %rsp, %rbp - .endif - orq $0x1, %rbp + leaq 1+\ptregs_offset(%rsp), %rbp #endif .endm
tip-bot for Josh Poimboeuf
merge bot[tip:x86/pti] x86/entry/64: Simplify ENCODE_FRAME_POINTER
In reply to Josh Poimboeufx86/entry/64: Simplify ENCODE_FRAME_POINTER
Commit-ID: 0ca7d5baa1787e5f2a7abd6bfca3303b1bbb48ac Gitweb: https://git.kernel.org/tip/0ca7d5baa1787e5f2a7abd6bfca3303b1bbb48ac Author: Josh Poimboeuf <[email protected]> AuthorDate: Tue, 20 Feb 2018 20:42:14 -0600 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 16:54:07 +0100
x86/entry/64: Simplify ENCODE_FRAME_POINTER
On 64-bit, the stack pointer is always aligned on interrupt, so instead of setting the LSB of the pt_regs address, we can just add 1 to it.
--- arch/x86/entry/calling.h | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
arch/x86/entry/calling.h +1 -6
@@ -181,12 +181,7 @@ For 32-bit we have the following conventions - kernel is built with*/ .macro ENCODE_FRAME_POINTER ptregs_offset=0#ifdef CONFIG_FRAME_POINTER - .if \ptregs_offset - leaq \ptregs_offset(%rsp), %rbp - .else - mov %rsp, %rbp - .endif - orq $0x1, %rbp + leaq 1+\ptregs_offset(%rsp), %rbp #endif .endm
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
In reply to Dominik Brodowskix86/entry/64: Move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
Commit-ID: 6014809adf04188c4bce4a5d231f13a602203c95 Gitweb: https://git.kernel.org/tip/6014809adf04188c4bce4a5d231f13a602203c95 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:08 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 10:04:46 +0100
x86/entry/64: Move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
The PUSH_AND_CLEAR_REGS macro is able to insert the GP registers "above" the original return address. This allows us to move a sizeable part of the interrupt entry macro to an interrupt entry helper function:
text data bss dec hex filename
21088 0 0 21088 5260 entry_64.o-orig 18006 0 0 18006 4656 entry_64.o
--- arch/x86/entry/entry_64.S | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +9 -2
@@ -525,6 +525,14 @@ END(irq_entries_start) * * Entry runs with interrupts off. */ +ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + + PUSH_AND_CLEAR_REGS save_ret=1 + ENCODE_FRAME_POINTER 8 + + ret +END(interrupt_entry) /* 0(%rsp): ~(interrupt number) */ .macro interrupt func
@@ -536,8 +544,7 @@ END(irq_entries_start) call switch_to_thread_stack 1: - PUSH_AND_CLEAR_REGS - ENCODE_FRAME_POINTER + call interrupt_entry testb $3, CS(%rsp) jz 1f
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
In reply to Dominik Brodowskix86/entry/64: Move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
Commit-ID: 0e34d226342c27c4f96138b211547d423e4be8a1 Gitweb: https://git.kernel.org/tip/0e34d226342c27c4f96138b211547d423e4be8a1 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:08 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 16:54:03 +0100
x86/entry/64: Move PUSH_AND_CLEAR_REGS from interrupt macro to helper function
The PUSH_AND_CLEAR_REGS macro is able to insert the GP registers "above" the original return address. This allows us to move a sizeable part of the interrupt entry macro to an interrupt entry helper function:
text data bss dec hex filename
21088 0 0 21088 5260 entry_64.o-orig 18006 0 0 18006 4656 entry_64.o
--- arch/x86/entry/entry_64.S | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +9 -2
@@ -525,6 +525,14 @@ END(irq_entries_start) * * Entry runs with interrupts off. */ +ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + + PUSH_AND_CLEAR_REGS save_ret=1 + ENCODE_FRAME_POINTER 8 + + ret +END(interrupt_entry) /* 0(%rsp): ~(interrupt number) */ .macro interrupt func
@@ -536,8 +544,7 @@ END(irq_entries_start) call switch_to_thread_stack 1: - PUSH_AND_CLEAR_REGS - ENCODE_FRAME_POINTER + call interrupt_entry testb $3, CS(%rsp) jz 1f
RFC PATCH v3 2/6 x86/entry/64: move ENTER_IRQ_STACK from interrupt macro to interrupt_entry
In reply to Dominik BrodowskiMoving the switch to IRQ stack from the interrupt macro to the helper function requires some trickery: All ENTER_IRQ_STACK really cares about is where the "original" stack -- meaning the GP registers etc. -- is stored. Therefore, we need to offset the stored RSP value by 8 whenever ENTER_IRQ_STACK is called from within a function. In such cases, and after switching to the IRQ stack, we need to push the "original" return address (i.e. the return address from the call to the interrupt entry function) to the IRQ stack.
This trickery allows us to carve another .85k from the text size (it would be more except for the additional unwind hints):
text data bss dec hex filename
18006 0 0 18006 4656 entry_64.o-orig 17158 0 0 17158 4306 entry_64.o
--- arch/x86/entry/entry_64.S | 56 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 38 insertions(+), 18 deletions(-)
arch/x86/entry/entry_64.S +38 -18
@@ -448,9 +448,19 @@ END(irq_entries_start) * * The invariant is that, if irq_count != -1, then the IRQ stack is in use. */ -.macro ENTER_IRQ_STACK regs=1 old_rsp +.macro ENTER_IRQ_STACK regs=1 old_rsp save_ret=0 DEBUG_ENTRY_ASSERT_IRQS_OFF + + .if \save_ret + /* + * If save_ret is set, the original stack contains one additional + * entry -- the return address. Therefore, move the address one + * entry below %rsp to \old_rsp. + */ + leaq 8(%rsp), \old_rsp + .else movq %rsp, \old_rsp + .endif .if \regs UNWIND_HINT_REGS base=\old_rsp
@@ -496,6 +506,15 @@ END(irq_entries_start) .if \regs UNWIND_HINT_REGS indirect=1 .endif + + .if \save_ret + /* + * Push the return address to the stack. This return address can + * be found at the "real" original RSP, which was offset by 8 at + * the beginning of this macro. + */ + pushq -8(\old_rsp) + .endif .endm /*
@@ -531,22 +550,7 @@ ENTRY(interrupt_entry) PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8 - ret -END(interrupt_entry) - -/* 0(%rsp): ~(interrupt number) */ - .macro interrupt func - cld - - testb $3, CS-ORIG_RAX(%rsp) - jz 1f - SWAPGS - call switch_to_thread_stack -1: - - call interrupt_entry - - testb $3, CS(%rsp) + testb $3, CS+8(%rsp) jz 1f /*
@@ -564,10 +568,26 @@ END(interrupt_entry) CALL_enter_from_user_mode 1: - ENTER_IRQ_STACK old_rsp=%rdi + ENTER_IRQ_STACK old_rsp=%rdi save_ret=1 /* We entered an interrupt context - irqs are off: */ TRACE_IRQS_OFF + ret +END(interrupt_entry) + +/* 0(%rsp): ~(interrupt number) */ + .macro interrupt func + cld + + testb $3, CS-ORIG_RAX(%rsp) + jz 1f + SWAPGS + call switch_to_thread_stack +1: + + call interrupt_entry + + UNWIND_HINT_REGS indirect=1 call \func /* rdi points to pt_regs */ .endm
Signature · 1 line
2.16.2
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move ENTER_IRQ_STACK from interrupt macro to interrupt_entry
x86/entry/64: Move ENTER_IRQ_STACK from interrupt macro to interrupt_entry
Commit-ID: 484689299ece526dd75088946c2307c3a74d9d67 Gitweb: https://git.kernel.org/tip/484689299ece526dd75088946c2307c3a74d9d67 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:09 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 10:04:46 +0100
x86/entry/64: Move ENTER_IRQ_STACK from interrupt macro to interrupt_entry
Moving the switch to IRQ stack from the interrupt macro to the helper function requires some trickery: All ENTER_IRQ_STACK really cares about is where the "original" stack -- meaning the GP registers etc. -- is stored. Therefore, we need to offset the stored RSP value by 8 whenever ENTER_IRQ_STACK is called from within a function. In such cases, and after switching to the IRQ stack, we need to push the "original" return address (i.e. the return address from the call to the interrupt entry function) to the IRQ stack.
This trickery allows us to carve another .85k from the text size (it would be more except for the additional unwind hints):
text data bss dec hex filename
18006 0 0 18006 4656 entry_64.o-orig 17158 0 0 17158 4306 entry_64.o
--- arch/x86/entry/entry_64.S | 56 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 38 insertions(+), 18 deletions(-)
arch/x86/entry/entry_64.S +38 -18
@@ -448,9 +448,19 @@ END(irq_entries_start) * * The invariant is that, if irq_count != -1, then the IRQ stack is in use. */ -.macro ENTER_IRQ_STACK regs=1 old_rsp +.macro ENTER_IRQ_STACK regs=1 old_rsp save_ret=0 DEBUG_ENTRY_ASSERT_IRQS_OFF + + .if \save_ret + /* + * If save_ret is set, the original stack contains one additional + * entry -- the return address. Therefore, move the address one + * entry below %rsp to \old_rsp. + */ + leaq 8(%rsp), \old_rsp + .else movq %rsp, \old_rsp + .endif .if \regs UNWIND_HINT_REGS base=\old_rsp
@@ -496,6 +506,15 @@ END(irq_entries_start) .if \regs UNWIND_HINT_REGS indirect=1 .endif + + .if \save_ret + /* + * Push the return address to the stack. This return address can + * be found at the "real" original RSP, which was offset by 8 at + * the beginning of this macro. + */ + pushq -8(\old_rsp) + .endif .endm /*
@@ -531,22 +550,7 @@ ENTRY(interrupt_entry) PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8 - ret -END(interrupt_entry) - -/* 0(%rsp): ~(interrupt number) */ - .macro interrupt func - cld - - testb $3, CS-ORIG_RAX(%rsp) - jz 1f - SWAPGS - call switch_to_thread_stack -1: - - call interrupt_entry - - testb $3, CS(%rsp) + testb $3, CS+8(%rsp) jz 1f /*
@@ -564,10 +568,26 @@ END(interrupt_entry) CALL_enter_from_user_mode 1: - ENTER_IRQ_STACK old_rsp=%rdi + ENTER_IRQ_STACK old_rsp=%rdi save_ret=1 /* We entered an interrupt context - irqs are off: */ TRACE_IRQS_OFF + ret +END(interrupt_entry) + +/* 0(%rsp): ~(interrupt number) */ + .macro interrupt func + cld + + testb $3, CS-ORIG_RAX(%rsp) + jz 1f + SWAPGS + call switch_to_thread_stack +1: + + call interrupt_entry + + UNWIND_HINT_REGS indirect=1 call \func /* rdi points to pt_regs */ .endm
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move ENTER_IRQ_STACK from interrupt macro to interrupt_entry
In reply to Dominik Brodowskix86/entry/64: Move ENTER_IRQ_STACK from interrupt macro to interrupt_entry
Commit-ID: 2ba6474104a1132c4af9f6dc42c6bfe3ca71f8c7 Gitweb: https://git.kernel.org/tip/2ba6474104a1132c4af9f6dc42c6bfe3ca71f8c7 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:09 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 16:54:03 +0100
x86/entry/64: Move ENTER_IRQ_STACK from interrupt macro to interrupt_entry
Moving the switch to IRQ stack from the interrupt macro to the helper function requires some trickery: All ENTER_IRQ_STACK really cares about is where the "original" stack -- meaning the GP registers etc. -- is stored. Therefore, we need to offset the stored RSP value by 8 whenever ENTER_IRQ_STACK is called from within a function. In such cases, and after switching to the IRQ stack, we need to push the "original" return address (i.e. the return address from the call to the interrupt entry function) to the IRQ stack.
This trickery allows us to carve another .85k from the text size (it would be more except for the additional unwind hints):
text data bss dec hex filename
18006 0 0 18006 4656 entry_64.o-orig 17158 0 0 17158 4306 entry_64.o
--- arch/x86/entry/entry_64.S | 56 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 38 insertions(+), 18 deletions(-)
arch/x86/entry/entry_64.S +38 -18
@@ -448,9 +448,19 @@ END(irq_entries_start) * * The invariant is that, if irq_count != -1, then the IRQ stack is in use. */ -.macro ENTER_IRQ_STACK regs=1 old_rsp +.macro ENTER_IRQ_STACK regs=1 old_rsp save_ret=0 DEBUG_ENTRY_ASSERT_IRQS_OFF + + .if \save_ret + /* + * If save_ret is set, the original stack contains one additional + * entry -- the return address. Therefore, move the address one + * entry below %rsp to \old_rsp. + */ + leaq 8(%rsp), \old_rsp + .else movq %rsp, \old_rsp + .endif .if \regs UNWIND_HINT_REGS base=\old_rsp
@@ -496,6 +506,15 @@ END(irq_entries_start) .if \regs UNWIND_HINT_REGS indirect=1 .endif + + .if \save_ret + /* + * Push the return address to the stack. This return address can + * be found at the "real" original RSP, which was offset by 8 at + * the beginning of this macro. + */ + pushq -8(\old_rsp) + .endif .endm /*
@@ -531,22 +550,7 @@ ENTRY(interrupt_entry) PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8 - ret -END(interrupt_entry) - -/* 0(%rsp): ~(interrupt number) */ - .macro interrupt func - cld - - testb $3, CS-ORIG_RAX(%rsp) - jz 1f - SWAPGS - call switch_to_thread_stack -1: - - call interrupt_entry - - testb $3, CS(%rsp) + testb $3, CS+8(%rsp) jz 1f /*
@@ -564,10 +568,26 @@ END(interrupt_entry) CALL_enter_from_user_mode 1: - ENTER_IRQ_STACK old_rsp=%rdi + ENTER_IRQ_STACK old_rsp=%rdi save_ret=1 /* We entered an interrupt context - irqs are off: */ TRACE_IRQS_OFF + ret +END(interrupt_entry) + +/* 0(%rsp): ~(interrupt number) */ + .macro interrupt func + cld + + testb $3, CS-ORIG_RAX(%rsp) + jz 1f + SWAPGS + call switch_to_thread_stack +1: + + call interrupt_entry + + UNWIND_HINT_REGS indirect=1 call \func /* rdi points to pt_regs */ .endm
RFC PATCH v3 3/6 x86/entry/64: move switch_to_thread_stack to interrupt_entry
In reply to Dominik BrodowskiWe can also move the CLD, SWAPGS, and the switch_to_thread_stack to the interrupt_entry helper function. As we do not want call depths of two, convert switch_to_thread_stack to a macro.
However, switch_to_thread_stack has another user in entry_64_compat.S, which currently expects it to be a function. To keep the code changes in this patch minimal, create a wrapper function.
The switch to a macro means that there is some binary code duplication if CONFIG_IA32_EMULATION is enabled. Therefore, the size reduction differs whether CONFIG_IA32_EMULATION is enabled or not:
CONFIG_IA32_EMULATION=y (-0.13k):
text data bss dec hex filename
17158 0 0 17158 4306 entry_64.o-orig 17028 0 0 17028 4284 entry_64.o
CONFIG_IA32_EMULATION=n (-0.27k):
text data bss dec hex filename
17158 0 0 17158 4306 entry_64.o-orig 16882 0 0 16882 41f2 entry_64.o
--- arch/x86/entry/entry_64.S | 66 ++++++++++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 29 deletions(-)
arch/x86/entry/entry_64.S +37 -29
@@ -537,6 +537,31 @@ END(irq_entries_start) decl PER_CPU_VAR(irq_count) .endm +/* + * Switch to the thread stack. This is called with the IRET frame and + * orig_ax on the stack. (That is, RDI..R12 are not on the stack and + * space has not been allocated for them.) + */ +.macro DO_SWITCH_TO_THREAD_STACK + pushq %rdi + /* Need to switch before accessing the thread stack. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + movq %rsp, %rdi + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI + + pushq 7*8(%rdi) /* regs->ss */ + pushq 6*8(%rdi) /* regs->rsp */ + pushq 5*8(%rdi) /* regs->eflags */ + pushq 4*8(%rdi) /* regs->cs */ + pushq 3*8(%rdi) /* regs->ip */ + pushq 2*8(%rdi) /* regs->orig_ax */ + pushq 8(%rdi) /* return address */ + UNWIND_HINT_FUNC + + movq (%rdi), %rdi +.endm + /* * Interrupt entry/exit. *
@@ -544,8 +569,16 @@ END(irq_entries_start) * * Entry runs with interrupts off. */ +/* 8(%rsp): ~(interrupt number) */ ENTRY(interrupt_entry) UNWIND_HINT_FUNC + cld + + testb $3, CS-ORIG_RAX+8(%rsp) + jz 1f + SWAPGS + DO_SWITCH_TO_THREAD_STACK +1: PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -577,14 +610,6 @@ END(interrupt_entry) /* 0(%rsp): ~(interrupt number) */ .macro interrupt func - cld - - testb $3, CS-ORIG_RAX(%rsp) - jz 1f - SWAPGS - call switch_to_thread_stack -1: - call interrupt_entry UNWIND_HINT_REGS indirect=1
@@ -858,33 +883,16 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) -/* - * Switch to the thread stack. This is called with the IRET frame and - * orig_ax on the stack. (That is, RDI..R12 are not on the stack and - * space has not been allocated for them.) - */ +#if defined(CONFIG_IA32_EMULATION) +/* entry_64_compat.S::entry_INT80_compat expects this to be an ASM function */ ENTRY(switch_to_thread_stack) UNWIND_HINT_FUNC - pushq %rdi - /* Need to switch before accessing the thread stack. */ - SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi - movq %rsp, %rdi - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI - - pushq 7*8(%rdi) /* regs->ss */ - pushq 6*8(%rdi) /* regs->rsp */ - pushq 5*8(%rdi) /* regs->eflags */ - pushq 4*8(%rdi) /* regs->cs */ - pushq 3*8(%rdi) /* regs->ip */ - pushq 2*8(%rdi) /* regs->orig_ax */ - pushq 8(%rdi) /* return address */ - UNWIND_HINT_FUNC + DO_SWITCH_TO_THREAD_STACK - movq (%rdi), %rdi ret END(switch_to_thread_stack) +#endif .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 ENTRY(\sym)
Signature · 1 line
2.16.2
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move the switch_to_thread_stack() call to interrupt_entry()
x86/entry/64: Move the switch_to_thread_stack() call to interrupt_entry()
Commit-ID: 359cc2d9e0a810808a90adda3e612e2fef49f2cd Gitweb: https://git.kernel.org/tip/359cc2d9e0a810808a90adda3e612e2fef49f2cd Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:10 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 10:04:47 +0100
x86/entry/64: Move the switch_to_thread_stack() call to interrupt_entry()
We can also move the CLD, SWAPGS, and the switch_to_thread_stack() call to the interrupt_entry() helper function. As we do not want call depths of two, convert switch_to_thread_stack() to a macro.
However, switch_to_thread_stack() has another user in entry_64_compat.S, which currently expects it to be a function. To keep the code changes in this patch minimal, create a wrapper function.
The switch to a macro means that there is some binary code duplication if CONFIG_IA32_EMULATION=y is enabled. Therefore, the size reduction differs whether CONFIG_IA32_EMULATION is enabled or not:
CONFIG_IA32_EMULATION=y (-0.13k):
text data bss dec hex filename
17158 0 0 17158 4306 entry_64.o-orig 17028 0 0 17028 4284 entry_64.o
CONFIG_IA32_EMULATION=n (-0.27k):
text data bss dec hex filename
17158 0 0 17158 4306 entry_64.o-orig 16882 0 0 16882 41f2 entry_64.o
--- arch/x86/entry/entry_64.S | 66 ++++++++++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 29 deletions(-)
arch/x86/entry/entry_64.S +37 -29
@@ -538,14 +538,47 @@ END(irq_entries_start) .endm /* + * Switch to the thread stack. This is called with the IRET frame and + * orig_ax on the stack. (That is, RDI..R12 are not on the stack and + * space has not been allocated for them.) + */ +.macro DO_SWITCH_TO_THREAD_STACK + pushq %rdi + /* Need to switch before accessing the thread stack. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + movq %rsp, %rdi + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI + + pushq 7*8(%rdi) /* regs->ss */ + pushq 6*8(%rdi) /* regs->rsp */ + pushq 5*8(%rdi) /* regs->eflags */ + pushq 4*8(%rdi) /* regs->cs */ + pushq 3*8(%rdi) /* regs->ip */ + pushq 2*8(%rdi) /* regs->orig_ax */ + pushq 8(%rdi) /* return address */ + UNWIND_HINT_FUNC + + movq (%rdi), %rdi +.endm + +/* * Interrupt entry/exit. * * Interrupt entry points save only callee clobbered registers in fast path. * * Entry runs with interrupts off. */ +/* 8(%rsp): ~(interrupt number) */ ENTRY(interrupt_entry) UNWIND_HINT_FUNC + cld + + testb $3, CS-ORIG_RAX+8(%rsp) + jz 1f + SWAPGS + DO_SWITCH_TO_THREAD_STACK +1: PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -577,14 +610,6 @@ END(interrupt_entry) /* 0(%rsp): ~(interrupt number) */ .macro interrupt func - cld - - testb $3, CS-ORIG_RAX(%rsp) - jz 1f - SWAPGS - call switch_to_thread_stack -1: - call interrupt_entry UNWIND_HINT_REGS indirect=1
@@ -858,33 +883,16 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) -/* - * Switch to the thread stack. This is called with the IRET frame and - * orig_ax on the stack. (That is, RDI..R12 are not on the stack and - * space has not been allocated for them.) - */ +#if defined(CONFIG_IA32_EMULATION) +/* entry_64_compat.S::entry_INT80_compat expects this to be an ASM function */ ENTRY(switch_to_thread_stack) UNWIND_HINT_FUNC - pushq %rdi - /* Need to switch before accessing the thread stack. */ - SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi - movq %rsp, %rdi - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI - - pushq 7*8(%rdi) /* regs->ss */ - pushq 6*8(%rdi) /* regs->rsp */ - pushq 5*8(%rdi) /* regs->eflags */ - pushq 4*8(%rdi) /* regs->cs */ - pushq 3*8(%rdi) /* regs->ip */ - pushq 2*8(%rdi) /* regs->orig_ax */ - pushq 8(%rdi) /* return address */ - UNWIND_HINT_FUNC + DO_SWITCH_TO_THREAD_STACK - movq (%rdi), %rdi ret END(switch_to_thread_stack) +#endif .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 ENTRY(\sym)
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move the switch_to_thread_stack() call to interrupt_entry()
In reply to Dominik Brodowskix86/entry/64: Move the switch_to_thread_stack() call to interrupt_entry()
Commit-ID: 90a6acc4e7ebafa8672a7a1a5b23fbad3dd04130 Gitweb: https://git.kernel.org/tip/90a6acc4e7ebafa8672a7a1a5b23fbad3dd04130 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:10 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 16:54:04 +0100
x86/entry/64: Move the switch_to_thread_stack() call to interrupt_entry()
We can also move the CLD, SWAPGS, and the switch_to_thread_stack() call to the interrupt_entry() helper function. As we do not want call depths of two, convert switch_to_thread_stack() to a macro.
However, switch_to_thread_stack() has another user in entry_64_compat.S, which currently expects it to be a function. To keep the code changes in this patch minimal, create a wrapper function.
The switch to a macro means that there is some binary code duplication if CONFIG_IA32_EMULATION=y is enabled. Therefore, the size reduction differs whether CONFIG_IA32_EMULATION is enabled or not:
CONFIG_IA32_EMULATION=y (-0.13k):
text data bss dec hex filename
17158 0 0 17158 4306 entry_64.o-orig 17028 0 0 17028 4284 entry_64.o
CONFIG_IA32_EMULATION=n (-0.27k):
text data bss dec hex filename
17158 0 0 17158 4306 entry_64.o-orig 16882 0 0 16882 41f2 entry_64.o
--- arch/x86/entry/entry_64.S | 66 ++++++++++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 29 deletions(-)
arch/x86/entry/entry_64.S +37 -29
@@ -538,14 +538,47 @@ END(irq_entries_start) .endm /* + * Switch to the thread stack. This is called with the IRET frame and + * orig_ax on the stack. (That is, RDI..R12 are not on the stack and + * space has not been allocated for them.) + */ +.macro DO_SWITCH_TO_THREAD_STACK + pushq %rdi + /* Need to switch before accessing the thread stack. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + movq %rsp, %rdi + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI + + pushq 7*8(%rdi) /* regs->ss */ + pushq 6*8(%rdi) /* regs->rsp */ + pushq 5*8(%rdi) /* regs->eflags */ + pushq 4*8(%rdi) /* regs->cs */ + pushq 3*8(%rdi) /* regs->ip */ + pushq 2*8(%rdi) /* regs->orig_ax */ + pushq 8(%rdi) /* return address */ + UNWIND_HINT_FUNC + + movq (%rdi), %rdi +.endm + +/* * Interrupt entry/exit. * * Interrupt entry points save only callee clobbered registers in fast path. * * Entry runs with interrupts off. */ +/* 8(%rsp): ~(interrupt number) */ ENTRY(interrupt_entry) UNWIND_HINT_FUNC + cld + + testb $3, CS-ORIG_RAX+8(%rsp) + jz 1f + SWAPGS + DO_SWITCH_TO_THREAD_STACK +1: PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -577,14 +610,6 @@ END(interrupt_entry) /* 0(%rsp): ~(interrupt number) */ .macro interrupt func - cld - - testb $3, CS-ORIG_RAX(%rsp) - jz 1f - SWAPGS - call switch_to_thread_stack -1: - call interrupt_entry UNWIND_HINT_REGS indirect=1
@@ -858,33 +883,16 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) -/* - * Switch to the thread stack. This is called with the IRET frame and - * orig_ax on the stack. (That is, RDI..R12 are not on the stack and - * space has not been allocated for them.) - */ +#if defined(CONFIG_IA32_EMULATION) +/* entry_64_compat.S::entry_INT80_compat expects this to be an ASM function */ ENTRY(switch_to_thread_stack) UNWIND_HINT_FUNC - pushq %rdi - /* Need to switch before accessing the thread stack. */ - SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi - movq %rsp, %rdi - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI - - pushq 7*8(%rdi) /* regs->ss */ - pushq 6*8(%rdi) /* regs->rsp */ - pushq 5*8(%rdi) /* regs->eflags */ - pushq 4*8(%rdi) /* regs->cs */ - pushq 3*8(%rdi) /* regs->ip */ - pushq 2*8(%rdi) /* regs->orig_ax */ - pushq 8(%rdi) /* return address */ - UNWIND_HINT_FUNC + DO_SWITCH_TO_THREAD_STACK - movq (%rdi), %rdi ret END(switch_to_thread_stack) +#endif .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 ENTRY(\sym)
RFC PATCH v3 4/6 x86/entry/64: remove interrupt macro
In reply to Dominik BrodowskiIt is now trivial to call interrupt_entry and then the actual worker. Therefore, remove the interrupt macro.
--- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -10
@@ -608,14 +608,6 @@ ENTRY(interrupt_entry) ret END(interrupt_entry) -/* 0(%rsp): ~(interrupt number) */ - .macro interrupt func - call interrupt_entry - - UNWIND_HINT_REGS indirect=1 - call \func /* rdi points to pt_regs */ - .endm - /* * The interrupt stubs push (~vector+0x80) onto the stack and * then jump to common_interrupt.
@@ -624,7 +616,9 @@ END(interrupt_entry) common_interrupt: ASM_CLAC addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ - interrupt do_IRQ + call interrupt_entry + UNWIND_HINT_REGS indirect=1 + call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY)
@@ -820,7 +814,9 @@ ENTRY(\sym) ASM_CLAC pushq $~(\num) .Lcommon_\sym: - interrupt \do_sym + call interrupt_entry + UNWIND_HINT_REGS indirect=1 + call \do_sym /* rdi points to pt_regs */ jmp ret_from_intr END(\sym) .endm
Signature · 1 line
2.16.2
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Remove 'interrupt' macro
x86/entry/64: Remove 'interrupt' macro
Commit-ID: 9aaa78dee43a61febabddeab44ea9da90dd05ba8 Gitweb: https://git.kernel.org/tip/9aaa78dee43a61febabddeab44ea9da90dd05ba8 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:11 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 10:04:47 +0100
x86/entry/64: Remove 'interrupt' macro
It is now trivial to call interrupt_entry() and then the actual worker. Therefore, remove the interrupt macro and open code it all.
--- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -10
@@ -608,14 +608,6 @@ ENTRY(interrupt_entry) ret END(interrupt_entry) -/* 0(%rsp): ~(interrupt number) */ - .macro interrupt func - call interrupt_entry - - UNWIND_HINT_REGS indirect=1 - call \func /* rdi points to pt_regs */ - .endm - /* * The interrupt stubs push (~vector+0x80) onto the stack and * then jump to common_interrupt.
@@ -624,7 +616,9 @@ END(interrupt_entry) common_interrupt: ASM_CLAC addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ - interrupt do_IRQ + call interrupt_entry + UNWIND_HINT_REGS indirect=1 + call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY)
@@ -820,7 +814,9 @@ ENTRY(\sym) ASM_CLAC pushq $~(\num) .Lcommon_\sym: - interrupt \do_sym + call interrupt_entry + UNWIND_HINT_REGS indirect=1 + call \do_sym /* rdi points to pt_regs */ jmp ret_from_intr END(\sym) .endm
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Remove 'interrupt' macro
In reply to Dominik Brodowskix86/entry/64: Remove 'interrupt' macro
Commit-ID: 3aa99fc3e708b9cd9b4cfe2df0b7a66cf293e3cf Gitweb: https://git.kernel.org/tip/3aa99fc3e708b9cd9b4cfe2df0b7a66cf293e3cf Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:11 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 16:54:04 +0100
x86/entry/64: Remove 'interrupt' macro
It is now trivial to call interrupt_entry() and then the actual worker. Therefore, remove the interrupt macro and open code it all.
--- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -10
@@ -608,14 +608,6 @@ ENTRY(interrupt_entry) ret END(interrupt_entry) -/* 0(%rsp): ~(interrupt number) */ - .macro interrupt func - call interrupt_entry - - UNWIND_HINT_REGS indirect=1 - call \func /* rdi points to pt_regs */ - .endm - /* * The interrupt stubs push (~vector+0x80) onto the stack and * then jump to common_interrupt.
@@ -624,7 +616,9 @@ END(interrupt_entry) common_interrupt: ASM_CLAC addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ - interrupt do_IRQ + call interrupt_entry + UNWIND_HINT_REGS indirect=1 + call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY)
@@ -820,7 +814,9 @@ ENTRY(\sym) ASM_CLAC pushq $~(\num) .Lcommon_\sym: - interrupt \do_sym + call interrupt_entry + UNWIND_HINT_REGS indirect=1 + call \do_sym /* rdi points to pt_regs */ jmp ret_from_intr END(\sym) .endm
RFC PATCH v3 5/6 x86/entry/64: move ASM_CLAC to interrupt_entry
In reply to Dominik BrodowskiMoving ASM_CLAC to interrupt_entry means two instructions (addq / pushq and call interrupt_entry) are not covered by it. However, it offers a noticeable size reduction (-.2k):
text data bss dec hex filename
16882 0 0 16882 41f2 entry_64.o-orig 16623 0 0 16623 40ef entry_64.o
--- arch/x86/entry/entry_64.S | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/entry/entry_64.S +1 -2
@@ -572,6 +572,7 @@ END(irq_entries_start) /* 8(%rsp): ~(interrupt number) */ ENTRY(interrupt_entry) UNWIND_HINT_FUNC + ASM_CLAC cld testb $3, CS-ORIG_RAX+8(%rsp)
@@ -614,7 +615,6 @@ END(interrupt_entry) */ .p2align CONFIG_X86_L1_CACHE_SHIFT common_interrupt: - ASM_CLAC addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1
@@ -811,7 +811,6 @@ END(common_interrupt) .macro apicinterrupt3 num sym do_sym ENTRY(\sym) UNWIND_HINT_IRET_REGS - ASM_CLAC pushq $~(\num) .Lcommon_\sym: call interrupt_entry
Signature · 1 line
2.16.2
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move ASM_CLAC to interrupt_entry()
x86/entry/64: Move ASM_CLAC to interrupt_entry()
Commit-ID: a2bb5db1193effd900216a62a6d03d177c26ab02 Gitweb: https://git.kernel.org/tip/a2bb5db1193effd900216a62a6d03d177c26ab02 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:12 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 10:04:47 +0100
x86/entry/64: Move ASM_CLAC to interrupt_entry()
Moving ASM_CLAC to interrupt_entry means two instructions (addq / pushq and call interrupt_entry) are not covered by it. However, it offers a noticeable size reduction (-.2k):
text data bss dec hex filename
16882 0 0 16882 41f2 entry_64.o-orig 16623 0 0 16623 40ef entry_64.o
--- arch/x86/entry/entry_64.S | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/entry/entry_64.S +1 -2
@@ -572,6 +572,7 @@ END(irq_entries_start) /* 8(%rsp): ~(interrupt number) */ ENTRY(interrupt_entry) UNWIND_HINT_FUNC + ASM_CLAC cld testb $3, CS-ORIG_RAX+8(%rsp)
@@ -614,7 +615,6 @@ END(interrupt_entry) */ .p2align CONFIG_X86_L1_CACHE_SHIFT common_interrupt: - ASM_CLAC addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1
@@ -811,7 +811,6 @@ END(common_interrupt) .macro apicinterrupt3 num sym do_sym ENTRY(\sym) UNWIND_HINT_IRET_REGS - ASM_CLAC pushq $~(\num) .Lcommon_\sym: call interrupt_entry
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Move ASM_CLAC to interrupt_entry()
In reply to Dominik Brodowskix86/entry/64: Move ASM_CLAC to interrupt_entry()
Commit-ID: b2855d8d2de0fa15c1ff30c69ed7756b00c48b22 Gitweb: https://git.kernel.org/tip/b2855d8d2de0fa15c1ff30c69ed7756b00c48b22 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:12 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 16:54:05 +0100
x86/entry/64: Move ASM_CLAC to interrupt_entry()
Moving ASM_CLAC to interrupt_entry means two instructions (addq / pushq and call interrupt_entry) are not covered by it. However, it offers a noticeable size reduction (-.2k):
text data bss dec hex filename
16882 0 0 16882 41f2 entry_64.o-orig 16623 0 0 16623 40ef entry_64.o
--- arch/x86/entry/entry_64.S | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/entry/entry_64.S +1 -2
@@ -572,6 +572,7 @@ END(irq_entries_start) /* 8(%rsp): ~(interrupt number) */ ENTRY(interrupt_entry) UNWIND_HINT_FUNC + ASM_CLAC cld testb $3, CS-ORIG_RAX+8(%rsp)
@@ -614,7 +615,6 @@ END(interrupt_entry) */ .p2align CONFIG_X86_L1_CACHE_SHIFT common_interrupt: - ASM_CLAC addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1
@@ -811,7 +811,6 @@ END(common_interrupt) .macro apicinterrupt3 num sym do_sym ENTRY(\sym) UNWIND_HINT_IRET_REGS - ASM_CLAC pushq $~(\num) .Lcommon_\sym: call interrupt_entry
RFC PATCH v3 6/6 x86/entry/64: open-code switch_to_thread_stack
In reply to Dominik BrodowskiOpen-code the two instances which used switch_to_thread_stack. This allows us to remove the wrapper around DO_SWITCH_TO_THREAD_STACK.
While at it, update the UNWIND hint to reflect where the IRET frame is, and update the commentary to reflect what we are actually doing here.
--- arch/x86/entry/entry_64.S | 76 +++++++++++++++++++++------------------- arch/x86/entry/entry_64_compat.S | 17 +++++++-- 2 files changed, 55 insertions(+), 38 deletions(-)
arch/x86/entry/entry_64.S +40 -36
@@ -538,17 +538,48 @@ END(irq_entries_start) .endm /* - * Switch to the thread stack. This is called with the IRET frame and - * orig_ax on the stack. (That is, RDI..R12 are not on the stack and - * space has not been allocated for them.) + * Interrupt entry helper function. + * + * Entry runs with interrupts off. Stack layout at entry: + * +----------------------------------------------------+ + * | regs->ss | + * | regs->rsp | + * | regs->eflags | + * | regs->cs | + * | regs->ip | + * +----------------------------------------------------+ + * | regs->orig_ax = ~(interrupt number) | + * +----------------------------------------------------+ + * | return address | + * +----------------------------------------------------+ */ -.macro DO_SWITCH_TO_THREAD_STACK +ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + ASM_CLAC + cld + + testb $3, CS-ORIG_RAX+8(%rsp) + jz 1f + SWAPGS + + /* + * Switch to the thread stack. The IRET frame and orig_ax are + * on the stack, as well as the return address. RDI..R12 are + * not (yet) on the stack and space has not (yet) been + * allocated for them. + */ pushq %rdi + /* Need to switch before accessing the thread stack. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi movq %rsp, %rdi movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI + + /* + * We have RDI, return address, and orig_ax on the stack on + * top of the IRET frame. That means offset=24 + */ + UNWIND_HINT_IRET_REGS base=%rdi offset=24 pushq 7*8(%rdi) /* regs->ss */ pushq 6*8(%rdi) /* regs->rsp */
@@ -560,25 +591,6 @@ END(irq_entries_start) UNWIND_HINT_FUNC movq (%rdi), %rdi -.endm - -/* - * Interrupt entry/exit. - * - * Interrupt entry points save only callee clobbered registers in fast path. - * - * Entry runs with interrupts off. - */ -/* 8(%rsp): ~(interrupt number) */ -ENTRY(interrupt_entry) - UNWIND_HINT_FUNC - ASM_CLAC - cld - - testb $3, CS-ORIG_RAX+8(%rsp) - jz 1f - SWAPGS - DO_SWITCH_TO_THREAD_STACK 1: PUSH_AND_CLEAR_REGS save_ret=1
@@ -592,7 +604,7 @@ ENTRY(interrupt_entry) * * We need to tell lockdep that IRQs are off. We can't do this until * we fix gsbase, and we should do it before enter_from_user_mode - * (which can take locks). Since TRACE_IRQS_OFF idempotent, + * (which can take locks). Since TRACE_IRQS_OFF is idempotent, * the simplest way to handle it is to just call it twice if * we enter from user mode. There's no reason to optimize this since * TRACE_IRQS_OFF is a no-op if lockdep is off.
@@ -609,6 +621,9 @@ ENTRY(interrupt_entry) ret END(interrupt_entry) + +/* Interrupt entry/exit. */ + /* * The interrupt stubs push (~vector+0x80) onto the stack and * then jump to common_interrupt.
@@ -878,17 +893,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) -#if defined(CONFIG_IA32_EMULATION) -/* entry_64_compat.S::entry_INT80_compat expects this to be an ASM function */ -ENTRY(switch_to_thread_stack) - UNWIND_HINT_FUNC - - DO_SWITCH_TO_THREAD_STACK - - ret -END(switch_to_thread_stack) -#endif - .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 ENTRY(\sym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8
arch/x86/entry/entry_64_compat.S +15 -2
@@ -347,10 +347,23 @@ ENTRY(entry_INT80_compat) */ movl %eax, %eax + /* switch to thread stack expects orig_ax and rdi to be pushed */ pushq %rax /* pt_regs->orig_ax */ + pushq %rdi /* pt_regs->di */ + + /* Need to switch before accessing the thread stack. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + movq %rsp, %rdi + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + + pushq 6*8(%rdi) /* regs->ss */ + pushq 5*8(%rdi) /* regs->rsp */ + pushq 4*8(%rdi) /* regs->eflags */ + pushq 3*8(%rdi) /* regs->cs */ + pushq 2*8(%rdi) /* regs->ip */ + pushq 1*8(%rdi) /* regs->orig_ax */ - /* switch to thread stack expects orig_ax to be pushed */ - call switch_to_thread_stack + movq (%rdi), %rdi /* restore %rdi */ pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */
Signature · 1 line
2.16.2
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Open-code switch_to_thread_stack()
x86/entry/64: Open-code switch_to_thread_stack()
Commit-ID: 179efbb14ced86ce0ec6495b85fa971aa3466be3 Gitweb: https://git.kernel.org/tip/179efbb14ced86ce0ec6495b85fa971aa3466be3 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:13 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 10:04:47 +0100
x86/entry/64: Open-code switch_to_thread_stack()
Open-code the two instances which called switch_to_thread_stack(). This allows us to remove the wrapper around DO_SWITCH_TO_THREAD_STACK.
While at it, update the UNWIND hint to reflect where the IRET frame is, and update the commentary to reflect what we are actually doing here.
--- arch/x86/entry/entry_64.S | 76 +++++++++++++++++++++------------------- arch/x86/entry/entry_64_compat.S | 17 +++++++-- 2 files changed, 55 insertions(+), 38 deletions(-)
arch/x86/entry/entry_64.S +40 -36
@@ -538,17 +538,48 @@ END(irq_entries_start) .endm /* - * Switch to the thread stack. This is called with the IRET frame and - * orig_ax on the stack. (That is, RDI..R12 are not on the stack and - * space has not been allocated for them.) + * Interrupt entry helper function. + * + * Entry runs with interrupts off. Stack layout at entry: + * +----------------------------------------------------+ + * | regs->ss | + * | regs->rsp | + * | regs->eflags | + * | regs->cs | + * | regs->ip | + * +----------------------------------------------------+ + * | regs->orig_ax = ~(interrupt number) | + * +----------------------------------------------------+ + * | return address | + * +----------------------------------------------------+ */ -.macro DO_SWITCH_TO_THREAD_STACK +ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + ASM_CLAC + cld + + testb $3, CS-ORIG_RAX+8(%rsp) + jz 1f + SWAPGS + + /* + * Switch to the thread stack. The IRET frame and orig_ax are + * on the stack, as well as the return address. RDI..R12 are + * not (yet) on the stack and space has not (yet) been + * allocated for them. + */ pushq %rdi + /* Need to switch before accessing the thread stack. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi movq %rsp, %rdi movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI + + /* + * We have RDI, return address, and orig_ax on the stack on + * top of the IRET frame. That means offset=24 + */ + UNWIND_HINT_IRET_REGS base=%rdi offset=24 pushq 7*8(%rdi) /* regs->ss */ pushq 6*8(%rdi) /* regs->rsp */
@@ -560,25 +591,6 @@ END(irq_entries_start) UNWIND_HINT_FUNC movq (%rdi), %rdi -.endm - -/* - * Interrupt entry/exit. - * - * Interrupt entry points save only callee clobbered registers in fast path. - * - * Entry runs with interrupts off. - */ -/* 8(%rsp): ~(interrupt number) */ -ENTRY(interrupt_entry) - UNWIND_HINT_FUNC - ASM_CLAC - cld - - testb $3, CS-ORIG_RAX+8(%rsp) - jz 1f - SWAPGS - DO_SWITCH_TO_THREAD_STACK 1: PUSH_AND_CLEAR_REGS save_ret=1
@@ -592,7 +604,7 @@ ENTRY(interrupt_entry) * * We need to tell lockdep that IRQs are off. We can't do this until * we fix gsbase, and we should do it before enter_from_user_mode - * (which can take locks). Since TRACE_IRQS_OFF idempotent, + * (which can take locks). Since TRACE_IRQS_OFF is idempotent, * the simplest way to handle it is to just call it twice if * we enter from user mode. There's no reason to optimize this since * TRACE_IRQS_OFF is a no-op if lockdep is off.
@@ -609,6 +621,9 @@ ENTRY(interrupt_entry) ret END(interrupt_entry) + +/* Interrupt entry/exit. */ + /* * The interrupt stubs push (~vector+0x80) onto the stack and * then jump to common_interrupt.
@@ -878,17 +893,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) -#if defined(CONFIG_IA32_EMULATION) -/* entry_64_compat.S::entry_INT80_compat expects this to be an ASM function */ -ENTRY(switch_to_thread_stack) - UNWIND_HINT_FUNC - - DO_SWITCH_TO_THREAD_STACK - - ret -END(switch_to_thread_stack) -#endif - .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 ENTRY(\sym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8
arch/x86/entry/entry_64_compat.S +15 -2
@@ -347,10 +347,23 @@ ENTRY(entry_INT80_compat) */ movl %eax, %eax + /* switch to thread stack expects orig_ax and rdi to be pushed */ pushq %rax /* pt_regs->orig_ax */ + pushq %rdi /* pt_regs->di */ + + /* Need to switch before accessing the thread stack. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + movq %rsp, %rdi + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + + pushq 6*8(%rdi) /* regs->ss */ + pushq 5*8(%rdi) /* regs->rsp */ + pushq 4*8(%rdi) /* regs->eflags */ + pushq 3*8(%rdi) /* regs->cs */ + pushq 2*8(%rdi) /* regs->ip */ + pushq 1*8(%rdi) /* regs->orig_ax */ - /* switch to thread stack expects orig_ax to be pushed */ - call switch_to_thread_stack + movq (%rdi), %rdi /* restore %rdi */ pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */
tip-bot for Dominik Brodowski
merge bot[tip:x86/pti] x86/entry/64: Open-code switch_to_thread_stack()
In reply to Dominik Brodowskix86/entry/64: Open-code switch_to_thread_stack()
Commit-ID: f3d415ea46968ae1f9cbb9e201601d7207ce74c7 Gitweb: https://git.kernel.org/tip/f3d415ea46968ae1f9cbb9e201601d7207ce74c7 Author: Dominik Brodowski <[email protected]> AuthorDate: Tue, 20 Feb 2018 22:01:13 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 21 Feb 2018 16:54:05 +0100
x86/entry/64: Open-code switch_to_thread_stack()
Open-code the two instances which called switch_to_thread_stack(). This allows us to remove the wrapper around DO_SWITCH_TO_THREAD_STACK.
While at it, update the UNWIND hint to reflect where the IRET frame is, and update the commentary to reflect what we are actually doing here.
--- arch/x86/entry/entry_64.S | 76 +++++++++++++++++++++------------------- arch/x86/entry/entry_64_compat.S | 17 +++++++-- 2 files changed, 55 insertions(+), 38 deletions(-)
arch/x86/entry/entry_64.S +40 -36
@@ -538,17 +538,48 @@ END(irq_entries_start) .endm /* - * Switch to the thread stack. This is called with the IRET frame and - * orig_ax on the stack. (That is, RDI..R12 are not on the stack and - * space has not been allocated for them.) + * Interrupt entry helper function. + * + * Entry runs with interrupts off. Stack layout at entry: + * +----------------------------------------------------+ + * | regs->ss | + * | regs->rsp | + * | regs->eflags | + * | regs->cs | + * | regs->ip | + * +----------------------------------------------------+ + * | regs->orig_ax = ~(interrupt number) | + * +----------------------------------------------------+ + * | return address | + * +----------------------------------------------------+ */ -.macro DO_SWITCH_TO_THREAD_STACK +ENTRY(interrupt_entry) + UNWIND_HINT_FUNC + ASM_CLAC + cld + + testb $3, CS-ORIG_RAX+8(%rsp) + jz 1f + SWAPGS + + /* + * Switch to the thread stack. The IRET frame and orig_ax are + * on the stack, as well as the return address. RDI..R12 are + * not (yet) on the stack and space has not (yet) been + * allocated for them. + */ pushq %rdi + /* Need to switch before accessing the thread stack. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi movq %rsp, %rdi movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI + + /* + * We have RDI, return address, and orig_ax on the stack on + * top of the IRET frame. That means offset=24 + */ + UNWIND_HINT_IRET_REGS base=%rdi offset=24 pushq 7*8(%rdi) /* regs->ss */ pushq 6*8(%rdi) /* regs->rsp */
@@ -560,25 +591,6 @@ END(irq_entries_start) UNWIND_HINT_FUNC movq (%rdi), %rdi -.endm - -/* - * Interrupt entry/exit. - * - * Interrupt entry points save only callee clobbered registers in fast path. - * - * Entry runs with interrupts off. - */ -/* 8(%rsp): ~(interrupt number) */ -ENTRY(interrupt_entry) - UNWIND_HINT_FUNC - ASM_CLAC - cld - - testb $3, CS-ORIG_RAX+8(%rsp) - jz 1f - SWAPGS - DO_SWITCH_TO_THREAD_STACK 1: PUSH_AND_CLEAR_REGS save_ret=1
@@ -592,7 +604,7 @@ ENTRY(interrupt_entry) * * We need to tell lockdep that IRQs are off. We can't do this until * we fix gsbase, and we should do it before enter_from_user_mode - * (which can take locks). Since TRACE_IRQS_OFF idempotent, + * (which can take locks). Since TRACE_IRQS_OFF is idempotent, * the simplest way to handle it is to just call it twice if * we enter from user mode. There's no reason to optimize this since * TRACE_IRQS_OFF is a no-op if lockdep is off.
@@ -609,6 +621,9 @@ ENTRY(interrupt_entry) ret END(interrupt_entry) + +/* Interrupt entry/exit. */ + /* * The interrupt stubs push (~vector+0x80) onto the stack and * then jump to common_interrupt.
@@ -878,17 +893,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) -#if defined(CONFIG_IA32_EMULATION) -/* entry_64_compat.S::entry_INT80_compat expects this to be an ASM function */ -ENTRY(switch_to_thread_stack) - UNWIND_HINT_FUNC - - DO_SWITCH_TO_THREAD_STACK - - ret -END(switch_to_thread_stack) -#endif - .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 ENTRY(\sym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8
arch/x86/entry/entry_64_compat.S +15 -2
@@ -347,10 +347,23 @@ ENTRY(entry_INT80_compat) */ movl %eax, %eax + /* switch to thread stack expects orig_ax and rdi to be pushed */ pushq %rax /* pt_regs->orig_ax */ + pushq %rdi /* pt_regs->di */ + + /* Need to switch before accessing the thread stack. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + movq %rsp, %rdi + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + + pushq 6*8(%rdi) /* regs->ss */ + pushq 5*8(%rdi) /* regs->rsp */ + pushq 4*8(%rdi) /* regs->eflags */ + pushq 3*8(%rdi) /* regs->cs */ + pushq 2*8(%rdi) /* regs->ip */ + pushq 1*8(%rdi) /* regs->orig_ax */ - /* switch to thread stack expects orig_ax to be pushed */ - call switch_to_thread_stack + movq (%rdi), %rdi /* restore %rdi */ pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */