x86: simplify interrupt dispatch table
PATCH 1/1 x86: simplify interrupt dispatch table
Interrupt entry points are handled with the following code:
Each 32-byte code block contains seven entry points
... [push][jump 22] // 4 bytes [push][jump 18] // 4 bytes [push][jump 14] // 4 bytes [push][jump 10] // 4 bytes [push][jump 6] // 4 bytes [push][jump 2] // 4 bytes [push][jump common_interrupt][padding] // 8 bytes [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump common_interrupt][padding] [padding_2] common_interrupt:
And there is a table which holds pointers to every entry point,
IOW: to every push.
In cold cache, two jumps are still costlier than one, even though we get the benefit of them residing in the same cacheline.
This change replaces short jumps with near ones to common_interrupt, and pads every push+jump pair to 8 bytes. This way, each interrupt takes only one jump.
This change replaces ".p2align CONFIG_X86_L1_CACHE_SHIFT" before dispatch table with ".align 8" - we do not need anything stronger than that.
The table of entry addresses (the interrupt[] array) is no longer necessary, the address of entries can be easily calculated as (irq_entries_start + i*8).
text data bss dec hex filename
12546 0 0 12546 3102 entry_64.o.before 11626 0 0 11626 2d6a entry_64.o
The size decrease is because 1656 bytes of .init.rodata are gone. That's initdata, though. The resident size does go up a bit.
Run-tested (32 and 64 bits).
--- arch/x86/include/asm/hw_irq.h | 5 ++--- arch/x86/kernel/entry_32.S | 41 ++++++++++------------------------------- arch/x86/kernel/entry_64.S | 41 ++++++++++------------------------------- arch/x86/kernel/irqinit.c | 3 ++- arch/x86/lguest/boot.c | 3 ++- 5 files changed, 26 insertions(+), 67 deletions(-)
arch/x86/include/asm/hw_irq.h +2 -3
@@ -181,10 +181,9 @@ extern __visible void smp_call_function_single_interrupt(struct pt_regs *); extern __visible void smp_invalidate_interrupt(struct pt_regs *);#endif -extern void (*__initconst interrupt[FIRST_SYSTEM_VECTOR - -FIRST_EXTERNAL_VECTOR ])(void ); +extern char irq_entries_start[];#ifdef CONFIG_TRACING -#define trace_interrupt interrupt +#define trace_irq_entries_start irq_entries_start#endif #define VECTOR_UNDEFINED (-1)
arch/x86/kernel/entry_32.S +10 -31
53 lines not on this page · raw message
arch/x86/kernel/entry_64.S +10 -31
54 lines not on this page · raw message
arch/x86/kernel/irqinit.c +2 -1
@@ -178,7 +178,8 @@ void __init native_init_IRQ(void)#endif for_each_clear_bit_from (i, used_vectors, first_system_vector) {/* IA32_SYSCALL_VECTOR could be used in trap_init already. */ -set_intr_gate (i, interrupt[i -FIRST_EXTERNAL_VECTOR ]); +set_intr_gate (i, irq_entries_start + +8 * (i -FIRST_EXTERNAL_VECTOR )); }#ifdef CONFIG_X86_LOCAL_APIC for_each_clear_bit_from (i, used_vectors,NR_VECTORS )
arch/x86/lguest/boot.c +2 -1
@@ -868,7 +868,8 @@ static void __init lguest_init_IRQ(void)/* Some systems map "vectors" to interrupts weirdly. Not us! */ __this_cpu_write (vector_irq[i], i -FIRST_EXTERNAL_VECTOR );if (i !=SYSCALL_VECTOR ) -set_intr_gate (i, interrupt[i -FIRST_EXTERNAL_VECTOR ]); +set_intr_gate (i, irq_entries_start + +8 * (i -FIRST_EXTERNAL_VECTOR )); }/*
Signature · 1 line
1.8.1.4
Quoted 39 lines · Denys Vlasenko
Interrupt entry points are handled with the following code:
Each 32-byte code block contains seven entry points... [push][jump 22] // 4 bytes [push][jump 18] // 4 bytes [push][jump 14] // 4 bytes [push][jump 10] // 4 bytes [push][jump 6] // 4 bytes [push][jump 2] // 4 bytes [push][jump common_interrupt][padding] // 8 bytes [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump common_interrupt][padding] [padding_2] common_interrupt:And there is a table which holds pointers to every entry point,
IOW: to every push.In cold cache, two jumps are still costlier than one, even though we get the benefit of them residing in the same cacheline.
This change replaces short jumps with near ones to
common_interrupt, and pads every push+jump pair to 8 bytes. This way, each interrupt takes only one jump.This change replaces ".p2align
CONFIG_X86_L1_CACHE_SHIFT" before dispatch table with ".align 8" - we do not need anything stronger than that.The table of entry addresses (the interrupt[] array) is no longer necessary, the address of entries can be easily calculated as (
irq_entries_start+ i*8).
Quoted 6 lines
text data bss dec hex filename
12546 0 0 12546 3102 entry_64.o.before 11626 0 0 11626 2d6a entry_64.oThe size decrease is because 1656 bytes of .init.rodata are gone. That's initdata, though. The resident size does go up a bit.
So I like this a lot, as it's straight, simple and obvious, both to hardware and to humans as well. (This is btw. quite close to the irq entry code layout we used to have historically.)
We could do three other changes that would probably help a lot more in practice than the addition or elimination of a single instruction:
1)
We could try to not spread vectors, as modern APICs seem to handle clustered vectors a lot better and we don't actually use irq priority levels like other OSs so we are free to choose our vectors.
This compresses the I$ footprint a bit more if lots of related irq sources are firing towards the same CPUs that share one or more caches (HT threads, cores, node local siblings).
Even on single-node systems this would still compress the IDT and the entry code cache footprint a bit.
2)
We could allocate the IDT per CPU (or per node), lowering the D$ cache miss costs on NUMA systems. (This, if we allowed the IDTs to diverge, would also allow more irq sources sent to separate vectors.)
The simplest model of this, where each IDT is just a copy of each other, is relatively easy to implement, as the IDT is page aligned and ro mapped already.
3)
We could allocate the entry code itself too per cpu (or per node), lowering the I$ cache miss costs on NUMA systems. This would be a bit trickier to implement, as that part of the image has to be relinked during bootup, but is doable.
I'd do 3) only once we are done with the current audit/cleanup/rewrite of the entry code.
Thanks,
Ingo
Borislav Petkov
maintainerQuoted 62 lines · Denys Vlasenko
Interrupt entry points are handled with the following code:
Each 32-byte code block contains seven entry points... [push][jump 22] // 4 bytes [push][jump 18] // 4 bytes [push][jump 14] // 4 bytes [push][jump 10] // 4 bytes [push][jump 6] // 4 bytes [push][jump 2] // 4 bytes [push][jump common_interrupt][padding] // 8 bytes [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump common_interrupt][padding] [padding_2] common_interrupt:And there is a table which holds pointers to every entry point,
IOW: to every push.In cold cache, two jumps are still costlier than one, even though we get the benefit of them residing in the same cacheline.
This change replaces short jumps with near ones to
common_interrupt, and pads every push+jump pair to 8 bytes. This way, each interrupt takes only one jump.This change replaces ".p2align
CONFIG_X86_L1_CACHE_SHIFT" before dispatch table with ".align 8" - we do not need anything stronger than that.The table of entry addresses (the interrupt[] array) is no longer necessary, the address of entries can be easily calculated as (
irq_entries_start+ i*8).text data bss dec hex filename
12546 0 0 12546 3102 entry_64.o.before 11626 0 0 11626 2d6a entry_64.oThe size decrease is because 1656 bytes of .init.rodata are gone. That's initdata, though. The resident size does go up a bit.
Run-tested (32 and 64 bits).
Signed-off-by Denys Vlasenko <[email protected]>CC Linus Torvalds <[email protected]>CC Steven Rostedt <[email protected]>CC Ingo Molnar <[email protected]>CC Borislav Petkov <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Andy Lutomirski <[email protected]>CC Oleg Nesterov <[email protected]>CC Frederic Weisbecker <[email protected]>CC Alexei Starovoitov <[email protected]>CC Will Drewry <[email protected]>CC Kees Cook <[email protected]>
Quoted 7 lines
--- arch/x86/include/asm/hw_irq.h | 5 ++--- arch/x86/kernel/entry_32.S | 41 ++++++++++------------------------------- arch/x86/kernel/entry_64.S | 41 ++++++++++------------------------------- arch/x86/kernel/irqinit.c | 3 ++- arch/x86/lguest/boot.c | 3 ++- 5 files changed, 26 insertions(+), 67 deletions(-)
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--tip-bot for Denys Vlasenko
merge bot[tip:x86/asm] x86/asm/entry/irq: Simplify interrupt dispatch table (IDT) layout
In reply to Denys Vlasenkox86/asm/entry/irq: Simplify interrupt dispatch table (IDT) layout
Commit-ID: 3304c9c37bef30ebd2ef71d986e6568372ce80f8 Gitweb: http://git.kernel.org/tip/3304c9c37bef30ebd2ef71d986e6568372ce80f8 Author: Denys Vlasenko <[email protected]> AuthorDate: Fri, 3 Apr 2015 21:49:13 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 8 Apr 2015 09:02:13 +0200
x86/asm/entry/irq: Simplify interrupt dispatch table (IDT) layout
Interrupt entry points are handled with the following code, each 32-byte code block contains seven entry points:
... [push][jump 22] // 4 bytes [push][jump 18] // 4 bytes [push][jump 14] // 4 bytes [push][jump 10] // 4 bytes [push][jump 6] // 4 bytes [push][jump 2] // 4 bytes [push][jump common_interrupt][padding] // 8 bytes [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump] [push][jump common_interrupt][padding] [padding_2] common_interrupt:
And there is a table which holds pointers to every entry point,
IOW: to every push.
In cold cache, two jumps are still costlier than one, even though we get the benefit of them residing in the same cacheline.
This change replaces short jumps with near ones to 'common_interrupt', and pads every push+jump pair to 8 bytes. This way, each interrupt takes only one jump.
This change replaces ".p2align CONFIG_X86_L1_CACHE_SHIFT" before dispatch table with ".align 8" - we do not need anything stronger than that.
The table of entry addresses (the interrupt[] array) is no longer necessary, the address of entries can be easily calculated as (irq_entries_start + i*8).
text data bss dec hex filename
12546 0 0 12546 3102 entry_64.o.before 11626 0 0 11626 2d6a entry_64.o
The size decrease is because 1656 bytes of .init.rodata are gone. That's initdata, though. The resident size does go up a bit.
Run-tested (32 and 64 bits).
--- arch/x86/include/asm/hw_irq.h | 5 ++--- arch/x86/kernel/entry_32.S | 41 ++++++++++------------------------------- arch/x86/kernel/entry_64.S | 41 ++++++++++------------------------------- arch/x86/kernel/irqinit.c | 3 ++- arch/x86/lguest/boot.c | 3 ++- 5 files changed, 26 insertions(+), 67 deletions(-)
arch/x86/include/asm/hw_irq.h +2 -3
@@ -181,10 +181,9 @@ extern __visible void smp_call_function_single_interrupt(struct pt_regs *); extern __visible void smp_invalidate_interrupt(struct pt_regs *);#endif -extern void (*__initconst interrupt[FIRST_SYSTEM_VECTOR - -FIRST_EXTERNAL_VECTOR ])(void ); +extern char irq_entries_start[];#ifdef CONFIG_TRACING -#define trace_interrupt interrupt +#define trace_irq_entries_start irq_entries_start#endif #define VECTOR_UNDEFINED (-1)
arch/x86/kernel/entry_32.S +10 -31
53 lines not on this page · raw message
arch/x86/kernel/entry_64.S +10 -31
54 lines not on this page · raw message
arch/x86/kernel/irqinit.c +2 -1
@@ -178,7 +178,8 @@ void __init native_init_IRQ(void)#endif for_each_clear_bit_from (i, used_vectors, first_system_vector) {/* IA32_SYSCALL_VECTOR could be used in trap_init already. */ -set_intr_gate (i, interrupt[i -FIRST_EXTERNAL_VECTOR ]); +set_intr_gate (i, irq_entries_start + +8 * (i -FIRST_EXTERNAL_VECTOR )); }#ifdef CONFIG_X86_LOCAL_APIC for_each_clear_bit_from (i, used_vectors,NR_VECTORS )
arch/x86/lguest/boot.c +2 -1
@@ -868,7 +868,8 @@ static void __init lguest_init_IRQ(void)/* Some systems map "vectors" to interrupts weirdly. Not us! */ __this_cpu_write (vector_irq[i], i -FIRST_EXTERNAL_VECTOR );if (i !=SYSCALL_VECTOR ) -set_intr_gate (i, interrupt[i -FIRST_EXTERNAL_VECTOR ]); +set_intr_gate (i, irq_entries_start + +8 * (i -FIRST_EXTERNAL_VECTOR )); }/*