apic: core part for eoi optimization support
Subsystems (3)
v5 cover apic: core part for eoi optimization support
Hi Ingo, please cosider the following patches for tip.git and for 3.5.
This patchset includes just the core bits for the eoi optimization. Please note that even though I numbered it PATCHv5 to stress this is the latest version, in fact there are absolutely no changes in the patches included here since v1. the only change is that I split out kvm bits out to a separate patch.
Avi indicated the preferred merging path is to merge core bits throug tip.git and the kvm bits through kvm. So I'd like it for it to be merged in tip and merged for 3.5: this will make sure APIs for kvm are in place, minor fixes included here will also be helpful. Also in my testing this gives a tiny speedup on some benchmarks, these will get wider testing this way. The kvm parts, posted separately, will be subsequently merged through kvm.git: they reduce the # of exits by about half, details are in the kvm patchset which was posted separately.
---
I'm looking at reducing the interrupt overhead for virtualized guests:
some workloads spend a large part of their time processing interrupts. This patchset supplies infrastructure to reduce the IRQ ack overhead on
x86: the idea is to add an eoi_write callback that we can then optimize without touching other apic functionality.
The main user weill be kvm: on kvm, an EOI write from the guest causes an expensive exit to host; we can avoid this using shared memory as the last patches in the series demonstrate.
But I also wrote a micro-optimized version for the regular x2apic: this shaves off a branch and about 9 instructions from EOI when x2apic is used, and a comment in ack_APIC_irq implies that someone counted instructions there, at some point.
That's patch 4 in the series - if someone's unhappy with this patch specifically this patch can be dropped as nothing else in the series depends on it.
Also included in the patchset are a couple of trivial macro fixes.
The patches work fine on my boxes. See individual patches for perf tests. You need to patch qemu to whitelist the kvm feature. qemu patch will be sent as a reply to this mail.
The patches are against 3.4-rc7 - let me know if
I need to rebase.
Please review, and consider for linux 3.5.
Thanks,
MST
Michael S. Tsirkin (4):
apic: fix typo EIO_ACK -> EOI_ACK and document
apic: use symbolic APIC_EOI_ACK
x86: add apic->eoi_write callback
x86: eoi micro-optimization
arch/x86/include/asm/apic.h | 22 +++++++++++++++++++--- arch/x86/include/asm/apicdef.h | 2 +- arch/x86/kernel/apic/apic_flat_64.c | 2 ++ arch/x86/kernel/apic/apic_noop.c | 1 + arch/x86/kernel/apic/apic_numachip.c | 1 + arch/x86/kernel/apic/bigsmp_32.c | 1 + arch/x86/kernel/apic/es7000_32.c | 2 ++ arch/x86/kernel/apic/numaq_32.c | 1 + arch/x86/kernel/apic/probe_32.c | 1 + arch/x86/kernel/apic/summit_32.c | 1 + arch/x86/kernel/apic/x2apic_cluster.c | 1 + arch/x86/kernel/apic/x2apic_phys.c | 1 + arch/x86/kernel/apic/x2apic_uv_x.c | 1 + arch/x86/platform/visws/visws_quirks.c | 2 +- 14 files changed, 34 insertions(+), 5 deletions(-)
Signature · 1 line
MST
PATCH v5 1/4 apic: fix typo EIO_ACK -> EOI_ACK and document
Fix typo in the macro name and document the
reason it has this value. Update users.
--- arch/x86/include/asm/apicdef.h | 2 +- arch/x86/platform/visws/visws_quirks.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/apicdef.h +1 -1
@@ -37,7 +37,7 @@#define APIC_ARBPRI_MASK 0xFFu#define APIC_PROCPRI 0xA0#define APIC_EOI 0xB0 -#define APIC_EIO_ACK 0x0 +#define APIC_EOI_ACK 0x0/* Docs say 0 for future compat. */ #define APIC_RRR 0xC0#define APIC_LDR 0xD0#define APIC_LDR_MASK (0xFFu << 24)
arch/x86/platform/visws/visws_quirks.c +1 -1
@@ -445,7 +445,7 @@ static void ack_cobalt_irq(struct irq_data *data)spin_lock_irqsave (&cobalt_lock, flags);disable_cobalt_irq (data); -apic_write (APIC_EOI ,APIC_EIO_ACK ); +apic_write (APIC_EOI ,APIC_EOI_ACK );spin_unlock_irqrestore (&cobalt_lock, flags); }
Signature · 1 line
MST
[tip:x86/apic] x86/apic: Fix typo EIO_ACK -> EOI_ACK and document it
Commit-ID: c8f64bf7df9e4858c051b6c1c09582359b97677d Gitweb: http://git.kernel.org/tip/c8f64bf7df9e4858c051b6c1c09582359b97677d Author: Michael S. Tsirkin <[email protected]> AuthorDate: Wed, 16 May 2012 19:03:25 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 18 May 2012 09:46:07 +0200
x86/apic: Fix typo EIO_ACK -> EOI_ACK and document it
Fix typo in the macro name and document the reason it has this value. Update users.
--- arch/x86/include/asm/apicdef.h | 2 +- arch/x86/platform/visws/visws_quirks.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/apicdef.h +1 -1
@@ -37,7 +37,7 @@#define APIC_ARBPRI_MASK 0xFFu#define APIC_PROCPRI 0xA0#define APIC_EOI 0xB0 -#define APIC_EIO_ACK 0x0 +#define APIC_EOI_ACK 0x0/* Docs say 0 for future compat. */ #define APIC_RRR 0xC0#define APIC_LDR 0xD0#define APIC_LDR_MASK (0xFFu << 24)
arch/x86/platform/visws/visws_quirks.c +1 -1
@@ -445,7 +445,7 @@ static void ack_cobalt_irq(struct irq_data *data)spin_lock_irqsave (&cobalt_lock, flags);disable_cobalt_irq (data); -apic_write (APIC_EOI ,APIC_EIO_ACK ); +apic_write (APIC_EOI ,APIC_EOI_ACK );spin_unlock_irqrestore (&cobalt_lock, flags); }
PATCH v5 2/4 apic: use symbolic APIC_EOI_ACK
In reply to Michael S. TsirkinUse the symbol instead of hard-coded numbers, now that the reason for the value is documented where the constant is defined we don't need to duplicate this explanation in code.
--- arch/x86/include/asm/apic.h | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-)
arch/x86/include/asm/apic.h +1 -3
@@ -463,9 +463,7 @@ static inline void ack_APIC_irq(void)* ack_APIC_irq() actually gets compiled as a single instruction * ... yummie. */ - -/* Docs say use 0 for future compatibility */ -apic_write (APIC_EOI ,0 ); +apic_write (APIC_EOI ,APIC_EOI_ACK ); }static inline unsigned default_get_apic_id (unsigned long x)
Signature · 1 line
MST
[tip:x86/apic] x86/apic: Use symbolic APIC_EOI_ACK
Commit-ID: 4ebcc243901c48ee3baba6bdf179c7315fa8806f Gitweb: http://git.kernel.org/tip/4ebcc243901c48ee3baba6bdf179c7315fa8806f Author: Michael S. Tsirkin <[email protected]> AuthorDate: Wed, 16 May 2012 19:03:44 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 18 May 2012 09:46:08 +0200
x86/apic: Use symbolic APIC_EOI_ACK
Use the symbol instead of hard-coded numbers, now that the reason for the value is documented where the constant is defined we don't need to duplicate this explanation in code.
--- arch/x86/include/asm/apic.h | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-)
arch/x86/include/asm/apic.h +1 -3
@@ -463,9 +463,7 @@ static inline void ack_APIC_irq(void)* ack_APIC_irq() actually gets compiled as a single instruction * ... yummie. */ - -/* Docs say use 0 for future compatibility */ -apic_write (APIC_EOI ,0 ); +apic_write (APIC_EOI ,APIC_EOI_ACK ); }static inline unsigned default_get_apic_id (unsigned long x)
PATCH v5 3/4 x86: add apic->eoi_write callback
In reply to Michael S. TsirkinAdd eoi_write callback so that kvm can override eoi accesses without touching the rest of the apic. As a side-effect, this will enable a micro-optimization for apics using msr.
--- arch/x86/include/asm/apic.h | 15 ++++++++++++++- arch/x86/kernel/apic/apic_flat_64.c | 2 ++ arch/x86/kernel/apic/apic_noop.c | 1 + arch/x86/kernel/apic/apic_numachip.c | 1 + arch/x86/kernel/apic/bigsmp_32.c | 1 + arch/x86/kernel/apic/es7000_32.c | 2 ++ arch/x86/kernel/apic/numaq_32.c | 1 + arch/x86/kernel/apic/probe_32.c | 1 + arch/x86/kernel/apic/summit_32.c | 1 + arch/x86/kernel/apic/x2apic_cluster.c | 1 + arch/x86/kernel/apic/x2apic_phys.c | 1 + arch/x86/kernel/apic/x2apic_uv_x.c | 1 + 12 files changed, 27 insertions(+), 1 deletions(-)
arch/x86/include/asm/apic.h +14 -1
@@ -351,6 +351,13 @@ struct apic {/* apic ops */ u32 (*read)(u32 reg);void (*write)(u32 reg,u32 v); +/* +* eoi_write has the same signature as write. +* Drivers can support both eoi_write and write by passing the same +* callback value. Kernel can override eoi_write and fall back +* on write for EOI. +*/ +void (*eoi_write)(u32 reg,u32 v);u64 (*icr_read)(void);void (*icr_write)(u32 low,u32 high);void (*wait_icr_idle)(void );
@@ -426,6 +433,11 @@ static inline void apic_write(u32 reg, u32 val) apic->write (reg, val); } +static inline void apic_eoi (void ) +{ + apic->eoi_write (APIC_EOI ,APIC_EOI_ACK ); +} +static inline u64 apic_icr_read (void ) {return apic->icr_read ();
@@ -450,6 +462,7 @@ static inline u32 safe_apic_wait_icr_idle(void)static inline u32 apic_read (u32 reg) {return 0 ; }static inline void apic_write (u32 reg,u32 val) { } +static inline void apic_eoi (void ) { }static inline u64 apic_icr_read (void ) {return 0 ; }static inline void apic_icr_write (u32 low,u32 high) { }static inline void apic_wait_icr_idle (void ) { }
@@ -463,7 +476,7 @@ static inline void ack_APIC_irq(void)* ack_APIC_irq() actually gets compiled as a single instruction * ... yummie. */ -apic_write (APIC_EOI ,APIC_EOI_ACK ); +apic_eoi (); }static inline unsigned default_get_apic_id (unsigned long x)
arch/x86/kernel/apic/apic_flat_64.c +2 -0
@@ -227,6 +227,7 @@ static struct apic apic_flat = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
@@ -386,6 +387,7 @@ static struct apic apic_physflat = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/apic_noop.c +1 -0
@@ -181,6 +181,7 @@ struct apic apic_noop = { .read = noop_apic_read, .write = noop_apic_write, + .eoi_write = noop_apic_write, .icr_read = noop_apic_icr_read, .icr_write = noop_apic_icr_write, .wait_icr_idle = noop_apic_wait_icr_idle,
arch/x86/kernel/apic/apic_numachip.c +1 -0
@@ -295,6 +295,7 @@ static struct apic apic_numachip __refconst = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/bigsmp_32.c +1 -0
@@ -248,6 +248,7 @@ static struct apic apic_bigsmp = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/es7000_32.c +2 -0
@@ -678,6 +678,7 @@ static struct apic __refdata apic_es7000_cluster = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
@@ -742,6 +743,7 @@ static struct apic __refdata apic_es7000 = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/numaq_32.c +1 -0
@@ -530,6 +530,7 @@ static struct apic __refdata apic_numaq = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/probe_32.c +1 -0
@@ -142,6 +142,7 @@ static struct apic apic_default = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/summit_32.c +1 -0
@@ -546,6 +546,7 @@ static struct apic apic_summit = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_cluster.c +1 -0
@@ -260,6 +260,7 @@ static struct apic apic_x2apic_cluster = { .read = native_apic_msr_read, .write = native_apic_msr_write, + .eoi_write = native_apic_msr_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_phys.c +1 -0
@@ -172,6 +172,7 @@ static struct apic apic_x2apic_phys = { .read = native_apic_msr_read, .write = native_apic_msr_write, + .eoi_write = native_apic_msr_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_uv_x.c +1 -0
@@ -404,6 +404,7 @@ static struct apic __refdata apic_x2apic_uv_x = { .read = native_apic_msr_read, .write = native_apic_msr_write, + .eoi_write = native_apic_msr_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
Signature · 1 line
MST
[tip:x86/apic] x86/apic: Add apic->eoi_write() callback
Commit-ID: 2a43195d831997551da93e6b3c22c965e93fe9cc Gitweb: http://git.kernel.org/tip/2a43195d831997551da93e6b3c22c965e93fe9cc Author: Michael S. Tsirkin <[email protected]> AuthorDate: Wed, 16 May 2012 19:03:52 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 18 May 2012 09:46:08 +0200
x86/apic: Add apic->eoi_write() callback
Add eoi_write callback so that kvm can override eoi accesses without touching the rest of the apic. As a side-effect, this will enable a micro-optimization for apics using msr.
--- arch/x86/include/asm/apic.h | 16 +++++++++++++++- arch/x86/kernel/apic/apic_flat_64.c | 2 ++ arch/x86/kernel/apic/apic_noop.c | 1 + arch/x86/kernel/apic/apic_numachip.c | 1 + arch/x86/kernel/apic/bigsmp_32.c | 1 + arch/x86/kernel/apic/es7000_32.c | 2 ++ arch/x86/kernel/apic/numaq_32.c | 1 + arch/x86/kernel/apic/probe_32.c | 1 + arch/x86/kernel/apic/summit_32.c | 1 + arch/x86/kernel/apic/x2apic_cluster.c | 1 + arch/x86/kernel/apic/x2apic_phys.c | 1 + arch/x86/kernel/apic/x2apic_uv_x.c | 1 + 12 files changed, 28 insertions(+), 1 deletions(-)
arch/x86/include/asm/apic.h +15 -1
@@ -351,6 +351,14 @@ struct apic {/* apic ops */ u32 (*read)(u32 reg);void (*write)(u32 reg,u32 v); +/* +* ->eoi_write() has the same signature as ->write(). +* +* Drivers can support both ->eoi_write() and ->write() by passing the same +* callback value. Kernel can override ->eoi_write() and fall back +* on write for EOI. +*/ +void (*eoi_write)(u32 reg,u32 v);u64 (*icr_read)(void);void (*icr_write)(u32 low,u32 high);void (*wait_icr_idle)(void );
@@ -426,6 +434,11 @@ static inline void apic_write(u32 reg, u32 val) apic->write (reg, val); } +static inline void apic_eoi (void ) +{ + apic->eoi_write (APIC_EOI ,APIC_EOI_ACK ); +} +static inline u64 apic_icr_read (void ) {return apic->icr_read ();
@@ -450,6 +463,7 @@ static inline u32 safe_apic_wait_icr_idle(void)static inline u32 apic_read (u32 reg) {return 0 ; }static inline void apic_write (u32 reg,u32 val) { } +static inline void apic_eoi (void ) { }static inline u64 apic_icr_read (void ) {return 0 ; }static inline void apic_icr_write (u32 low,u32 high) { }static inline void apic_wait_icr_idle (void ) { }
@@ -463,7 +477,7 @@ static inline void ack_APIC_irq(void)* ack_APIC_irq() actually gets compiled as a single instruction * ... yummie. */ -apic_write (APIC_EOI ,APIC_EOI_ACK ); +apic_eoi (); }static inline unsigned default_get_apic_id (unsigned long x)
arch/x86/kernel/apic/apic_flat_64.c +2 -0
@@ -227,6 +227,7 @@ static struct apic apic_flat = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
@@ -386,6 +387,7 @@ static struct apic apic_physflat = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/apic_noop.c +1 -0
@@ -181,6 +181,7 @@ struct apic apic_noop = { .read = noop_apic_read, .write = noop_apic_write, + .eoi_write = noop_apic_write, .icr_read = noop_apic_icr_read, .icr_write = noop_apic_icr_write, .wait_icr_idle = noop_apic_wait_icr_idle,
arch/x86/kernel/apic/apic_numachip.c +1 -0
@@ -295,6 +295,7 @@ static struct apic apic_numachip __refconst = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/bigsmp_32.c +1 -0
@@ -248,6 +248,7 @@ static struct apic apic_bigsmp = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/es7000_32.c +2 -0
@@ -678,6 +678,7 @@ static struct apic __refdata apic_es7000_cluster = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
@@ -742,6 +743,7 @@ static struct apic __refdata apic_es7000 = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/numaq_32.c +1 -0
@@ -530,6 +530,7 @@ static struct apic __refdata apic_numaq = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/probe_32.c +1 -0
@@ -142,6 +142,7 @@ static struct apic apic_default = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/summit_32.c +1 -0
@@ -546,6 +546,7 @@ static struct apic apic_summit = { .read = native_apic_mem_read, .write = native_apic_mem_write, + .eoi_write = native_apic_mem_write, .icr_read = native_apic_icr_read, .icr_write = native_apic_icr_write, .wait_icr_idle = native_apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_cluster.c +1 -0
@@ -260,6 +260,7 @@ static struct apic apic_x2apic_cluster = { .read = native_apic_msr_read, .write = native_apic_msr_write, + .eoi_write = native_apic_msr_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_phys.c +1 -0
@@ -172,6 +172,7 @@ static struct apic apic_x2apic_phys = { .read = native_apic_msr_read, .write = native_apic_msr_write, + .eoi_write = native_apic_msr_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_uv_x.c +1 -0
@@ -404,6 +404,7 @@ static struct apic __refdata apic_x2apic_uv_x = { .read = native_apic_msr_read, .write = native_apic_msr_write, + .eoi_write = native_apic_msr_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
PATCH v5 4/4 x86: eoi micro-optimization
In reply to Michael S. TsirkinWe know both register and value for eoi beforehand, so there's no need to check it and no need to do math to calculate the msr. Saves instructions/branches on each EOI when using x2apic.
I'm not sure what kind of tests should one run to check whether this patch is good for performance.
Some data below - in case it's insufficient, this patch can be dropped from the series for now:
I looked at the objdump output to verify that the generated code looks right and actually is shorter.
Some benchmark results below show a tiny but measureable improvement. The tests were run on an AMD box with 24 cpus.
- A clean kernel build after reboot shows
a tiny but measureable improvement in system time which means lower CPU overhead (though not measureable in total time - that is dominated by user time and fluctuates too much):
linux# reboot -f
...
linux# make clean
linux# time make -j 64 LOCALVERSION= 2>&1 > /dev/null
Before:
real 2m52.244s user 35m53.833s sys 6m7.194s
After:
real 2m52.827s user 35m48.916s sys 6m2.305s
- perf micro-benchmarks seem to consistently show a tiny improvement in total time as well but it's below the confidence level of 3 std deviations:
...
0.414666797 seconds time elapsed ( +- 1.29% )
Performance counter stats for 'perf bench sched messaging' (100 runs):
0.395370891 seconds time elapsed ( +- 1.04% )
0.307019664 seconds time elapsed ( +- 0.10% )
0.304738024 seconds time elapsed ( +- 0.08% )--- arch/x86/include/asm/apic.h | 5 +++++ arch/x86/kernel/apic/x2apic_cluster.c | 2 +- arch/x86/kernel/apic/x2apic_phys.c | 2 +- arch/x86/kernel/apic/x2apic_uv_x.c | 2 +- 4 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/include/asm/apic.h +5 -0
@@ -138,6 +138,11 @@ static inline void native_apic_msr_write(u32 reg, u32 v)wrmsr (APIC_BASE_MSR + (reg >>4 ), v,0 ); } +static inline void native_apic_msr_eoi_write (u32 reg,u32 v) +{ +wrmsr (APIC_BASE_MSR + (APIC_EOI >>4 ),APIC_EOI_ACK ,0 ); +} +static inline u32 native_apic_msr_read (u32 reg) {u64 msr;
arch/x86/kernel/apic/x2apic_cluster.c +1 -1
@@ -260,7 +260,7 @@ static struct apic apic_x2apic_cluster = { .read = native_apic_msr_read, .write = native_apic_msr_write, - .eoi_write = native_apic_msr_write, + .eoi_write = native_apic_msr_eoi_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_phys.c +1 -1
@@ -172,7 +172,7 @@ static struct apic apic_x2apic_phys = { .read = native_apic_msr_read, .write = native_apic_msr_write, - .eoi_write = native_apic_msr_write, + .eoi_write = native_apic_msr_eoi_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_uv_x.c +1 -1
@@ -404,7 +404,7 @@ static struct apic __refdata apic_x2apic_uv_x = { .read = native_apic_msr_read, .write = native_apic_msr_write, - .eoi_write = native_apic_msr_write, + .eoi_write = native_apic_msr_eoi_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
Signature · 1 line
MST
[tip:x86/apic] x86/apic: Implement EIO micro-optimization
Commit-ID: 0ab711ae6ab0db7696b43c74f9ba9de4d7fc1deb Gitweb: http://git.kernel.org/tip/0ab711ae6ab0db7696b43c74f9ba9de4d7fc1deb Author: Michael S. Tsirkin <[email protected]> AuthorDate: Wed, 16 May 2012 19:03:58 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 18 May 2012 09:46:09 +0200
x86/apic: Implement EIO micro-optimization
We know both register and value for eoi beforehand, so there's no need to check it and no need to do math to calculate the msr. Saves instructions/branches on each EOI when using x2apic.
I looked at the objdump output to verify that the generated code looks right and actually is shorter.
The real improvemements will be on the KVM guest side though, those come in a later patch.
--- arch/x86/include/asm/apic.h | 5 +++++ arch/x86/kernel/apic/x2apic_cluster.c | 2 +- arch/x86/kernel/apic/x2apic_phys.c | 2 +- arch/x86/kernel/apic/x2apic_uv_x.c | 2 +- 4 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/include/asm/apic.h +5 -0
@@ -138,6 +138,11 @@ static inline void native_apic_msr_write(u32 reg, u32 v)wrmsr (APIC_BASE_MSR + (reg >>4 ), v,0 ); } +static inline void native_apic_msr_eoi_write (u32 reg,u32 v) +{ +wrmsr (APIC_BASE_MSR + (APIC_EOI >>4 ),APIC_EOI_ACK ,0 ); +} +static inline u32 native_apic_msr_read (u32 reg) {u64 msr;
arch/x86/kernel/apic/x2apic_cluster.c +1 -1
@@ -260,7 +260,7 @@ static struct apic apic_x2apic_cluster = { .read = native_apic_msr_read, .write = native_apic_msr_write, - .eoi_write = native_apic_msr_write, + .eoi_write = native_apic_msr_eoi_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_phys.c +1 -1
@@ -172,7 +172,7 @@ static struct apic apic_x2apic_phys = { .read = native_apic_msr_read, .write = native_apic_msr_write, - .eoi_write = native_apic_msr_write, + .eoi_write = native_apic_msr_eoi_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,
arch/x86/kernel/apic/x2apic_uv_x.c +1 -1
@@ -404,7 +404,7 @@ static struct apic __refdata apic_x2apic_uv_x = { .read = native_apic_msr_read, .write = native_apic_msr_write, - .eoi_write = native_apic_msr_write, + .eoi_write = native_apic_msr_eoi_write, .icr_read = native_x2apic_icr_read, .icr_write = native_x2apic_icr_write, .wait_icr_idle = native_x2apic_wait_icr_idle,