SMP alternatives
Subsystems (2)
PATCH 1/1 SMP alternatives
Hi Andrew,
Can you include the patch in -mm to give it some testing? Then merge maybe for 2.6.17? Posted last time in december, with nobody complaining any more about the most recent version. The patch is almost unmodified since, I've only had to add a small chunk due to the mutex merge. Description is below, the patch (against -rc1-git4 snapshot) is attached.
cheers,
Gerd
=====================================================================
This patch implements SMP alternatives, i.e. switching at runtime between different code versions for UP and SMP. The code can patch both SMP->UP and UP->SMP. The UP->SMP case is useful for CPU hotplug.
With CONFIG_CPU_HOTPLUG enabled the code switches to UP at boot time and when the number of CPUs goes down to 1, and switches to SMP when the number of CPUs goes up to 2.
Without CONFIG_CPU_HOTPLUG or on non-SMP-capable systems the code is patched once at boot time (if needed) and the tables are released afterwards.
The changes in detail:
* The current alternatives bits are moved to a separate file, the SMP alternatives code is added there.
* The patch adds some new elf sections to the kernel:
.smp_altinstructions
like .altinstructions, also contains a list
of alt_instr structs.
.smp_altinstr_replacement
like .altinstr_replacement, but also has some space to save original instruction before replaving it.
.smp_locks
list of pointers to lock prefixes which can be nop'ed out on UP.
The first two are used to replace more complex instruction sequences such as spinlocks and semaphores. It would be possible to deal with the lock prefixes with that as well, but by handling them as special case the table sizes become much smaller.
* The sections are page-aligned and padded up to page size, so they can be free if they are not needed.
* Splitted the code to release init pages to a separate function and use it to release the elf sections if they are unused.
Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]> I'm the hacker formerly known as Gerd Knorr. http://www.suse.de/~kraxel/just-married.jpeg
This patch implements SMP alternatives, i.e. switching at runtime between different code versions for UP and SMP. The code can patch both SMP->UP and UP->SMP. The UP->SMP case is useful for CPU hotplug.
With CONFIG_CPU_HOTPLUG enabled the code switches to UP at boot time and when the number of CPUs goes down to 1, and switches to SMP when the number of CPUs goes up to 2.
Without CONFIG_CPU_HOTPLUG or on non-SMP-capable systems the code is patched once at boot time (if needed) and the tables are released afterwards.
The changes in detail:
* The current alternatives bits are moved to a separate file, the SMP alternatives code is added there.
* The patch adds some new elf sections to the kernel:
.smp_altinstructions
like .altinstructions, also contains a list
of alt_instr structs.
.smp_altinstr_replacement
like .altinstr_replacement, but also has some space to save original instruction before replaving it.
.smp_locks
list of pointers to lock prefixes which can be nop'ed out on UP.
The first two are used to replace more complex instruction sequences such as spinlocks and semaphores. It would be possible to deal with the lock prefixes with that as well, but by handling them as special case the table sizes become much smaller.
* The sections are page-aligned and padded up to page size, so they can be free if they are not needed.
* Splitted the code to release init pages to a separate function and use it to release the elf sections if they are unused.
arch/i386/kernel/Makefile +1 -1
@@ -7,7 +7,7 @@ extra-y := head.o init_task.o vmlinux.ld obj-y := process.o semaphore.o signal.o entry.o traps.o irq.o \ ptrace.o time.o ioport.o ldt.o setup.o i8259.o sys_i386.o \ pci-dma.o i386_ksyms.o i387.o dmi_scan.o bootflag.o \ - quirks.o i8237.o + quirks.o i8237.o alternative.o obj-y += cpu/ obj-y += timers/
arch/i386/kernel/alternative.c new file +320 -0
320 lines not on this page · raw message
arch/i386/kernel/module.c +22 -10
@@ -104,26 +104,38 @@ int apply_relocate_add(Elf32_Shdr *sechdreturn -ENOEXEC ; } -extern void apply_alternatives (void *start,void *end); -int module_finalize (const Elf_Ehdr *hdr,const Elf_Shdr *sechdrs,struct module *me) { -const Elf_Shdr *s; +const Elf_Shdr *s, *text =NULL , *alt =NULL , *locks =NULL ;char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; -/* look for .altinstructions to patch */ for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) { -void *seg; -if (strcmp (".altinstructions" , secstrings + s->sh_name)) -continue ; - seg = (void *)s->sh_addr; -apply_alternatives (seg, seg + s->sh_size); - } +if (!strcmp (".text" , secstrings + s->sh_name)) + text = s; +if (!strcmp (".altinstructions" , secstrings + s->sh_name)) + alt = s; +if (!strcmp (".smp_locks" , secstrings + s->sh_name)) + locks= s; + } + +if (alt) { +/* patch .altinstructions */ +void *aseg = (void *)alt->sh_addr; +apply_alternatives (aseg, aseg + alt->sh_size); + } +if (locks && text) { +void *lseg = (void *)locks->sh_addr; +void *tseg = (void *)text->sh_addr; +alternatives_smp_module_add (me, me->name, + lseg, lseg + locks->sh_size, + tseg, tseg + text->sh_size); + }return 0 ; }void module_arch_cleanup (struct module *mod) { +alternatives_smp_module_del (mod); }
arch/i386/kernel/semaphore.c +4 -4
@@ -110,11 +110,11 @@ asm(".align 4\n" ".globl __write_lock_failed\n" "__write_lock_failed:\n\t" -LOCK "addl $" RW_LOCK_BIAS_STR ",(%eax)\n" +LOCK_PREFIX "addl $" RW_LOCK_BIAS_STR ",(%eax)\n" "1: rep; nop\n\t" "cmpl $" RW_LOCK_BIAS_STR ",(%eax)\n\t" "jne 1b\n\t" -LOCK "subl $" RW_LOCK_BIAS_STR ",(%eax)\n\t" +LOCK_PREFIX "subl $" RW_LOCK_BIAS_STR ",(%eax)\n\t" "jnz __write_lock_failed\n\t" "ret" );
@@ -124,11 +124,11 @@ asm(".align 4\n" ".globl __read_lock_failed\n" "__read_lock_failed:\n\t" -LOCK "incl (%eax)\n" +LOCK_PREFIX "incl (%eax)\n" "1: rep; nop\n\t" "cmpl $1,(%eax)\n\t" "js 1b\n\t" -LOCK "decl (%eax)\n\t" +LOCK_PREFIX "decl (%eax)\n\t" "js __read_lock_failed\n\t" "ret" );
arch/i386/kernel/setup.c +0 -95
101 lines not on this page · raw message
arch/i386/kernel/smpboot.c +3 -0
@@ -909,6 +909,7 @@ static int __devinit do_boot_cpu(int api } ++cpucount; +alternatives_smp_switch (1 );/* * We can't use kernel_thread since we must avoid to
@@ -1368,6 +1369,8 @@ void __cpu_die(unsigned int cpu)/* They ack this in play_dead by setting CPU_DEAD */ if (per_cpu (cpu_state, cpu) ==CPU_DEAD ) {printk ("CPU %d is now offline\n" , cpu); +if (1 ==num_online_cpus ()) +alternatives_smp_switch (0 );return ; }msleep (100 );
arch/i386/kernel/vmlinux.lds.S +20 -0
@@ -68,6 +68,26 @@ SECTIONS *(.data.init_task) } + /* might get freed after init */ + . = ALIGN(4096); + __smp_alt_begin = .; + __smp_alt_instructions = .; + .smp_altinstructions : AT(ADDR(.smp_altinstructions) - LOAD_OFFSET) { + *(.smp_altinstructions) + } + __smp_alt_instructions_end = .; + . = ALIGN(4); + __smp_locks = .; + .smp_locks : AT(ADDR(.smp_locks) - LOAD_OFFSET) { + *(.smp_locks) + } + __smp_locks_end = .; + .smp_altinstr_replacement : AT(ADDR(.smp_altinstr_replacement) - LOAD_OFFSET) { + *(.smp_altinstr_replacement) + } + . = ALIGN(4096); + __smp_alt_end = .; + /* will be freed after init */ . = ALIGN(4096); /* Init code and data */ __init_begin = .;
arch/i386/mm/init.c +22 -23
60 lines not on this page · raw message
include/asm-i386/alternative.h new file +129 -0
129 lines not on this page · raw message
include/asm-i386/atomic.h +11 -17
@@ -10,12 +10,6 @@* resource counting etc.. */ -#ifdef CONFIG_SMP -#define LOCK "lock ; " -#else -#define LOCK "" -#endif -/* * Make sure gcc doesn't try to be clever and move things around * on us. We need to use _exactly_ the address the user gave us,
@@ -52,7 +46,7 @@ typedef struct { volatile int counter; }static __inline__void atomic_add (int i,atomic_t *v) { __asm__ __volatile__( -LOCK "addl %1,%0" +LOCK_PREFIX "addl %1,%0" :"=m" (v->counter) :"ir" (i),"m" (v->counter)); }
@@ -67,7 +61,7 @@ static __inline__ void atomic_add(int i,static __inline__void atomic_sub (int i,atomic_t *v) { __asm__ __volatile__( -LOCK "subl %1,%0" +LOCK_PREFIX "subl %1,%0" :"=m" (v->counter) :"ir" (i),"m" (v->counter)); }
@@ -86,7 +80,7 @@ static __inline__ int atomic_sub_and_tesunsigned char c; __asm__ __volatile__( -LOCK "subl %2,%0; sete %1" +LOCK_PREFIX "subl %2,%0; sete %1" :"=m" (v->counter),"=qm" (c) :"ir" (i),"m" (v->counter) :"memory" );return c;
@@ -101,7 +95,7 @@ static __inline__ int atomic_sub_and_tesstatic __inline__void atomic_inc (atomic_t *v) { __asm__ __volatile__( -LOCK "incl %0" +LOCK_PREFIX "incl %0" :"=m" (v->counter) :"m" (v->counter)); }
@@ -115,7 +109,7 @@ static __inline__ void atomic_inc(atomicstatic __inline__void atomic_dec (atomic_t *v) { __asm__ __volatile__( -LOCK "decl %0" +LOCK_PREFIX "decl %0" :"=m" (v->counter) :"m" (v->counter)); }
@@ -133,7 +127,7 @@ static __inline__ int atomic_dec_and_tesunsigned char c; __asm__ __volatile__( -LOCK "decl %0; sete %1" +LOCK_PREFIX "decl %0; sete %1" :"=m" (v->counter),"=qm" (c) :"m" (v->counter) :"memory" );return c !=0 ;
@@ -152,7 +146,7 @@ static __inline__ int atomic_inc_and_tesunsigned char c; __asm__ __volatile__( -LOCK "incl %0; sete %1" +LOCK_PREFIX "incl %0; sete %1" :"=m" (v->counter),"=qm" (c) :"m" (v->counter) :"memory" );return c !=0 ;
@@ -172,7 +166,7 @@ static __inline__ int atomic_add_negativunsigned char c; __asm__ __volatile__( -LOCK "addl %2,%0; sets %1" +LOCK_PREFIX "addl %2,%0; sets %1" :"=m" (v->counter),"=qm" (c) :"ir" (i),"m" (v->counter) :"memory" );return c;
@@ -195,7 +189,7 @@ static __inline__ int atomic_add_return(/* Modern 486+ processor */ __i = i; __asm__ __volatile__( -LOCK "xaddl %0, %1;" +LOCK_PREFIX "xaddl %0, %1;" :"=r" (i) :"m" (v->counter),"0" (i));return i + __i;
@@ -242,11 +236,11 @@ static __inline__ int atomic_sub_return(/* These are x86-specific, used by some header files */ #define atomic_clear_mask (mask, addr) \ -__asm__ __volatile__(LOCK "andl %0,%1" \ +__asm__ __volatile__(LOCK_PREFIX "andl %0,%1" \ : : "r" (~(mask)),"m" (*addr) : "memory")#define atomic_set_mask (mask, addr) \ -__asm__ __volatile__(LOCK "orl %0,%1" \ +__asm__ __volatile__(LOCK_PREFIX "orl %0,%1" \ : : "r" (mask),"m" (*(addr)) : "memory")/* Atomic operations are already serializing on x86 */
include/asm-i386/bitops.h +1 -6
@@ -7,6 +7,7 @@#include <linux/config.h> #include <linux/compiler.h> +#include <asm/alternative.h> /* * These have to be done with inline assembly: that way the bit-setting
@@ -16,12 +17,6 @@* bit 0 is the LSB of addr; bit 32 is the LSB of (addr+1). */ -#ifdef CONFIG_SMP -#define LOCK_PREFIX "lock ; " -#else -#define LOCK_PREFIX "" -#endif -#define ADDR (*(volatile long *) addr)/**
include/asm-i386/cpufeature.h +2 -0
@@ -71,6 +71,8 @@#define X86_FEATURE_P4 (3*32+ 7)/* P4 */ #define X86_FEATURE_CONSTANT_TSC (3*32+ 8)/* TSC ticks at a constant rate */ +#define X86_FEATURE_UP (3*32+ 8)/* smp kernel running on up */ +/* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */ #define X86_FEATURE_XMM3 (4*32+ 0)/* Streaming SIMD Extensions-3 */ #define X86_FEATURE_MWAIT (4*32+ 3)/* Monitor/Mwait support */
include/asm-i386/rwlock.h +30 -26
72 lines not on this page · raw message
include/asm-i386/semaphore.h +4 -4
@@ -99,7 +99,7 @@ static inline void down(struct semaphoremight_sleep (); __asm__ __volatile__("# atomic down operation\n\t" -LOCK "decl %0\n\t" /* --sem->count */ +LOCK_PREFIX "decl %0\n\t" /* --sem->count */ "js 2f\n" "1:\n" LOCK_SECTION_START("")
@@ -123,7 +123,7 @@ static inline int down_interruptible(strmight_sleep (); __asm__ __volatile__("# atomic interruptible down operation\n\t" -LOCK "decl %1\n\t" /* --sem->count */ +LOCK_PREFIX "decl %1\n\t" /* --sem->count */ "js 2f\n\t" "xorl %0,%0\n" "1:\n"
@@ -148,7 +148,7 @@ static inline int down_trylock(struct se __asm__ __volatile__("# atomic interruptible down operation\n\t" -LOCK "decl %1\n\t" /* --sem->count */ +LOCK_PREFIX "decl %1\n\t" /* --sem->count */ "js 2f\n\t" "xorl %0,%0\n" "1:\n"
@@ -173,7 +173,7 @@ static inline void up(struct semaphore * { __asm__ __volatile__("# atomic up operation\n\t" -LOCK "incl %0\n\t" /* ++sem->count */ +LOCK_PREFIX "incl %0\n\t" /* ++sem->count */ "jle 2f\n" "1:\n" LOCK_SECTION_START("")
include/asm-i386/spinlock.h +13 -8
@@ -48,18 +48,23 @@"jmp 1b\n" \ "4:\n\t" +#define __raw_spin_lock_string_up \ + "\n\tdecb %0" +static inline void __raw_spin_lock (raw_spinlock_t *lock) { - __asm__ __volatile__( - __raw_spin_lock_string - :"=m" (lock->slock) : : "memory"); +alternative_smp ( + __raw_spin_lock_string, + __raw_spin_lock_string_up, + "=m" (lock->slock) : : "memory"); }static inline void __raw_spin_lock_flags (raw_spinlock_t *lock,unsigned long flags) { - __asm__ __volatile__( - __raw_spin_lock_string_flags - :"=m" (lock->slock) : "r" (flags) : "memory"); +alternative_smp ( + __raw_spin_lock_string_flags, + __raw_spin_lock_string_up, + "=m" (lock->slock) : "r" (flags) : "memory"); } static inline int __raw_spin_trylock(raw_spinlock_t *lock)
@@ -178,12 +183,12 @@ static inline int __raw_write_trylock(rastatic inline void __raw_read_unlock (raw_rwlock_t *rw) { - asmvolatile ("lock ; incl %0" :"=m" (rw->lock) : :"memory" ); + asmvolatile (LOCK_PREFIX "incl %0" :"=m" (rw->lock) : :"memory" ); }static inline void __raw_write_unlock (raw_rwlock_t *rw) { - asmvolatile ("lock ; addl $" RW_LOCK_BIAS_STR ", %0" + asmvolatile (LOCK_PREFIX "addl $" RW_LOCK_BIAS_STR ", %0" :"=m" (rw->lock) : :"memory" ); }
include/asm-i386/system.h +0 -61
67 lines not on this page · raw message
arch/um/kernel/um_arch.c +11 -1
@@ -477,6 +477,16 @@ void __init check_bugs(void)check_devanon (); } -void apply_alternatives (void *start,void *end) +void apply_alternatives (struct alt_instr *start,struct alt_instr *end) +{ +} + +void alternatives_smp_module_add (struct module *mod,char *name, +void *locks,void *locks_end, +void *text,void *text_end) +{ +} + +void alternatives_smp_module_del (struct module *mod) { }
include/asm-um/alternative.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef __UM_ALTERNATIVE_H +#define __UM_ALTERNATIVE_H + +#include "asm/arch/alternative.h" + +#endif
include/asm-i386/mutex.h +4 -2
@@ -9,6 +9,8 @@#ifndef _ASM_MUTEX_H#define _ASM_MUTEX_H +#include "asm/alternative.h" +/** * __mutex_fastpath_lock - try to take the lock by moving the count * from 1 to a 0 value
@@ -27,7 +29,7 @@ do { \ typecheck_fn(fastcall void (*)(atomic_t *), fail_fn); \ \ __asm__ __volatile__( \ -LOCK " decl (%%eax) \n" \ +LOCK_PREFIX " decl (%%eax) \n" \" js 2f \n" \ "1: \n" \ \
@@ -83,7 +85,7 @@ do { \ typecheck_fn(fastcall void (*)(atomic_t *), fail_fn); \ \ __asm__ __volatile__( \ -LOCK " incl (%%eax) \n" \ +LOCK_PREFIX " incl (%%eax) \n" \" jle 2f \n" \ "1: \n" \ \
Quoted 7 lines · Gerd Hoffmann
Hi Andrew,
Can you include the patch in -mm to give it some testing? Then merge maybe for 2.6.17? Posted last time in december, with nobody complaining any more about the most recent version. The patch is almost unmodified since, I've only had to add a small chunk due to the mutex merge. Description is below, the patch (against -rc1-git4 snapshot) is attached.
FYI, I have this being used in Ubuntu's kernel right now. It's pretty stable. I have it implemented for x86_64 aswell. I can send you that patch when I get a chance to pull it from the repo cleanly. I did enable a kconfig option and command line option so it can be enabled/disabled by default and also at boot.
Signature · 2 lines
Ben Collins Kernel Developer - Ubuntu Linux
Quoted 5 lines · Ben Collins
FYI, I have this being used in Ubuntu's kernel right now. It's pretty stable. I have it implemented for
x86_64aswell. I can send you that patch when I get a chance to pull it from the repo cleanly. I did enable a kconfig option and command line option so it can be enabled/disabled by default and also at boot.
The x86_64 bits would be very nice. Linus' didn't like the idea to make that a config option, commented "either it works or it doesn't", so I left it out. The code is small anyway, and the data tables can be freed, so making it an option doesn't make much sense. IMHO there are way to many config options anyway ...
cheers,
Gerd
Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]> I'm the hacker formerly known as Gerd Knorr. http://www.suse.de/~kraxel/just-married.jpeg
Hi!
Quoted 6 lines
Can you include the patch in -mm to give it some testing? Then merge maybe for 2.6.17? Posted last time in december, with nobody complaining any more about the most recent version. The patch is almost unmodified since, I've only had to add a small chunk due to the mutex merge. Description is below, the patch (against -rc1-git4 snapshot) is attached.
Well, I'm not 100% convinced this is really good idea.. It increases complexity quite a lot.
Oh and please inline patches.
alternatives.c misses header (copyright, GPL).
Pavel
Signature · 1 line
Thanks, Sharp!
Quoted 11 lines · Pavel Machek
Hi!
Quoted 6 lines
Can you include the patch in -mm to give it some testing? Then merge maybe for 2.6.17? Posted last time in december, with nobody complaining any more about the most recent version. The patch is almost unmodified since, I've only had to add a small chunk due to the mutex merge. Description is below, the patch (against -rc1-git4 snapshot) is attached.
Well, I'm not 100% convinced this is really good idea.. It increases complexity quite a lot.
Well, we have alternatives for quite some time already, this is just an extension of the existing bits ...
Quoted 1 line
Oh and please inline patches.
Whats wrong with "Content-Disposition: inline" attachments? The risk they get whitespace-mangeled is much lower then. Also mailers display them inline and also quote them on reply so you can easily comment them.
At least mutt and thunderbird do that. If your mailer doesn't file a
bug ;)
cheers,
Gerd
Signature · 3 lines
Gerd 'just married' Hoffmann <[email protected]> I'm the hacker formerly known as Gerd Knorr. http://www.suse.de/~kraxel/just-married.jpeg
Quoted 10 lines · Gerd Hoffmann
Quoted 9 lines · Pavel Machek
Hi!
Quoted 6 lines
Can you include the patch in -mm to give it some testing? Then merge maybe for 2.6.17? Posted last time in december, with nobody complaining any more about the most recent version. The patch is almost unmodified since, I've only had to add a small chunk due to the mutex merge. Description is below, the patch (against -rc1-git4 snapshot) is attached.
Quoted 5 lines
Quoted 2 lines
Well, I'm not 100% convinced this is really good idea.. It increases complexity quite a lot.
Well, we have alternatives for quite some time already, this is just an extension of the existing bits ...
Like... during suspend we hot-unplug all but one cpu. Patching code at that point is quite unneccessary...
Quoted 7 lines
Quoted 1 line
Oh and please inline patches.
Whats wrong with "Content-Disposition: inline" attachments? The risk they get whitespace-mangeled is much lower then. Also mailers display them inline and also quote them on reply so you can easily comment them.
At least mutt and thunderbird do that. If your mailer doesn't file a
bug ;)
Consensus on lkml is to inline patches. Content-disposition: inline is commonly accepted as not-too-evil, and my mailer (mutt) usually honours that, but something in your mail tripped it.
Pavel
Signature · 1 line
Thanks, Sharp!