New per-cpu patch v2.5.1
Hi Rusty,
I appreciate the noble gesture of allowing NUMA people to locate the per-cpu areas in different places in memory ;-)
That said, memcpy of static per-cpu areas doesn't seem right. Why copy the same area to all the dynamically allocated per-cpu areas ? Also, shouldn't the size be &__per_cpu_end - &__per_cpu_start ? And how do we use this with per-cpu data used right
from smp boot, say apic_timer_irqs[] ?
Thanks
Dipankar
Signature · 2 lines
Dipankar Sarma <[email protected]> http://lse.sourceforge.net Linux Technology Center, IBM Software Lab, Bangalore, India.
Quoted 35 lines · In article <[email protected]> Rusty Russell wrote:
After some discussion, this may be a more sane (untested) per-cpu area patch. It dynamically allocated the sections (and discards the original), which would allow (future) NUMA people to make sure their CPU area is allocated near them.
Comments welcome,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
/* Called by boot processor to activate the rest. */ static void__initsmp_init(void)
{(quoted diff) +13
-0+ unsigned int i; + size_t per_cpu_size; + /* Get other processors into their bootup holding patterns. */ smp_boot_cpus(); wait_init_idle = cpu_online_map;@@ -324,6 +328,16 @@ barrier(); } printk("All processors have done init_idle\n"); + + /* Set up per-CPU section pointers. Page align to be safe. */ + per_cpu_size = ((&__per_cpu_end - &__per_cpu_start) + PAGE_SIZE-1) + & ~(PAGE_SIZE-1); + per_cpu_off[0] = kmalloc(per_cpu_size * smp_num_cpus, GFP_KERNEL); + for (i = 0; i < smp_num_cpus; i++) { + per_cpu_off[i] = per_cpu_off[0] + per_cpu_size; + memcpy(per_cpu_off[i], &__per_cpu_start, + __per_cpu_end - &__per_cpu_start); + } } #endif
Quoted 8 lines · On Thu, 20 Dec 2001 17:43:35 +0530 Dipankar Sarma <[email protected]> wrote:
Hi Rusty,
I appreciate the noble gesture of allowing NUMA people to locate the per-cpu areas in different places in memory ;-)
That said, memcpy of static per-cpu areas doesn't seem right. Why copy the same area to all the dynamically allocated per-cpu areas ?
Initialization. Sure, we can't do it with the current macros...
Quoted 1 line
Also, shouldn't the size be
&__per_cpu_end-&__per_cpu_start?
Oops... Yes. (I wrote this just before my Xmas break, and didn't have time to test it).
Quoted 2 lines
And how do we use this with per-cpu data used right from smp boot, say apic_timer_irqs[] ?
No. You can't have everthing (without going into the arch-specific code).
Cheers,
Rusty.
Signature · 1 line
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
In message <[email protected]> you write:
Quoted 10 lines
Hi Rusty,
I appreciate the noble gesture of allowing NUMA people to locate the per-cpu areas in different places in memory ;-)
That said, memcpy of static per-cpu areas doesn't seem right. Why copy the same area to all the dynamically allocated per-cpu areas ? Also, shouldn't the size be
&__per_cpu_end-&__per_cpu_start? And how do we use this with per-cpu data used right
from smp boot, say apic_timer_irqs[] ?
New patch... this one is better, but runs into the same gcc assumptions about doing pointer arith outside the object. Now, noone is putting string literals in the per-cpu areas, but medium-term we probably need the arch-specific asm hack to stop bogosity.
By Anton's request, an arch can define __HAVE_ARCH_CPU_OFFSET if it wants to store each CPU's offset in a spare register to save a mem access.
If no complaints, I'd like this in 2.5 soon, so we can start using it.
Thanks!
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
arch/i386/vmlinux.lds +4 -0
@@ -50,6 +50,10 @@ __initcall_start = .; .initcall.init : { *(.initcall.init) } __initcall_end = .; + . = ALIGN(32); + __per_cpu_start = .; + .data.percpu : { *(.data.percpu) } + __per_cpu_end = .; . = ALIGN(4096); __init_end = .;
arch/ppc/vmlinux.lds +4 -0
@@ -96,6 +96,10 @@ __initcall_start = .; .initcall.init : { *(.initcall.init) } __initcall_end = .; + . = ALIGN(32); + __per_cpu_start = .; + .data.percpu : { *(.data.percpu) } + __per_cpu_end = .; . = ALIGN(4096); __init_end = .;
include/linux/smp.h +15 -1
@@ -71,7 +71,17 @@#define MSG_RESCHEDULE 0x0003/* Reschedule request from master CPU*/ #define MSG_CALL_FUNCTION 0x0004/* Call function on all other CPUs */ -#else +#define __per_cpu_data __attribute__((section(".data.percpu"))) + +#ifndef __HAVE_ARCH_CPU_OFFSET +#define per_cpu_offset (cpu) (__per_cpu_offset[(cpu)]) +#endif + +#define per_cpu (var, cpu) \ +(*(__typeof__(&var)((void *)&var + per_cpu_offset(cpu)))) + +extern unsigned long __per_cpu_offset[NR_CPUS ]; +#else /* !SMP */ /* * These macros fold the SMP functionality into a single CPU system
@@ -88,6 +98,10 @@#define cpu_online_map 1static inline void smp_send_reschedule (int cpu) { }static inline void smp_send_reschedule_all (void ) { } +#define __per_cpu_data +#define per_cpu (var, cpu) var#endif + +#define this_cpu (var) per_cpu(var,smp_processor_id())#endif
init/main.c +26 -0
@@ -305,10 +305,36 @@#else +unsigned long __per_cpu_offset[NR_CPUS ]; +/* Created by linker magic */ +extern char __per_cpu_start, __per_cpu_end; + +static void setup_per_cpu_areas (void ) +{ +unsigned int i; +size_t per_cpu_size; +char *region; + +/* Set up per-CPU offset pointers. Page align to be safe. */ + per_cpu_size = ((&__per_cpu_end - &__per_cpu_start) +PAGE_SIZE -1 ) + & ~(PAGE_SIZE -1 ); + region =kmalloc (per_cpu_size * smp_num_cpus,GFP_KERNEL ); +if (!region) +panic ("Could not allocate per-cpu regions: %u bytes\n" , + per_cpu_size * smp_num_cpus); +for (i =0 ; i < smp_num_cpus; i++) { +memcpy (region + per_cpu_size*i, &__per_cpu_start, + &__per_cpu_end - &__per_cpu_start); + __per_cpu_offset[i] + = (region + per_cpu_size*i) - &__per_cpu_start; + } +}/* Called by boot processor to activate the rest. */ static void __init smp_init (void ) { +setup_per_cpu_areas (); +/* Get other processors into their bootup holding patterns. */ smp_boot_cpus (); wait_init_idle = cpu_online_map;