Add SYSTEM_BOOTING_KMALLOC_AVAIL system_state
Subsystems (8)
There are a few places that check the system_state variable to determine if they should use the bootmem or kmalloc allocator. However, this is not accurate as system_state transitions from SYSTEM_BOOTING to SYSTEM_RUNNING well after the bootmem allocator is no longer usable. Introduce the SYSTEM_BOOTING_KMALLOC_AVAIL state which indicates the kmalloc allocator is available for use.
arch/arm/kernel/setup.c +1 -1
@@ -378,7 +378,7 @@ void cpu_init(void)BUG (); } -if (system_state ==SYSTEM_BOOTING ) +if (system_state <SYSTEM_RUNNING )dump_cpu_info (cpu);/*
arch/powerpc/kernel/smp.c +1 -1
@@ -540,7 +540,7 @@ int __devinit start_secondary(void *unusif (smp_ops->take_timebase) smp_ops->take_timebase (); -if (system_state >SYSTEM_BOOTING ) +if (system_state >=SYSTEM_RUNNING )snapshot_timebase ();spin_lock (&call_lock);
include/linux/kernel.h +1 -0
@@ -189,6 +189,7 @@ extern void add_taint(unsigned);/* Values used for system_state */ extern enum system_states { SYSTEM_BOOTING, + SYSTEM_BOOTING_KMALLOC_AVAIL, SYSTEM_RUNNING, SYSTEM_HALT, SYSTEM_POWER_OFF,
init/main.c +1 -0
@@ -513,6 +513,7 @@ asmlinkage void __init start_kernel(voidcpuset_init_early ();mem_init ();kmem_cache_init (); + system_state =SYSTEM_BOOTING_KMALLOC_AVAIL ;setup_per_cpu_pageset ();numa_policy_init ();if (late_time_init)
kernel/sched.c +1 -1
@@ -5917,7 +5917,7 @@ static void calibrate_migration_costs(co-1 #endif ); -if (system_state ==SYSTEM_BOOTING ) { +if (system_state <SYSTEM_RUNNING ) {printk ("migration_cost=" );for (distance =0 ; distance <= max_distance; distance++) {if (distance)
kernel/sys.c +2 -2
@@ -259,7 +259,7 @@ int blocking_notifier_chain_register(str* not yet working and interrupts must remain disabled. At * such times we must not call down_write(). */ -if (unlikely (system_state ==SYSTEM_BOOTING )) +if (unlikely (system_state <SYSTEM_RUNNING ))return notifier_chain_register (&nh->head, n);down_write (&nh->rwsem);
@@ -290,7 +290,7 @@ int blocking_notifier_chain_unregister(s* not yet working and interrupts must remain disabled. At * such times we must not call down_write(). */ -if (unlikely (system_state ==SYSTEM_BOOTING )) +if (unlikely (system_state <SYSTEM_RUNNING ))return notifier_chain_unregister (&nh->head, n);down_write (&nh->rwsem);
mm/page_alloc.c +2 -2
@@ -1782,7 +1782,7 @@ static int __meminit __build_all_zonelisvoid __meminit build_all_zonelists (void ) { -if (system_state ==SYSTEM_BOOTING ) { +if (system_state <SYSTEM_RUNNING ) {__build_all_zonelists (0 );cpuset_init_current_mems_allowed (); }else {
@@ -2136,7 +2136,7 @@ int zone_wait_table_init(struct zone *zo alloc_size = zone->wait_table_hash_nr_entries *sizeof (wait_queue_head_t); -if (system_state ==SYSTEM_BOOTING ) { +if (system_state <SYSTEM_RUNNING ) { zone->wait_table = (wait_queue_head_t *)alloc_bootmem_node (pgdat, alloc_size); }else {
mm/sparse.c +1 -1
@@ -32,7 +32,7 @@ static struct mem_section *sparse_index_unsigned long array_size =SECTIONS_PER_ROOT *sizeof (struct mem_section ); -if (system_state ==SYSTEM_RUNNING ) +if (system_state >=SYSTEM_BOOTING_KMALLOC_AVAIL ) section =kmalloc_node (array_size,GFP_KERNEL , nid);else section =alloc_bootmem_node (NODE_DATA (nid), array_size);
mm/vmscan.c +1 -1
@@ -1477,7 +1477,7 @@ int kswapd_run(int nid) pgdat->kswapd =kthread_run (kswapd, pgdat,"kswapd%d" , nid);if (IS_ERR (pgdat->kswapd)) {/* failure at boot is fatal */ -BUG_ON (system_state ==SYSTEM_BOOTING ); +BUG_ON (system_state <SYSTEM_RUNNING );printk ("Failed to start kswapd on node %d\n" ,nid); ret =-1 ; }
Quoted 7 lines · Mike Kravetz
There are a few places that check the
system_statevariable to
determine if they should use the bootmem or kmalloc allocator. However, this is not accurate assystem_statetransitions fromSYSTEM_BOOTINGtoSYSTEM_RUNNINGwell after the bootmem allocator is no longer usable. Introduce theSYSTEM_BOOTING_KMALLOC_AVAILstate which indicates the kmalloc allocator is available for use.
Let's not do this - system_state is getting out of control.
How about some private boolean in slab.c, and some special allocation function like
void __init *alloc_memory_early(size_t size, gfp_t gfp_flags)
{
if (slab_is_available)
return kmalloc(size, gfp_flags);
return alloc_bootmem(size);
} ?
Quoted 9 lines · Andrew Morton
Quoted 3 lines
SYSTEM_BOOTINGtoSYSTEM_RUNNINGwell after the bootmem allocator is no longer usable. Introduce theSYSTEM_BOOTING_KMALLOC_AVAILstate which indicates the kmalloc allocator is available for use.Let's not do this -
system_stateis getting out of control.How about some private boolean in
slab.c, and some special allocation function like
Quoted 6 lines
void
__init*alloc_memory_early(size_t size,gfp_tgfp_flags){ if (slab_is_available) return kmalloc(size, gfp_flags); return alloc_bootmem(size); }
You may use g_cpucache_up to check the state of the slab allocator.
Quoted 14 lines · Andrew Morton
Quoted 7 lines · Mike Kravetz
There are a few places that check the
system_statevariable to
determine if they should use the bootmem or kmalloc allocator. However, this is not accurate assystem_statetransitions fromSYSTEM_BOOTINGtoSYSTEM_RUNNINGwell after the bootmem allocator is no longer usable. Introduce theSYSTEM_BOOTING_KMALLOC_AVAILstate which indicates the kmalloc allocator is available for use.Let's not do this -
system_stateis getting out of control.How about some private boolean in
slab.c, and some special allocation function like
Quoted 6 lines
void
__init*alloc_memory_early(size_t size,gfp_tgfp_flags){ if (slab_is_available) return kmalloc(size, gfp_flags); return alloc_bootmem(size); }
One issue with that approach is that you can't use it for larger allocations (which we have a lot of at boot-time). Would it be OK to fall back to the raw page allocator for things where kmalloc() fails? Oh, and do we want to make it explicitly NUMA aware?
-- Dave
Quoted 12 lines · Dave Hansen
Quoted 10 lines · Andrew Morton
How about some private boolean in
slab.c, and some special allocation function likevoid
__init*alloc_memory_early(size_t size,gfp_tgfp_flags){ if (slab_is_available) return kmalloc(size, gfp_flags); return alloc_bootmem(size); }
Quoted 4 lines
One issue with that approach is that you can't use it for larger allocations (which we have a lot of at boot-time). Would it be OK to fall back to the raw page allocator for things where
kmalloc()fails? Oh, and do we want to make it explicitly NUMA aware?
Well, I am making it NUMA aware as the first identified user of such a routine does want to make node specific allocations. I haven't thought about the 'large' allocations.
Any thoughts about also including free_memory_early() routines? It seems like a good idea. However, I'm somewhat afraid of these gaining widespread use and the potential mis-use. Using free in the same routine as the alloc (such as in error paths) is generally ok. But, trying to do a free at a later time requires the attention of the coder. Just don't want people to think these wrappers will be a substitute for that analysis.
Signature · 1 line
Mike
PATCH 1/1 alloc_memory_early() routines
In reply to Andrew MortonAdd alloc_memory_early() routines so that code needing to allocate space during boot need not be aware of which allocator is in use. Includes first use of such routine by the SPARSEMEM code.
I did not include support for 'large' allocations as suggested by Dave, or corresponding free_memory_early() routines. The only immediate need is for NUMA/node aware allocation. Others can be added as the needs arise.
include/linux/slab.h +6 -0
@@ -150,10 +150,12 @@ static inline void *kcalloc(size_t n, siextern void kfree (const void *);extern unsigned int ksize (const void *); +extern void *alloc_memory_early (size_t size,gfp_t flags);#ifdef CONFIG_NUMA extern void *kmem_cache_alloc_node (kmem_cache_t *,gfp_t flags,int node);extern void *kmalloc_node (size_t size,gfp_t flags,int node); +extern void *alloc_memory_early_node (size_t size,gfp_t flags,int node);#else static inline void *kmem_cache_alloc_node (kmem_cache_t *cachep,gfp_t flags,int node) {
@@ -163,6 +165,10 @@ static inline void *kmalloc_node(size_t {return kmalloc (size, flags); } +static inline void *alloc_memory_early_node (size_t size,gfp_t flags,int node) +{ +return alloc_memory_early (size, flags); +}#endif extern int FASTCALL (kmem_cache_reap (int ));
mm/slab.c +17 -0
@@ -108,6 +108,7 @@#include <linux/mempolicy.h> #include <linux/mutex.h> #include <linux/rtmutex.h> +#include <linux/bootmem.h> #include <asm/uaccess.h> #include <asm/cacheflush.h>
@@ -3266,8 +3267,24 @@ void *kmalloc_node(size_t size, gfp_t fl return kmem_cache_alloc_node(cachep, flags, node); }EXPORT_SYMBOL (kmalloc_node); + +void *__init alloc_memory_early_node (size_t size,gfp_t flags,int node) +{ +if (g_cpucache_up ==FULL ) +return kmalloc_node (size, flags, node); +else +return alloc_bootmem_node (NODE_DATA (node), size); +}#endif +void *__init alloc_memory_early (size_t size,gfp_t flags) +{ +if (g_cpucache_up ==FULL ) +return kmalloc (size, flags); +else +return alloc_bootmem (size); +} +/** * kmalloc - allocate memory * @size: how many bytes of memory are required.
mm/sparse.c +1 -5
@@ -32,11 +32,7 @@ static struct mem_section *sparse_index_unsigned long array_size =SECTIONS_PER_ROOT *sizeof (struct mem_section ); -if (system_state ==SYSTEM_RUNNING ) - section =kmalloc_node (array_size,GFP_KERNEL , nid); -else - section =alloc_bootmem_node (NODE_DATA (nid), array_size); - + section =alloc_memory_early_node (array_size,GFP_KERNEL , nid);if (section)memset (section,0 , array_size);
Re: [PATCH] alloc_memory_early() routines
Hi Mike,
Quoted 3 lines · Mike Kravetz
diff -Naupr
linux-2.6.17-rc3-mm1/mm/slab.clinux-2.6.17-rc3-mm1.work3/mm/slab.cmm/slab.c 2006-05-09 21:38:23.000000000 +0000 renamed from mm/slab.c 2006-05-03 22:19:16.000000000 +0000 +0
-0
[snip]
mm/slab.c 2006-05-09 21:38:23.000000000 +0000
Quoted 7 lines
mm/slab.c 2006-05-09 21:38:23.000000000 +0000 renamed from mm/slab.c 2006-05-03 22:19:16.000000000 +0000 +7
-0+void * __init alloc_memory_early_node(size_t size, gfp_t flags, int node) +{ + if (g_cpucache_up == FULL) + return kmalloc_node(size, flags, node); + else + return alloc_bootmem_node(NODE_DATA(node), size); +}
I'd prefer you put this in mm/bootmem.c and added a
int slab_is_available(void)
{
return g_cpucache_up == FULL;
}to mm/slab.c instead.
Pekka
Re: [PATCH] alloc_memory_early() routines
Quoted 10 lines · Pekka Enberg
mm/slab.c void *kmalloc_node(size_t size, gfp_t fl · line 3277
Quoted 7 lines
mm/slab.c +7
-0+void *__init alloc_memory_early_node (size_t size,gfp_t flags,int node) +{ +if (g_cpucache_up ==FULL ) +return kmalloc_node (size, flags, node); +else +return alloc_bootmem_node (NODE_DATA (node), size); +}I'd prefer you put this in
mm/bootmem.cand added a
Quoted 6 lines
int
slab_is_available(void){ return g_cpucache_up == FULL; }to
mm/slab.cinstead.
Does slab not available mean that bootmem can be used?
Re: [PATCH] alloc_memory_early() routines
Quoted 16 lines · Pekka Enberg
Quoted 16 lines
mm/slab.c void *kmalloc_node(size_t size, gfp_t fl · line 3277
Quoted 7 lines
mm/slab.c +7
-0+void *__init alloc_memory_early_node (size_t size,gfp_t flags,int node) +{ +if (g_cpucache_up ==FULL ) +return kmalloc_node (size, flags, node); +else +return alloc_bootmem_node (NODE_DATA (node), size); +}I'd prefer you put this in
mm/bootmem.cand added aint
slab_is_available(void){ return g_cpucache_up == FULL; }to
mm/slab.cinstead.
Quoted 1 line · Christoph Lameter
Does slab not available mean that bootmem can be used?
Yes.
Pekka
Re: [PATCH] alloc_memory_early() routines
Quoted 16 lines · Pekka J Enberg
Quoted 15 lines · Pekka Enberg
Quoted 15 lines
mm/slab.c void *kmalloc_node(size_t size, gfp_t fl · line 3277
Quoted 7 lines
mm/slab.c +7
-0+void *__init alloc_memory_early_node (size_t size,gfp_t flags,int node) +{ +if (g_cpucache_up ==FULL ) +return kmalloc_node (size, flags, node); +else +return alloc_bootmem_node (NODE_DATA (node), size); +}I'd prefer you put this in
mm/bootmem.cand added aint
slab_is_available(void){ return g_cpucache_up == FULL; }
Quoted 6 lines
Quoted 1 line
Quoted 1 line
to
mm/slab.cinstead.Quoted 1 line · Christoph Lameter
Does slab not available mean that bootmem can be used?
Yes.
Actually it doesn't - in early boot up there is a phase where even bootmem doesn't work yet.
-Andi
Re: [PATCH] alloc_memory_early() routines
In reply to Christoph LameterQuoted 10 lines · Christoph Lameter
Quoted 8 lines
I'd prefer you put this in
mm/bootmem.cand added aint
slab_is_available(void){ return g_cpucache_up == FULL; }to
mm/slab.cinstead.Does slab not available mean that bootmem can be used?
I like the 'slab_is_available()' check. How about if we simply add this routine and let the people doing the allocation determine what allocator to use?
As has already been stated, slab not available does NOT imply that bootmem can be used. Heck, on POWER there is even an allocator used before bootmem. I doubt we could provide an 'intelligent' routine to works in all cases. So, for right now we could/should just provide the slab not available() check. There is only one piece of code in SPARSEMEM that cares about this.
Sound reasonable?
Signature · 1 line
Mike
Re: [PATCH] alloc_memory_early() routines
Quoted 3 lines · Mike Kravetz
I like the '
slab_is_available()' check. How about if we simply add this routine and let the people doing the allocation determine what allocator to use?
I'm fine with that.
Pekka
Re: [PATCH] alloc_memory_early() routines
In reply to Mike KravetzQuoted 4 lines · Mike Kravetz
I did not include support for 'large' allocations as suggested by Dave, or corresponding
free_memory_early()routines. The only immediate need is for NUMA/node aware allocation. Others can be added as the needs arise.
Sorry if this was already discussed, but you're not supposed to free the memory allocated by alloc_memory_early() ever? If so, please add a kerneldoc stating that.
Pekka