CONFIG_NR_CPUS, redux
Subsystems (5)
PATCH 1/1 CONFIG_NR_CPUS, redux
Quoted 2 lines · Andrew Morton
In which case,
CONFIG_NR_CPUSis the only way to get the memory back...
Alright, so here we go. Andrew introduced this idea last week and here is a revised, perhaps final, version. We have two goals:
(a) allow NR_CPUS to be set at configure time, to save on memory and gain an unmeasurable boost in speed
(b) let each architecture define a default NR_CPUS, i.e. 64-bit architectures can have an NR_CPUS of 64
We do this fairly easily. In include/linux/threads.h:
#ifdef CONFIG_SMP #define NR_CPUS CONFIG_NR_CPUS #else #define NR_CPUS 1 #endif
and in each arch/xxx/config.in add a configure entry, conditional on CONFIG_SMP, for CONFIG_NR_CPUS with a proper default. If CONFIG_SMP is set in defconfig, set CONFIG_NR_CPUS properly there, too.
Here are the defaults I picked:
CONFIG_NR_CPUS=32: i386, mips, parisc, ppc, sparc
CONFIG_NR_CPUS=64: alpha, ia64, mips64, ppc64, s390, s390x, sparc64, x86-64
No CONFIG_NR_CPUS: arm, cris, sh
Andrew has pointed out some architectures may need minor tweaks to work with NR_CPUS < 32. He discovered and fixed a minor issue on i386... other architectures, please verify non-standard options work. Also make sure 64 works!
Patch is against 2.5.21, enjoy...
Robert Love
arch/alpha/config.in +1 -0
@@ -232,6 +232,7 @@ if [ "$CONFIG_SMP" = "y" ]; then define_bool CONFIG_HAVE_DEC_LOCK y + int 'Maximum number of CPUs (2-64)' CONFIG_NR_CPUS 64 fi if [ "$CONFIG_EXPERIMENTAL" = "y" ]; then
arch/i386/Config.help +8 -0
@@ -25,6 +25,14 @@ If you don't know what to do here, say N. +CONFIG_NR_CPUS + This allows you to specify the maximum number of CPUs which this + kernel will support. The maximum supported value is 32 and the + mimimum value which makes sense is 2. + + This is purely to save memory - each supported CPU adds + approximately eight kilobytes to the kernel image. + CONFIG_PREEMPT This option reduces the latency of the kernel when reacting to real-time or interactive events by allowing a low priority process to
arch/i386/config.in +2 -1
@@ -185,8 +185,8 @@ bool 'Math emulation' CONFIG_MATH_EMULATION bool 'MTRR (Memory Type Range Register) support' CONFIG_MTRR -bool 'Symmetric multi-processing support' CONFIG_SMP bool 'Preemptible Kernel' CONFIG_PREEMPT +bool 'Symmetric multi-processing support' CONFIG_SMP if [ "$CONFIG_SMP" != "y" ]; then bool 'Local APIC support on uniprocessors' CONFIG_X86_UP_APIC dep_bool 'IO-APIC support on uniprocessors' CONFIG_X86_UP_IOAPIC $CONFIG_X86_UP_APIC
@@ -197,6 +197,7 @@ define_bool CONFIG_X86_IO_APIC y fi else + int 'Maximum number of CPUs (2-32)' CONFIG_NR_CPUS 32 bool 'Multiquad NUMA system' CONFIG_MULTIQUAD fi
arch/i386/defconfig +1 -0
@@ -74,6 +74,7 @@ # CONFIG_PREEMPT is not set # CONFIG_MULTIQUAD is not set CONFIG_HAVE_DEC_LOCK=y +CONFIG_NR_CPUS=32 # # General options
arch/i386/kernel/smpboot.c +2 -2
@@ -54,7 +54,7 @@static int smp_b_stepping;/* Setup configured maximum number of CPUs to activate */ -static int max_cpus =-1 ; +static int max_cpus =NR_CPUS ;/* Total count of live CPUs */ int smp_num_cpus =1 ;
@@ -1145,7 +1145,7 @@if (!(phys_cpu_present_map & (1 << bit)))continue ; -if ((max_cpus >=0 ) && (max_cpus <= cpucount+1 )) +if (max_cpus <= cpucount+1 )continue ;do_boot_cpu (apicid);
arch/ia64/config.in +4 -0
@@ -95,6 +95,10 @@ tristate '/proc/pal support' CONFIG_IA64_PALINFO tristate '/proc/efi/vars support' CONFIG_EFI_VARS +if [ "$CONFIG_SMP" = "y" ]; then + int 'Maximum number of CPUs (2-64)' CONFIG_NR_CPUS 64 +fi + tristate 'Kernel support for ELF binaries' CONFIG_BINFMT_ELF tristate 'Kernel support for MISC binaries' CONFIG_BINFMT_MISC
arch/ia64/defconfig +1 -0
@@ -61,6 +61,7 @@ CONFIG_EFI_VARS=y CONFIG_BINFMT_ELF=y # CONFIG_BINFMT_MISC is not set +CONFIG_NR_CPUS=64 # # ACPI Support
arch/mips/config.in +2 -0
@@ -500,6 +500,8 @@ bool 'Magic SysRq key' CONFIG_MAGIC_SYSRQ if [ "$CONFIG_SMP" != "y" ]; then bool 'Run uncached' CONFIG_MIPS_UNCACHED +else + int 'Maximum number of CPUs (2-32)' CONFIG_NR_CPUS 32 fi endmenu
arch/mips64/config.in +2 -0
@@ -247,6 +247,8 @@ bool 'Magic SysRq key' CONFIG_MAGIC_SYSRQ if [ "$CONFIG_SMP" != "y" ]; then bool 'Run uncached' CONFIG_MIPS_UNCACHED +else + int 'Maximum number of CPUs (2-64)' CONFIG_NR_CPUS 64 fi endmenu
arch/mips64/defconfig +1 -0
@@ -32,6 +32,7 @@ # CONFIG_EISA is not set # CONFIG_MCA is not set # CONFIG_SBUS is not set +CONFIG_NR_CPUS=64 # # CPU selection
arch/parisc/config.in +4 -0
@@ -18,6 +18,10 @@ # bool 'Symmetric multi-processing support' CONFIG_SMP define_bool CONFIG_SMP n +if [ "$CONFIG_SMP" = "y" ]; then + int 'Maximum number of CPUs (2-32)' CONFIG_NR_CPUS 32 +fi + bool 'Kernel Debugger support' CONFIG_KWDB # define_bool CONFIG_KWDB n
arch/ppc/config.in +1 -0
@@ -172,6 +172,7 @@ bool 'Symmetric multi-processing support' CONFIG_SMP if [ "$CONFIG_SMP" = "y" ]; then bool ' Distribute interrupts on all CPUs by default' CONFIG_IRQ_ALL_CPUS + int 'Maximum number of CPUs (2-32)' CONFIG_NR_CPUS 32 fi if [ "$CONFIG_SMP" != "y" ]; then bool 'Preemptible Kernel' CONFIG_PREEMPT
arch/ppc64/config.in +1 -0
@@ -21,6 +21,7 @@ bool 'Symmetric multi-processing support' CONFIG_SMP if [ "$CONFIG_SMP" = "y" ]; then bool ' Distribute interrupts on all CPUs by default' CONFIG_IRQ_ALL_CPUS + int 'Maximum number of CPUs (2-64)' CONFIG_NR_CPUS 64 if [ "$CONFIG_PPC_PSERIES" = "y" ]; then bool ' Hardware multithreading' CONFIG_HMT fi
arch/ppc64/defconfig +1 -0
@@ -38,6 +38,7 @@ CONFIG_IRQ_ALL_CPUS=y # CONFIG_HMT is not set # CONFIG_PREEMPT is not set +CONFIG_NR_CPUS=64 # # General setup
arch/s390/config.in +3 -0
@@ -20,6 +20,9 @@ comment 'Processor type and features' bool 'Symmetric multi-processing support' CONFIG_SMP bool 'IEEE FPU emulation' CONFIG_MATHEMU +if [ "$CONFIG_SMP" = "y" ]; then + int 'Maximum number of CPUs (2-32)' CONFIG_NR_CPUS 32 +fi endmenu mainmenu_option next_comment
arch/s390/defconfig +1 -0
@@ -35,6 +35,7 @@ # CONFIG_SMP=y CONFIG_MATHEMU=y +CONFIG_NR_CPUS=64 # # Base setup
arch/s390x/config.in +3 -0
@@ -19,6 +19,9 @@ mainmenu_option next_comment comment 'Processor type and features' bool 'Symmetric multi-processing support' CONFIG_SMP +if [ "$CONFIG_SMP" = "y" ]; then + int 'Maximum number of CPUs (2-64)' CONFIG_NR_CPUS 64 +fi bool 'Kernel support for 31 bit emulation' CONFIG_S390_SUPPORT if [ "$CONFIG_S390_SUPPORT" = "y" ]; then tristate 'Kernel support for 31 bit ELF binaries' CONFIG_BINFMT_ELF32
arch/s390x/defconfig +1 -0
@@ -36,6 +36,7 @@ CONFIG_SMP=y CONFIG_S390_SUPPORT=y CONFIG_BINFMT_ELF32=y +CONFIG_NR_CPUS=64 # # Base setup
arch/sparc/config.in +4 -0
@@ -17,6 +17,10 @@ bool 'Symmetric multi-processing support (does not work on sun4/sun4c)' CONFIG_SMP +if [ "$CONFIG_SMP" = "y" ]; then + int 'Maximum number of CPUs (2-32)' CONFIG_NR_CPUS 32 +fi + # Identify this as a Sparc32 build define_bool CONFIG_SPARC32 y
arch/sparc64/config.in +4 -0
@@ -17,6 +17,10 @@ bool 'Symmetric multi-processing support' CONFIG_SMP bool 'Preemptible kernel' CONFIG_PREEMPT +if [ "$CONFIG_SMP" = "y" ]; then + int 'Maximum number of CPUs (2-64)' CONFIG_NR_CPUS 64 +fi + # Identify this as a Sparc64 build define_bool CONFIG_SPARC64 y
arch/sparc64/defconfig +1 -0
@@ -63,6 +63,7 @@ CONFIG_BINFMT_MISC=m # CONFIG_SUNOS_EMUL is not set CONFIG_SOLARIS_EMUL=m +CONFIG_NR_CPUS=64 # # Parallel port support
arch/x86_64/config.in +5 -2
@@ -45,8 +45,11 @@ #bool 'MTRR (Memory Type Range Register) support' CONFIG_MTRR bool 'Symmetric multi-processing support' CONFIG_SMP bool 'Preemptible Kernel' CONFIG_PREEMPT -if [ "$CONFIG_SMP" = "y" -a "$CONFIG_X86_CMPXCHG" = "y" ]; then - define_bool CONFIG_HAVE_DEC_LOCK y +if [ "$CONFIG_SMP" = "y" ]; then + int 'Maximum number of CPUs (2-64)' CONFIG_NR_CPUS 64 + if [ "$CONFIG_X86_CMPXCHG" = "y" ]; then + define_bool CONFIG_HAVE_DEC_LOCK y + fi fi define_bool CONFIG_X86_MCE y
arch/x86_64/defconfig +1 -0
@@ -51,6 +51,7 @@ CONFIG_HAVE_DEC_LOCK=y CONFIG_X86_MCE=y # CONFIG_X86_MCE_NONFATAL is not set +CONFIG_NR_CPUS=64 # # General options
include/linux/threads.h +8 -2
@@ -8,10 +8,16 @@* /proc/sys/kernel/threads-max. */ +/* +* Maximum supported processors that can run under SMP. This value is +* set via configure setting. The maximum is equal to the size of the +* bitmasks used on that platform, i.e. 32 or 64. Setting this smaller +* saves quite a bit of memory. +*/ #ifdef CONFIG_SMP -#define NR_CPUS 32/* Max processors that can be running in SMP */ +#define NR_CPUS CONFIG_NR_CPUS#else -#define NR_CPUS 1 +#define NR_CPUS 1#endif #define MIN_THREADS_LEFT_FOR_ROOT 4
On 11 Jun 2002, Robert Love wrote:
| Here are the defaults I picked: | | CONFIG_NR_CPUS=32: i386, mips, parisc, ppc, sparc
I don't know what is "typical" for non-x86, but for x86, why not use something more like a 'typical' NR_CPUS for SMP, like 8 (?)... why still waste all of that memory?
| CONFIG_NR_CPUS=64: alpha, ia64, mips64, ppc64, s390, s390x, sparc64, x86-64 | | No CONFIG_NR_CPUS: arm, cris, sh | | Andrew has pointed out some architectures may need minor tweaks to work | with NR_CPUS < 32. He discovered and fixed a minor issue on i386...
What was this problem? I missed it but would like to see it. (or do you know what the Subject: was?)
One spello (typo) below.
| diff -urN linux-2.5.21/arch/i386/Config.help linux/arch/i386/Config.help | --- linux-2.5.21/arch/i386/Config.help Sat Jun 8 22:27:21 2002 | +++ linux/arch/i386/Config.help Sun Jun 9 13:13:02 2002 | @@ -25,6 +25,14 @@ | | If you don't know what to do here, say N. | | +CONFIG_NR_CPUS | + This allows you to specify the maximum number of CPUs which this | + kernel will support. The maximum supported value is 32 and the | + mimimum value which makes sense is 2. --- minimum | + | + This is purely to save memory - each supported CPU adds | + approximately eight kilobytes to the kernel image.
Thanks,
Signature · 1 line
~Randy
Quoted 3 lines · Randy.Dunlap
I don't know what is "typical" for non-x86, but for x86, why not use something more like a 'typical'
NR_CPUSfor SMP, like 8 (?)... why still waste all of that memory?
Well right now we set it to 32...
I think out-of-the-box Linux, with SMP set, should support the maximum number of CPUs. While we do save some memory I do not think it is going to be huge with 8 vs 32.
But whatever you, Linus, and the arch maintainers say... all my boxen are 2-way so I don't care ;)
Quoted 2 lines
What was this problem? I missed it but would like to see it. (or do you know what the Subject: was?)
Not sure to be honest. The subject was "[patch] CONFIG_NR_CPUS"
Or ask Andrew...
Quoted 3 lines
One spello (typo) below.
| + mimimum value which makes sense is 2.
(cough) someone else wrote that ;)
Fixed, thanks.
Robert Love
Quoted 9 lines · Randy.Dunlap
On 11 Jun 2002, Robert Love wrote:
| Here are the defaults I picked: | | CONFIG_NR_CPUS=32: i386, mips, parisc, ppc, sparcI don't know what is "typical" for non-x86, but for x86, why not use something more like a 'typical'
NR_CPUSfor SMP, like 8 (?)... why still waste all of that memory?
Perhaps it's just because I'm coming in late, but I cannot understand why NR_CPUS cannot be as low as 4 by default, for all archs, and then in the kernel boot messages, should more be found than is configured for a message is emitted to say "reconfigure your kernel", and continue with the number it was configured for. I personally only rarely see 2-way boxes, 4-way is pretty rare, and anything more must surely count as very specialized.
Let the defaults be reasonable for 99% of users (IMO 99.9%), and let the rest have to think about config options...
Ruth
Signature · 2 lines
Ruth Ivimey-Cook Software engineer and technical writer.
Quoted 6 lines · Ruth Ivimey-Cook
Perhaps it's just because I'm coming in late, but I cannot understand why
NR_CPUScannot be as low as 4 by default, for all archs, and then in the kernel boot messages, should more be found than is configured for a message is emitted to say "reconfigure your kernel", and continue with the number it was configured for. I personally only rarely see 2-way boxes, 4-way is pretty rare, and anything more must surely count as very specialized.
Ugh let's stop this thread now. Two points:
(a) imo, the kernel should support out-of-the-box the maximum number of CPUs it can handle. Be lucky we now have a configure option to change that. But that does not matter..
(b) Right now it is 32. Now you can change it... if you want to change the current behavior by _default_ why don't we suggest that _after_ this is accepted into 2.5? I.e., one battle at a time.
Thanks,
Robert Love
Quoted 10 lines
Ugh let's stop this thread now. Two points:
(a) imo, the kernel should support out-of-the-box the maximum number of CPUs it can handle. Be lucky we now have a configure option to change that. But that does not matter..
(b) Right now it is 32. Now you can change it... if you want to change the current behavior by _default_ why don't we suggest that _after_ this is accepted into 2.5? I.e., one battle at a time.
Indeed. And before that gets changed, it would be necessary to change the bootstrap procedure not to crash if you have more than NR_CPUS cpus (as Andrew reported it did), but instead to just not configure them ... much less troublesome.
M.
Quoted 5 lines · Martin J. Bligh
Indeed. And before that gets changed, it would be necessary to change the bootstrap procedure not to crash if you have more than
NR_CPUScpus (as Andrew reported it did), but instead to just not configure them ... much less troublesome. M.
There are also fun deadlocks on i386 with "too many cpu's" as it appears the kernel attempts to use logical APIC mode to get beyond the eigth cpu and seems unaware that it can't IPI them that way unless it's configured to always use the clustered hierarchical destination format, though that's perhaps a little beyond just NR_CPUS. Perhaps some kind of sanity checking is needed at the arch level for issues like this as well, not just the absolute maximum number of cpu's?
Cheers,
Bill
Quoted 14 lines · Użytkownik Ruth Ivimey-Cook
Quoted 9 lines · Randy.Dunlap
On 11 Jun 2002, Robert Love wrote:
| Here are the defaults I picked: | | CONFIG_NR_CPUS=32: i386, mips, parisc, ppc, sparcI don't know what is "typical" for non-x86, but for x86, why not use something more like a 'typical'
NR_CPUSfor SMP, like 8 (?)... why still waste all of that memory?
Quoted 8 lines
Perhaps it's just because I'm coming in late, but I cannot understand why
NR_CPUScannot be as low as 4 by default, for all archs, and then in the kernel boot messages, should more be found than is configured for a message is emitted to say "reconfigure your kernel", and continue with the number it was configured for. I personally only rarely see 2-way boxes, 4-way is pretty rare, and anything more must surely count as very specialized.Let the defaults be reasonable for 99% of users (IMO 99.9%), and let the rest have to think about config options...
Actually 2 would only make sense on Intel.
Well and then you have to account for the recent
HT additions so this becoumes 4.
On Sparc 4 is quite common.
But anything above is indeed very very rare.
Quoted 3 lines · Martin Dalecki
Quoted 2 lines
Let the defaults be reasonable for 99% of users (IMO 99.9%), and let the rest have to think about config options...
Actually 2 would only make sense on Intel.
I'm sure the many owners of dual Athlons would beg to differ.
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Quoted 11 lines · Robert Love
Quoted 6 lines · Ruth Ivimey-Cook
Perhaps it's just because I'm coming in late, but I cannot understand why
NR_CPUScannot be as low as 4 by default, for all archs, and then in the kernel boot messages, should more be found than is configured for a message is emitted to say "reconfigure your kernel", and continue with the number it was configured for. I personally only rarely see 2-way boxes, 4-way is pretty rare, and anything more must surely count as very specialized.Ugh let's stop this thread now. Two points:
Quoted 8 lines
(a) imo, the kernel should support out-of-the-box the maximum number of CPUs it can handle. Be lucky we now have a configure option to change that. But that does not matter..
(b) Right now it is 32. Now you can change it... if you want to change the current behavior by _default_ why don't we suggest that _after_ this is accepted into 2.5? I.e., one battle at a time.
By that logic CONFIG_SMP should be "y" by default.
Now i find the name NR_CPUS a bit misleading it seems that this should be MAX_CPUS but "legacy is as legacy does".
Using the names i prefer i would suggest in *config we replace CONFIG_SMP with CONFIG_MAX_CPUS and give it a default of 1. Then make CONFIG_SMP dependant on CONFIG_MAX_CPUS > 1. That way we avoid adding yet another option. KISS for the users.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Quoted 7 lines · Ruth Ivimey-Cook
Perhaps it's just because I'm coming in late, but I cannot understand why
NR_CPUScannot be as low as 4 by default, for all archs, and then in the kernel boot messages, should more be found than is configured for a message is emitted to say "reconfigure your kernel", and continue with the number it was configured for. I personally only rarely see 2-way boxes, 4-way is pretty rare, and anything more must surely count as very specialized.
Why not let the boot process select the highest of two numbers, the (default-low) NR_CPUS and the number of CPU's detected?
Boot with "too many" cpu's and you still get to use them - you merely can't hotplug even more.
Configuring a high NR_CPUS becomes something only hot-pluggers need to do, or those whose architecture doesn't support cpu detection in the early boot process. Those with a fixed number of detectable CPUs can simply go with a default of NR_CPUS=2 no matter what they actually have.
Helge Hafting
Quoted 2 lines · Helge Hafting
Why not let the boot process select the highest of two numbers, the (default-low)
NR_CPUSand the number of CPU's detected?
That seems to address the issue. It allows use of all CPUs in the most common case that they all are in and working at boot.
Quoted 2 lines
Boot with "too many" cpu's and you still get to use them - you merely can't hotplug even more.
My impression is that a fair number of users don't add CPUs anyway, they swap problem parts if runtime diagnostics indicate a failing {fan,VRM,other}.
Quoted 5 lines
Configuring a high
NR_CPUSbecomes something only hot-pluggers need to do, or those whose architecture doesn't support cpu detection in the early boot process. Those with a fixed number of detectable CPUs can simply go with a default of NR_CPUS=2 no matter what they actually have.
Since this would be init code the size is not an issue. I guess a boot-time option would be desirable in case a site wants to boot with fewer processors than are physically present (like mem=) for some reason like benchmarking.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 3 lines · Robert Love
Andrew has pointed out some architectures may need minor tweaks to work with
NR_CPUS< 32. He discovered and fixed a minor issue on i386... other architectures, please verify non-standard options work. Also make
On sparc, setting NR_CPUS to anything other than 32 with CONFIG_SMP set breaks. Simple, stupid patch is attached, but there's a more elegant fix to the actual functions abusing NR_CPUS that just seems much nicer ;-).
And yes, I know, sparc on 2.5 is badly broken, but this just might help keep the broken-ness down a little bit.
Regards,
Rob Radez
include/asm-sparc/smp.h +1 -1
@@ -31,7 +31,7 @@#include <asm/ptrace.h> #include <asm/asi.h> -extern struct prom_cpuinfo linux_cpus[NR_CPUS ]; +extern struct prom_cpuinfo linux_cpus[32 ];/* Per processor Sparc parameters we need. */