ia64: change defconfig to NR_CPUS==1024
PATCH 1/1 ia64: change defconfig to NR_CPUS==1024
Increase the generic ia64 config from its current max of 512 CPUs to a new max of 1024 CPUs.
The principal file change that implements this is arch/ia64/defconfig, which currently defines CONFIG_NR_CPUSQ2. The change to arch/ia64/Kconfig brings that file's declaration of the NR_CPUS default up to the new reality of 1024. That Kconfig currently says
default "64"
which is bogus -- it is currently 512.
The recent change to arch/ia64/configs/sn2_defconfig just raised that platform-specific defconfig to =1024.
The rationale for increasing the generic ia64 defconfig to 1024 is that the generic config needs to support the new upcoming max CPU count for ia64 platforms, which for ia64/sn ("Altix") platform is 1024, just as the current max of 512 supports the current max CPU count.
The downside is that the ia64 cpumask increases from 8 words to 16. I have tried various heavy workloads and have seen no significant measurable performance regression from this increase. The potential extra cachemiss seems to be lost in the noise. The for_each_*cpu() macros are relatively efficient in skipping past zeroed cpumask bits. Workloads that impose higher loads on the CPU Scheduler tend to bottleneck on non-Scheduler parts of the kernel, and it's the Scheduler which makes the principal use of the cpumask_t, so these extra cachemiss inefficiencies and extra CPU cycles to scan zero mask words just get lost in the general system overhead.
Index: linux/arch/ia64/Kconfig
=================================--- linux.orig/arch/ia64/Kconfig 2006-01-02 19:21:10.000000000 -0800
+++ linux/arch/ia64/Kconfig 2006-01-04 10:32:50.000000000 -0800
@@ -245,7 +245,7 @@
int "Maximum number of CPUs (2-1024)"
range 2 1024
depends on SMP
- default "64"
+ default "1024"
help
You should set this to the number of CPUs in your system, but keep in mind that a kernel compiled for, e.g., 2 CPUs will boot but
Index: linux/arch/ia64/defconfig
=================================--- linux.orig/arch/ia64/defconfig 2006-01-02 19:21:10.000000000 -0800
+++ linux/arch/ia64/defconfig 2006-01-04 10:32:50.000000000 -0800
@@ -98,7 +98,7 @@
# CONFIG_IA64_SGI_SN_XP is not set
CONFIG_FORCE_MAX_ZONEORDER
CONFIG_SMP=y
-CONFIG_NR_CPUSQ2
+CONFIG_NR_CPUS24
CONFIG_HOTPLUG_CPU=y
# CONFIG_SCHED_SMT is not set
# CONFIG_PREEMPT is not set
[email protected] wrote on Thursday, January 05, 2006 1:40 PM
Quoted 3 lines
The downside is that the ia64 cpumask increases from 8 words to 16. I have tried various heavy workloads and have seen no significant measurable performance regression from this increase.
What type of heavy workloads have you measured? Including db transaction processing and decision making workloads?
Quoted 8 lines
The potential
extra cachemiss seems to be lost in the noise. The for_each_*cpu() macros are relatively efficient in skipping past zeroed cpumask bits. Workloads that impose higher loads on the CPU Scheduler tend to bottleneck on non-Scheduler parts of the kernel, and it's the Scheduler which makes the principal use of thecpumask_t, so these extra cachemiss inefficiencies and extra CPU cycles to scan zero mask words just get lost in the general system overhead.
I found above claims are generally false for workload that puts tons of pressure on CPU cache, especially with db workload. Typically for db workload, the working set in user space is so large that making a trip into the kernel has far large secondary effect then the primary cache miss occurred in the kernel. In other word, cache lines evicted by the kernel code have far larger impact to the overall application performance and leads to lower overall lower system performance. So when you say "get lost in the general system overhead", did you consider the secondary effect it does to the application performance?
What we found is going from NR_CPU = 64 to 128, it has small performance impact to db transaction processing workload. Though I have not measured difference between 128 to 1024.
- Ken
From: "Chen, Kenneth W" <[email protected]>
Quoted 2 lines
What type of heavy workloads have you measured? Including db transaction processing and decision making workloads?
I haven't used a db transaction processing benchmark, but I have used other workloads with large process counts and high context-switch rates.
Quoted 9 lines
Quoted 8 lines
The potential
extra cachemiss seems to be lost in the noise. The for_each_*cpu() macros are relatively efficient in skipping past zeroed cpumask bits. Workloads that impose higher loads on the CPU Scheduler tend to bottleneck on non-Scheduler parts of the kernel, and it's the Scheduler which makes the principal use of thecpumask_t, so these extra cachemiss inefficiencies and extra CPU cycles to scan zero mask words just get lost in the general system overhead.
Quoted 9 lines
I found above claims are generally false for workload that puts tons of pressure on CPU cache, especially with db workload. Typically for db workload, the working set in user space is so large that making a trip into the kernel has far large secondary effect then the primary cache miss occurred in the kernel. In other word, cache lines evicted by the kernel code have far larger impact to the overall application performance and leads to lower overall lower system performance. So when you say "get lost in the general system overhead", did you consider the secondary effect it does to the application performance?
The current default is 512p, which is 8 words -- a cacheline. Increasing to 1024p adds an additional 8 words -- one cacheline -- to the cpumask_t. I doubt you're going to see a performance regression on your db transaction processing benchmark because of an additional cachemiss during active or passive load-balancing.
I agree that throughout the kernel we ought to be aware of increasing cachemisses and the lengthening code paths, but I don't believe this particular one is some evil that needs to be suppressed. We have far more micro-performance-impacting algorithms and data structures in the kernel right now that we ought to consider -- e.g., cache coloring conflicts with the struct runqueue -- as well as the obvious algorithm tweaks that greatly affect processor assignments -- e.g., whether or not to call wake_idle().
Quoted 3 lines
What we found is going from
NR_CPU= 64 to 128, it has small performance impact to db transaction processing workload. Though I have not measured difference between 128 to 1024.
Going from 64 (one word) to >64 (an array of words) produces a qualitative change to the emitted code in how the cpumask_t is passed in calling sequences and how it is manipulated. I completely understand that you can detect a small performance regression between 64 and 128. I just don't believe you can conclude that going from 512 to 1024 will exhibit a similar measurable regression.
John Hawkes
Quoted 2 lines · On Thu, 2006-01-05 at 13:39 -0800, [email protected] wrote:
Increase the generic ia64 config from its current max of 512 CPUs to a new max of 1024 CPUs.
I wonder why this would be seen as a sane thing...
Very few people have 1024 cpus, and the ones that do sure would know how to set 1024 in their kernel config. In fact I would argue to lower it. I can see the point of keeping it over 64, to give the more-cpus-than-a-long code more testing, but 512 is too high already.. most people who have ia64 will not have 512 cpus.
(and please don't say "but distributions.." because distributions don't use the defconfigs anyway)
When I tried to build a 2.6.15-mm2 kernel with NR_CPUS 1024, and turned on some of the LOCK DEBUG options:
1160c1160
< # CONFIG_DEBUG_SLAB is not set
---
> CONFIG_DEBUG_SLAB=y
1163,1164c1163,1164
< # CONFIG_DEBUG_SPINLOCK is not set
< # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
---
> CONFIG_DEBUG_SPINLOCK=y
> CONFIG_DEBUG_SPINLOCK_SLEEP=ythen the build failed:
LD .tmp_vmlinux1
arch/ia64/kernel/built-in.o(.init.text+0x68b2): In function `topology_init':
arch/ia64/kernel/ivt.S:1465: undefined reference to `__you_cannot_kmalloc_that_much'
make: *** [.tmp_vmlinux1] Error 1Backing off to 512 CPUs built ok.
There are a couple of kmalloc() calls in topology_init(). I didn't try to figure out which one blew up, nor did I try to investigate further.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 9 lines · Paul Jackson
LD .tmp_vmlinux1 arch/ia64/kernel/built-in.o(.init.text+0x68b2): In function `topology_init': arch/ia64/kernel/ivt.S:1465: undefined reference to `__you_cannot_kmalloc_that_much' make: *** [.tmp_vmlinux1] Error 1Backing off to 512 CPUs built ok.
There are a couple of
kmalloc()calls intopology_init(). I didn't try to figure out which one blew up, nor did I try to investigate further.
Here are two patches that might to cure it (used to be in 2.6.15-rc5-mm1 but the cure accepted for the workqueues was to use alloc_percpu):
From: Christoph Lameter <[email protected]>
The workqueue structure can grow larger than 128k under 2.6.14-rc2 (with all debugging features enabled on 64 bit platforms) which will make kzalloc for workqueue structure entries fail. This patch increases the maximum slab entry size to 256K.
---
include/linux/kmalloc_sizes.h | 2 +- 1 files changed, 1 insertion(+), 1 deletion(-)
include/linux/kmalloc_sizes.h renamed from include/linux/kmalloc_sizes.h~increase-maximum-kmalloc-size-to-256k +1 -1
@@ -19,8 +19,8 @@ CACHE(32768) CACHE(65536) CACHE(131072) -#ifndef CONFIG_MMU CACHE(262144) +#ifndef CONFIG_MMU CACHE(524288) CACHE(1048576)#ifdef CONFIG_LARGE_ALLOCS
_
From: Andrew Morton <[email protected]>
Fix instaoops.
---
mm/slab.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
mm/slab.c renamed from mm/slab.c~increase-maximum-kmalloc-size-to-256k-fix +2 -2
@@ -551,8 +551,8 @@ static void **dbg_userword(kmem_cache_t#define MAX_OBJ_ORDER 13/* up to 32Mb */ #define MAX_GFP_ORDER 13/* up to 32Mb */ #elif defined(CONFIG_MMU ) -#define MAX_OBJ_ORDER 5/* 32 pages */ -#define MAX_GFP_ORDER 5/* 32 pages */ +#define MAX_OBJ_ORDER 6/* 64 pages */ +#define MAX_GFP_ORDER 6/* 64 pages */ #else #define MAX_OBJ_ORDER 8/* up to 1Mb */ #define MAX_GFP_ORDER 8/* up to 1Mb */
_