X86_64 configurable stack size

Cleanup the allocation and freeing of stacks a bit by using a __GFP_ZERO flag instead of memset.

Cc [email protected]

Cc [email protected]

Signed-off-by Christoph Lameter <[email protected]>

---
 include/asm-x86/thread_info_64.h |   16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)

Index: linux-2.6/include/asm-x86/thread_info_64.h
===================================================================

include/asm-x86/thread_info_64.h +1 -10

@@ -74,20 +74,14 @@ static inline struct thread_info *stack_
 
 /* thread information allocation */
 #ifdef CONFIG_DEBUG_STACK_USAGE
-#define alloc_thread_info(tsk)					\
-    ({								\
-	struct thread_info *ret;				\
-								\
-	ret = ((struct thread_info *) __get_free_pages(GFP_KERNEL,THREAD_ORDER)); \
-	if (ret)						\
-		memset(ret, 0, THREAD_SIZE);			\
-	ret;							\
-    })
+#define THREAD_FLAGS (GFP_KERNEL | __GFP_ZERO)
 #else
-#define alloc_thread_info(tsk) \

Make the stack size configurable necessary. SGI NUMA configurations may need more stack because cpumasks and nodemasks are at times kept on the stack. This patch allows to run with 16k or 32k kernel stacks.

Cc [email protected]

Cc [email protected]

Signed-off-by Christoph Lameter <[email protected]>

---
 arch/x86/Kconfig.x86_64          |    6 ++++++
 include/asm-x86/page_64.h        |    3 +--
 include/asm-x86/thread_info_64.h |    4 ++--
 3 files changed, 9 insertions(+), 4 deletions(-)

Index: linux-2.6/arch/x86/Kconfig.x86_64
===================================================================

arch/x86/Kconfig.x86_64 +6 -0

@@ -378,6 +378,12 @@ config NODES_SHIFT
 	default "6"
 	depends on NEED_MULTIPLE_NODES
 
+config THREAD_ORDER
+	int "Kernel stack size (in page order)"
+	default "1"
+	help
+	  Page order for the thread stack.
+
 # Dummy CONFIG option to select ACPI_NUMA from drivers/acpi/Kconfig.
 
 config X86_64_ACPI_NUMA

Index: linux-2.6/include/asm-x86/page_64.h
===================================================================

include/asm-x86/page_64.h +1 -2

@@ -9,8 +9,7 @@
 #define PAGE_MASK	(~(PAGE_SIZE-1))
 #define PHYSICAL_PAGE_MASK	(~(PAGE_SIZE-1) & __PHYSICAL_MASK)
 
-#define THREAD_ORDER 1 
-#define THREAD_SIZE  (PAGE_SIZE << THREAD_ORDER)
+#define THREAD_SIZE  (PAGE_SIZE << CONFIG_THREAD_ORDER)
 #define CURRENT_MASK (~(THREAD_SIZE-1))
 
 #define EXCEPTION_STACK_ORDER 0

Index: linux-2.6/include/asm-x86/thread_info_64.h
===================================================================

We seem to be growing two different mechanisms here for 32bit and 64bit. This does seem a better option than that in 32bit CONFIG_4KSTACKS etc. IMO when these two merge we should consolidate on this version.

Quoted 4 lines

We seem to be growing two different mechanisms here for 32bit and 64bit. This does seem a better option than that in 32bit CONFIG_4KSTACKS etc. IMO when these two merge we should consolidate on this version.

Best would be to not change it at all for 64bit for now.

We can worry about the 16k CPU systems when they appear, but shorter term it would just lead to other crappy kernel code relying on large stacks when it shouldn't.

Quoted 5 lines · Andi Kleen

Quoted 4 lines

We seem to be growing two different mechanisms here for 32bit and 64bit. This does seem a better option than that in 32bit CONFIG_4KSTACKS etc. IMO when these two merge we should consolidate on this version.

Quoted 5 lines

Best would be to not change it at all for 64bit for now.

We can worry about the 16k CPU systems when they appear, but shorter term it would just lead to other crappy kernel code relying on large stacks when it shouldn't.

Well we cannot really test these systems without these patches and when they become officially available then its too late for merging.

otoh, I doubt if anyone will actually ship an NR_CPUS=16384 kernel, so it isn't terribly pointful.

So I'm wobbly. Could we please examine the alternatives before proceeding?

Quoted 2 lines · Andrew Morton

otoh, I doubt if anyone will actually ship an NR_CPUS=16384 kernel, so it isn't terribly pointful.

Our competition (Cray) just announced a product featuring up to 21k cpus although that is a cluster. We are definitely getting there...

Quoted 1 line

So I'm wobbly. Could we please examine the alternatives before proceeding?

This works fine with a 32k stack on IA64 with 4k processors.

So I tend to
think of this as a solution that is already working on another platform. An 8k stack is also going to be tough with 4k processors on x86_64 which we will have soon.

I'm talking about software, not hardware. I'd expect that you'll have trouble talking RH/suse/etc into general shipping of an NR_CPUS=16384 kernel.

Quoted 1 line

If I'm correct than I'd have thought that this will be a significant

problem for SGI, so we should find other solutions.

Quoted 3 lines

Quoted 1 line

So I'm wobbly. Could we please examine the alternatives before proceeding?

This works fine with a 32k stack on IA64 with 4k processors.

yeah, but that's an order-1 allocation on ia64, not an order-3.

Quoted 3 lines · Andrew Morton

I'm talking about software, not hardware. I'd expect that you'll have trouble talking RH/suse/etc into general shipping of an NR_CPUS=16384 kernel.

Yeah that is one reason why I removed all percpu arrays from the kernel with the cpu_alloc patchset. I think we can get to a point where this does not hurt that much. We want to be as close as possible to a distro kernel as possible.

Quoted 2 lines

Quoted 1 line

If I'm correct than I'd have thought that this will be a significant

problem for SGI, so we should find other solutions.

Maybe a special kernel from the distros is unavoidable but then they have done that in the past for us too. Certainly we do not want to have the kernel patches just for HPC apps. This is an option after all and not a default. Mike Travis is working on reducing the per cpu overhead in the x86_64 arch code. So we should be getting to a pretty good situation even if we have to leave the cpumasks alone.

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

So I'm wobbly. Could we please examine the alternatives before proceeding?

Quoted 3 lines

Quoted 1 line

This works fine with a 32k stack on IA64 with 4k processors.

yeah, but that's an order-1 allocation on ia64, not an order-3.

Well the default is also an order-1 allocation on x86_64. The order-3 alloc is not going to be that much of a problem if you have a system with several terabytes of RAM.

Did you consider making the stack size a calculated-in-Kconfig-arithmetic thing rather than an offered-to-humans thing? Derive it from CONFIG_NR_CPUS?

What else can we do? Change all sites to do some dynamic allocation if (NR_CPUS >= lots), I guess.

As for timing: we might as well merge it now so that 2.6.25 has at least a chance of running on 16384-way.

otoh, I doubt if anyone will actually ship an NR_CPUS=16384 kernel, so it isn't terribly pointful.