Skip to contentSkip to conversation

core kernel update

GIT PULL Ingo Molnar 4 participants 8 messages Nov 19, 2013 – Nov 20, 2013 Library code

lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest core-urgent-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core-urgent-for-linus

# HEAD: 9dd1220114e00d8ec5cdc20085bbe198b21e1985 smp/cpumask: Make CONFIG_CPUMASK_OFFSTACK=y usable without debug dependency

A late change/fix decoupling CPUMASK_OFFSTACK from DEBUG_PER_CPU_MAPS, fallout of:

b53b5eda8194 x86/cpu: Increase max CPU count to 8192 bb61ccc7dbcb x86/cpu: Allow higher NR_CPUS values

Thanks,

Ingo

------------------>
Josh Boyer (1):
smp/cpumask: Make CONFIG_CPUMASK_OFFSTACK=y usable without debug dependency

 lib/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/Kconfig +2 -1
index b3c8be0..50b47cd 100644
@@ -342,7 +342,8 @@ config CHECK_SIGNATURE
 	bool
 
 config CPUMASK_OFFSTACK
-	bool "Force CPU masks off stack" if DEBUG_PER_CPU_MAPS
+	bool "Force CPU masks off stack"
+	depends on SMP
 	help
 	  Use dynamic allocation for cpumask_var_t, instead of putting
 	  them on the stack.  This is a bit more expensive, but avoids
#1 lore raw
Quoted 3 lines · Ingo Molnar

Josh Boyer (1):
smp/cpumask: Make CONFIG_CPUMASK_OFFSTACK=y usable without debug dependency

This seems to be just pure stupid.

Why the hell should we ever ask the user whethr they want to force CPUMASK offstack?

Seriously, it only leaves room for mistakes and stupidities. There is no reason for any normal use-case to ask the user about this. Even the help message is pure and utter garbage ("This is a bit more expensive, but avoids stack overflow").

The fact is, an on-stack CPUMASK is *smaller* than a pointer to an offstack one when NR_CPU is small. The question makes no sense then. And when NR_CPU is huge, the question makes no sense _either_, since we can't have the cpumasks on stack.

Asking the user questions that make no f*cking sense to ask is stupid. And I'm not knowingly pulling stupid crap.

Linus

#2 lore raw
Quoted 15 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar

Josh Boyer (1):
smp/cpumask: Make CONFIG_CPUMASK_OFFSTACK=y usable without debug dependency

This seems to be just pure stupid.

Why the hell should we ever ask the user whethr they want to force CPUMASK offstack?

Seriously, it only leaves room for mistakes and stupidities. There is no reason for any normal use-case to ask the user about this. Even the help message is pure and utter garbage ("This is a bit more expensive, but avoids stack overflow").

Quoted 4 lines

The fact is, an on-stack CPUMASK is *smaller* than a pointer to an offstack one when NR_CPU is small. The question makes no sense then. And when NR_CPU is huge, the question makes no sense _either_, since we can't have the cpumasks on stack.

You are completely right. I missed that, will fix it up.

So we have these CPU count ranges right now:

  config NR_CPUS
        int "Maximum number of CPUs" if SMP && !MAXSMP
        range 2 8 if SMP && X86_32 && !X86_BIGSMP
        range 2 512 if SMP && !MAXSMP && !CPUMASK_OFFSTACK
        range 2 8192 if SMP && !MAXSMP && CPUMASK_OFFSTACK && X86_64

As you pointed out offstack is definitely nonsensical for NR_CPUS <= 64. (on 64-bit CPUs) On-stack is known to crash with NR_CPUS >= 1024.

The 128..512 CPUs range is somewhat of an unknown.

So the fixed code would always use on-stack cpumasks up to a limit (which limit falls into the 128...512 range) and it would always use off-stack cpumasks above that limit. No user configurability in the regular case. [*]

The actual value of the limit - here's the on-stack cpumask sizes of the candidate range:

128 CPUs: 16 byte cpumasks
256 CPUs: 32 byte cpumasks
512 CPUs: 64 byte cpumasks

512 definitely 'feels' dangerous, so I'd say the limit should be either 128 or 256. Both are ridiculously high counts for regular systems, so it doesn't really matter which one we pick.

I'd guess we should pick the higher limit for the time being (256 CPUs
- maybe even 512 CPUs), and lower it on contrary evidence (evidence of stack overflows)?

Thoughts?

Thanks,

Ingo

[*] Plus perhaps allow offstack to be configurable arbitrarily if
debugging is enabled [DEBUG_PER_CPU_MAP=y], to allow easy experiments/measurements?

#3 lore raw
Quoted 6 lines · Ingo Molnar

The actual value of the limit - here's the on-stack cpumask sizes of the candidate range:

128 CPUs: 16 byte cpumasks
256 CPUs: 32 byte cpumasks
512 CPUs: 64 byte cpumasks

So 512 / 64bytes is a single cacheline and feels like a nice cut-off before requiring an extra indirection and more cachelines.

64 bytes also doesn't sound _that_ big to have on-stack.

So I'd go for having the cut-off on >512, unless of course theres evidence 64bytes is already too much.

#5 lore raw
In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar

As you pointed out offstack is definitely nonsensical for NR_CPUS <= 64. (on 64-bit CPUs) On-stack is known to crash with NR_CPUS >= 1024.

The 128..512 CPUs range is somewhat of an unknown.

Agreed. And we could even decide to ask somewhere in that range.

I think 128 bits is still safely "don't bother with an external pointer" (it's just two words, it's like a "struct list_head" - there's no way that should be unsafe on the stack). Once we get to 256 bits I start going "Hmm, that's 32 bytes, maybe an external allocation makes sense..."

I'd personally put the cut-off point at just keeping it on-stack if it's smaller than or equal to 256. But I agree that at that point it's really just a judgement call.

Linus

#4 lore raw
In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Ingo Molnar

The actual value of the limit - here's the on-stack cpumask sizes of the candidate range:

Quoted 8 lines
Quoted 3 lines

128 CPUs: 16 byte cpumasks
256 CPUs: 32 byte cpumasks
512 CPUs: 64 byte cpumasks

So 512 / 64bytes is a single cacheline and feels like a nice cut-off before requiring an extra indirection and more cachelines.

64 bytes also doesn't sound _that_ big to have on-stack.

The cacheline size itself isn't necessarily super meaningful for on-stack variables: they are rarely cacheline aligned so they will take part in two cachelines.

Quoted 2 lines

So I'd go for having the cut-off on >512, unless of course theres evidence 64bytes is already too much.

I'm fine with that in any case, for the other reason I outlined: it's the highest one and we can iterate down if it proves to be bad. If we start out too low we'll probably never know it was too low.

Thanks,

Ingo

#6 lore raw
In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

[*] Plus perhaps allow offstack to be configurable arbitrarily if
debugging is enabled [DEBUG_PER_CPU_MAP=y], to allow easy experiments/measurements?

Yes, it was good for i386 testing.

But now it seems that DEBUG_KERNEL is on for every distribution (AFAICT), it's no longer an obscure option. I'm happy to kill it.

Cheers,
Rusty.

#7 lore raw
Quoted 9 lines · Rusty Russell
Quoted 3 lines · Ingo Molnar

[*] Plus perhaps allow offstack to be configurable arbitrarily if
debugging is enabled [DEBUG_PER_CPU_MAP=y], to allow easy experiments/measurements?

Yes, it was good for i386 testing.

But now it seems that DEBUG_KERNEL is on for every distribution (AFAICT), it's no longer an obscure option. I'm happy to kill it.

It would depend on DEBUG_PER_CPU_MAP=y, not DEBUG_KERNEL=y.

Thanks,

Ingo