tlb flush_data: replace per_cpu with an array
PATCH 1/1 tlb flush_data: replace per_cpu with an array
Hi,
On x86_64 flush tlb data is stored in per_cpu variables. This is unnecessary because only the first NUM_INVALIDATE_TLB_VECTORS entries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntu x86_64 has NR_CPUS=64 by default).
Regards,
Frederik
arch/x86/kernel/tlb_64.c +6 -6
@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now.
@@ -54,7 +54,7 @@ union smp_flush_state {/* State is put into the per CPU data section, but padded to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];/* * We cannot call mmdrop() because we are in interrupt context,
@@ -129,7 +129,7 @@ asmlinkage void smp_invalidate_interrupt(struct pt_regs *regs)* Use that to determine where the sender put the data. */ sender = ~regs->orig_ax -INVALIDATE_TLB_VECTOR_START ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];if (!cpu_isset (cpu, f->flush_cpumask)) goto out;
@@ -169,7 +169,7 @@ void native_flush_tlb_others(const cpumask_t *cpumaskp, struct mm_struct *mm,/* Caller has disabled preemption */ sender =smp_processor_id () %NUM_INVALIDATE_TLB_VECTORS ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];/* * Could avoid this lock when
@@ -205,8 +205,8 @@ static int __cpuinit init_smp_flush(void) {int i; -for_each_possible_cpu (i) -spin_lock_init (&per_cpu (flush_state, i).tlbstate_lock); +for (i =0 ; i <ARRAY_SIZE (flush_state); i++) +spin_lock_init (&flush_state[i].tlbstate_lock);return 0 ; }
Quoted 8 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).
Nope it doesn't save memory on most systems because per cpu is only allocated based on the CPUs that are actually there. And if you have more than 8 cores you can likely afford a few bytes per CPU.
You would need to cache line pad each entry then, otherwise you risk false sharing. That would make the array 1K on 128 bytes cache line system. This means on small systems this would actually waste much more memory.
per cpu avoids that problem completely.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 8 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).
Quoted 3 lines
Nope it doesn't save memory on most systems because per cpu is only allocated based on the CPUs that are actually there. And if you have more than 8 cores you can likely afford a few bytes per CPU.
I did not understand that, thanks for clarifiying
Quoted 7 lines
You would need to cache line pad each entry then, otherwise you risk false sharing. That would make the array 1K on 128 bytes cache line system. This means on small systems this would actually waste much more memory.
per cpu avoids that problem completely.
It is also slower (or so percpu.h says), and confusing I'd say.
Regards,
Frederik
Quoted 16 lines · Frederik Deweerdt
Quoted 10 lines · Andi Kleen
Quoted 8 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).Quoted 3 lines
Nope it doesn't save memory on most systems because per cpu is only allocated based on the CPUs that are actually there. And if you have more than 8 cores you can likely afford a few bytes per CPU.
I did not understand that, thanks for clarifiying
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
You would need to cache line pad each entry then, otherwise you risk false sharing. That would make the array 1K on 128 bytes cache line system. This means on small systems this would actually waste much more memory.
per cpu avoids that problem completely.
It is also slower (or so
percpu.hsays), and confusing I'd say.
Well it's something like 3 instructions versus one. You would have a hard time benchmarking it unless you run it in a very tight loop. It will be lost in the noise compared to all the other costs of the IPI.
Also why i don't like this patch is that on the typical small single/dual core system running a 128 byte cache line distro kernel you always pay the 1K cost now, while with per cpu it only needed one/two entries.
Admittedly it could have been better commented.
Not that it matters now unfortunately it's already applied. Sometimes wonder why I still bother to do patch review...
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 3 lines · Andi Kleen
Quoted 1 line
It is also slower (or so
percpu.hsays), and confusing I'd say.Well it's something like 3 instructions versus one. [...]
That's enough - micro-optimizations are done like that, instruction by instruction.
Quoted 8 lines
[...] You would have a hard time benchmarking it unless you run it in a very tight loop. It will be lost in the noise compared to all the other costs of the IPI.
Also why i don't like this patch is that on the typical small single/dual core system running a 128 byte cache line distro kernel you always pay the 1K cost now, while with per cpu it only needed one/two entries.
4 or 8 cores is the norm these days - by the time this change hits real Linux computers en masse 8 cores will be quite common.
Quoted 4 lines
Admittedly it could have been better commented.
Not that it matters now unfortunately it's already applied. Sometimes wonder why I still bother to do patch review...
Whether patches are already applied or not has no relevance - patches can still be undone or reverted of course, should your review feedback be correct.
Ingo
Quoted 7 lines · Andi Kleen
Quoted 6 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
Quoted 8 lines
Quoted 4 lines
This patch aims at making the code less confusing (there's nothing really "
per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).Nope it doesn't save memory on most systems because per cpu is only allocated based on the CPUs that are actually there. And if you have more than 8 cores you can likely afford a few bytes per CPU.
No distro kernel will build with less than 8 CPUs anyway so this point is moot.
Quoted 2 lines
You would need to cache line pad each entry then, otherwise you risk false sharing. [...]
They are already cache line padded.
Quoted 1 line
[...] That would make the array 1K on 128 bytes cache line system.
512 bytes.
Quoted 2 lines
[...] This means on small systems this would actually waste much more memory.
Really small systems will be UP and wont do cross-CPU TLB flushes, so if they are a worry the flush code can be UP optimized. (Nobody bothered so far.)
Ingo
H. Peter Anvin
reviewerQuoted 3 lines · Ingo Molnar
Quoted 1 line
Nope it doesn't save memory on most systems because per cpu is only allocated based on the CPUs that are actually there. And if you have more than 8 cores you can likely afford a few bytes per CPU.
No distro kernel will build with less than 8 CPUs anyway so this point is moot.
Except for embedded, which will build for exactly what's in the box.
-hpa
Quoted 2 lines
No distro kernel will build with less than 8 CPUs anyway so this point is moot.
It has nothing to do with what the distro kernel builds with. As I stated clearly in my review the per cpu data is sized based on the possible map, which is discovered from the BIOS at runtime. So if your system has two cores only you will only have two copies of per cpu data.
With this patch on the other hand you will always have 8 copies of this data; aka 1K no matter how many CPUs you have.
So the description that it saves memory is flat out wrong on any system with less than 8 threads (which is by far the biggest majority of systems currently and in the forseeable future)
Quoted 4 lines
Quoted 2 lines
You would need to cache line pad each entry then, otherwise you risk false sharing. [...]
They are already cache line padded.
Yes that's the problem here.
Quoted 4 lines
Quoted 2 lines
[...] That would make the array 1K on 128 bytes cache line system.
512 bytes.
8 * 128 = 1024
Ok the real waste is a little less because you need at least one copy, but still considerable. (896 bytes on UP, 768 bytes on 2C)
Quoted 7 lines
Quoted 3 lines
[...] This means on small systems this would actually waste much more memory.
Really small systems will be UP and wont do cross-CPU TLB flushes, so if they are a worry the flush code can be UP optimized. (Nobody bothered so far.)
The SMP flush code shouldn't be called at all on UP because the "other cpu" mask is always empty.
Just talking about the memory. Sure it's only 1K (or 896 bytes), but if you add up a lot of little 896byte wastes you eventually get a lot of waste all over.
Anyways if you wanted to do this without using per cpu data you could use alloc_percpu(), but that would be much more complicated code.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 7 lines · Andi Kleen
Quoted 2 lines
No distro kernel will build with less than 8 CPUs anyway so this point is moot.
It has nothing to do with what the distro kernel builds with. As I stated clearly in my review the per cpu data is sized based on the possible map, which is discovered from the BIOS at runtime. So if your system has two cores only you will only have two copies of per cpu data.
You ignored my observation that by the time this hits distro kernels the usual SMP hw size will be 8 or more cores.
Quoted 2 lines
With this patch on the other hand you will always have 8 copies of this data; aka 1K no matter how many CPUs you have.
Firstly, it's 512 bytes (see below), secondly, with the percpu approach we waste far more total memory as time goes on and the average core count increases.
Quoted 10 lines
So the description that it saves memory is flat out wrong on any system with less than 8 threads (which is by far the biggest majority of systems currently and in the forseeable future)
Quoted 4 lines
Quoted 2 lines
You would need to cache line pad each entry then, otherwise you risk false sharing. [...]
They are already cache line padded.
Yes that's the problem here.
I fail to see a problem. It has to be padded and aligned no matter where it is - and it is padded and aligned both before and after the patch. I dont know why you keep insisting that there's a problem where there is none.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
[...] That would make the array 1K on 128 bytes cache line system.
512 bytes.
8 * 128 = 1024
The default cacheline size in the x86 tree (for generic cpus) is 64 bytes, not 128 bytes - so the default size of the array is 512 bytes, not 1024 bytes. (This is a change we made yesterday, so you can not have known about it unless you follow the x86 tree closely.)
Ingo
Ingo Molnar
AppliedQuoted 9 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.This patch aims at making the code less confusing (there's nothing really "
per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).
indeed, well spotted!
This construct was indeed an abuse of the per_cpu facilities.
Note that beyond the obvious memory savings, your patch should make the code a bit faster and a bit smaller as well in the SMP TLB flush codepath:
the smp_flush_state data structure is 64 (or 128) bytes so static array arithmetics will be faster than the per_cpu indirection.
I have applied your patch to tip/x86/mm, thanks Frederik!
Ingo
Hi Frederik,
Quoted 17 lines · Frederik Deweerdt
Signed-off-by Frederik Deweerdt <[email protected]>arch/x86/kernel/tlb_64.c +1
-1@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now.
arch/x86/kernel/tlb_64.c union smp_flush_state { · line 57
Quoted 5 lines
arch/x86/kernel/tlb_64.c +1
-1@@ -54,7 +54,7 @@ union smp_flush_state {/* State is put into the per CPU data section, but padded to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];
Since flush_state has a spinlock, an array of flush_states would end up having multiples spinlocks on the same L1 cacheline. However, I see that the union smp_flush_state is already padded to SMP_CACHE_BYTES.
With per-cpu areas, locks belonging to different array elements did not end up on the same internode cacheline due to per-node allocation of per-cpu areas, but with a simple array, two locks could end up on the same internode cacheline.
Hence, can you please change the padding in smp_flush_state to INTERNODE_CACHE_BYTES (you have to derive this off INTERNODE_CACHE_SHIFT) and align it using ____cacheline_internodealigned_in_smp?
Quoted 21 lines · Ravikiran G Thirumalai
Hi Frederik,
Quoted 18 lines · Frederik Deweerdt
Signed-off-by Frederik Deweerdt <[email protected]>arch/x86/kernel/tlb_64.c +1
-1@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now. @@ -54,7 +54,7 @@ union smp_flush_state {/* State is put into the per CPU data section, but padded
Quoted 8 lines
arch/x86/kernel/tlb_64.c union smp_flush_state { · line 57
Quoted 4 lines
arch/x86/kernel/tlb_64.c +1
-1to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];Since
flush_statehas a spinlock, an array offlush_stateswould end up having multiples spinlocks on the same L1 cacheline. However, I see that the unionsmp_flush_stateis already padded toSMP_CACHE_BYTES.
Correct.
Quoted 9 lines
With per-cpu areas, locks belonging to different array elements did not end up on the same internode cacheline due to per-node allocation of per-cpu areas, but with a simple array, two locks could end up on the same internode cacheline.
Hence, can you please change the padding in
smp_flush_statetoINTERNODE_CACHE_BYTES(you have to derive this offINTERNODE_CACHE_SHIFT) and align it using____cacheline_internodealigned_in_smp?
Yes, that matters to vsmp, which has ... 4K node cache-lines.
Note that there's already CONFIG_X86_INTERNODE_CACHE_BYTES which is set properly on vsmp.
So ... something like the commit below?
Ingo
--------------->
Quoted 1 line
From
23d9dc8bffc759c131b09a48b5215cc2b37a5ac3Mon Sep 17 00:00:00 2001
From: Frederik Deweerdt <[email protected]>
Date: Mon, 12 Jan 2009 22:35:42 +0100
Subject: [PATCH] x86, tlb flush_data: replace per_cpu with an array
Impact: micro-optimization, memory reduction
On x86_64 flush tlb data is stored in per_cpu variables. This is unnecessary because only the first NUM_INVALIDATE_TLB_VECTORS entries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntu x86_64 has NR_CPUS=64 by default).
[ Ravikiran G Thirumalai also pointed out that the correct alignment
is ____cacheline_internodealigned_in_smp, so that there's no bouncing on vsmp. ]
--- arch/x86/kernel/tlb_64.c | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-)
arch/x86/kernel/tlb_64.c +8 -8
@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now.
@@ -48,13 +48,13 @@ union smp_flush_state {unsigned long flush_va;spinlock_t tlbstate_lock; }; -char pad[SMP_CACHE_BYTES ]; -}____cacheline_aligned ; +char pad[X86_INTERNODE_CACHE_BYTES ]; +}____cacheline_internodealigned_in_smp ;/* State is put into the per CPU data section, but padded to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];/* * We cannot call mmdrop() because we are in interrupt context,
@@ -129,7 +129,7 @@ asmlinkage void smp_invalidate_interrupt(struct pt_regs *regs)* Use that to determine where the sender put the data. */ sender = ~regs->orig_ax -INVALIDATE_TLB_VECTOR_START ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];if (!cpu_isset (cpu, f->flush_cpumask)) goto out;
@@ -169,7 +169,7 @@ void native_flush_tlb_others(const cpumask_t *cpumaskp, struct mm_struct *mm,/* Caller has disabled preemption */ sender =smp_processor_id () %NUM_INVALIDATE_TLB_VECTORS ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];/* * Could avoid this lock when
@@ -205,8 +205,8 @@ static int __cpuinit init_smp_flush(void) {int i; -for_each_possible_cpu (i) -spin_lock_init (&per_cpu (flush_state, i).tlbstate_lock); +for (i =0 ; i <ARRAY_SIZE (flush_state); i++) +spin_lock_init (&flush_state[i].tlbstate_lock);return 0 ; }
arch/x86/kernel/tlb_64.c union smp_flush_state { · line 51
Quoted 1 line · Ingo Molnar
arch/x86/kernel/tlb_64.c +1
-0+char pad[X86_INTERNODE_CACHE_BYTES ];
s/CONFIG_X86_INTERNODE_CACHE_BYTES
Ingo
Ravikiran G Thirumalai
Acked-byQuoted 12 lines · Ingo Molnar
Quoted 11 lines
With per-cpu areas, locks belonging to different array elements did not end up on the same internode cacheline due to per-node allocation of per-cpu areas, but with a simple array, two locks could end up on the same internode cacheline.
Hence, can you please change the padding in
smp_flush_statetoINTERNODE_CACHE_BYTES(you have to derive this offINTERNODE_CACHE_SHIFT) and align it using____cacheline_internodealigned_in_smp?
Quoted 7 lines
Yes, that matters to vsmp, which has ... 4K node cache-lines.
Note that there's already
CONFIG_X86_INTERNODE_CACHE_BYTESwhich is set properly on vsmp.So ... something like the commit below?
Yep. Looks good!
Thanks,
Kiran
Quoted 39 lines · Ingo Molnar
From
23d9dc8bffc759c131b09a48b5215cc2b37a5ac3Mon Sep 17 00:00:00 2001
From: Frederik Deweerdt <[email protected]>
Date: Mon, 12 Jan 2009 22:35:42 +0100
Subject: [PATCH] x86, tlbflush_data: replaceper_cpuwith an arrayImpact: micro-optimization, memory reduction
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.This patch aims at making the code less confusing (there's nothing really "
per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).[ Ravikiran G Thirumalai also pointed out that the correct alignment
is____cacheline_internodealigned_in_smp, so that there's no bouncing on vsmp. ]Signed-off-by Frederik Deweerdt <[email protected]>Signed-off-by Ingo Molnar <[email protected]>--- arch/x86/kernel/tlb_64.c | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-)arch/x86/kernel/tlb_64.c +1
-1@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now.
arch/x86/kernel/tlb_64.c union smp_flush_state { · line 52
Quoted 7 lines
arch/x86/kernel/tlb_64.c +2
-2@@ -48,13 +48,13 @@ union smp_flush_state {unsigned long flush_va;spinlock_t tlbstate_lock; }; -char pad[SMP_CACHE_BYTES ]; -}____cacheline_aligned ; +char pad[X86_INTERNODE_CACHE_BYTES ]; +}____cacheline_internodealigned_in_smp ;
That will make the below array 8*4096 bytes for VSMP, which pushes the limit for memory savings up to 256 cpus.
I'm really dubious this patch is really worth it.
Quoted 42 lines
arch/x86/kernel/tlb_64.c +5
-6/* State is put into the per CPU data section, but padded to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];/* * We cannot call mmdrop() because we are in interrupt context, @@ -129,7 +129,7 @@ asmlinkage void smp_invalidate_interrupt(struct pt_regs *regs)* Use that to determine where the sender put the data. */ sender = ~regs->orig_ax -INVALIDATE_TLB_VECTOR_START ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];if (!cpu_isset (cpu, f->flush_cpumask)) goto out;@@ -169,7 +169,7 @@ void native_flush_tlb_others(const cpumask_t *cpumaskp, struct mm_struct *mm,/* Caller has disabled preemption */ sender =smp_processor_id () %NUM_INVALIDATE_TLB_VECTORS ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];/* * Could avoid this lock when @@ -205,8 +205,8 @@ static int __cpuinit init_smp_flush(void) {int i; -for_each_possible_cpu (i) -spin_lock_init (&per_cpu (flush_state, i).tlbstate_lock); +for (i =0 ; i <ARRAY_SIZE (flush_state); i++) +spin_lock_init (&flush_state[i].tlbstate_lock);return 0 ; } --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Peter Zijlstra
arch/x86/kernel/tlb_64.c union smp_flush_state { · line 52
Quoted 4 lines
arch/x86/kernel/tlb_64.c +2
-2-char pad[SMP_CACHE_BYTES ]; -}____cacheline_aligned ; +char pad[X86_INTERNODE_CACHE_BYTES ]; +}____cacheline_internodealigned_in_smp ;That will make the below array 8*4096 bytes for VSMP, which pushes the limit for memory savings up to 256 cpus.
VSMP is a clustering solution (default-disabled) that pushes L1_CACHE_BYTES to 4096 bytes (4K). That is an extremely large alignment that pushes up the BSS size ten-fold (!), so no generic Linux distribution enables it. 32K compared to 9MB bloat caused by 4K cachelines is a drop in the ocean.
Ingo
Quoted 3 lines · Ingo Molnar
VSMP is a clustering solution (default-disabled)
Small correction :)
Well, it is a virtualization solution creating single VM across multiple systems. Since it is shared memory, we would rather refer to it as SMP or Virtual SMP rather than a clustering solution.
Quoted 9 lines · Ravikiran G Thirumalai
Quoted 3 lines · Ingo Molnar
VSMP is a clustering solution (default-disabled)
Small correction :)
Well, it is a virtualization solution creating single VM across multiple systems. Since it is shared memory, we would rather refer to it as SMP or Virtual SMP rather than a clustering solution.
You do have hardware help too to do efficient cross-node memory traffic, dont you?
Ingo
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Ravikiran G Thirumalai
Quoted 3 lines · Ingo Molnar
VSMP is a clustering solution (default-disabled)
Quoted 8 lines
Quoted 4 lines
Small correction :)
Well, it is a virtualization solution creating single VM across multiple systems. Since it is shared memory, we would rather refer to it as SMP or Virtual SMP rather than a clustering solution.You do have hardware help too to do efficient cross-node memory traffic, dont you?
Of course. A fast backplane interconnect.
Quoted 14 lines · Ravikiran G Thirumalai
Quoted 13 lines · Ingo Molnar
Quoted 9 lines · Ravikiran G Thirumalai
Quoted 3 lines · Ingo Molnar
VSMP is a clustering solution (default-disabled)
Small correction :)
Well, it is a virtualization solution creating single VM across multiple systems. Since it is shared memory, we would rather refer to it as SMP or Virtual SMP rather than a clustering solution.
Quoted 4 lines
Quoted 2 lines
You do have hardware help too to do efficient cross-node memory traffic, dont you?
Of course. A fast backplane interconnect.
... so it's PCs clustered together, and called vSMP, right? ;-)
a neat hack in any case.
Ingo
Quoted 10 lines · Ingo Molnar
Quoted 4 lines · Ravikiran G Thirumalai
Quoted 2 lines
You do have hardware help too to do efficient cross-node memory traffic, dont you?
Of course. A fast backplane interconnect.
... so it's PCs clustered together, and called vSMP, right? ;-)
Well, it depends on how you define a cluster. You can even call a SMP mother board a cluster of cpus :). In that case even Sequent NUMA were clusters, and so are some of the large core count shared memory systems made of multiple similar boards/bricks/books (based on vendor terminology) with a custom interconnect for cache coherency.
A vSMP system is multiple similar cpu mother boards, usually dual socket, each board having multiple cpus and a commodity fast interconnect for cache coherency. There is a VMM which among other tasks also takes care of the coherency, and each 'board' or 'node' can access physical memory from another board. There is one instance of OS running on the entire system unlike the classical cluster where there are multiple instances of OS and the apps under the OS have to work together as a cluster.
Quoted 3 lines
Quoted 3 lines
It also would save some memory
on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).
As pointed out several times this claim in the changelog is incorrect. Please fix it.
A correct description would be:
- This patch wastes memory of one configured cache line for each CPU on systems with less than 8 cores (e.g. 896 bytes on a 1 core system with CONFIG_GENERIC_CPU/ cache line size 128)
It saves some memory on systems with more than 8 cores (one cache line size for each core above 8)
-Andi
Quoted 7 lines · Andi Kleen
Quoted 3 lines
Quoted 3 lines
It also would save some memory
on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).As pointed out several times this claim in the changelog is incorrect. Please fix it.
Quoted 7 lines
A correct description would be:
- This patch wastes memory of one configured cache line for each CPU on systems with less than 8 cores (e.g. 896 bytes on a 1 core system with CONFIG_GENERIC_CPU/ cache line size 128)
It saves some memory on systems with more than 8 cores (one cache line size for each core above 8)
To be fair, it also depends on whenever HOTPLUG_CPU is on or not (if it is we allocate NR_CPUS percpu sections). In that case, which is Ubuntu's and Fedora's, we do save memory.
This would make for a fairly long changelog, how about a link to this thread, and removing the NR_CPUS bit?
Regards,
Frederik
Quoted 3 lines
-Andi
Quoted 2 lines
To be fair, it also depends on whenever
HOTPLUG_CPUis on or not (if it is we allocateNR_CPUSpercpu sections).
not true to my knowledge. Even a !HOTPLUG_CPU kernel doesn't allocate more than what is in the possible map. That would be dumb if it was true, but it isn't fortunately.
I don't think it does.
In that case, which is Ubuntu's
Quoted 1 line
and Fedora's,
You're saying that Ubuntu and Fedora do not support suspend to ram? (suspend to ram requires CPU hotplug). At least my Fedora box does S3 just fine so that cannot be correct.
we do save memory.
Quoted 2 lines
This would make for a fairly long changelog, how about a link to this thread, and removing the
NR_CPUSbit?
My original paragraph is correct and it's not long. At least your claim is just wrong.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 6 lines · Andi Kleen
Quoted 2 lines
To be fair, it also depends on whenever
HOTPLUG_CPUis on or not (if it is we allocateNR_CPUSpercpu sections).not true to my knowledge. Even a !HOTPLUG_CPU kernel doesn't allocate more than what is in the possible map. That would be dumb if it was true, but it isn't fortunately.
My point is that possible map == NR_CPUS if HOTPLUG_CPU
Quoted 20 lines
I don't think it does.
In that case, which is Ubuntu'sQuoted 1 line
and Fedora's,
You're saying that Ubuntu and Fedora do not support suspend to ram? (suspend to ram requires CPU hotplug). At least my Fedora box does S3 just fine so that cannot be correct.
we do save memory.
Quoted 2 lines
This would make for a fairly long changelog, how about a link to this thread, and removing the
NR_CPUSbit?My original paragraph is correct and it's not long. At least your claim is just wrong.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).
Quoted 4 lines
Regards,
FrederikSigned-off-by Frederik Deweerdt <[email protected]>
Here is the net change in memory usage with this patch on a allyesconfig with NR_CPUS=4096.
====== Data (-l 500)
1 - initial
2 - change-flush-tlb
.1. .2. .delta.
0 5120 +5120 . flush_state(.bss)
====== Sections (-l 500)
.1. .2. .delta.
12685496 12693688 +8192 +0.06% .bss
1910176 1909408 -768 -0.04% .data.percpuQuoted 57 lines
arch/x86/kernel/tlb_64.c +6
-7@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now. @@ -54,7 +54,7 @@ union smp_flush_state {/* State is put into the per CPU data section, but padded to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];/* * We cannot call mmdrop() because we are in interrupt context, @@ -129,7 +129,7 @@ asmlinkage void smp_invalidate_interrupt(struct pt_regs *regs)* Use that to determine where the sender put the data. */ sender = ~regs->orig_ax -INVALIDATE_TLB_VECTOR_START ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];if (!cpu_isset (cpu, f->flush_cpumask)) goto out;@@ -169,7 +169,7 @@ void native_flush_tlb_others(const cpumask_t *cpumaskp, struct mm_struct *mm,/* Caller has disabled preemption */ sender =smp_processor_id () %NUM_INVALIDATE_TLB_VECTORS ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];/* * Could avoid this lock when @@ -205,8 +205,8 @@ static int __cpuinit init_smp_flush(void) {int i; -for_each_possible_cpu (i) -spin_lock_init (&per_cpu (flush_state, i).tlbstate_lock); +for (i =0 ; i <ARRAY_SIZE (flush_state); i++) +spin_lock_init (&flush_state[i].tlbstate_lock);return 0 ; } --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines · Mike Travis
Quoted 9 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).
Quoted 7 lines
Quoted 4 lines
Regards,
FrederikSigned-off-by Frederik Deweerdt <[email protected]>Here is the net change in memory usage with this patch on a allyesconfig with NR_CPUS=4096.
Yes, this point wrt. memory was based on my flawed understanding of how per_cpu actually allocates the data. There is however 1) a confusing use of per_cpu removed, 2) faster access to the flush data.
Quoted 6 lines
====== Data (-l 500)
1 - initial
2 - change-flush-tlb
Quoted 9 lines
.1. .2. .delta.
0 5120 +5120 . flush_state(.bss)====== Sections (-l 500)
.1. .2. .delta. 12685496 12693688 +8192 +0.06% .bss 1910176 1909408 -768 -0.04% .data.percpu
I get :
Initial
size ./arch/x86/kernel/tlb_64.o
text data bss dec hex filename
1667 136 8 1811 713 ./arch/x86/kernel/tlb_64.o
After
size ./arch/x86/kernel/tlb_64.o
text data bss dec hex filename
1598 8 1088 2694 a86 ./arch/x86/kernel/tlb_64.o
-69 -128 +1080 +883
But I'm not sure those numbers are that relevant, as the percpu part will be allocated at runtime.
Regards,
Frederik
Quoted 58 lines
Quoted 57 lines
arch/x86/kernel/tlb_64.c +6
-7@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now. @@ -54,7 +54,7 @@ union smp_flush_state {/* State is put into the per CPU data section, but padded to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];/* * We cannot call mmdrop() because we are in interrupt context, @@ -129,7 +129,7 @@ asmlinkage void smp_invalidate_interrupt(struct pt_regs *regs)* Use that to determine where the sender put the data. */ sender = ~regs->orig_ax -INVALIDATE_TLB_VECTOR_START ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];if (!cpu_isset (cpu, f->flush_cpumask)) goto out;@@ -169,7 +169,7 @@ void native_flush_tlb_others(const cpumask_t *cpumaskp, struct mm_struct *mm,/* Caller has disabled preemption */ sender =smp_processor_id () %NUM_INVALIDATE_TLB_VECTORS ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];/* * Could avoid this lock when @@ -205,8 +205,8 @@ static int __cpuinit init_smp_flush(void) {int i; -for_each_possible_cpu (i) -spin_lock_init (&per_cpu (flush_state, i).tlbstate_lock); +for (i =0 ; i <ARRAY_SIZE (flush_state); i++) +spin_lock_init (&flush_state[i].tlbstate_lock);return 0 ; } --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 14 lines · Frederik Deweerdt
Quoted 13 lines · Mike Travis
Quoted 12 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).Regards,
Frederik
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Frederik Deweerdt <[email protected]>Here is the net change in memory usage with this patch on a allyesconfig with NR_CPUS=4096.
Yes, this point wrt. memory was based on my flawed understanding of how
per_cpuactually allocates the data. There is however 1) a confusing use ofper_cpuremoved, 2) faster access to the flush data.
Is this true? On a widely separated NUMA system, requiring all CPU's to access memory on NODE 0 for every tlb flush would seem expensive. That's another benefit of per_cpu data, it's local to the node's cpus.
(And was it determined yet, that a cacheline has to be tossed around as well?)
Thanks,
Mike
Quoted 95 lines
Quoted 6 lines
====== Data (-l 500)
1 - initial
2 - change-flush-tlbQuoted 9 lines
.1. .2. .delta.
0 5120 +5120 . flush_state(.bss)====== Sections (-l 500)
.1. .2. .delta. 12685496 12693688 +8192 +0.06% .bss 1910176 1909408 -768 -0.04% .data.percpuI get :
Initial
size./arch/x86/kernel/tlb_64.o
text data bss dec hex filename
1667 136 8 1811 713./arch/x86/kernel/tlb_64.oAfter
size./arch/x86/kernel/tlb_64.o
text data bss dec hex filename
1598 8 1088 2694 a86./arch/x86/kernel/tlb_64.o-69 -128 +1080 +883But I'm not sure those numbers are that relevant, as the percpu part will be allocated at runtime.
Regards,
FrederikQuoted 49 lines
Quoted 49 lines
arch/x86/kernel/tlb_64.c +6
-6@@ -33,7 +33,7 @@* To avoid global state use 8 different call vectors. * Each CPU uses a specific vector to trigger flushes on other * CPUs. Depending on the received vector the target CPUs look into -* the right per cpu variable for the flush data. +* the right array slot for the flush data. * * With more than 8 CPUs they are hashed to the 8 available * vectors. The limited global vector space forces us to this right now. @@ -54,7 +54,7 @@ union smp_flush_state {/* State is put into the per CPU data section, but padded to a full cache line because other CPUs can access it and we don't want false sharing in the per cpu data segment. */ -static DEFINE_PER_CPU(union smp_flush_state, flush_state); +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS ];/* * We cannot call mmdrop() because we are in interrupt context, @@ -129,7 +129,7 @@ asmlinkage void smp_invalidate_interrupt(struct pt_regs *regs)* Use that to determine where the sender put the data. */ sender = ~regs->orig_ax -INVALIDATE_TLB_VECTOR_START ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];if (!cpu_isset (cpu, f->flush_cpumask)) goto out;@@ -169,7 +169,7 @@ void native_flush_tlb_others(const cpumask_t *cpumaskp, struct mm_struct *mm,/* Caller has disabled preemption */ sender =smp_processor_id () %NUM_INVALIDATE_TLB_VECTORS ; - f = &per_cpu (flush_state, sender); + f = &flush_state[sender];/* * Could avoid this lock when @@ -205,8 +205,8 @@ static int __cpuinit init_smp_flush(void) {int i; -for_each_possible_cpu (i) -spin_lock_init (&per_cpu (flush_state, i).tlbstate_lock); +for (i =0 ; i <ARRAY_SIZE (flush_state); i++) +spin_lock_init (&flush_state[i].tlbstate_lock);Quoted 7 lines
Quoted 7 lines
arch/x86/kernel/tlb_64.c +0
-1return 0 ; } --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 19 lines · Mike Travis
Quoted 17 lines · Frederik Deweerdt
Quoted 16 lines · Mike Travis
Quoted 9 lines · Frederik Deweerdt
Hi,
On
x86_64flush tlb data is stored inper_cpuvariables. This is unnecessary because only the firstNUM_INVALIDATE_TLB_VECTORSentries are accessed.
This patch aims at making the code less confusing (there's nothing really "per_cpu") by using a plain array. It also would save some memory on most distros out there (Ubuntux86_64has NR_CPUS=64 by default).Quoted 4 lines
Regards,
FrederikSigned-off-by Frederik Deweerdt <[email protected]>Here is the net change in memory usage with this patch on a allyesconfig with NR_CPUS=4096.
Quoted 7 lines
Quoted 3 lines
Yes, this point wrt. memory was based on my flawed understanding of how
per_cpuactually allocates the data. There is however 1) a confusing use ofper_cpuremoved, 2) faster access to the flush data.Is this true? On a widely separated NUMA system, requiring all CPU's to access memory on NODE 0 for every tlb flush would seem expensive. That's another benefit of
per_cpudata, it's local to the node's cpus.
That's irrelevant here: we only have 8 IPI slots for the TLB flush, so we'll use the first 8 per_cpu areas (note how all access is cpu-nr modulo 8 in essence). That is going to be node 0 anyway in most NUMA setups.
Quoted 2 lines
(And was it determined yet, that a cacheline has to be tossed around as well?)
No, that assertion of Andi is wrong.
Ingo
Quoted 4 lines
Quoted 2 lines
(And was it determined yet, that a cacheline has to be tossed around as well?)
No, that assertion of Andi is wrong.
For the IPI you always have to toss one cache line around to pass the information.
Anyways the padding is just to avoid false sharing.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines · Andi Kleen
Quoted 4 lines
Quoted 2 lines
(And was it determined yet, that a cacheline has to be tossed around as well?)
No, that assertion of Andi is wrong.
For the IPI you always have to toss one cache line around to pass the information.
Anyways the padding is just to avoid false sharing.
Which is a red herring - the padding was there before the patch, and is there after the patch as well.
Ingo