x86: avoid tlbstate lock if no enough cpus
RFC 4/4 x86: avoid tlbstate lock if no enough cpus
This one isn't related to previous patch. If online cpus are below NUM_INVALIDATE_TLB_VECTORS, we don't need the lock. The comments in the code declares we don't need the check, but a hot lock still needs an atomic operation and expensive, so add the check here.
--- arch/x86/mm/tlb.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
arch/x86/mm/tlb.c +7 -7
@@ -174,17 +174,16 @@ static void flush_tlb_others_ipi(const s {unsigned int sender;union smp_flush_state *f; +bool do_lock = false;/* Caller has disabled preemption */ sender =this_cpu_read (tlb_vector_offset); f = &flush_state[sender]; -/* -* Could avoid this lock when -* num_online_cpus() <= NUM_INVALIDATE_TLB_VECTORS, but it is -* probably not worth checking this for a cache-hot lock. -*/ -raw_spin_lock (&f->tlbstate_lock); +if (num_online_cpus () >NUM_INVALIDATE_TLB_VECTORS ) { + do_lock = true; +raw_spin_lock (&f->tlbstate_lock); + } f->flush_mm = mm; f->flush_va = va;
@@ -202,7 +201,8 @@ static void flush_tlb_others_ipi(const s f->flush_mm =NULL ; f->flush_va =0 ; -raw_spin_unlock (&f->tlbstate_lock); +if (do_lock) +raw_spin_unlock (&f->tlbstate_lock); } void native_flush_tlb_others(const struct cpumask *cpumask,
Quoted 21 lines · Shaohua Li
This one isn't related to previous patch. If online cpus are below
NUM_INVALIDATE_TLB_VECTORS, we don't need the lock. The comments in the code declares we don't need the check, but a hot lock still needs an atomic operation and expensive, so add the check here.Signed-off-by Shaohua Li <[email protected]>--- arch/x86/mm/tlb.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)Index:
linux/arch/x86/mm/tlb.c
===================================================================arch/x86/mm/tlb.c +1
-0@@ -174,17 +174,16 @@ static void flush_tlb_others_ipi(const s {unsigned int sender;union smp_flush_state *f; +bool do_lock = false;/* Caller has disabled preemption */
arch/x86/mm/tlb.c static void flush_tlb_others_ipi(const s · line 183
Quoted 10 lines
arch/x86/mm/tlb.c +1
-6sender =this_cpu_read (tlb_vector_offset); f = &flush_state[sender]; -/* -* Could avoid this lock when -* num_online_cpus() <= NUM_INVALIDATE_TLB_VECTORS, but it is -* probably not worth checking this for a cache-hot lock. -*/ -raw_spin_lock (&f->tlbstate_lock); +if (num_online_cpus () >NUM_INVALIDATE_TLB_VECTORS ) {
Ouch, you remove a comment that pretty well explained the problem.
Last time I checked, num_online_cpus() was pretty expensive on a 4096 cpus machine, since 4096 bits array is 512 bytes long.
Are you sure you didnt want to use nr_cpu_ids here ?
Quoted 17 lines
arch/x86/mm/tlb.c +5
-1+ do_lock = true; +raw_spin_lock (&f->tlbstate_lock); + } f->flush_mm = mm; f->flush_va = va;@@ -202,7 +201,8 @@ static void flush_tlb_others_ipi(const s f->flush_mm =NULL ; f->flush_va =0 ; -raw_spin_unlock (&f->tlbstate_lock); +if (do_lock) +raw_spin_unlock (&f->tlbstate_lock); } void native_flush_tlb_others(const struct cpumask *cpumask,
Quoted 35 lines · Eric Dumazet
Quoted 21 lines · Shaohua Li
This one isn't related to previous patch. If online cpus are below
NUM_INVALIDATE_TLB_VECTORS, we don't need the lock. The comments in the code declares we don't need the check, but a hot lock still needs an atomic operation and expensive, so add the check here.Signed-off-by Shaohua Li <[email protected]>--- arch/x86/mm/tlb.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)Index:
linux/arch/x86/mm/tlb.c
===================================================================arch/x86/mm/tlb.c +1
-0@@ -174,17 +174,16 @@ static void flush_tlb_others_ipi(const s {unsigned int sender;union smp_flush_state *f; +bool do_lock = false;/* Caller has disabled preemption */ arch/x86/mm/tlb.c static void flush_tlb_others_ipi(const s · line 183
Quoted 10 lines
arch/x86/mm/tlb.c +1
-6sender =this_cpu_read (tlb_vector_offset); f = &flush_state[sender]; -/* -* Could avoid this lock when -* num_online_cpus() <= NUM_INVALIDATE_TLB_VECTORS, but it is -* probably not worth checking this for a cache-hot lock. -*/ -raw_spin_lock (&f->tlbstate_lock); +if (num_online_cpus () >NUM_INVALIDATE_TLB_VECTORS ) {Ouch, you remove a comment that pretty well explained the problem.
Quoted 2 lines
Last time I checked,
num_online_cpus()was pretty expensive on a 4096 cpus machine, since 4096 bits array is 512 bytes long.
ok
Quoted 1 line
Are you sure you didnt want to use
nr_cpu_idshere ?
just don't want to include the non-present cpus here. I wonder why we haven't a variable to record online cpu number.
Thanks,
Shaohua
Quoted 2 lines · Shaohua Li
just don't want to include the non-present cpus here. I wonder why we haven't a variable to record online cpu number.
What prevents a 256 cpus machine, to have 8 online cpus that all use the same TLB vector ?
(Max 32 vectors, so 8 cpus share each vector, settled at boot time)
Forget about 'online', and think 'possible' ;)
Quoted 10 lines · Eric Dumazet
Quoted 2 lines · Shaohua Li
just don't want to include the non-present cpus here. I wonder why we haven't a variable to record online cpu number.
What prevents a 256 cpus machine, to have 8 online cpus that all use the same TLB vector ?
(Max 32 vectors, so 8 cpus share each vector, settled at boot time)
Forget about 'online', and think 'possible' ;)
Hmm, the spread vectors to node already merged, how could the 8 cpus share a vector?
Quoted 5 lines · Shaohua Li
Quoted 4 lines · Eric Dumazet
Quoted 2 lines · Shaohua Li
just don't want to include the non-present cpus here. I wonder why we haven't a variable to record online cpu number.
Quoted 9 lines
Quoted 6 lines
What prevents a 256 cpus machine, to have 8 online cpus that all use the same TLB vector ?
(Max 32 vectors, so 8 cpus share each vector, settled at boot time)
Forget about 'online', and think 'possible' ;)
Hmm, the spread vectors to node already merged, how could the 8 cpus share a vector?
You boot a machine with 256 cpu.
They are online and very well.
Each vector is shared by at least 8 cpus, because 256/32 = 8. OK ?
Now you off-line 256-8 cpus, because you have HOTPLUG capability in your kernel and you have some policy to bring them up later if needed.
What happens ? Do you rebalance TLB vectors to make sure each cpu has its own vector ?
Quoted 20 lines · Eric Dumazet
Quoted 5 lines · Shaohua Li
Quoted 4 lines · Eric Dumazet
Quoted 2 lines · Shaohua Li
just don't want to include the non-present cpus here. I wonder why we haven't a variable to record online cpu number.
Quoted 9 lines
Quoted 6 lines
What prevents a 256 cpus machine, to have 8 online cpus that all use the same TLB vector ?
(Max 32 vectors, so 8 cpus share each vector, settled at boot time)
Forget about 'online', and think 'possible' ;)
Hmm, the spread vectors to node already merged, how could the 8 cpus share a vector?
You boot a machine with 256 cpu.
They are online and very well.
Quoted 9 lines
Each vector is shared by at least 8 cpus, because 256/32 = 8. OK ?
Now you off-line 256-8 cpus, because you have HOTPLUG capability in your kernel and you have some policy to bring them up later if needed.
What happens ? Do you rebalance TLB vectors to make sure each cpu has its own vector ?
It seems you do that since commit 932967202182743 in calculate_tlb_offset()
So just add in this function some logic to tell if each tlb vector is used by one or several cpu.
We can avoid the lock for each TLB vector used by exactly one cpu.
A message that is not in the archive (sent off-list or not yet received): 1288773716.23014.124.camel@sli10-conroe
Quoted 4 lines · Shaohua Li
yes, this is ok. we might need avoid some cpu hotplug race too. I'll post a new patch later.
Hmm, maybe only set the variable "must take the lock", never unset it.
Quoted 8 lines · Eric Dumazet
Quoted 4 lines · Shaohua Li
yes, this is ok. we might need avoid some cpu hotplug race too. I'll post a new patch later.
Hmm, maybe only set the variable "must take the lock", never unset it.
I followed your suggestions to use nr_cpu_ids, it should be good enough.
Thanks,
Shaohua
This one isn't related to previous patch. If online cpus are below NUM_INVALIDATE_TLB_VECTORS, we don't need the lock. The comments in the code declares we don't need the check, but a hot lock still needs an atomic operation and expensive, so add the check here.
Uses nr_cpu_ids here as suggested by Eric Dumazet.
--- arch/x86/mm/tlb.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
arch/x86/mm/tlb.c +4 -7
@@ -179,12 +179,8 @@ static void flush_tlb_others_ipi(const s sender =this_cpu_read (tlb_vector_offset); f = &flush_state[sender]; -/* -* Could avoid this lock when -* num_online_cpus() <= NUM_INVALIDATE_TLB_VECTORS, but it is -* probably not worth checking this for a cache-hot lock. -*/ -raw_spin_lock (&f->tlbstate_lock); +if (nr_cpu_ids >NUM_INVALIDATE_TLB_VECTORS ) +raw_spin_lock (&f->tlbstate_lock); f->flush_mm = mm; f->flush_va = va;
@@ -202,7 +198,8 @@ static void flush_tlb_others_ipi(const s f->flush_mm =NULL ; f->flush_va =0 ; -raw_spin_unlock (&f->tlbstate_lock); +if (nr_cpu_ids >NUM_INVALIDATE_TLB_VECTORS ) +raw_spin_unlock (&f->tlbstate_lock); } void native_flush_tlb_others(const struct cpumask *cpumask,
Eric Dumazet
Acked-byQuoted 11 lines · Shaohua Li
Quoted 8 lines · Eric Dumazet
Quoted 4 lines · Shaohua Li
yes, this is ok. we might need avoid some cpu hotplug race too. I'll post a new patch later.
Hmm, maybe only set the variable "must take the lock", never unset it.
I followed your suggestions to use
nr_cpu_ids, it should be good enough.
Yes, unfortunately not on HP machines, because of their tendency to have holes in CPU numberings :)
This can probably can improved later.
Thanks
Quoted 9 lines
Thanks,
ShaohuaThis one isn't related to previous patch. If online cpus are below
NUM_INVALIDATE_TLB_VECTORS, we don't need the lock. The comments in the code declares we don't need the check, but a hot lock still needs an atomic operation and expensive, so add the check here.
Quoted 4 lines
Uses
nr_cpu_idshere as suggested by Eric Dumazet.Signed-off-by Shaohua Li <[email protected]>---