Skip to contentSkip to conversation

x86: allocate up to 32 tlb invalidate vectors

RFC In review Shaohua Li 2 participants 3 messages Nov 3, 2010 – Nov 15, 2010

lore
Share & tools
lore

Shaohua Li

author
#0 lore raw

cover x86: allocate up to 32 tlb invalidate vectors

Hi,
In workload with heavy page reclaim, flush_tlb_page() is frequently used. We currently have 8 vectors for tlb flush, which is fine for small machines. But for big machines with a lot of CPUs, the 8 vectors are shared by all CPUs and we need lock to protect them. This will cause a lot of lock contentions. please see the patch 3 for detailed number of the lock contention.
Andi Kleen suggests we can use 32 vectors for tlb flush, which should be fine for even 8 socket machines. Test shows this reduces lock contention dramatically (see patch 3 for number).
One might argue if this will waste too many vectors and leave less vectors for devices. This could be a problem. But even we use 32 vectors, we still leave 78 vectors for devices. And we now have per-cpu vector, vector isn't scarce any more, but I'm open if anybody has objections.

Thanks,
Shaohua

author
#1 lore raw
Quoted 7 lines · Shaohua Li

Hi,
In workload with heavy page reclaim, flush_tlb_page() is frequently used. We currently have 8 vectors for tlb flush, which is fine for small machines. But for big machines with a lot of CPUs, the 8 vectors are shared by all CPUs and we need lock to protect them. This will cause a lot of lock contentions. please see the patch 3 for detailed number of the lock contention.

Quoted 9 lines

Andi Kleen suggests we can use 32 vectors for tlb flush, which should be fine for even 8 socket machines. Test shows this reduces lock contention dramatically (see patch 3 for number).
One might argue if this will waste too many vectors and leave less vectors for devices. This could be a problem. But even we use 32 vectors, we still leave 78 vectors for devices. And we now have per-cpu vector, vector isn't scarce any more, but I'm open if anybody has objections.

Hi Ingo & hpa, any comments about this series?

Thanks,
Shaohua

#2 lore raw
Quoted 17 lines · Shaohua Li
Quoted 7 lines · Shaohua Li

Hi,
In workload with heavy page reclaim, flush_tlb_page() is frequently used. We currently have 8 vectors for tlb flush, which is fine for small machines. But for big machines with a lot of CPUs, the 8 vectors are shared by all CPUs and we need lock to protect them. This will cause a lot of lock contentions. please see the patch 3 for detailed number of the lock contention.

Quoted 9 lines

Andi Kleen suggests we can use 32 vectors for tlb flush, which should be fine for even 8 socket machines. Test shows this reduces lock contention dramatically (see patch 3 for number).
One might argue if this will waste too many vectors and leave less vectors for devices. This could be a problem. But even we use 32 vectors, we still leave 78 vectors for devices. And we now have per-cpu vector, vector isn't scarce any more, but I'm open if anybody has objections.

Quoted 2 lines

Hi Ingo & hpa, any comments about this series?

Hi Shaohua,

It looks good... I need to do a more thorough review and put it in; I just have been consumed a bit too much by a certain internal project.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.