prefetch the mmap_sem in the fault path
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/3 prefetch the mmap_sem in the fault path
In a micro-benchmark that stresses the pagefault path, the down_read_trylock on the mmap_sem showed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, the down_read_trylock basically fell out of the top of profile.
--- arch/x86_64/mm/fault.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86_64/mm/fault.c +4 -2
@@ -312,6 +312,10 @@ asmlinkage void __kprobes do_page_fault(unsigned long flags;siginfo_t info; + tsk = current; + mm = tsk->mm; +prefetchw (&mm->mmap_sem); +/* get the address */ __asm__("movq %%cr2,%0" :"=r" (address)); if (notify_die(DIE_PAGE_FAULT, "page fault", regs, error_code, 14,
@@ -325,8 +329,6 @@ asmlinkage void __kprobes do_page_fault(printk ("pagefault rip:%lx rsp:%lx cs:%lu ss:%lu address %lx error %lx\n" , regs->rip,regs->rsp,regs->cs,regs->ss,address,error_code); - tsk = current; - mm = tsk->mm; info.si_code =SEGV_MAPERR ;
Quoted 6 lines · Arjan van de Ven
In a micro-benchmark that stresses the pagefault path, the
down_read_trylockon themmap_semshowed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, thedown_read_trylockbasically fell out of the top of profile.
It is hard to believe because you effectively didn't do the prefetch very early
(e.g. the patch from your prefetch to taking the lock is quite short)
-Andi
Quoted 10 lines · Andi Kleen
Quoted 6 lines · Arjan van de Ven
In a micro-benchmark that stresses the pagefault path, the
down_read_trylockon themmap_semshowed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, thedown_read_trylockbasically fell out of the top of profile.It is hard to believe because you effectively didn't do the prefetch very early
all you need is a few dozen cycles though; there's a cr2 move and the entire notifier inbetween.... neither of those is really cheap.
(and after patch 3/3 also a page allocation/clear)
Quoted 9 lines · Arjan van de Ven
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Arjan van de Ven
In a micro-benchmark that stresses the pagefault path, the
down_read_trylockon themmap_semshowed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, thedown_read_trylockbasically fell out of the top of profile.
Quoted 5 lines
Quoted 2 lines
It is hard to believe because you effectively didn't do the prefetch very early
all you need is a few dozen cycles though; there's a cr2 move and the entire notifier inbetween.... neither of those is really cheap.
Ok. I added that patch.
-Andi
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arjan" == Arjan van de Ven <[email protected]> writes:
Arjan> In a micro-benchmark that stresses the pagefault path, the Arjan> down_read_trylock on the mmap_sem showed up quite high on the Arjan> profile. Turns out this lock is bouncing between cpus quite a Arjan> bit and thus is cache-cold a lot. This patch prefetches the Arjan> lock (for write) as early as possible (and before some other Arjan> somewhat expensive operations). With this patch, the Arjan> down_read_trylock basically fell out of the top of profile.
Out of curiousity, how big was the box used for testing? It might be worth investigating if anything can be done to reduce the number of times that lock is taken in the first place.
After all, what's a pain on a 4-way tends to be an utter nightmare on a 16-way ;(
Cheers,
Jes
Quoted 10 lines · Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arjan" == Arjan van de Ven <[email protected]> writes:
Arjan> In a micro-benchmark that stresses the pagefault path, the Arjan>
down_read_trylockon themmap_semshowed up quite high on the Arjan> profile. Turns out this lock is bouncing between cpus quite a Arjan> bit and thus is cache-cold a lot. This patch prefetches the Arjan> lock (for write) as early as possible (and before some other Arjan> somewhat expensive operations). With this patch, the Arjan>down_read_trylockbasically fell out of the top of profile.
Quoted 6 lines
Out of curiousity, how big was the box used for testing? It might be worth investigating if anything can be done to reduce the number of times that lock is taken in the first place.
After all, what's a pain on a 4-way tends to be an utter nightmare on a 16-way ;(
most of it was done on a 2 way, but some tests were done on a 4-way.
Quoted 18 lines · Arjan van de Ven
Quoted 10 lines · Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arjan" == Arjan van de Ven <[email protected]> writes:
Arjan> In a micro-benchmark that stresses the pagefault path, the Arjan>
down_read_trylockon themmap_semshowed up quite high on the Arjan> profile. Turns out this lock is bouncing between cpus quite a Arjan> bit and thus is cache-cold a lot. This patch prefetches the Arjan> lock (for write) as early as possible (and before some other Arjan> somewhat expensive operations). With this patch, the Arjan>down_read_trylockbasically fell out of the top of profile.Quoted 6 lines
Out of curiousity, how big was the box used for testing? It might be worth investigating if anything can be done to reduce the number of times that lock is taken in the first place.
After all, what's a pain on a 4-way tends to be an utter nightmare on a 16-way ;(
Quoted 6 lines
most of it was done on a 2 way, but some tests were done on a 4-way. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Could you share your microbenchmark with us (or point to the source) and we can give this a try on larger systems?
Thanks,
Signature · 3 lines
Ray Bryant AMD Performance Labs Austin, Tx 512-602-0038 (o) 512-507-7807 (c)