Skip to contentSkip to conversation

prefetch the mmap_sem in the fault path

PATCH In review Arjan van de Ven 5 participants 7 messages Feb 23, 2006 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH 1/3 prefetch the mmap_sem in the fault path

In a micro-benchmark that stresses the pagefault path, the down_read_trylock on the mmap_sem showed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, the down_read_trylock basically fell out of the top of profile.

---
 arch/x86_64/mm/fault.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86_64/mm/fault.c +4 -2
Index: linux-work/arch/x86_64/mm/fault.c
===================================================================
@@ -312,6 +312,10 @@ asmlinkage void __kprobes do_page_fault(
 	unsigned long flags;
 	siginfo_t info;
 
+	tsk = current;
+	mm = tsk->mm;
+	prefetchw(&mm->mmap_sem);
+
 	/* get the address */
 	__asm__("movq %%cr2,%0":"=r" (address));
 	if (notify_die(DIE_PAGE_FAULT, "page fault", regs, error_code, 14,
@@ -325,8 +329,6 @@ asmlinkage void __kprobes do_page_fault(
 		printk("pagefault rip:%lx rsp:%lx cs:%lu ss:%lu address %lx error %lx\n",
 		       regs->rip,regs->rsp,regs->cs,regs->ss,address,error_code); 
 
-	tsk = current;
-	mm = tsk->mm;
 	info.si_code = SEGV_MAPERR;
 
 
#2 lore raw
Quoted 6 lines · Arjan van de Ven

In a micro-benchmark that stresses the pagefault path, the down_read_trylock on the mmap_sem showed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, the down_read_trylock basically fell out of the top of profile.

It is hard to believe because you effectively didn't do the prefetch very early

(e.g. the patch from your prefetch to taking the lock is quite short)

-Andi

#3 lore raw
Quoted 10 lines · Andi Kleen
Quoted 6 lines · Arjan van de Ven

In a micro-benchmark that stresses the pagefault path, the down_read_trylock on the mmap_sem showed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, the down_read_trylock basically fell out of the top of profile.

It is hard to believe because you effectively didn't do the prefetch very early

all you need is a few dozen cycles though; there's a cr2 move and the entire notifier inbetween.... neither of those is really cheap.

(and after patch 3/3 also a page allocation/clear)

#4 lore raw
Quoted 9 lines · Arjan van de Ven
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Arjan van de Ven

In a micro-benchmark that stresses the pagefault path, the down_read_trylock on the mmap_sem showed up quite high on the profile. Turns out this lock is bouncing between cpus quite a bit and thus is cache-cold a lot. This patch prefetches the lock (for write) as early as possible (and before some other somewhat expensive operations). With this patch, the down_read_trylock basically fell out of the top of profile.

Quoted 5 lines
Quoted 2 lines

It is hard to believe because you effectively didn't do the prefetch very early

all you need is a few dozen cycles though; there's a cr2 move and the entire notifier inbetween.... neither of those is really cheap.

Ok. I added that patch.

-Andi

#5 lore raw
In reply to Arjan van de Ven
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arjan" == Arjan van de Ven <[email protected]> writes:

Arjan> In a micro-benchmark that stresses the pagefault path, the Arjan> down_read_trylock on the mmap_sem showed up quite high on the Arjan> profile. Turns out this lock is bouncing between cpus quite a Arjan> bit and thus is cache-cold a lot. This patch prefetches the Arjan> lock (for write) as early as possible (and before some other Arjan> somewhat expensive operations). With this patch, the Arjan> down_read_trylock basically fell out of the top of profile.

Out of curiousity, how big was the box used for testing? It might be worth investigating if anything can be done to reduce the number of times that lock is taken in the first place.

After all, what's a pain on a 4-way tends to be an utter nightmare on a 16-way ;(

Cheers,
Jes

#6 lore raw
Quoted 10 lines · Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arjan" == Arjan van de Ven <[email protected]> writes:

Arjan> In a micro-benchmark that stresses the pagefault path, the Arjan> down_read_trylock on the mmap_sem showed up quite high on the Arjan> profile. Turns out this lock is bouncing between cpus quite a Arjan> bit and thus is cache-cold a lot. This patch prefetches the Arjan> lock (for write) as early as possible (and before some other Arjan> somewhat expensive operations). With this patch, the Arjan> down_read_trylock basically fell out of the top of profile.

Quoted 6 lines

Out of curiousity, how big was the box used for testing? It might be worth investigating if anything can be done to reduce the number of times that lock is taken in the first place.

After all, what's a pain on a 4-way tends to be an utter nightmare on a 16-way ;(

most of it was done on a 2 way, but some tests were done on a 4-way.

#7 lore raw
Quoted 18 lines · Arjan van de Ven
Quoted 10 lines · Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arjan" == Arjan van de Ven <[email protected]> writes:

Arjan> In a micro-benchmark that stresses the pagefault path, the Arjan> down_read_trylock on the mmap_sem showed up quite high on the Arjan> profile. Turns out this lock is bouncing between cpus quite a Arjan> bit and thus is cache-cold a lot. This patch prefetches the Arjan> lock (for write) as early as possible (and before some other Arjan> somewhat expensive operations). With this patch, the Arjan> down_read_trylock basically fell out of the top of profile.

Quoted 6 lines

Out of curiousity, how big was the box used for testing? It might be worth investigating if anything can be done to reduce the number of times that lock is taken in the first place.

After all, what's a pain on a 4-way tends to be an utter nightmare on a 16-way ;(

Quoted 6 lines

most of it was done on a 2 way, but some tests were done on a 4-way. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Could you share your microbenchmark with us (or point to the source) and we can give this a try on larger systems?

Thanks,

Signature · 3 lines
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)