lazy freeing of memory through MADV_FREE

Restore MADV_DONTNEED to its original Linux behaviour. This is still not the same behaviour as POSIX, but applications may be depending on the Linux behaviour already. Besides, glibc catches POSIX_MADV_DONTNEED and makes sure nothing is done...

OK, we need to flesh this out a lot please. People often get confused about what our MADV_DONTNEED behaviour is.

Just for reference: the MADV_CURRENT behavior is to throw away data in the range.

The POSIX_MADV_DONTNEED behavior is to never lose data. I.e., file backed data is written back, anon data is at most swapped out.

But the Linux MADV_DONTNEED does throw away
data from a PROT_WRITE,MAP_PRIVATE mapping (or brk or stack) - those changes are discarded, and a subsequent access will revert to zeroes or the underlying mapped file. Been like that since before 2.4.0.

Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating them, to just marking them freeable. If the application wants to reuse them before the kernel needs the memory, not even a page fault will happen.

This patch, together with Ulrich's glibc change, increases MySQL sysbench performance by a factor of 2 on my quad core test system.

Signed-off-by Rik van Riel <[email protected]>

---
Ulrich Drepper has test glibc RPMS for this functionality at:

http://people.redhat.com/drepper/rpms

Andrew, I have stress tested this patch for a few days now and have not been able to find any more bugs. I believe it is ready to be merged in -mm, and upstream at the next merge window.

When the patch goes upstream, I will submit a small follow-up patch to revert MADV_DONTNEED behaviour to what it did previously and have the new behaviour trigger only on MADV_FREE: at that point people will have to get new test RPMs of glibc.

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

- Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when combined with Nick's patch.

It could well be that the combination of the two is even better, but it would be nice to firm that up a bit.

I do go on about that. But we're adding page flags at about one per year, and when we run out we're screwed - we'll need to grow the pageframe.

- I need to update your patch for Nick's patch. Please confirm that down_read(mmap_sem) is sufficient for MADV_FREE.

Stylistic nit:

mm/rmap.c static int try_to_unmap_one(struct page · line 717

Quoted 7 lines

mm/rmap.c renamed from mm/rmap.c.madv_free +7 -0

+	if (PageLazyFree(page) && !migration) {
+		/* There is new data in the page.  Reinstate it. */
+		if (unlikely(pte_dirty(pteval))) {
+			set_pte_at(mm, address, pte, pteval);
+			ret = SWAP_FAIL;
+			goto out_unmap;
+		}

The comment should be inside the second `if' statement. As it is, It looks like we reinstate the page if (PageLazyFree(page) && !migration).

Quoted 7 lines · Andrew Morton

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

- Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when combined with Nick's patch.

It could well be that the combination of the two is even better, but it would be nice to firm that up a bit.

I'll test that.

Quoted 3 lines

I do go on about that. But we're adding page flags at about one per year, and when we run out we're screwed - we'll need to grow the pageframe.

If you want, I can take a look at folding this into the ->mapping pointer. I can guarantee you it won't be pretty, though :)

Quoted 2 lines

- I need to update your patch for Nick's patch. Please confirm that down_read(mmap_sem) is sufficient for MADV_FREE.

It is. MADV_FREE needs no more protection than MADV_DONTNEED.

Quoted 12 lines

Stylistic nit:

mm/rmap.c static int try_to_unmap_one(struct page · line 717

Quoted 7 lines

mm/rmap.c renamed from mm/rmap.c.madv_free +7 -0

+	if (PageLazyFree(page) && !migration) {
+		/* There is new data in the page.  Reinstate it. */
+		if (unlikely(pte_dirty(pteval))) {
+			set_pte_at(mm, address, pte, pteval);
+			ret = SWAP_FAIL;
+			goto out_unmap;
+		}

The comment should be inside the second `if' statement. As it is, It looks like we reinstate the page if (PageLazyFree(page) && !migration).

Want me to move it?

Quoted 4 lines · On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[email protected]> wrote:

Quoted 2 lines · Andrew Morton

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

Quoted 6 lines

Quoted 5 lines

- Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when combined with Nick's patch.

It could well be that the combination of the two is even better, but it would be nice to firm that up a bit.

I'll test that.

Thanks.

Quoted 11 lines · Andrew Morton

Quoted 8 lines · On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[email protected]> wrote:

Quoted 6 lines · Andrew Morton

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

- Nick's patch also will help this problem. It could be that your patch
no longer offers a 2x speedup when combined with Nick's patch.

Quoted 6 lines

Quoted 4 lines

Quoted 3 lines

It could well be that the combination of the two is even better,
but it
would be nice to firm that up a bit.

I'll test that.

Thanks.

Well, good news.

It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU!

We _definately_ want both patches, there's a huge benefit in having them both.

Here are the transactions/seconds for each combination:

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

1 610 609 596 545

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

Rik van Riel a A(C)crit :

Quoted 30 lines

Quoted 11 lines · Andrew Morton

Quoted 8 lines · On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[email protected]> wrote:

Quoted 6 lines · Andrew Morton

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

- Nick's patch also will help this problem. It could be that your patch
no longer offers a 2x speedup when combined with Nick's patch.

Quoted 6 lines

Quoted 4 lines

Quoted 3 lines

It could well be that the combination of the two is even better,
but it
would be nice to firm that up a bit.

I'll test that.

Thanks.

Well, good news.

It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU!

We _definately_ want both patches, there's a huge benefit in having them both.

Here are the transactions/seconds for each combination:

Quoted 4 lines

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

1 610 609 596 545

545 tps versus 610 tps for one thread ? It seems quite bad, no ?

Could you please find an explanation for this ?

Quoted 4 lines

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU!

We _definately_ want both patches, there's a huge benefit in having them both.

Here are the transactions/seconds for each combination:

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

1 610 609 596 545

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

Quoted 9 lines · Rik van Riel

It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU!

We _definately_ want both patches, there's a huge benefit in having them both.

Here are the transactions/seconds for each combination:

Quoted 8 lines

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

1 610 609 596 545

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

FYI, I have uploaded a testing glibc that uses MADV_FREE and falls back to MADV_DONTUSE if MADV_FREE is not available, to
http://people.redhat.com/jakub/glibc/2.5.90-21.1/

Quoted 11 lines · Andrew Morton

Quoted 8 lines · On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[email protected]> wrote:

Quoted 6 lines · Andrew Morton

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

- Nick's patch also will help this problem. It could be that your patch
no longer offers a 2x speedup when combined with Nick's patch.

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

It could well be that the combination of the two is even better,
but it
would be nice to firm that up a bit.

I'll test that.

Thanks.

Well, good news.

It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU!

We _definately_ want both patches, there's a huge benefit in having them both.

Here are the transactions/seconds for each combination:

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

1 610 609 596 545

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

Quoted 37 lines · Rik van Riel

Quoted 16 lines · Andrew Morton

Quoted 13 lines · On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[email protected]> wrote:

Quoted 9 lines · Andrew Morton

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

- Nick's patch also will help this problem. It could be that your patch
no longer offers a 2x speedup when combined with Nick's patch.

It could well be that the combination of the two is even better,
but it
would be nice to firm that up a bit.

Quoted 5 lines

Quoted 1 line

I'll test that.

Thanks.

Well, good news.

It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU!

We _definately_ want both patches, there's a huge benefit in having them both.

Here are the transactions/seconds for each combination:

Quoted 8 lines

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

1 610 609 596 545

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

Is "new glibc" meaning MADV_DONTNEED + kernel with mmap_sem patch?

The strange thing with your madv_free kernel is that it doesn't help single-threaded performance at all. So that work to avoid zeroing the new page is not a win at all there (maybe due to the cache effects I was worried about?).

However MADV_FREE does improve scalability, which is interesting. The most likely reason I can see why that may be the case is that it avoids mmap_sem when faulting pages back in (I doubt it is due to avoiding the page allocator, but maybe?).

So where is the down_write coming from in this workload, I wonder? Heap management? What syscalls?

x86_64's rwsems are crap under heavy parallelism (even read-only), as I fixed in my recent generic rwsems patch. I don't expect MySQL to be such a mmap_sem microbenchmark, but I wonder how much this would help?

What if we ran the private futexes patch to further cut down mmap_sem contention?

Quoted 37 lines · Rik van Riel

Quoted 11 lines · Andrew Morton

Quoted 8 lines · On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[email protected]> wrote:

Quoted 6 lines · Andrew Morton

I've also merged Nick's "mm: madvise avoid exclusive mmap_sem".

- Nick's patch also will help this problem. It could be that your patch
no longer offers a 2x speedup when combined with Nick's patch.

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

It could well be that the combination of the two is even better,
but it
would be nice to firm that up a bit.

I'll test that.

Thanks.

Well, good news.

It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU!

We _definately_ want both patches, there's a huge benefit in having them both.

Here are the transactions/seconds for each combination:

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

Quoted 5 lines

1 610 609 596 545

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

Is "new glibc" meaning MADV_DONTNEED + kernel with mmap_sem patch?

The strange thing with your madv_free kernel is that it doesn't help single-threaded performance at all. So that work to avoid zeroing the new page is not a win at all there (maybe due to the cache effects I was worried about?).

However MADV_FREE does improve scalability, which is interesting. The most likely reason I can see why that may be the case is that it avoids mmap_sem when faulting pages back in (I doubt it is due to avoiding the page allocator, but maybe?).

So where is the down_write coming from in this workload, I wonder? Heap management? What syscalls?

Quoted 15 lines · Nick Piggin

Quoted 10 lines · Rik van Riel

Here are the transactions/seconds for each combination:

vanilla new glibc madv_free kernel madv_free + mmap_sem
threads

1 610 609 596 545

2    1032        1136            1196               1200
4    1070        1128            2014               2024
8    1000        1088            1665               2087
16    779        1073            1310               1999

Quoted 1 line

Is "new glibc" meaning MADV_DONTNEED + kernel with mmap_sem patch?

No, that's just the glibc change, with a vanilla kernel.

The third column is glibc change + mmap_sem patch.

The fourth column has your patch in it, too.

Quoted 4 lines

The strange thing with your madv_free kernel is that it doesn't help single-threaded performance at all. So that work to avoid zeroing the new page is not a win at all there (maybe due to the cache effects I was worried about?).

Well, your patch causes the performance to drop from 596 transactions/second to 545. Your patch is the only difference between the third and the fourth column.

Quoted 7 lines

However MADV_FREE does improve scalability, which is interesting. The most likely reason I can see why that may be the case is that it avoids mmap_sem when faulting pages back in (I doubt it is due to avoiding the page allocator, but maybe?).

So where is the down_write coming from in this workload, I wonder? Heap management? What syscalls?

I wonder if the increased parallelism simply caused more cache line bouncing, with bounces happening in some inner loop instead of an outer loop.

Btw, it is quite possible that the MySQL sysbench thing gives different results on your system. It would be good to know what it does on a real SMP system, vs. a single quad-core chip :)

Other architectures would be interesting to know, too.

Quoted 5 lines · Rik van Riel

Quoted 3 lines · Nick Piggin

Quoted 1 line · Rik van Riel

Here are the transactions/seconds for each combination:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

vanilla new glibc madv_free kernel madv_free + mmap_sem mmap_sem
threads

1 610 609 596 545 534

2    1032        1136            1196               1200        1180
4    1070        1128            2014               2024        2027
8    1000        1088            1665               2087        2089
16    779        1073            1310               1999        2012

Yeah. That's funny, because it means either there is some contention on the mmap_sem (or ptl) at 1 thread, or that my patch alters the uncontended performance.

Quoted 3 lines · Nick Piggin

Quoted 2 lines · Rik van Riel

Nick Piggin wrote:

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 1 line · Rik van Riel

Here are the transactions/seconds for each combination:

I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails.

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

vanilla new glibc madv_free kernel madv_free + mmap_sem
mmap_sem
threads

1 610 609 596 545 534

2    1032        1136            1196               1200        1180
4    1070        1128            2014               2024        2027
8    1000        1088            1665               2087        2089
16    779        1073            1310               1999        2012

Quoted 4 lines

I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails.

Now that I think about it - this is all with the rawhide kernel configuration, which has an ungodly number of debug config options enabled.

I should try this with a more normal kernel, on various different systems.

Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system.

Signed-off-by Rik van Riel <[email protected]>

---

Quoted 4 lines · Rik van Riel

Quoted 4 lines

I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails.

With the attached patch to make MADV_FREE use tlb batching, not only do we gain an additional 10-15% performance but Nick's mmap_sem patch also shows the performance increase that we expected to see.

It looks like the tlb flushes (and IPIs) from zap_pte_range() could have been the problem. They're gone now.

The second column from the right has Nick's patch and my own two patches. Performance with 16 threads is almost triple what it used to be...

vanilla   glibc  glibc      glibc        glibc      glibc      glibc
                 madv_free  madv_free               madv_free madv_free
                            mmap_sem     mmap_sem   mmap_sem

tlb batch tlb_batch
threads

1 610 609 596 545 534 547 537

 2    1032    1136    1196        1200        1180    1293    1194
 4    1070    1128    2014        2024        2027    2248    2040
 8    1000    1088    1665        2087        2089    2314    1869
 16    779    1073    1310        1999        2012    2214    1557

Quoted 6 lines

Now that I think about it - this is all with the rawhide kernel configuration, which has an ungodly number of debug config options enabled.

I should try this with a more normal kernel, on various different systems.

This is for another day. :)

First some ebizzy runs...

------------------------------------------------------------------------

mm/memory.c renamed from mm/memory.c.orig +6 -2

@@ -677,11 +677,15 @@ static unsigned long zap_pte_range(struc
 						remove_exclusive_swap_page(page);
 						unlock_page(page);
 					}
-					ptep_clear_flush_dirty(vma, addr, pte);
-					ptep_clear_flush_young(vma, addr, pte);
 					SetPageLazyFree(page);
 					if (PageActive(page))
 						deactivate_tail_page(page);
+					ptent = *pte;
+					set_pte_at(mm, addr, pte,
+						pte_mkclean(pte_mkold(ptent)));
+					/* tlb_remove_page frees it again */
+					get_page(page);
+					tlb_remove_page(tlb, page);
 					continue;
 				}
 			}

Quoted 2 lines

It looks like the tlb flushes (and IPIs) from zap_pte_range() could have been the problem. They're gone now.

I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after another thread already accesses it again, or after it subsequently gets removed from the address space via another CPU?

Quoted 9 lines · Nick Piggin

Quoted 2 lines

It looks like the tlb flushes (and IPIs) from zap_pte_range() could have been the problem. They're gone now.

I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after another thread already accesses it again, or after it subsequently gets removed from the address space via another CPU?

I have thought about this a lot tonight, and have come to the conclusion that they are ok.

The reason is simple:

1) we do the TLB flush before we return from the
madvise(MADV_FREE) syscall.

2) anything that accessess the pages between the start and end of the MADV_FREE procedure does not know in which order we go through the pages, so it could hit a page either before or after we get to processing it

3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka undefined behaviour territory and we do not need to worry about them

4) because we flush the tlb before releasing the page table lock, other CPUs cannot remove this page from the address space - they will block on the page table lock before looking at this pte

Quoted 4 lines

3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka undefined behaviour territory and we do not need to worry about them

Yes, but I'm wondering if it is legal in all architectures.

Quoted 4 lines

4) because we flush the tlb before releasing the page table lock, other CPUs cannot remove this page from the address space - they will block on the page table lock before looking at this pte

We don't when the ptl is split.

What the tlb flush used to be able to assume is that the page has been removed from the pagetables when they are put in the tlb flush batch.

I'm not saying there is any bugs, but just suggesting there might be.

Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system.

Signed-off-by Rik van Riel <[email protected]>

---

Quoted 6 lines · Nick Piggin

Quoted 4 lines

3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka undefined behaviour territory and we do not need to worry about them

Quoted 1 line

Yes, but I'm wondering if it is legal in all architectures.

It's similar to trying to access memory during an munmap.

You may be able to for a short time, but it'll come back to haunt you.

Quoted 6 lines

Quoted 4 lines

4) because we flush the tlb before releasing the page table lock, other CPUs cannot remove this page from the address space - they will block on the page table lock before looking at this pte

Quoted 1 line

We don't when the ptl is split.

Even then we do. Each invocation of zap_pte_range() only touches one page table page, and it flushes the TLB before releasing the page table lock.

Quoted 11 lines

Quoted 7 lines · Nick Piggin

Quoted 4 lines

3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka undefined behaviour territory and we do not need to worry about them

Yes, but I'm wondering if it is legal in all architectures.

Quoted 4 lines

It's similar to trying to access memory during an munmap.

You may be able to for a short time, but it'll come back to haunt you.

The question is whether the architecture specific tlb flushing code will break or not.

Quoted 3 lines

Even then we do. Each invocation of zap_pte_range() only touches one page table page, and it flushes the TLB before releasing the page table lock.

What kernel are you looking at? -rc7 and rc6-mm1 don't, AFAIKS.

This should fix the MADV_FREE code for PPC's hashed tlb.

Signed-off-by Rik van Riel <[email protected]>

---

Quoted 13 lines · Nick Piggin

Quoted 13 lines

Quoted 8 lines · Nick Piggin

Quoted 4 lines

3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka undefined behaviour territory and we do not need to worry about them

Yes, but I'm wondering if it is legal in all architectures.

Quoted 8 lines

Quoted 4 lines

It's similar to trying to access memory during an munmap.

You may be able to for a short time, but it'll come back to haunt you.

The question is whether the architecture specific tlb flushing code will break or not.

I guess we'll need to call tlb_remove_tlb_entry() inside the MADV_FREE code to keep powerpc happy.

Thanks for pointing this one out.

Quoted 6 lines

Quoted 3 lines

Even then we do. Each invocation of zap_pte_range() only touches one page table page, and it flushes the TLB before releasing the page table lock.

What kernel are you looking at? -rc7 and rc6-mm1 don't, AFAIKS.

Oh dear. I see it now...

The tlb end things inside zap_pte_range() are actually noops and the actual tlb flush only happens inside zap_page_range().

I guess the fact that munmap gets the mmap_sem for writing should save us, though...

------------------------------------------------------------------------

mm/memory.c renamed from mm/memory.c.noppc +1 -0

@@ -679,6 +679,7 @@ static unsigned long zap_pte_range(struc
 					}
 					ptep_test_and_clear_dirty(vma, addr, pte);
 					ptep_test_and_clear_young(vma, addr, pte);
+					tlb_remove_tlb_entry(tlb, pte, addr);
 					SetPageLazyFree(page);
 					if (PageActive(page))
 						deactivate_tail_page(page);

I guess we'll need to call tlb_remove_tlb_entry() inside the MADV_FREE code to keep powerpc happy.

Quoted 2 lines · Rik van Riel

I guess we'll need to call tlb_remove_tlb_entry() inside the MADV_FREE code to keep powerpc happy.

I don't see why; once ptep_test_and_clear_young has returned, the entry in the hash table has already been removed.

What the tlb flush used to be able to assume is that the page has been removed from the pagetables when they are put in the tlb flush batch.

I don't see why we need the attached, but in case you find a good reason, here's my signed-off-by line for Andrew :)

I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after another thread already accesses it again, or after it subsequently gets removed from the address space via another CPU?

First some ebizzy runs...

I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails.

vanilla new glibc madv_free mmap_sem both
threads

1 610 609 596 534 545

2    1032        1136        1196        1180        1200
4    1070        1128        2014        2027        2024
8    1000        1088        1665        2089        2087
16    779        1073        1310        2012        1999

Not doing the mprotect calls is the big one I guess, especially the fact that we don't need to take the mmap_sem for writing.

With both our patches, single and two thread performance with MySQL sysbench is somewhat better than with just your patch, 4 and 8 thread performance are basically the same and just your patch gives a slight benefit with 16 threads.

I guess I should benchmark up to 64 or 128 threads tomorrow, to see if this is just luck or if the cache benefit of doing the page faults and reusing hot pages is faster than not having page faults at all.

I should run some benchmarks on other systems, too. Some of these results could be an artifact of my quad core CPU. The results could be very different on other systems...

Quoted 4 lines · Rik van Riel

I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails.

Thanks! (I edited slightly so it doesn't wrap)

Quoted 4 lines

vanilla new glibc madv_free mmap_sem both
threads

1 610 609 596 534 545

Quoted 8 lines

2    1032        1136        1196        1180        1200
4    1070        1128        2014        2027        2024
8    1000        1088        1665        2089        2087
16    779        1073        1310        2012        1999

Not doing the mprotect calls is the big one I guess, especially the fact that we don't need to take the mmap_sem for writing.

Yes.

Quoted 10 lines

With both our patches, single and two thread performance with MySQL sysbench is somewhat better than with just your patch, 4 and 8 thread performance are basically the same and just your patch gives a slight benefit with 16 threads.

I guess I should benchmark up to 64 or 128 threads tomorrow, to see if this is just luck or if the cache benefit of doing the page faults and reusing hot pages is faster than not having page faults at all.

Quoted 3 lines

I should run some benchmarks on other systems, too. Some of these results could be an artifact of my quad core CPU. The results could be very different on other systems...

I'm getting the 16 core box out of retirement as we speak :)

I haven't tested your MADV_FREE patch yet.

So where is the down_write coming from in this workload, I wonder? Heap management? What syscalls?

Quoted 3 lines · Andrew Morton

I do go on about that. But we're adding page flags at about one per year, and when we run out we're screwed - we'll need to grow the pageframe.

If you want, I can take a look at folding this into the ->mapping pointer. I can guarantee you it won't be pretty, though :)

Quoted 5 lines · Rik van Riel

Quoted 3 lines · Andrew Morton

I do go on about that. But we're adding page flags at about one per year, and when we run out we're screwed - we'll need to grow the pageframe.

Quoted 3 lines

If you want, I can take a look at folding this into the ->mapping pointer. I can guarantee you it won't be pretty, though :)

Please don't. If we're going to stuff another pageflag into there, let it be PageSwapCache the natural partner of PageAnon, rather than whatever our latest pageflag happens to be.

Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating them, to just marking them freeable. If the application wants to reuse them before the kernel needs the memory, not even a page fault will happen.

This patch, together with Ulrich's glibc change, increases MySQL sysbench performance by a factor of 2 on my quad core test system.

Quoted 10 lines · On Tue, 17 Apr 2007 03:15:51 -0400 Rik van Riel <[email protected]> wrote:

Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating them, to just marking them freeable. If the application wants to reuse them before the kernel needs the memory, not even a page fault will happen.

This patch, together with Ulrich's glibc change, increases MySQL sysbench performance by a factor of 2 on my quad core test system.

In file included from include/linux/mman.h:4,

                 from arch/sparc64/kernel/sys_sparc.c:19:
include/asm/mman.h:36:1: "MADV_FREE" redefined
In file included from include/asm/mman.h:5,
                 from include/linux/mman.h:4,
                 from arch/sparc64/kernel/sys_sparc.c:19:

include/asm-generic/mman.h:32:1: this is the location of the previous definition

sparc32 and sparc64 already defined MADV_FREE:

#define MADV_FREE 0x5 /* (Solaris) contents can be freed */

I'll remove the sparc definitions for now, but we need to work out what we're going to do here. Your patch changes the values of MADV_FREE on sparc.

Perhaps this should be renamed to MADV_FREE_LINUX and given a different number. It depends on how close your proposed behaviour is to Solaris's.

Why isn't MADV_FREE defined to 5 for linux? It's our first free madv value? Also the behaviour should better match the one in solaris or BSD, the last thing we need is slightly different behaviour from operating systems supporting this for ages.