Add madvise(..., MADV_WILLWRITE)

My application fallocates and mmaps (shared, writable) a lot (several GB) of data at startup. Those mappings are mlocked, and they live on ext4. The first write to any given page is slow because ext4_da_get_block_prep can block. This means that, to get decent performance, I need to write something to all of these pages at startup. This, in turn, causes a giant IO storm as several GB of zeros get pointlessly written to disk.

This series is an attempt to add madvise(..., MADV_WILLWRITE) to signal to the kernel that I will eventually write to the referenced pages. It should cause any expensive operations that happen on the first write to happen immediately, but it should not result in dirtying the pages.

madvice(addr, len, MADV_WILLWRITE) returns the number of bytes that the operation succeeded on or a negative error code if there was an actual failure. A return value of zero signifies that the kernel doesn't know how to "willwrite" the range and that userspace should implement a fallback.

For now, it only works on shared writable ext4 mappings. Eventually it should support other filesystems as well as private pages (it should COW the pages but not cause swap IO) and anonymous pages (it should COW the zero page if applicable).

The implementation leaves much to be desired. In particular, it generates dirty buffer heads on a clean page, and this scares me.

Thoughts?

Andy Lutomirski (3):
mm: Add MADV_WILLWRITE to indicate that a range will be written to
fs: Add block_willwrite
ext4: Implement willwrite for the delalloc case

 fs/buffer.c                            | 57 ++++++++++++++++++++++++++++++++++
 fs/ext4/ext4.h                         |  2 ++
 fs/ext4/file.c                         |  1 +
 fs/ext4/inode.c                        | 22 +++++++++++++
 include/linux/buffer_head.h            |  3 ++
 include/linux/mm.h                     | 12 +++++++
 include/uapi/asm-generic/mman-common.h |  3 ++
 mm/madvise.c                           | 28 +++++++++++++++--
 8 files changed, 126 insertions(+), 2 deletions(-)

Signature · 6 lines

1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 25 lines · Andy Lutomirski

My application fallocates and mmaps (shared, writable) a lot (several GB) of data at startup. Those mappings are mlocked, and they live on ext4. The first write to any given page is slow because ext4_da_get_block_prep can block. This means that, to get decent performance, I need to write something to all of these pages at startup. This, in turn, causes a giant IO storm as several GB of zeros get pointlessly written to disk.

This series is an attempt to add madvise(..., MADV_WILLWRITE) to signal to the kernel that I will eventually write to the referenced pages. It should cause any expensive operations that happen on the first write to happen immediately, but it should not result in dirtying the pages.

madvice(addr, len, MADV_WILLWRITE) returns the number of bytes that the operation succeeded on or a negative error code if there was an actual failure. A return value of zero signifies that the kernel doesn't know how to "willwrite" the range and that userspace should implement a fallback.

For now, it only works on shared writable ext4 mappings. Eventually it should support other filesystems as well as private pages (it should COW the pages but not cause swap IO) and anonymous pages (it should COW the zero page if applicable).

Quoted 4 lines

The implementation leaves much to be desired. In particular, it generates dirty buffer heads on a clean page, and this scares me.

Thoughts?

One question before I look at the patches: Why don't you use fallocate()
in your application? The functionality you require seems to be pretty similar to it - writing to an already allocated block is usually quick.

One question before I look at the patches: Why don't you use fallocate()
in your application? The functionality you require seems to be pretty similar to it - writing to an already allocated block is usually quick.

Quoted 3 lines · Jan Kara

One question before I look at the patches: Why don't you use fallocate()
in your application? The functionality you require seems to be pretty similar to it - writing to an already allocated block is usually quick.

One problem I've seen is that it still costs you a fault per-page to get the PTEs in to a state where you can write to the memory. MADV_WILLNEED will do readahead to get the page cache filled, but it still leaves the pages unmapped. Those faults get expensive when you're trying to do a couple hundred million of them all at once.

Quoted 10 lines · Dave Hansen

Quoted 3 lines · Jan Kara

One question before I look at the patches: Why don't you use fallocate()
in your application? The functionality you require seems to be pretty similar to it - writing to an already allocated block is usually quick.

One problem I've seen is that it still costs you a fault per-page to get the PTEs in to a state where you can write to the memory. MADV_WILLNEED will do readahead to get the page cache filled, but it still leaves the pages unmapped. Those faults get expensive when you're trying to do a couple hundred million of them all at once.

I have grand plans to teach the kernel to use hardware dirty tracking so that (some?) pages can be left clean and writable for long periods of time. This will be hard.

Even so, the second write fault to a page tends to take only a few microseconds, while the first one often blocks in fs code.

(mmap_sem is a different story, but I see it as a separate issue.)

Quoted 12 lines · Andy Lutomirski

Quoted 10 lines · Dave Hansen

Quoted 3 lines · Jan Kara

One question before I look at the patches: Why don't you use fallocate()
in your application? The functionality you require seems to be pretty similar to it - writing to an already allocated block is usually quick.

One problem I've seen is that it still costs you a fault per-page to get the PTEs in to a state where you can write to the memory. MADV_WILLNEED will do readahead to get the page cache filled, but it still leaves the pages unmapped. Those faults get expensive when you're trying to do a couple hundred million of them all at once.

Quoted 3 lines

I have grand plans to teach the kernel to use hardware dirty tracking so that (some?) pages can be left clean and writable for long periods of time. This will be hard.

Right that will be tough... Although with your application you could
require such pages to be mlocked and then I could imagine we would get away at least from problems with dirty page accounting.

Quoted 3 lines

Even so, the second write fault to a page tends to take only a few microseconds, while the first one often blocks in fs code.

So you wrote blocks are already preallocated with fallocate(). If you
also preload pages in memory with MADV_WILLNEED is there still big difference between the first and subsequent write fault?

Quoted 13 lines · Jan Kara

Quoted 12 lines · Andy Lutomirski

Quoted 10 lines · Dave Hansen

Quoted 3 lines · Jan Kara

One question before I look at the patches: Why don't you use fallocate()
in your application? The functionality you require seems to be pretty similar to it - writing to an already allocated block is usually quick.

One problem I've seen is that it still costs you a fault per-page to get the PTEs in to a state where you can write to the memory. MADV_WILLNEED will do readahead to get the page cache filled, but it still leaves the pages unmapped. Those faults get expensive when you're trying to do a couple hundred million of them all at once.

Quoted 6 lines

Quoted 3 lines

I have grand plans to teach the kernel to use hardware dirty tracking so that (some?) pages can be left clean and writable for long periods of time. This will be hard.

Right that will be tough... Although with your application you could
require such pages to be mlocked and then I could imagine we would get away at least from problems with dirty page accounting.

True. The nasty part will be all the code that assumes that the acts of un-write-protecting and dirtying are the same thing, for example __block_write_begin, which is why I don't really believe in my willwrite patches...

Quoted 1 line

Quoted 5 lines

Quoted 2 lines

Even so, the second write fault to a page tends to take only a few microseconds, while the first one often blocks in fs code.

So you wrote blocks are already preallocated with fallocate(). If you
also preload pages in memory with MADV_WILLNEED is there still big difference between the first and subsequent write fault?

I haven't measured it yet, because I suspect that my patches are rather buggy in their current form.

Quoted 27 lines · Andy Lutomirski

Quoted 13 lines · Jan Kara

Quoted 12 lines · Andy Lutomirski

Quoted 10 lines · Dave Hansen

Quoted 3 lines · Jan Kara

One question before I look at the patches: Why don't you use fallocate()
in your application? The functionality you require seems to be pretty similar to it - writing to an already allocated block is usually quick.

One problem I've seen is that it still costs you a fault per-page to get the PTEs in to a state where you can write to the memory. MADV_WILLNEED will do readahead to get the page cache filled, but it still leaves the pages unmapped. Those faults get expensive when you're trying to do a couple hundred million of them all at once.

Quoted 6 lines

Quoted 3 lines

I have grand plans to teach the kernel to use hardware dirty tracking so that (some?) pages can be left clean and writable for long periods of time. This will be hard.

Right that will be tough... Although with your application you could
require such pages to be mlocked and then I could imagine we would get away at least from problems with dirty page accounting.

True. The nasty part will be all the code that assumes that the acts of un-write-protecting and dirtying are the same thing, for example __block_write_begin, which is why I don't really believe in my willwrite patches...

Quoted 1 line

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

Even so, the second write fault to a page tends to take only a few microseconds, while the first one often blocks in fs code.

So you wrote blocks are already preallocated with fallocate(). If you
also preload pages in memory with MADV_WILLNEED is there still big difference between the first and subsequent write fault?

I haven't measured it yet, because I suspect that my patches are rather buggy in their current form.

I think you're mistaking MADV_WILLNEED with your MADV_WILLWRITE.
MADV_WILLNEED will just trigger readahead for the range - thus you should have all pages with their block mappings set up in memory. Now the first writeable fault will still have to do some work, namely converting unwritten extents in extent tree to written ones. So there is going to be some difference between the first and subsequent writeable faults. But I'd like to see whether the difference is really worth the effort with new MADV_... call.

Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR

     2.39%   +2.34%  [kernel.kallsyms]      [k] __set_page_dirty_buffers               
             +2.50%  [kernel.kallsyms]      [k] __block_write_begin                    
             +2.16%  [kernel.kallsyms]      [k] __block_commit_write                   

I was coincidentally tracking down what I thought was a scalability problem (turned out to be full disks :). I noticed, though, that ext4 is about 20% slower than ext2/3 at doing write page faults (x-axis is number of tasks):

http://www.sr71.net/~dave/intel/page-fault-exts/cmp.html?1=ext3&2=ext4&hide=linear,threads,threads_idle,processes_idle&rollPeriod=5

The test case is:

https://github.com/antonblanchard/will-it-scale/blob/master/tests/page_fault3.c

Quoted 9 lines · Dave Hansen

I was coincidentally tracking down what I thought was a scalability problem (turned out to be full disks :). I noticed, though, that ext4 is about 20% slower than ext2/3 at doing write page faults (x-axis is number of tasks):

http://www.sr71.net/~dave/intel/page-fault-exts/cmp.html?1=ext3&2=ext4&hide=linear,threads,threads_idle,processes_idle&rollPeriod=5

The test case is:

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/page_fault3.c

The reason is that ext2/ext3 do almost nothing in their write fault
handler - they are about as fast as it can get. ext4 OTOH needs to reserve blocks for delayed allocation, setup buffers under a page etc. This is necessary if you want to make sure that if data are written via mmap, they also have space available on disk to be written to (ext2 / ext3 do not care and will just drop the data on the floor if you happen to hit ENOSPC during writeback).

I'm not saying ext4 write fault path cannot possibly be optimized (noone seriously looked into that AFAIK so there may well be some low hanging fruit) but it will always be slower than ext2/3. A more meaningful comparison would be with filesystems like XFS which make similar guarantees regarding data safety.

Quoted 10 lines · Jan Kara

Quoted 9 lines · Dave Hansen

I was coincidentally tracking down what I thought was a scalability problem (turned out to be full disks :). I noticed, though, that ext4 is about 20% slower than ext2/3 at doing write page faults (x-axis is number of tasks):

http://www.sr71.net/~dave/intel/page-fault-exts/cmp.html?1=ext3&2=ext4&hide=linear,threads,threads_idle,processes_idle&rollPeriod=5

The test case is:

Quoted 8 lines

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/page_fault3.c

The reason is that ext2/ext3 do almost nothing in their write fault
handler - they are about as fast as it can get. ext4 OTOH needs to reserve blocks for delayed allocation, setup buffers under a page etc. This is necessary if you want to make sure that if data are written via mmap, they also have space available on disk to be written to (ext2 / ext3 do not care and will just drop the data on the floor if you happen to hit ENOSPC during writeback).

Out of curiosity, why does ext4 need to set up buffers? That is, as long as the fs can guarantee that there is reserved space to write out the page, why isn't it sufficient to just mark the page dirty and let the writeback code set up the buffers?

Quoted 9 lines · Dave Hansen

Quoted 9 lines

I was coincidentally tracking down what I thought was a scalability problem (turned out to be full disks :). I noticed, though, that ext4 is about 20% slower than ext2/3 at doing write page faults (x-axis is number of tasks):

http://www.sr71.net/~dave/intel/page-fault-exts/cmp.html?1=ext3&2=ext4&hide=linear,threads,threads_idle,processes_idle&rollPeriod=5

The test case is:

Quoted 1 line

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/page_fault3.c

The reason is that ext2/ext3 do almost nothing in their write fault
handler - they are about as fast as it can get. ext4 OTOH needs to reserve blocks for delayed allocation, setup buffers under a page etc. This is necessary if you want to make sure that if data are written via mmap, they also have space available on disk to be written to (ext2 / ext3 do not care and will just drop the data on the floor if you happen to hit ENOSPC during writeback).

Quoted 10 lines · Jan Kara

Quoted 9 lines · Dave Hansen

Quoted 9 lines

I was coincidentally tracking down what I thought was a scalability problem (turned out to be full disks :). I noticed, though, that ext4 is about 20% slower than ext2/3 at doing write page faults (x-axis is number of tasks):

http://www.sr71.net/~dave/intel/page-fault-exts/cmp.html?1=ext3&2=ext4&hide=linear,threads,threads_idle,processes_idle&rollPeriod=5

The test case is:

Quoted 8 lines

Quoted 1 line

Quoted 1 line

https://github.com/antonblanchard/will-it-scale/blob/master/tests/page_fault3.c

The reason is that ext2/ext3 do almost nothing in their write fault
handler - they are about as fast as it can get. ext4 OTOH needs to reserve blocks for delayed allocation, setup buffers under a page etc. This is necessary if you want to make sure that if data are written via mmap, they also have space available on disk to be written to (ext2 / ext3 do not care and will just drop the data on the floor if you happen to hit ENOSPC during writeback).

I did try throwing a fallocate() in there to see if it helped. It didn't appear to help. Should it have?

Quoted 10 lines · Dave Hansen

I was coincidentally tracking down what I thought was a scalability problem (turned out to be full disks :). I noticed, though, that ext4 is about 20% slower than ext2/3 at doing write page faults (x-axis is number of tasks):

http://www.sr71.net/~dave/intel/page-fault-exts/cmp.html?1=ext3&2=ext4&hide=linear,threads,threads_idle,processes_idle&rollPeriod=5

The test case is:

https://github.com/antonblanchard/will-it-scale/blob/master/tests/page_fault3.c

The reason is that ext2/ext3 do almost nothing in their write fault
handler - they are about as fast as it can get. ext4 OTOH needs to reserve blocks for delayed allocation, setup buffers under a page etc. This is necessary if you want to make sure that if data are written via mmap, they also have space available on disk to be written to (ext2 / ext3 do not care and will just drop the data on the floor if you happen to hit ENOSPC during writeback).

I'm not saying ext4 write fault path cannot possibly be optimized (noone seriously looked into that AFAIK so there may well be some low hanging fruit) but it will always be slower than ext2/3. A more meaningful comparison would be with filesystems like XFS which make similar guarantees regarding data safety.

Whoops -- I read your email too quickly. I haven't tried MADV_WILLNEED, but I think I tried reading each page to fault them in.
Is there any reason to expect MADV_WILLNEED to do any better? I'll
try to do some new tests to see how well this all works.

(I imagine that freshly fallocated files are somehow different when read, since there aren't zeros on the disk backing them until they get written.)