Support for RWF_UNCACHED
Subsystems (9)
Jens Axboe
maintainerv2 cover Support for RWF_UNCACHED
ecently someone asked me how io_uring buffered IO compares to mmaped IO in terms of performance. So I ran some tests with buffered IO, and found the experience to be somewhat painful. The test case is pretty basic, random reads over a dataset that's 10x the size of RAM. Performance starts out fine, and then the page cache fills up and we hit a throughput cliff. CPU usage of the IO threads go up, and we have kswapd spending 100% of a core trying to keep up. Seeing that, I was reminded of the many complaints I here about buffered IO, and the fact that most of the folks complaining will ultimately bite the bullet and move to O_DIRECT to just get the kernel out of the way.
But I don't think it needs to be like that. Switching to O_DIRECT isn't always easily doable. The buffers have different life times, size and alignment constraints, etc. On top of that, mixing buffered and O_DIRECT can be painful.
Seems to me that we have an opportunity to provide something that sits somewhere in between buffered and O_DIRECT, and this is where RWF_UNCACHED enters the picture. If this flag is set on IO, we get the following behavior:
- If the data is in cache, it remains in cache and the copy (in or out) is served to/from that.
- If the data is NOT in cache, we add it while performing the IO. When the IO is done, we remove it again.
With this, I can do 100% smooth buffered reads or writes without pushing the kernel to the state where kswapd is sweating bullets. In fact it doesn't even register.
Comments appreciated! This should work on any standard file system, using either the generic helpers or iomap. Patches are against current git, and can also be found here:
https://git.kernel.dk/cgit/linux-block/log/?h=buffered-uncached
fs/ceph/file.c | 2 +- fs/dax.c | 2 +- fs/ext4/file.c | 2 +- fs/iomap/apply.c | 2 +- fs/iomap/buffered-io.c | 89 +++++++++++++++++++------ fs/iomap/direct-io.c | 3 +- fs/iomap/fiemap.c | 5 +- fs/iomap/seek.c | 6 +- fs/iomap/swapfile.c | 2 +- fs/nfs/file.c | 2 +- include/linux/fs.h | 11 +++- include/linux/iomap.h | 6 +- include/uapi/linux/fs.h | 5 +- mm/filemap.c | 139 ++++++++++++++++++++++++++++++++++++---- 14 files changed, 230 insertions(+), 46 deletions(-)
Changes since v1:
- Switch to pagevecs for write_drop_cached_pages()
- Use page_offset() instead of manual shift
- Ensure we hold a reference on the page between calling ->write_end() and checking the mapping on the locked page
- Fix XFS multi-page streamed writes, we'd drop the UNCACHED flag after the first page
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v2 1/5 fs: add read support for RWF_UNCACHED
If RWF_UNCACHED is set for io_uring (or preadv2(2)), we'll drop the cache for buffered reads if we are the ones instantiating it. If the data is already cached, we leave it cached.
--- include/linux/fs.h | 3 +++ include/uapi/linux/fs.h | 5 ++++- mm/filemap.c | 46 ++++++++++++++++++++++++++++++++++++----- 3 files changed, 48 insertions(+), 6 deletions(-)
include/linux/fs.h +3 -0
@@ -314,6 +314,7 @@ enum rw_hint {#define IOCB_SYNC (1 << 5)#define IOCB_WRITE (1 << 6)#define IOCB_NOWAIT (1 << 7) +#define IOCB_UNCACHED (1 << 8)struct kiocb {struct file *ki_filp;
@@ -3418,6 +3419,8 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags) ki->ki_flags |= (IOCB_DSYNC |IOCB_SYNC );if (flags &RWF_APPEND ) ki->ki_flags |=IOCB_APPEND ; +if (flags &RWF_UNCACHED ) + ki->ki_flags |=IOCB_UNCACHED ;return 0 ; }
include/uapi/linux/fs.h +4 -1
@@ -299,8 +299,11 @@ typedef int __bitwise __kernel_rwf_t;/* per-IO O_APPEND */ #define RWF_APPEND ((__force __kernel_rwf_t)0x00000010) +/* drop cache after reading or writing data */ +#define RWF_UNCACHED ((__force __kernel_rwf_t)0x00000040) +/* mask of flags supported by the kernel */ #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ - RWF_APPEND) + RWF_APPEND | RWF_UNCACHED)#endif /* _UAPI_LINUX_FS_H */
mm/filemap.c +41 -5
84 lines not on this page · raw message
Signature · 1 line
2.24.0
Jens Axboe
maintainerPATCH v2 2/5 mm: make generic_perform_write() take a struct kiocb
In reply to Jens AxboeRight now all callers pass in iocb->ki_pos, just pass in the iocb. This is in preparation for using the iocb flags in generic_perform_write().
--- fs/ceph/file.c | 2 +- fs/ext4/file.c | 2 +- fs/nfs/file.c | 2 +- include/linux/fs.h | 3 ++- mm/filemap.c | 8 +++++--- 5 files changed, 10 insertions(+), 7 deletions(-)
fs/ceph/file.c +1 -1
@@ -1538,7 +1538,7 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from)* are pending vmtruncate. So write and vmtruncate * can not run at the same time */ - written =generic_perform_write (file, from, pos); + written =generic_perform_write (file, from, iocb);if (likely (written >=0 )) iocb->ki_pos = pos + written;ceph_end_io_write (inode);
fs/ext4/file.c +1 -1
@@ -249,7 +249,7 @@ static ssize_t ext4_buffered_write_iter(struct kiocb *iocb, goto out; current->backing_dev_info =inode_to_bdi (inode); - ret =generic_perform_write (iocb->ki_filp, from, iocb->ki_pos); + ret =generic_perform_write (iocb->ki_filp, from, iocb); current->backing_dev_info =NULL ; out:
fs/nfs/file.c +1 -1
@@ -624,7 +624,7 @@ ssize_t nfs_file_write(struct kiocb *iocb, struct iov_iter *from) result =generic_write_checks (iocb, from);if (result >0 ) { current->backing_dev_info =inode_to_bdi (inode); - result =generic_perform_write (file, from, iocb->ki_pos); + result =generic_perform_write (file, from, iocb); current->backing_dev_info =NULL ; }nfs_end_io_write (inode);
include/linux/fs.h +2 -1
@@ -3103,7 +3103,8 @@ extern ssize_t generic_file_read_iter(struct kiocb *, struct iov_iter *);extern ssize_t __generic_file_write_iter (struct kiocb *,struct iov_iter *);extern ssize_t generic_file_write_iter (struct kiocb *,struct iov_iter *);extern ssize_t generic_file_direct_write (struct kiocb *,struct iov_iter *); -extern ssize_t generic_perform_write (struct file *,struct iov_iter *,loff_t ); +extern ssize_t generic_perform_write (struct file *,struct iov_iter *, +struct kiocb *);ssize_t vfs_iter_read (struct file *file,struct iov_iter *iter,loff_t *ppos,rwf_t flags);
mm/filemap.c +5 -3
@@ -3302,10 +3302,11 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping,EXPORT_SYMBOL (grab_cache_page_write_begin);ssize_t generic_perform_write (struct file *file, -struct iov_iter *i,loff_t pos) +struct iov_iter *i,struct kiocb *iocb) {struct address_space *mapping = file->f_mapping;const struct address_space_operations *a_ops = mapping->a_ops; +loff_t pos = iocb->ki_pos;long status =0 ;ssize_t written =0 ;unsigned int flags =0 ;
@@ -3439,7 +3440,8 @@ ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)if (written <0 || !iov_iter_count (from) ||IS_DAX (inode)) goto out; - status =generic_perform_write (file, from, pos = iocb->ki_pos); + pos = iocb->ki_pos; + status =generic_perform_write (file, from, iocb);/* * If generic_perform_write() returned a synchronous error * then we want to return the number of bytes which were
@@ -3471,7 +3473,7 @@ ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)*/ } }else { - written =generic_perform_write (file, from, iocb->ki_pos); + written =generic_perform_write (file, from, iocb);if (likely (written >0 )) iocb->ki_pos += written; }
Signature · 1 line
2.24.0
Jens Axboe
maintainerPATCH v2 3/5 mm: make buffered writes work with RWF_UNCACHED
In reply to Jens AxboeIf RWF_UNCACHED is set for io_uring (or pwritev2(2)), we'll drop the cache instantiated for buffered writes. If new pages aren't instantiated, we leave them alone. This provides similar semantics to reads with RWF_UNCACHED set.
--- include/linux/fs.h | 5 +++ mm/filemap.c | 85 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 85 insertions(+), 5 deletions(-)
include/linux/fs.h +5 -0
@@ -285,6 +285,7 @@ enum positive_aop_returns {#define AOP_FLAG_NOFS 0x0002/* used by filesystem to direct * helper code (eg buffer layer) * to clear GFP_FS from alloc */ +#define AOP_FLAG_UNCACHED 0x0004/* * oh the beauties of C type declarations.
@@ -3106,6 +3107,10 @@ extern ssize_t generic_file_direct_write(struct kiocb *, struct iov_iter *);extern ssize_t generic_perform_write (struct file *,struct iov_iter *,struct kiocb *); +struct pagevec ; +extern void write_drop_cached_pages (struct pagevec *pvec, +struct address_space *mapping); +ssize_t vfs_iter_read (struct file *file,struct iov_iter *iter,loff_t *ppos,rwf_t flags); ssize_t vfs_iter_write(struct file *file, struct iov_iter *iter, loff_t *ppos,
mm/filemap.c +80 -5
151 lines not on this page · raw message
Signature · 1 line
2.24.0
Andrew Morton
maintainerRe: [PATCH 3/5] mm: make buffered writes work with RWF_UNCACHED
Quoted 5 lines · On Tue, 10 Dec 2019 13:43:02 -0700 Jens Axboe <[email protected]> wrote:
If
RWF_UNCACHEDis set forio_uring(or pwritev2(2)), we'll drop the cache instantiated for buffered writes. If new pages aren't instantiated, we leave them alone. This provides similar semantics to reads withRWF_UNCACHEDset.
Wouid be nice to see a description of the proposed userspace API(s) for exploiting this feature.
Quoted 67 lines
include/linux/fs.h +5
-0@@ -285,6 +285,7 @@ enum positive_aop_returns {#define AOP_FLAG_NOFS 0x0002/* used by filesystem to direct * helper code (eg buffer layer) * to clear GFP_FS from alloc */ +#define AOP_FLAG_UNCACHED 0x0004/* * oh the beauties of C type declarations. @@ -3106,6 +3107,10 @@ extern ssize_t generic_file_direct_write(struct kiocb *, struct iov_iter *);extern ssize_t generic_perform_write (struct file *,struct iov_iter *,struct kiocb *); +struct pagevec ; +extern void write_drop_cached_pages (struct pagevec *pvec, +struct address_space *mapping); +ssize_t vfs_iter_read (struct file *file,struct iov_iter *iter,loff_t *ppos,rwf_t flags); ssize_t vfs_iter_write(struct file *file, struct iov_iter *iter, loff_t *ppos,mm/filemap.c +27
-1@@ -3287,10 +3287,12 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping, pgoff_t index, unsigned flags) {struct page *page; -int fgp_flags =FGP_LOCK |FGP_WRITE |FGP_CREAT ; +int fgp_flags =FGP_LOCK |FGP_WRITE ;if (flags &AOP_FLAG_NOFS ) fgp_flags |=FGP_NOFS ; +if (!(flags &AOP_FLAG_UNCACHED )) + fgp_flags |=FGP_CREAT ; page =pagecache_get_page (mapping, index, fgp_flags,mapping_gfp_mask (mapping));@@ -3301,21 +3303,65 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping, }EXPORT_SYMBOL (grab_cache_page_write_begin); +/* +* Start writeback on the pages in pgs[], and then try and remove those pages +* from the page cached. Used with RWF_UNCACHED. +*/ +void write_drop_cached_pages (struct pagevec *pvec, +struct address_space *mapping) +{ +loff_t start, end; +int i; + + end =0 ; + start =LLONG_MAX ; +for (i =0 ; i <pagevec_count (pvec); i++) { +loff_t off =page_offset (pvec->pages[i]); +if (off < start) + start = off; +if (off > end) + end = off; + } + +__filemap_fdatawrite_range (mapping, start, end,WB_SYNC_NONE ); + +for (i =0 ; i <pagevec_count (pvec); i++) { +struct page *page = pvec->pages[i];
mm/filemap.c struct page *grab_cache_page_write_begin(struct address_space *mapping, · line 3339
Quoted 10 lines
mm/filemap.c +10
-0+ +lock_page (page); +if (page->mapping == mapping) { +wait_on_page_writeback (page); +if (!page_has_private (page) || +try_to_release_page (page,0 )) +remove_mapping (mapping, page); + } +unlock_page (page); + }
This is kinda invalidate_inode_pages2_range(), only much less so? Why doesn't this code need to do all the things which invalidate_inode_pages2_range() does? What happens if these pages are mmapped, faulted in? Not faulted in?
Quoted 2 lines
mm/filemap.c +2
-0+pagevec_release (pvec); +}
Jens Axboe
maintainerPATCH v2 4/5 iomap: pass in the write_begin/write_end flags to iomap_actor
In reply to Jens AxboeThis is in preparation for passing in a flag to the iomap_actor, which currently doesn't support that.
No functional changes in this patch.
--- fs/dax.c | 2 +- fs/iomap/apply.c | 2 +- fs/iomap/buffered-io.c | 17 ++++++++++------- fs/iomap/direct-io.c | 3 ++- fs/iomap/fiemap.c | 5 +++-- fs/iomap/seek.c | 6 ++++-- fs/iomap/swapfile.c | 2 +- include/linux/iomap.h | 5 +++-- 8 files changed, 25 insertions(+), 17 deletions(-)
fs/dax.c +1 -1
@@ -1091,7 +1091,7 @@ EXPORT_SYMBOL_GPL(__dax_zero_page_range);static loff_t dax_iomap_actor (struct inode *inode,loff_t pos,loff_t length,void *data, -struct iomap *iomap,struct iomap *srcmap) +unsigned flags,struct iomap *iomap,struct iomap *srcmap) {struct block_device *bdev = iomap->bdev;struct dax_device *dax_dev = iomap->dax_dev;
fs/iomap/apply.c +1 -1
@@ -77,7 +77,7 @@ iomap_apply(struct inode *inode, loff_t pos, loff_t length, unsigned flags,* iomap into the actors so that they don't need to have special * handling for the two cases. */ - written =actor (inode, pos, length, data, &iomap, + written =actor (inode, pos, length, data, flags, &iomap, srcmap.type !=IOMAP_HOLE ? &srcmap : &iomap);/*
fs/iomap/buffered-io.c +10 -7
@@ -249,7 +249,7 @@ static inline bool iomap_block_needs_zeroing(struct inode *inode,static loff_t iomap_readpage_actor (struct inode *inode,loff_t pos,loff_t length,void *data, -struct iomap *iomap,struct iomap *srcmap) +unsigned flags,struct iomap *iomap,struct iomap *srcmap) {struct iomap_readpage_ctx *ctx = data;struct page *page = ctx->cur_page;
@@ -397,7 +397,8 @@ iomap_next_page(struct inode *inode, struct list_head *pages, loff_t pos,static loff_t iomap_readpages_actor (struct inode *inode,loff_t pos,loff_t length, -void *data,struct iomap *iomap,struct iomap *srcmap) +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap) {struct iomap_readpage_ctx *ctx = data;loff_t done, ret;
@@ -417,7 +418,7 @@ iomap_readpages_actor(struct inode *inode, loff_t pos, loff_t length, ctx->cur_page_in_bio = false; } ret =iomap_readpage_actor (inode, pos + done, length - done, - ctx, iomap, srcmap); + ctx,0 , iomap, srcmap); }return done;
@@ -797,7 +798,7 @@ iomap_write_end(struct inode *inode, loff_t pos, unsigned len, unsigned copied,static loff_t iomap_write_actor (struct inode *inode,loff_t pos,loff_t length,void *data, -struct iomap *iomap,struct iomap *srcmap) +unsigned flags,struct iomap *iomap,struct iomap *srcmap) {struct iov_iter *i = data;long status =0 ;
@@ -897,7 +898,7 @@ EXPORT_SYMBOL_GPL(iomap_file_buffered_write);static loff_t iomap_unshare_actor (struct inode *inode,loff_t pos,loff_t length,void *data, -struct iomap *iomap,struct iomap *srcmap) +unsigned flags,struct iomap *iomap,struct iomap *srcmap) {long status =0 ;ssize_t written =0 ;
@@ -983,7 +984,8 @@ static int iomap_dax_zero(loff_t pos, unsigned offset, unsigned bytes,static loff_t iomap_zero_range_actor (struct inode *inode,loff_t pos,loff_t count, -void *data,struct iomap *iomap,struct iomap *srcmap) +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap) {bool *did_zero = data;loff_t written =0 ;
@@ -1053,7 +1055,8 @@ EXPORT_SYMBOL_GPL(iomap_truncate_page);static loff_t iomap_page_mkwrite_actor (struct inode *inode,loff_t pos,loff_t length, -void *data,struct iomap *iomap,struct iomap *srcmap) +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap) {struct page *page = data;int ret;
fs/iomap/direct-io.c +2 -1
@@ -365,7 +365,8 @@ iomap_dio_inline_actor(struct inode *inode, loff_t pos, loff_t length,static loff_t iomap_dio_actor (struct inode *inode,loff_t pos,loff_t length, -void *data,struct iomap *iomap,struct iomap *srcmap) +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap) {struct iomap_dio *dio = data;
fs/iomap/fiemap.c +3 -2
@@ -44,7 +44,7 @@ static int iomap_to_fiemap(struct fiemap_extent_info *fi,static loff_t iomap_fiemap_actor (struct inode *inode,loff_t pos,loff_t length,void *data, -struct iomap *iomap,struct iomap *srcmap) +unsigned flags,struct iomap *iomap,struct iomap *srcmap) {struct fiemap_ctx *ctx = data;loff_t ret = length;
@@ -111,7 +111,8 @@ EXPORT_SYMBOL_GPL(iomap_fiemap);static loff_t iomap_bmap_actor (struct inode *inode,loff_t pos,loff_t length, -void *data,struct iomap *iomap,struct iomap *srcmap) +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap) {sector_t *bno = data, addr;
fs/iomap/seek.c +4 -2
@@ -119,7 +119,8 @@ page_cache_seek_hole_data(struct inode *inode, loff_t offset, loff_t length,static loff_t iomap_seek_hole_actor (struct inode *inode,loff_t offset,loff_t length, -void *data,struct iomap *iomap,struct iomap *srcmap) +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap) {switch (iomap->type) {case IOMAP_UNWRITTEN :
@@ -165,7 +166,8 @@ EXPORT_SYMBOL_GPL(iomap_seek_hole);static loff_t iomap_seek_data_actor (struct inode *inode,loff_t offset,loff_t length, -void *data,struct iomap *iomap,struct iomap *srcmap) +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap) {switch (iomap->type) {case IOMAP_HOLE :
fs/iomap/swapfile.c +1 -1
@@ -76,7 +76,7 @@ static int iomap_swapfile_add_extent(struct iomap_swapfile_info *isi)* distinction between written and unwritten extents. */ static loff_t iomap_swapfile_activate_actor (struct inode *inode,loff_t pos, -loff_t count,void *data,struct iomap *iomap, +loff_t count,void *data,unsigned flags,struct iomap *iomap,struct iomap *srcmap) {struct iomap_swapfile_info *isi = data;
include/linux/iomap.h +3 -2
@@ -113,7 +113,7 @@ struct iomap_page_ops { };/* -* Flags for iomap_begin / iomap_end. No flag implies a read. +* Flags for iomap_begin / iomap_end / factor. No flag implies a read. */ #define IOMAP_WRITE (1 << 0)/* writing, must allocate blocks */ #define IOMAP_ZERO (1 << 1)/* zeroing operation, may skip holes */
@@ -146,7 +146,8 @@ struct iomap_ops {* Main iomap iterator function. */ typedef loff_t (*iomap_actor_t )(struct inode *inode,loff_t pos,loff_t len, -void *data,struct iomap *iomap,struct iomap *srcmap); +void *data,unsigned flags,struct iomap *iomap, +struct iomap *srcmap); loff_t iomap_apply(struct inode *inode, loff_t pos, loff_t length, unsigned flags, const struct iomap_ops *ops, void *data,
Signature · 1 line
2.24.0
Jens Axboe
maintainerPATCH v2 5/5 iomap: support RWF_UNCACHED for buffered writes
In reply to Jens AxboeThis adds support for RWF_UNCACHED for file systems using iomap to perform buffered writes. We use the generic infrastructure for this, by tracking pages we created and calling write_drop_cached_pages() to issue writeback and prune those pages.
--- fs/iomap/buffered-io.c | 72 +++++++++++++++++++++++++++++++++++------- include/linux/iomap.h | 1 + 2 files changed, 62 insertions(+), 11 deletions(-)
fs/iomap/buffered-io.c +61 -11
@@ -17,6 +17,7 @@#include <linux/bio.h> #include <linux/sched/signal.h> #include <linux/migrate.h> +#include <linux/pagevec.h> #include "trace.h" #include "../internal.h"
@@ -566,6 +567,7 @@ EXPORT_SYMBOL_GPL(iomap_migrate_page);enum {IOMAP_WRITE_F_UNSHARE = (1 <<0 ), +IOMAP_WRITE_F_UNCACHED = (1 <<1 ), }; static void
@@ -643,6 +645,7 @@ iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, unsigned flags, struct page **pagep, struct iomap *iomap, struct iomap *srcmap) { const struct iomap_page_ops *page_ops = iomap->page_ops; +unsigned aop_flags;struct page *page;int status =0 ;
@@ -659,8 +662,11 @@ iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, unsigned flags,return status; } + aop_flags =AOP_FLAG_NOFS ; +if (flags &IOMAP_UNCACHED ) + aop_flags |=AOP_FLAG_UNCACHED ; page =grab_cache_page_write_begin (inode->i_mapping, pos >>PAGE_SHIFT , -AOP_FLAG_NOFS ); + aop_flags);if (!page) { status = -ENOMEM ; goto out_no_page;
@@ -670,9 +676,14 @@ iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, unsigned flags,iomap_read_inline_data (inode, page, srcmap); else if (iomap->flags & IOMAP_F_BUFFER_HEAD) status =__block_write_begin_int (page, pos, len,NULL , srcmap); - else - status =__iomap_write_begin (inode, pos, len, flags, page, + else { +unsigned wb_flags =0 ; + +if (flags &IOMAP_UNCACHED ) + wb_flags =IOMAP_WRITE_F_UNCACHED ; + status =__iomap_write_begin (inode, pos, len, wb_flags, page, srcmap); + }if (unlikely (status)) goto out_unlock;
@@ -796,19 +807,27 @@ iomap_write_end(struct inode *inode, loff_t pos, unsigned len, unsigned copied,return ret; } +#define GPW_PAGE_BATCH 16 + static loff_t iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data, unsigned flags, struct iomap *iomap, struct iomap *srcmap) { + struct address_space *mapping = inode->i_mapping; struct iov_iter *i = data; + struct pagevec pvec; long status = 0; ssize_t written = 0; + pagevec_init(&pvec); + do { struct page *page; unsigned long offset; /* Offset into pagecache page */ unsigned long bytes; /* Bytes to write to page */ size_t copied; /* Bytes copied from user */ + bool drop_page = false; /* drop page after IO */ + unsigned lflags = flags; offset = offset_in_page(pos); bytes = min_t(unsigned long, PAGE_SIZE - offset,
@@ -832,10 +851,17 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,break ; } - status =iomap_write_begin (inode, pos, bytes,0 , &page, iomap, - srcmap); -if (unlikely (status)) +retry: + status =iomap_write_begin (inode, pos, bytes, lflags, &page, + iomap, srcmap); +if (unlikely (status)) { +if (status == -ENOMEM && (lflags &IOMAP_UNCACHED )) { + drop_page = true; + lflags &= ~IOMAP_UNCACHED ; + goto retry; + }break ; + }if (mapping_writably_mapped (inode->i_mapping))flush_dcache_page (page);
@@ -844,10 +870,16 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,flush_dcache_page (page); +if (drop_page) +get_page (page); + status =iomap_write_end (inode, pos, bytes, copied, page, iomap, srcmap); -if (unlikely (status <0 )) +if (unlikely (status <0 )) { +if (drop_page) +put_page (page);break ; + } copied = status;cond_resched ();
@@ -864,15 +896,29 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,*/ bytes = min_t(unsigned long, PAGE_SIZE - offset,iov_iter_single_seg_count (i)); +if (drop_page) +put_page (page); goto again; } + +if (drop_page && + ((pos >>PAGE_SHIFT ) != ((pos + copied) >>PAGE_SHIFT ))) { +if (!pagevec_add (&pvec, page)) +write_drop_cached_pages (&pvec, mapping); + }else { +if (drop_page) +put_page (page); +balance_dirty_pages_ratelimited (inode->i_mapping); + } + pos += copied; written += copied; length -= copied; - -balance_dirty_pages_ratelimited (inode->i_mapping); }while (iov_iter_count (i) && length); +if (pagevec_count (&pvec)) +write_drop_cached_pages (&pvec, mapping); +return written ? written : status; }
@@ -882,10 +928,14 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *iter, {struct inode *inode = iocb->ki_filp->f_mapping->host;loff_t pos = iocb->ki_pos, ret =0 , written =0 ; +unsigned flags =IOMAP_WRITE ; + +if (iocb->ki_flags &IOCB_UNCACHED ) + flags |=IOMAP_UNCACHED ;while (iov_iter_count (iter)) { - ret =iomap_apply (inode, pos,iov_iter_count (iter), -IOMAP_WRITE , ops, iter, iomap_write_actor); + ret =iomap_apply (inode, pos,iov_iter_count (iter), flags, + ops, iter, iomap_write_actor);if (ret <=0 )break ; pos += ret;
include/linux/iomap.h +1 -0
@@ -121,6 +121,7 @@ struct iomap_page_ops {#define IOMAP_FAULT (1 << 3)/* mapping for page fault */ #define IOMAP_DIRECT (1 << 4)/* direct I/O */ #define IOMAP_NOWAIT (1 << 5)/* do not block */ +#define IOMAP_UNCACHED (1 << 6)struct iomap_ops {/*
Signature · 1 line
2.24.0
Re: [PATCH 5/5] iomap: support RWF_UNCACHED for buffered writes
Quoted 6 lines · Jens Axboe
This adds support for
RWF_UNCACHEDfor file systems using iomap to perform buffered writes. We use the generic infrastructure for this, by tracking pages we created and callingwrite_drop_cached_pages()to issue writeback and prune those pages.Signed-off-by Jens Axboe <[email protected]>
.....
fs/iomap/buffered-io.c iomap_write_end(struct inode *inode, loff_t pos, unsigned len, unsigned copied, · line 823
Quoted 12 lines
static
loff_t
iomap_write_actor(struct inode *inode,loff_tpos,loff_tlength, void *data,fs/iomap/buffered-io.c +4
-0unsigned flags, struct iomap *iomap, struct iomap *srcmap) { + struct address_space *mapping = inode->i_mapping;struct iov_iter *i = data; +struct pagevec pvec;long status =0 ;ssize_t written =0 ; +pagevec_init (&pvec); +
Ok, so the actor is called after we've already mapped and allocated an extent of arbitrary length. It may be a delalloc extent, it might be unwritten, it could be a COW mapping, etc.
Quoted 57 lines
fs/iomap/buffered-io.c +21
-4do { struct page *page; unsigned long offset; /* Offset into pagecache page */ unsigned long bytes; /* Bytes to write to page */ size_t copied; /* Bytes copied from user */ + bool drop_page = false; /* drop page after IO */ + unsigned lflags = flags; offset = offset_in_page(pos); bytes = min_t(unsigned long, PAGE_SIZE - offset,@@ -832,10 +851,17 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,break ; } - status =iomap_write_begin (inode, pos, bytes,0 , &page, iomap, - srcmap); -if (unlikely (status)) +retry: + status =iomap_write_begin (inode, pos, bytes, lflags, &page, + iomap, srcmap); +if (unlikely (status)) { +if (status == -ENOMEM && (lflags &IOMAP_UNCACHED )) { + drop_page = true; + lflags &= ~IOMAP_UNCACHED ; + goto retry; + }break ; + }if (mapping_writably_mapped (inode->i_mapping))flush_dcache_page (page);@@ -844,10 +870,16 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,flush_dcache_page (page); +if (drop_page) +get_page (page); + status =iomap_write_end (inode, pos, bytes, copied, page, iomap, srcmap); -if (unlikely (status <0 )) +if (unlikely (status <0 )) { +if (drop_page) +put_page (page);break ; + } copied = status;cond_resched ();@@ -864,15 +896,29 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,*/ bytes = min_t(unsigned long, PAGE_SIZE - offset,iov_iter_single_seg_count (i)); +if (drop_page) +put_page (page); goto again; }
fs/iomap/buffered-io.c iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data, · line 912
Quoted 10 lines
fs/iomap/buffered-io.c +10
-0+ +if (drop_page && + ((pos >>PAGE_SHIFT ) != ((pos + copied) >>PAGE_SHIFT ))) { +if (!pagevec_add (&pvec, page)) +write_drop_cached_pages (&pvec, mapping); + }else { +if (drop_page) +put_page (page); +balance_dirty_pages_ratelimited (inode->i_mapping); + }
This looks like it's a problem: this is going to write the data, which can cause the extent mapping of the file to change beyond the range that was written (e.g. due to speculative delayed allocation) and so the iomap we have already cached to direct write behaviour may now be stale.
IOWs, to be safe we need to terminate the write loop at this point, return to iomap_apply() and remap the range we are writing into so that we don't end up using a stale iomap. That kinda defeats the purpose of iomap - we are trying to do a single extent mapping per IO instead of per-page, and this pulls it back to an iomap per 16 pages for large user IOs. And it has the issues with breaking delayed allocation optimisations, too.
Hence, IMO, this is the wrong layer in iomap to be dealing with writeback and cache residency for uncached IO. We should be caching residency/invalidation at a per-IO level, not a per-page level.
Sure, have the write actor return a flag (e.g. in the iomap) to say that it encountered cached pages so that we can decide whether or not to invalidate the entire range we just wrote in iomap_apply, but doing it between mappings in iomap_apply means that the writeback is done once per user IO, and cache invalidation only occurs if no cached pages were encountered during that IO. i.e. add this to iomap_apply() after ops->iomap_end() has been called:
if (flags & RWF_UNCACHED) {
ret = filemap_write_and_wait_range(mapping, start, end);
if (ret)
goto out;
if (!drop_cache)
goto out;
/*
* Try to invalidate cache pages for the range we
* just wrote. We don't care if invalidation fails
* as the write has still worked and leaving clean
* uptodate pages * in the page cache isn't a
* corruption vector for uncached IO.
*/
invalidate_inode_pages2_range(mapping,
start >> PAGE_SHIFT, end >> PAGE_SHIFT);
}
out:
return written ? written : ret;
}Note that this doesn't solve the write error return issue. i.e. if filemap_write_and_wait_range() fails, should that error be returned or ignored?
And that leads to my next question: what data integrity guarantees does RWF_UNCACHED give? What if the underlying device has a volatile write cache or we dirtied metadata during block allocation? i.e. to a user, "UNCACHED" kinda implies that the write has ended up on stable storage because they are saying "do not cache this data". To me, none of this implementation guarantees data integrity, and users would still need O_DSYNC or fsync() with RWF_UNCACHED IO. That seems sane to me (same as direct io requirements) but whatever is decided here, it will need to be spelled out clearly in the man page so that users don't get it wrong.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jens Axboe
maintainerRe: [PATCH 5/5] iomap: support RWF_UNCACHED for buffered writes
Quoted 59 lines · Dave Chinner
Quoted 8 lines
fs/iomap/buffered-io.c +2
-0@@ -864,15 +896,29 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data,*/ bytes = min_t(unsigned long, PAGE_SIZE - offset,iov_iter_single_seg_count (i)); +if (drop_page) +put_page (page); goto again; }fs/iomap/buffered-io.c iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data, · line 912
Quoted 10 lines
fs/iomap/buffered-io.c +10
-0+ +if (drop_page && + ((pos >>PAGE_SHIFT ) != ((pos + copied) >>PAGE_SHIFT ))) { +if (!pagevec_add (&pvec, page)) +write_drop_cached_pages (&pvec, mapping); + }else { +if (drop_page) +put_page (page); +balance_dirty_pages_ratelimited (inode->i_mapping); + }This looks like it's a problem: this is going to write the data, which can cause the extent mapping of the file to change beyond the range that was written (e.g. due to speculative delayed allocation) and so the iomap we have already cached to direct write behaviour may now be stale.
IOWs, to be safe we need to terminate the write loop at this point, return to
iomap_apply()and remap the range we are writing into so that we don't end up using a stale iomap. That kinda defeats the purpose of iomap - we are trying to do a single extent mapping per IO instead of per-page, and this pulls it back to an iomap per 16 pages for large user IOs. And it has the issues with breaking delayed allocation optimisations, too.Hence, IMO, this is the wrong layer in iomap to be dealing with writeback and cache residency for uncached IO. We should be caching residency/invalidation at a per-IO level, not a per-page level.
Sure, have the write actor return a flag (e.g. in the iomap) to say that it encountered cached pages so that we can decide whether or not to invalidate the entire range we just wrote in
iomap_apply, but doing it between mappings iniomap_applymeans that the writeback is done once per user IO, and cache invalidation only occurs if no cached pages were encountered during that IO. i.e. add this toiomap_apply()afterops->iomap_end()has been called:if (flags & RWF_UNCACHED) { ret = filemap_write_and_wait_range(mapping, start, end); if (ret) goto out; if (!drop_cache) goto out; /* * Try to invalidate cache pages for the range we * just wrote. We don't care if invalidation fails * as the write has still worked and leaving clean * uptodate pages * in the page cache isn't a
Quoted 8 lines
* corruption vector for uncached IO. */ invalidate_inode_pages2_range(mapping, start >> PAGE_SHIFT, end >> PAGE_SHIFT); } out: return written ? written : ret; }
I really like this, and some variant of that solution can also be applied to the non-iomap case. It'll make for a cleaner implementation as well.
Once we do it per-write we'll have solved the performance and iomap weirdness, but it does mean that we need to make a decision on when to invalidate. For the per-page case it's clear - if the page is already there, leave it. If not, (attempt to) kill it. For the full write range, what if just one page was not there? Do we invalidate then? What if one page was there?
I'm going to move forward with the notion that if we had to allocate any page in the range, we invalidate the range. Only ranges that were fully cached already will remain in the cache. I think those semantics make the most sense.
Quoted 3 lines
Note that this doesn't solve the write error return issue. i.e. if
filemap_write_and_wait_range()fails, should that error be returned or ignored?
I think we should handle this exactly like we do the O_DIRECT case on buffered IO write-and-wait.
Quoted 10 lines
And that leads to my next question: what data integrity guarantees does
RWF_UNCACHEDgive? What if the underlying device has a volatile write cache or we dirtied metadata during block allocation? i.e. to a user, "UNCACHED" kinda implies that the write has ended up on stable storage because they are saying "do not cache this data". To me, none of this implementation guarantees data integrity, and users would still needO_DSYNCorfsync()withRWF_UNCACHEDIO. That seems sane to me (same as direct io requirements) but whatever is decided here, it will need to be spelled out clearly in the man page so that users don't get it wrong.
Fully agree, this isn't about data integrity guarantees. You will still need the appropriate sync magic to make it safe. This is exactly like O_DIRECT, and I think we should retain those exact same semantics for the RWF_UNCACHED case.
Thanks for taking a look at this! I'll respin (and re-test) with the write side changed.
Signature · 1 line
Jens Axboe