iov_iter: Improve page extraction (ref, pin or just list)
Subsystems (8)
v7 cover iov_iter: Improve page extraction (ref, pin or just list)
Hi Al, Christoph,
Here are patches to provide support for extracting pages from an iov_iter and a patch to use the primary extraction function in the block layer bio code.
The patches make the following changes:
(1) Add a function, iov_iter_extract_pages() to replace iov_iter_get_pages*() that gets refs, pins or just lists the pages as appropriate to the iterator type and the I/O direction.
Add a function, iov_iter_extract_mode() that will indicate from the iterator type and the I/O direction how the cleanup is to be performed, returning FOLL_GET, FOLL_PIN or 0.
(2) Add a function, folio_put_unpin(), and a wrapper, page_put_unpin(), that take a page and the return from iov_iter_extract_mode() and do the right thing to clean up the page.
(3) Make the bio struct carry a pair of flags to indicate the cleanup mode. BIO_NO_PAGE_REF is replaced with BIO_PAGE_REFFED (equivalent to FOLL_GET) and BIO_PAGE_PINNED (equivalent to BIO_PAGE_PINNED) is added.
(4) Add a function, bio_release_page(), to release a page appropriately to the cleanup mode indicated by the BIO_PAGE_* flags.
(5) Make the iter-to-bio code use iov_iter_extract_pages() to retain the pages appropriately and clean them up later.
(6) Fix bio_flagged() so that it doesn't prevent a gcc optimisation.
(7) Renumber FOLL_GET and FOLL_PIN down so that they're at bits 0 and 1 and coincident with BIO_PAGE_REFFED and BIO_PAGE_PINNED. The compiler can then optimise on that. Also, it's probably going to be necessary to embed these in the page pointer in sk_buff fragments. This patch can go independently through the mm tree.
Changes:
========
ver #7)
- For now, drop the parts to pass the I/O direction to iov_iter_*pages*() as it turned out to be a lot more complicated, with places not setting IOCB_WRITE when they should, for example.
- Drop all the patches that changed things other then the block layer's bio handling. The netfslib and cifs changes can go into a separate patchset.
- Add support for extracting pages from KVEC-type iterators.
- When extracting from BVEC/KVEC, skip over empty vecs at the front.
ver #6)
- Fix write() syscall and co. not setting IOCB_WRITE.
- Added iocb_is_read() and iocb_is_write() to check IOCB_WRITE.
- Use op_is_write() in bio_copy_user_iov().
- Drop the iterator direction checks from smbd_recv().
- Define FOLL_SOURCE_BUF and FOLL_DEST_BUF and pass them in as part of gup_flags to iov_iter_get/extract_pages*().
- Replace iov_iter_get_pages*2() with iov_iter_get_pages*() and remove.
- Add back the function to indicate the cleanup mode.
- Drop the cleanup_mode return arg to iov_iter_extract_pages().
- Provide a helper to clean up a page.
- Renumbered FOLL_GET and FOLL_PIN and made BIO_PAGE_REFFED/PINNED have the same numerical values, enforced with an assertion.
- Converted AF_ALG, SCSI vhost, generic DIO, FUSE, splice to pipe, 9P and NFS.
- Added in the patches to make CIFS do top-to-bottom iterators and use various of the added extraction functions.
- Added a pair of work-in-progess patches to make sk_buff fragments store FOLL_GET and FOLL_PIN.
ver #5)
- Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED and split into own patch.
- Transcribe FOLL_GET/PIN into BIO_PAGE_REFFED/PINNED flags.
- Add patch to allow bio_flagged() to be combined by gcc.
ver #4)
- Drop the patch to move the FOLL_* flags to linux/mm_types.h as they're no longer referenced by linux/uio.h.
- Add ITER_SOURCE/DEST cleanup patches.
- Make iov_iter/netfslib iter extraction patches use ITER_SOURCE/DEST.
- Allow additional gup_flags to be passed into iov_iter_extract_pages().
- Add struct bio patch.
ver #3)
- Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1].
ver #2)
- Rolled the extraction cleanup mode query function into the extraction function, returning the indication through the argument list.
- Fixed patch 4 (extract to scatterlist) to actually use the new extraction API.
I've pushed the patches (excluding the two WIP networking patches) here also:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=iov-extract
David
David Howells (8):
iov_iter: Define flags to qualify page extraction.
iov_iter: Add a function to extract a page list from an iterator
mm: Provide a helper to drop a pin/ref on a page
block: Rename BIO_NO_PAGE_REF to BIO_PAGE_REFFED and invert the
meaning
block: Add BIO_PAGE_PINNED
block: Make bio structs pin pages rather than ref'ing if appropriate
block: Fix bio_flagged() so that gcc can better optimise it
mm: Renumber FOLL_GET and FOLL_PIN down
block/bio.c | 43 ++-- block/blk-map.c | 26 +-- block/blk.h | 29 +++ fs/iomap/direct-io.c | 1 - include/linux/bio.h | 5 +- include/linux/blk_types.h | 3 +- include/linux/mm.h | 17 +- include/linux/uio.h | 35 ++- lib/iov_iter.c | 438 +++++++++++++++++++++++++++++++++++++- mm/gup.c | 22 ++ 10 files changed, 571 insertions(+), 48 deletions(-)
PATCH v7 1/8 iov_iter: Define flags to qualify page extraction.
Define flags to qualify page extraction to pass into iov_iter_*_pages*() rather than passing in FOLL_* flags.
For now only a flag to allow peer-to-peer DMA is allowed; but in future, additional flags will be provided to indicate the I/O direction.
---
Notes:
ver #7)
- Don't use FOLL_* as a parameter, but rather define constants specifically to use with iov_iter_*_pages*().
- Drop the I/O direction constants for now.
block/bio.c | 6 +++--- block/blk-map.c | 8 ++++---- include/linux/uio.h | 7 +++++-- lib/iov_iter.c | 14 ++++++++------ 4 files changed, 20 insertions(+), 15 deletions(-)
block/bio.c +3 -3
@@ -1249,7 +1249,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;struct page **pages = (struct page **)bv; -unsigned int gup_flags =0 ; +unsigned int extract_flags =0 ;ssize_t size, left;unsigned len, i =0 ;size_t offset, trim;
@@ -1264,7 +1264,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) pages += entries_left * (PAGE_PTRS_PER_BVEC -1 );if (bio->bi_bdev &&blk_queue_pci_p2pdma (bio->bi_bdev->bd_disk->queue)) - gup_flags |=FOLL_PCI_P2PDMA ; + extract_flags |=ITER_ALLOW_P2PDMA ;/* * Each segment in the iov is required to be a block size multiple.
@@ -1275,7 +1275,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)*/ size =iov_iter_get_pages (iter, pages,UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, gup_flags); + nr_pages, &offset, extract_flags);if (unlikely (size <=0 ))return size ? size : -EFAULT ;
block/blk-map.c +4 -4
@@ -267,7 +267,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, {unsigned int max_sectors =queue_max_hw_sectors (rq->q);unsigned int nr_vecs =iov_iter_npages (iter,BIO_MAX_VECS ); -unsigned int gup_flags =0 ; +unsigned int extract_flags =0 ;struct bio *bio;int ret;int j;
@@ -280,7 +280,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,return -ENOMEM ;if (blk_queue_pci_p2pdma (rq->q)) - gup_flags |=FOLL_PCI_P2PDMA ; + extract_flags |=ITER_ALLOW_P2PDMA ;while (iov_iter_count (iter)) {struct page **pages, *stack_pages[UIO_FASTIOV ];
@@ -291,10 +291,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (nr_vecs <=ARRAY_SIZE (stack_pages)) { pages = stack_pages; bytes =iov_iter_get_pages (iter, pages,LONG_MAX , - nr_vecs, &offs, gup_flags); + nr_vecs, &offs, extract_flags); }else { bytes =iov_iter_get_pages_alloc (iter, &pages, -LONG_MAX , &offs, gup_flags); +LONG_MAX , &offs, extract_flags); }if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ;
include/linux/uio.h +5 -2
@@ -252,12 +252,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count);ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start, -unsigned gup_flags); +unsigned extract_flags);ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start);ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start, -unsigned gup_flags); +unsigned extract_flags);ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages);
@@ -360,4 +360,7 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +/* Flags for iov_iter_get/extract_pages*() */ +#define ITER_ALLOW_P2PDMA 0x01/* Allow P2PDMA on the extracted pages */ +#endif
lib/iov_iter.c +8 -6
@@ -1432,9 +1432,9 @@ static struct page *first_bvec_segment(const struct iov_iter *i,static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages,size_t *start, -unsigned int gup_flags) +unsigned int extract_flags) { -unsigned int n; +unsigned int n, gup_flags =0 ;if (maxsize > i->count) maxsize = i->count;
@@ -1442,6 +1442,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return 0 ;if (maxsize >MAX_RW_COUNT ) maxsize =MAX_RW_COUNT ; +if (extract_flags &ITER_ALLOW_P2PDMA ) + gup_flags |=FOLL_PCI_P2PDMA ;if (likely (user_backed_iter (i))) {unsigned long addr;
@@ -1495,14 +1497,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages, -size_t *start,unsigned gup_flags) +size_t *start,unsigned extract_flags) {if (!maxpages)return 0 ;BUG_ON (!pages);return __iov_iter_get_pages_alloc (i, &pages, maxsize, maxpages, - start, gup_flags); + start, extract_flags); }EXPORT_SYMBOL_GPL (iov_iter_get_pages);
@@ -1515,14 +1517,14 @@ EXPORT_SYMBOL(iov_iter_get_pages2);ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize, -size_t *start,unsigned gup_flags) +size_t *start,unsigned extract_flags) {ssize_t len; *pages =NULL ; len =__iov_iter_get_pages_alloc (i, pages, maxsize, ~0U , start, - gup_flags); + extract_flags);if (len <=0 ) {kvfree (*pages); *pages =NULL ;
Re: [PATCH v7 1/8] iov_iter: Define flags to qualify page extraction.
Quoted 2 lines · David Howells
For now only a flag to allow peer-to-peer DMA is allowed; but in future, additional flags will be provided to indicate the I/O direction.
Al still doesn't seem to be sold on the latter part, so maybe just drop this sentence for now.
PATCH v7 2/8 iov_iter: Add a function to extract a page list from an iterator
In reply to David HowellsAdd a function, iov_iter_extract_pages(), to extract a list of pages from an iterator. The pages may be returned with a reference added or a pin added or neither, depending on the type of iterator and the direction of transfer. The caller must pass FOLL_READ_FROM_MEM or FOLL_WRITE_TO_MEM as part of gup_flags to indicate how the iterator contents are to be used.
Add a second function, iov_iter_extract_mode(), to determine how the cleanup should be done.
There are three cases:
(1) Transfer *into* an ITER_IOVEC or ITER_UBUF iterator.
Extracted pages will have pins obtained on them (but not references) so that fork() doesn't CoW the pages incorrectly whilst the I/O is in progress.
iov_iter_extract_mode() will return FOLL_PIN for this case. The caller should use something like unpin_user_page() to dispose of the page.
(2) Transfer is *out of* an ITER_IOVEC or ITER_UBUF iterator.
Extracted pages will have references obtained on them, but not pins.
iov_iter_extract_mode() will return FOLL_GET. The caller should use something like put_page() for page disposal.
(3) Any other sort of iterator.
No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention. ITER_ALLOW_P2PDMA is not permitted.
iov_iter_extract_mode() will return 0. The pages don't need additional disposal.
---
Notes:
ver #7)
- Switch to passing in iter-specific flags rather than FOLL_* flags.
- Drop the direction flags for now.
- Use ITER_ALLOW_P2PDMA to request FOLL_PCI_P2PDMA.
- Disallow use of ITER_ALLOW_P2PDMA with non-user-backed iter.
- Add support for extraction from KVEC-type iters.
- Use iov_iter_advance() rather than open-coding it.
- Make BVEC- and KVEC-type skip over initial empty vectors.
ver #6)
- Add back the function to indicate the cleanup mode.
- Drop the cleanup_mode return arg to iov_iter_extract_pages().
- Pass FOLL_SOURCE/DEST_BUF in gup_flags. Check this against the iter data_source.
ver #4)
- Use ITER_SOURCE/DEST instead of WRITE/READ.
- Allow additional FOLL_* flags, such as FOLL_PCI_P2PDMA to be passed in.
ver #3)
- Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1].
include/linux/uio.h | 28 +++ lib/iov_iter.c | 424 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 452 insertions(+)
include/linux/uio.h +28 -0
@@ -363,4 +363,32 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction,/* Flags for iov_iter_get/extract_pages*() */ #define ITER_ALLOW_P2PDMA 0x01/* Allow P2PDMA on the extracted pages */ +ssize_t iov_iter_extract_pages (struct iov_iter *i,struct page ***pages, +size_t maxsize,unsigned int maxpages, +unsigned int extract_flags,size_t *offset0); + +/** +* iov_iter_extract_mode - Indicate how pages from the iterator will be retained +* @iter: The iterator +* @extract_flags: How the iterator is to be used +* +* Examine the iterator and @extract_flags and indicate by returning FOLL_PIN, +* FOLL_GET or 0 as to how, if at all, pages extracted from the iterator will +* be retained by the extraction function. +* +* FOLL_GET indicates that the pages will have a reference taken on them that +* the caller must put. This can be done for DMA/async DIO write from a page. +* +* FOLL_PIN indicates that the pages will have a pin placed in them that the +* caller must unpin. This is must be done for DMA/async DIO read to a page to +* avoid CoW problems in fork. +* +* 0 indicates that no measures are taken and that it's up to the caller to +* retain the pages. +*/ +#define iov_iter_extract_mode (iter, extract_flags) \ + (user_backed_iter(iter) ? \ + (iter->data_source == ITER_SOURCE) ? \ + FOLL_GET : FOLL_PIN : 0) +#endif
lib/iov_iter.c +424 -0
427 lines not on this page · raw message
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
include/linux/uio.h static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, · line 392
Quoted 4 lines
include/linux/uio.h +4
-0+#define iov_iter_extract_mode (iter, extract_flags) \ + (user_backed_iter(iter) ? \ + (iter->data_source == ITER_SOURCE) ? \ + FOLL_GET : FOLL_PIN : 0)
The extract_flags argument is unused here.
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
In reply to David HowellsQuoted 5 lines · David Howells
(3) Any other sort of iterator.
No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention.
ITER_ALLOW_P2PDMAis not permitted.
The ITER_ALLOW_P2PDMA check here is fundamentally wrong and will break things like io_uring to fixed buffers on NVMe. ITER_ALLOW_P2PDMA should strictly increase the group of acceptable iters, it does must not restrict anything.
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · David Howells
(3) Any other sort of iterator.
No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention.
ITER_ALLOW_P2PDMAis not permitted.The
ITER_ALLOW_P2PDMAcheck here is fundamentally wrong and will break things likeio_uringto fixed buffers on NVMe.ITER_ALLOW_P2PDMAshould strictly increase the group of acceptable iters, it does must not restrict anything.
So just drop the check? Or do I actually need to do something to the pages to make it work? If I understand the code correctly, it's not actually operable right now on any page that doesn't have an appropriately marked PTE.
David
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 3 lines · David Howells
So just drop the check? Or do I actually need to do something to the pages to make it work? If I understand the code correctly, it's not actually operable right now on any page that doesn't have an appropriately marked PTE.
Yes, just drop the check. The flag just signals that the caller can deal with p2p pages. If it gets any or not is a totally different story.
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
In reply to David HowellsQuoted 106 lines · David Howells
Add a function,
iov_iter_extract_pages(), to extract a list of pages from an iterator. The pages may be returned with a reference added or a pin added or neither, depending on the type of iterator and the direction of transfer. The caller must passFOLL_READ_FROM_MEMorFOLL_WRITE_TO_MEMas part ofgup_flagsto indicate how the iterator contents are to be used.Add a second function,
iov_iter_extract_mode(), to determine how the cleanup should be done.There are three cases:
(1) Transfer *into* an
ITER_IOVECorITER_UBUFiterator.Extracted pages will have pins obtained on them (but not references) so that
fork()doesn't CoW the pages incorrectly whilst the I/O is in progress.
iov_iter_extract_mode()will returnFOLL_PINfor this case. The caller should use something likeunpin_user_page()to dispose of the page.(2) Transfer is *out of* an
ITER_IOVECorITER_UBUFiterator.Extracted pages will have references obtained on them, but not pins.
iov_iter_extract_mode()will returnFOLL_GET. The caller should use something likeput_page()for page disposal.(3) Any other sort of iterator.
No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention.
ITER_ALLOW_P2PDMAis not permitted.
iov_iter_extract_mode()will return 0. The pages don't need additional disposal.Signed-off-by David Howells <[email protected]>cc Al Viro <[email protected]>cc Christoph Hellwig <[email protected]>cc John Hubbard <[email protected]>cc Matthew Wilcox <[email protected]>Link https://lore.kernel.org/r/166920903885.1461876.692029808682876184.stgit@warthog.procyon.org.uk/ # v2 open hereLink https://lore.kernel.org/r/166997421646.9475.14837976344157464997.stgit@warthog.procyon.org.uk/ # v3 open hereLink https://lore.kernel.org/r/167305163883.1521586.10777155475378874823.stgit@warthog.procyon.org.uk/ # v4 open here---
Notes:
ver #7)
- Switch to passing in iter-specific flags rather than FOLL_* flags.
- Drop the direction flags for now.
- UseITER_ALLOW_P2PDMAto requestFOLL_PCI_P2PDMA.
- Disallow use ofITER_ALLOW_P2PDMAwith non-user-backed iter.
- Add support for extraction from KVEC-type iters.
- Useiov_iter_advance()rather than open-coding it.
- Make BVEC- and KVEC-type skip over initial empty vectors.ver #6)
- Add back the function to indicate the cleanup mode.
- Drop thecleanup_modereturn arg toiov_iter_extract_pages().
- Pass FOLL_SOURCE/DEST_BUF ingup_flags. Check this against the iterdata_source.ver #4)
- Use ITER_SOURCE/DEST instead of WRITE/READ.
- Allow additional FOLL_* flags, such asFOLL_PCI_P2PDMAto be passed in.ver #3)
- Switch to usingEXPORT_SYMBOL_GPLto prevent indirect 3rd-party access to get/pin_user_pages_fast()[1].include/linux/uio.h | 28 +++ lib/iov_iter.c | 424 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 452 insertions(+)include/linux/uio.h +19
-0@@ -363,4 +363,32 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction,/* Flags for iov_iter_get/extract_pages*() */ #define ITER_ALLOW_P2PDMA 0x01/* Allow P2PDMA on the extracted pages */ +ssize_t iov_iter_extract_pages (struct iov_iter *i,struct page ***pages, +size_t maxsize,unsigned int maxpages, +unsigned int extract_flags,size_t *offset0); + +/** +* iov_iter_extract_mode - Indicate how pages from the iterator will be retained +* @iter: The iterator +* @extract_flags: How the iterator is to be used +* +* Examine the iterator and @extract_flags and indicate by returning FOLL_PIN, +* FOLL_GET or 0 as to how, if at all, pages extracted from the iterator will +* be retained by the extraction function. +* +* FOLL_GET indicates that the pages will have a reference taken on them that +* the caller must put. This can be done for DMA/async DIO write from a page. +* +* FOLL_PIN indicates that the pages will have a pin placed in them that the +* caller must unpin. This is must be done for DMA/async DIO read to a page to +* avoid CoW problems in fork.
include/linux/uio.h static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, · line 394
Quoted 10 lines
include/linux/uio.h +9
-0+* +* 0 indicates that no measures are taken and that it's up to the caller to +* retain the pages. +*/ +#define iov_iter_extract_mode (iter, extract_flags) \ + (user_backed_iter(iter) ? \ + (iter->data_source == ITER_SOURCE) ? \ + FOLL_GET : FOLL_PIN : 0) +
How does this work align with the goal of no longer using FOLL_GET for O_DIRECT? We should get rid of any FOLL_GET usage for accessing page content.
@John, any comments?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 2 lines · David Hildenbrand
How does this work align with the goal of no longer using
FOLL_GETforO_DIRECT? We should get rid of anyFOLL_GETusage for accessing page content.
Would that run the risk of changes being made by the child being visible to the a DIO write if the parent changes the buffer first?
PARENT CHILD ====== ===== start-DIO-write fork() = pid fork() = 0 alter-buffer CoW happens page copied original page retained alter-buffer <DMA-happens>
David
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 7 lines · David Howells
Quoted 2 lines
How does this work align with the goal of no longer using
FOLL_GETforO_DIRECT? We should get rid of anyFOLL_GETusage for accessing page content.Would that run the risk of changes being made by the child being visible to the a DIO write if the parent changes the buffer first?
Quoted 9 lines
PARENT CHILD ====== ===== start-DIO-write fork() = pid fork() = 0 alter-buffer CoW happens page copied original page retained alter-buffer <DMA-happens>
Ah, I think I might have got the wrong end of the stick. A pinned page is *always* copied on fork() if I understand copy_present_pte() correctly.
David
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
In reply to David HowellsQuoted 9 lines · David Howells
Quoted 2 lines · David Hildenbrand
How does this work align with the goal of no longer using
FOLL_GETforO_DIRECT? We should get rid of anyFOLL_GETusage for accessing page content.Would that run the risk of changes being made by the child being visible to the a DIO write if the parent changes the buffer first?
Quoted 9 lines
PARENT CHILD ====== ===== start-DIO-write fork() = pid fork() = 0 alter-buffer CoW happens page copied original page retained alter-buffer <DMA-happens>
FOLL_PIN users are fine in that regard, because we properly detect "maybe pinned" during fork() and copy the page. See tools/testing/selftests/mm/cow.c (still called tools/testing/selftests/vm/cow.c upstream IIRC) for some test cases for that handling.
FOLL_GET does not work as expected in that regard: pages can't be detected as pinned and we won't be copying them during fork(). We'll end up COW-sharing them, which can result in trouble later.
Switching from FOLL_GET to FOLL_PIN was in the works by John H. Not sure what the status is. Interestingly, Documentation/core-api/pin_user_pages.rst already documents that "CASE 1: Direct IO (DIO)" uses FOLL_PIN ... which does, unfortunately, no reflect reality yet.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 4 lines · David Hildenbrand
Switching from
FOLL_GETtoFOLL_PINwas in the works by John H. Not sure what the status is. Interestingly,Documentation/core-api/pin_user_pages.rstalready documents that "CASE 1: Direct IO (DIO)" usesFOLL_PIN... which does, unfortunately, no reflect reality yet.
Yeah - I just came across that.
Should iov_iter.c then switch entirely to using pin_user_pages(), rather than get_user_pages()? In which case my patches only need keep track of pinned/not-pinned and never "got".
David
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 7 lines · David Howells
Quoted 4 lines · David Hildenbrand
Switching from
FOLL_GETtoFOLL_PINwas in the works by John H. Not sure what the status is. Interestingly,Documentation/core-api/pin_user_pages.rstalready documents that "CASE 1: Direct IO (DIO)" usesFOLL_PIN... which does, unfortunately, no reflect reality yet.
Quoted 5 lines
Yeah - I just came across that.
Should
iov_iter.cthen switch entirely to usingpin_user_pages(), rather thanget_user_pages()? In which case my patches only need keep track of pinned/not-pinned and never "got".
That would be the ideal case: whenever intending to access page content, use FOLL_PIN instead of FOLL_GET.
The issue that John was trying to sort out was that there are plenty of callsites that do a simple put_page() instead of calling unpin_user_page(). IIRC, handling that correctly in existing code -- what was pinned must be released via unpin_user_page() -- was the biggest workitem.
Not sure how that relates to your work here (that's why I was asking): if you could avoid FOLL_GET, that would be great :)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 10 lines · David Hildenbrand
That would be the ideal case: whenever intending to access page content, use
FOLL_PINinstead ofFOLL_GET.The issue that John was trying to sort out was that there are plenty of callsites that do a simple
put_page()instead of callingunpin_user_page(). IIRC, handling that correctly in existing code -- what was pinned must be released viaunpin_user_page()-- was the biggest workitem.Not sure how that relates to your work here (that's why I was asking): if you could avoid
FOLL_GET, that would be great :)
Well, it simplifies things a bit.
I can make the new iov_iter_extract_pages() just do "pin" or "don't pin" and do no ref-getting at all. Things can be converted over to "unpin the pages or doing nothing" as they're converted over to using iov_iter_extract_pages() from iov_iter_get_pages*().
The block bio code then only needs a single bit of state: pinned or not pinned.
For cifs RDMA, do I need to make it pass in FOLL_LONGTERM? And does that need a special cleanup?
sk_buff fragment handling could still be tricky. I'm thinking that in that code I'll need to store FOLL_GET/PIN in the bottom two bits of the frag page pointer. Sometimes it allocates a new page and attaches it (have ref); sometimes it does zerocopy to/from a page (have pin) and sometimes it may be pointing to a kernel buffer (don't pin or ref).
David
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 15 lines · David Howells
Quoted 10 lines · David Hildenbrand
That would be the ideal case: whenever intending to access page content, use
FOLL_PINinstead ofFOLL_GET.The issue that John was trying to sort out was that there are plenty of callsites that do a simple
put_page()instead of callingunpin_user_page(). IIRC, handling that correctly in existing code -- what was pinned must be released viaunpin_user_page()-- was the biggest workitem.Not sure how that relates to your work here (that's why I was asking): if you could avoid
FOLL_GET, that would be great :)Well, it simplifies things a bit.
Quoted 7 lines
I can make the new
iov_iter_extract_pages()just do "pin" or "don't pin" and do no ref-getting at all. Things can be converted over to "unpin the pages or doing nothing" as they're converted over to usingiov_iter_extract_pages()from iov_iter_get_pages*().The block bio code then only needs a single bit of state: pinned or not pinned.
Unfortunately, I'll have to let BIO experts comment on that :) I only know the MM side of things here.
Quoted 3 lines
For cifs RDMA, do I need to make it pass in
FOLL_LONGTERM? And does that need a special cleanup?
Anything that holds pins "possibly forever" should that. vmsplice() is another example that should use it, once properly using FOLL_PIN. [FOLL_GET | FOLL_LONGTERM is not really used/defined with semantics]
Quoted 9 lines
sk_bufffragment handling could still be tricky. I'm thinking that in that code I'll need to store FOLL_GET/PIN in the bottom two bits of the frag page pointer. Sometimes it allocates a new page and attaches it (have ref); sometimes it does zerocopy to/from a page (have pin) and sometimes it may be pointing to a kernel buffer (don't pin or ref).David
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 5 lines · David Hildenbrand
Quoted 2 lines
The block bio code then only needs a single bit of state: pinned or not pinned.
Unfortunately, I'll have to let BIO experts comment on that :) I only know the MM side of things here.
It can work for bios.
Jan Kara
reviewerRe: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
In reply to David HowellsQuoted 15 lines · David Howells
Quoted 10 lines · David Hildenbrand
That would be the ideal case: whenever intending to access page content, use
FOLL_PINinstead ofFOLL_GET.The issue that John was trying to sort out was that there are plenty of callsites that do a simple
put_page()instead of callingunpin_user_page(). IIRC, handling that correctly in existing code -- what was pinned must be released viaunpin_user_page()-- was the biggest workitem.Not sure how that relates to your work here (that's why I was asking): if you could avoid
FOLL_GET, that would be great :)Well, it simplifies things a bit.
Quoted 7 lines
I can make the new
iov_iter_extract_pages()just do "pin" or "don't pin" and do no ref-getting at all. Things can be converted over to "unpin the pages or doing nothing" as they're converted over to usingiov_iter_extract_pages()from iov_iter_get_pages*().The block bio code then only needs a single bit of state: pinned or not pinned.
I'm all for using only pin/unpin in the end. But you'd still need to handle the 'put' for the intermediate time when there are still FOLL_GET users of the bio infrastructure, wouldn't you?
Quoted 2 lines
For cifs RDMA, do I need to make it pass in
FOLL_LONGTERM? And does that need a special cleanup?
FOLL_LONGTERM doesn't need a special cleanup AFAIK. It should be used whenever there isn't reasonably bound time after which the page is unpinned. So in case CIFS sets up RDMA and then it is up to userspace how long the RDMA is going to be running it should be using FOLL_LONGTERM. The thing is that pins can block e.g. truncate for DAX inodes and so longterm pins are not supported for DAX backed pages.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 3 lines · Jan Kara
I'm all for using only pin/unpin in the end. But you'd still need to handle the 'put' for the intermediate time when there are still
FOLL_GETusers of the bio infrastructure, wouldn't you?
Yes, we need it for the transition. But Dave already has a patch to convert the legacy direct-io code as well, at which point we can the retire the get bit.
John Hubbard
reviewerRe: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
In reply to Jan KaraQuoted 7 lines · Jan Kara
Quoted 2 lines
For cifs RDMA, do I need to make it pass in
FOLL_LONGTERM? And does that need a special cleanup?
FOLL_LONGTERMdoesn't need a special cleanup AFAIK. It should be used whenever there isn't reasonably bound time after which the page is unpinned. So in case CIFS sets up RDMA and then it is up to userspace how long the RDMA is going to be running it should be usingFOLL_LONGTERM. The
Yes, we have been pretty consistently deciding that RDMA generally implies FOLL_LONGTERM. (And furthermore, FOLL_LONGTERM implies FOLL_PIN--that one is actually enforced by the gup/pup APIs.)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 4 lines · John Hubbard
Quoted 3 lines · Jan Kara
Quoted 2 lines
For cifs RDMA, do I need to make it pass in
FOLL_LONGTERM? And does that need a special cleanup?
Quoted 8 lines
Quoted 4 lines
FOLL_LONGTERMdoesn't need a special cleanup AFAIK. It should be used whenever there isn't reasonably bound time after which the page is unpinned. So in case CIFS sets up RDMA and then it is up to userspace how long the RDMA is going to be running it should be usingFOLL_LONGTERM. TheYes, we have been pretty consistently deciding that RDMA generally implies
FOLL_LONGTERM. (And furthermore,FOLL_LONGTERMimplies FOLL_PIN--that one is actually enforced by the gup/pup APIs.)
That's weird. For storage or file systems, pages are pinnen just as long when using RDMA as when using local DMA, in fact if you do RDMA to really fast remote media vs slow local media (e.g. SSD vs disk) you might pin it shorter when using RDMA.
I think FOLL_LONGTERM makes sense for non-ODP user space memory registrations for RDMA, which will last basically forever. It does not really make much sense at all for in-kernel memory registration for RDMA that are very short term.
John Hubbard
reviewerRe: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · John Hubbard
Quoted 7 lines · Jan Kara
Quoted 2 lines
For cifs RDMA, do I need to make it pass in
FOLL_LONGTERM? And does that need a special cleanup?
FOLL_LONGTERMdoesn't need a special cleanup AFAIK. It should be used whenever there isn't reasonably bound time after which the page is unpinned. So in case CIFS sets up RDMA and then it is up to userspace how long the RDMA is going to be running it should be usingFOLL_LONGTERM. The
Quoted 8 lines
Quoted 3 lines
Yes, we have been pretty consistently deciding that RDMA generally implies
FOLL_LONGTERM. (And furthermore,FOLL_LONGTERMimplies FOLL_PIN--that one is actually enforced by the gup/pup APIs.)That's weird. For storage or file systems, pages are pinnen just as long when using RDMA as when using local DMA, in fact if you do RDMA to really fast remote media vs slow local media (e.g. SSD vs disk) you might pin it shorter when using RDMA.
ah OK, it is possible that I've been thinking too much about the HPC cases that set up RDMA and leave it there, and not enough about storage.
Quoted 5 lines
I think
FOLL_LONGTERMmakes sense for non-ODP user space memory registrations for RDMA, which will last basically forever. It does not really make much sense at all for in-kernel memory registration for RDMA that are very short term.
Agreed.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
John Hubbard
reviewerRe: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
In reply to David HildenbrandQuoted 8 lines · David Hildenbrand
Quoted 7 lines · David Howells
Quoted 4 lines · David Hildenbrand
Switching from
FOLL_GETtoFOLL_PINwas in the works by John H. Not sure what the status is. Interestingly,Documentation/core-api/pin_user_pages.rstalready documents that "CASE 1: Direct IO (DIO)" usesFOLL_PIN... which does, unfortunately, no reflect reality yet.
Yes, that part of the documentation is...aspirational. :) But this series is taking us there, so good. Let me go review v8 of the series in actual detail to verify but it sounds very promising.
Quoted 6 lines
Quoted 5 lines
Yeah - I just came across that.
Should
iov_iter.cthen switch entirely to usingpin_user_pages(), rather than get_user_pages()? In which case my patches only need keep track of pinned/not-pinned and never "got".
Quoted 6 lines
That would be the ideal case: whenever intending to access page content, use
FOLL_PINinstead ofFOLL_GET.The issue that John was trying to sort out was that there are plenty of callsites that do a simple
put_page()instead of callingunpin_user_page(). IIRC, handling that correctly in existing code -- what was pinned must be released viaunpin_user_page()-- was the biggest workitem.Not sure how that relates to your work here (that's why I was asking): if you could avoid
FOLL_GET, that would be great :)
The largest part of this problem has been: __iov_iter_get_pages_alloc() calls get_user_pages_fast() (so, FOLL_GET), as part of the Direct IO path. And that __iov_iter_get_pages_alloc() is also called by a wide variety of things that are not Direct IO: networking, crytpo, RDS.
So splitting out a variant that only Direct IO uses is a great move and should allow conversion of Direct IO to FOLL_PIN. Again, let me go do an actual review to check on that.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Al Viro
maintainerRe: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
In reply to David HildenbrandQuoted 17 lines · David Hildenbrand
Quoted 7 lines · David Howells
Quoted 4 lines · David Hildenbrand
Switching from
FOLL_GETtoFOLL_PINwas in the works by John H. Not sure what the status is. Interestingly,Documentation/core-api/pin_user_pages.rstalready documents that "CASE 1: Direct IO (DIO)" usesFOLL_PIN... which does, unfortunately, no reflect reality yet.Quoted 5 lines
Yeah - I just came across that.
Should
iov_iter.cthen switch entirely to usingpin_user_pages(), rather thanget_user_pages()? In which case my patches only need keep track of pinned/not-pinned and never "got".That would be the ideal case: whenever intending to access page content, use
FOLL_PINinstead ofFOLL_GET.
Quoted 7 lines
The issue that John was trying to sort out was that there are plenty of callsites that do a simple
put_page()instead of callingunpin_user_page(). IIRC, handling that correctly in existing code -- what was pinned must be released viaunpin_user_page()-- was the biggest workitem.Not sure how that relates to your work here (that's why I was asking): if you could avoid
FOLL_GET, that would be great :)
Take a good look at iter_to_pipe(). It does *not* need to pin anything (we have an ITER_SOURCE there); with this approach it will. And it will stuff those pinned references into a pipe, where they can sit indefinitely.
IOW, I don't believe it's a usable approach.
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 26 lines · Al Viro
Quoted 17 lines · David Hildenbrand
Quoted 7 lines · David Howells
Quoted 4 lines · David Hildenbrand
Switching from
FOLL_GETtoFOLL_PINwas in the works by John H. Not sure what the status is. Interestingly,Documentation/core-api/pin_user_pages.rstalready documents that "CASE 1: Direct IO (DIO)" usesFOLL_PIN... which does, unfortunately, no reflect reality yet.Quoted 5 lines
Yeah - I just came across that.
Should
iov_iter.cthen switch entirely to usingpin_user_pages(), rather thanget_user_pages()? In which case my patches only need keep track of pinned/not-pinned and never "got".That would be the ideal case: whenever intending to access page content, use
FOLL_PINinstead ofFOLL_GET.Quoted 7 lines
The issue that John was trying to sort out was that there are plenty of callsites that do a simple
put_page()instead of callingunpin_user_page(). IIRC, handling that correctly in existing code -- what was pinned must be released viaunpin_user_page()-- was the biggest workitem.Not sure how that relates to your work here (that's why I was asking): if you could avoid
FOLL_GET, that would be great :)
Quoted 7 lines
Take a good look at
iter_to_pipe(). It does *not* need to pin anything (we have anITER_SOURCEthere); with this approach it will. And it will stuff those pinned references into a pipe, where they can sit indefinitely.IOW, I don't believe it's a usable approach.
Not sure what makes you believe that FOLL_GET is any better for this long-term pinning, I'd like to learn about that.
As raised already somewhere in the whole discussion by me, the right way to take such a long-term ping as vmsplice() does is to use FOLL_PIN|FOLL_LONGTERM. As also raised, that will fix the last remaining vmsplice()+hugetlb COW issue as tested by the cow.c vm selftest and make sure to migrate that memory off of MIGRATE_MOVABLE/CMA memory where we cannot tolerate to have long-term unmovable memory sitting around.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 3 lines · David Hildenbrand
As raised already somewhere in the whole discussion by me, the right way to take such a long-term ping as
vmsplice()does is to use FOLL_PIN|FOLL_LONGTERM.
So the pipe infrastructure would have to be able to pin pages instead of carrying refs on them? What about pages just allocated and added to the pipe ring in normal pipe use?
David
Re: [PATCH v7 2/8] iov_iter: Add a function to extract a page list from an iterator
Quoted 9 lines · David Howells
Quoted 3 lines · David Hildenbrand
As raised already somewhere in the whole discussion by me, the right way to take such a long-term ping as
vmsplice()does is to use FOLL_PIN|FOLL_LONGTERM.So the pipe infrastructure would have to be able to pin pages instead of carrying refs on them? What about pages just allocated and added to the pipe ring in normal pipe use?
Ordinary kernel allocations (alloc_page() ...) always have to be freed via put_page() and friends. Such allocations are unmovable as default and don't require any special care.
Pages mapped into user space are movable as default and might be placed on ZONE_MOVABLE/CMA memory (well, and might be swapped out). FOLL_LONGTERM makes sure to migrate these pages off of such problematic physical memory regions, such that we can safely pin them until eternity.
Signature · 3 lines
Thanks, David / dhildenb
PATCH v7 3/8 mm: Provide a helper to drop a pin/ref on a page
In reply to David HowellsProvide a helper in the get_user_pages code to drop a pin or a ref on a page based on being given FOLL_GET or FOLL_PIN in its flags argument or do nothing if neither is set.
--- include/linux/mm.h | 3 +++ mm/gup.c | 22 ++++++++++++++++++++++ 2 files changed, 25 insertions(+)
include/linux/mm.h +3 -0
@@ -1367,6 +1367,9 @@ static inline bool is_cow_mapping(vm_flags_t flags)#define SECTION_IN_PAGE_FLAGS #endif +void folio_put_unpin (struct folio *folio,unsigned int flags); +void page_put_unpin (struct page *page,unsigned int flags); +/* * The identification function is mainly used by the buddy allocator for * determining if two pages could be buddies. We are not really identifying
mm/gup.c +22 -0
@@ -191,6 +191,28 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags)folio_put_refs (folio, refs); } +/** +* folio_put_unpin - Unpin/put a folio as appropriate +* @folio: The folio to release +* @flags: gup flags indicating the mode of release (FOLL_*) +* +* Release a folio according to the flags. If FOLL_GET is set, the folio has a +* ref dropped; if FOLL_PIN is set, it is unpinned; otherwise it is left +* unaltered. +*/ +void folio_put_unpin (struct folio *folio,unsigned int flags) +{ +if (flags & (FOLL_GET |FOLL_PIN )) +gup_put_folio (folio,1 , flags); +} +EXPORT_SYMBOL_GPL (folio_put_unpin); + +void page_put_unpin (struct page *page,unsigned int flags) +{ +folio_put_unpin (page_folio (page), flags); +} +EXPORT_SYMBOL_GPL (page_put_unpin); +/** * try_grab_page() - elevate a page's refcount by a flag-dependent amount * @page: pointer to page to be grabbed
PATCH v7 4/8 block: Rename BIO_NO_PAGE_REF to BIO_PAGE_REFFED and invert the meaning
In reply to David HowellsRename BIO_NO_PAGE_REF to BIO_PAGE_REFFED and invert the meaning. In a following patch I intend to add a BIO_PAGE_PINNED flag to indicate that the page needs unpinning and this way both flags have the same logic.
---
Notes:
ver #5)
- Split from patch that uses iov_iter_extract_pages().
block/bio.c | 9 ++++++++- fs/iomap/direct-io.c | 1 - include/linux/bio.h | 2 +- include/linux/blk_types.h | 2 +- 4 files changed, 10 insertions(+), 4 deletions(-)
block/bio.c +8 -1
@@ -243,6 +243,10 @@ static void bio_free(struct bio *bio)* Users of this function have their own bio allocation. Subsequently, * they must remember to pair any call to bio_init() with bio_uninit() * when IO has completed, or when the bio is released. +* +* We set the initial assumption that pages attached to the bio will be +* released with put_page() by setting BIO_PAGE_REFFED; if the pages +* should not be put, this flag should be cleared. */ void bio_init (struct bio *bio,struct block_device *bdev,struct bio_vec *table,unsigned short max_vecs,blk_opf_t opf)
@@ -274,6 +278,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,#ifdef CONFIG_BLK_DEV_INTEGRITY bio->bi_integrity =NULL ;#endif +bio_set_flag (bio,BIO_PAGE_REFFED ); bio->bi_vcnt =0 ;atomic_set (&bio->__bi_remaining,1 );
@@ -302,6 +307,7 @@ void bio_reset(struct bio *bio, struct block_device *bdev, blk_opf_t opf) {bio_uninit (bio);memset (bio,0 ,BIO_RESET_BYTES ); +bio_set_flag (bio,BIO_PAGE_REFFED );atomic_set (&bio->__bi_remaining,1 ); bio->bi_bdev = bdev;if (bio->bi_bdev)
@@ -812,6 +818,7 @@ EXPORT_SYMBOL(bio_put);static int __bio_clone (struct bio *bio,struct bio *bio_src,gfp_t gfp) {bio_set_flag (bio,BIO_CLONED ); +bio_clear_flag (bio,BIO_PAGE_REFFED ); bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_iter = bio_src->bi_iter;
@@ -1198,7 +1205,7 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; bio->bi_iter.bi_size = size; -bio_set_flag (bio,BIO_NO_PAGE_REF ); +bio_clear_flag (bio,BIO_PAGE_REFFED );bio_set_flag (bio,BIO_CLONED ); }
fs/iomap/direct-io.c +0 -1
@@ -202,7 +202,6 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; -get_page (page);__bio_add_page (bio, page, len,0 );iomap_dio_submit_bio (iter, dio, bio, pos); }
include/linux/bio.h +1 -1
@@ -482,7 +482,7 @@ void zero_fill_bio(struct bio *bio);static inline void bio_release_pages (struct bio *bio,bool mark_dirty) { -if (!bio_flagged (bio,BIO_NO_PAGE_REF )) +if (bio_flagged (bio,BIO_PAGE_REFFED ))__bio_release_pages (bio, mark_dirty); }
include/linux/blk_types.h +1 -1
@@ -318,7 +318,7 @@ struct bio {* bio flags */ enum { -BIO_NO_PAGE_REF ,/* don't put release vec pages */ +BIO_PAGE_REFFED ,/* Pages need refs putting (equivalent to FOLL_GET) */ BIO_CLONED ,/* doesn't own data */ BIO_BOUNCED ,/* bio is a bounce bio */ BIO_QUIET ,/* Make BIO Quiet */
Re: [PATCH v7 4/8] block: Rename BIO_NO_PAGE_REF to BIO_PAGE_REFFED and invert the meaning
Quoted 6 lines · David Howells
Rename
BIO_NO_PAGE_REFtoBIO_PAGE_REFFEDand invert the meaning. In a following patch I intend to add aBIO_PAGE_PINNEDflag to indicate that the page needs unpinning and this way both flags have the same logic.Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>
As pointed out last time, we really should not set the flag by default. When a bio is allocated there are no pages to be released, that only happens when we add dio-like pages to the bio. Instead of explaining why I mean again, I've put together a version of this series that implements this and my other suggestions here:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dio-pin-pages
This also tests fine with xfs and btrfs and nvme passthrough I/O.
Re: [PATCH v7 4/8] block: Rename BIO_NO_PAGE_REF to BIO_PAGE_REFFED and invert the meaning
Quoted 6 lines · Christoph Hellwig
... I've put together a version of this series that implements this and my other suggestions here:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dio-pin-pages
This also tests fine with xfs and btrfs and nvme passthrough I/O.
That looks okay. Do you need to put a Co-developed-by tag on the "block:
invert BIO_NO_PAGE_REF" patch?
Should I take that set of patches and send it on to Linus when the window opens? Or should it go through the block tree?
David
Re: [PATCH v7 4/8] block: Rename BIO_NO_PAGE_REF to BIO_PAGE_REFFED and invert the meaning
Quoted 2 lines · David Howells
That looks okay. Do you need to put a Co-developed-by tag on the "block:
invertBIO_NO_PAGE_REF" patch?
I can do if you want, although there's really not much left of the original.
Quoted 2 lines
Should I take that set of patches and send it on to Linus when the window opens? Or should it go through the block tree?
I think the block tree would be better.
PATCH v7 5/8 block: Add BIO_PAGE_PINNED
In reply to David HowellsBIO_PAGE_PINNED to indicate that the pages in a bio were pinned (FOLL_PIN) and that the pin will need removing.
--- include/linux/blk_types.h | 1 + 1 file changed, 1 insertion(+)
include/linux/blk_types.h +1 -0
@@ -319,6 +319,7 @@ struct bio {*/ enum {BIO_PAGE_REFFED ,/* Pages need refs putting (equivalent to FOLL_GET) */ +BIO_PAGE_PINNED ,/* Pages need unpinning (equivalent to FOLL_PIN) */ BIO_CLONED ,/* doesn't own data */ BIO_BOUNCED ,/* bio is a bounce bio */ BIO_QUIET ,/* Make BIO Quiet */
Re: [PATCH v7 5/8] block: Add BIO_PAGE_PINNED
Quoted 2 lines · David Howells
BIO_PAGE_PINNEDto indicate that the pages in a bio were pinned (FOLL_PIN) and that the pin will need removing.
Just adding the flag without the infrastructure and users is a bit silly. See the link to my branch to see what I think is better split.
PATCH v7 6/8 block: Make bio structs pin pages rather than ref'ing if appropriate
In reply to David HowellsConvert the block layer's bio code to use iov_iter_extract_pages() instead of iov_iter_get_pages(). This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the source iterator.
The pages need to be pinned for DIO-read rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O would otherwise end up only visible to the child process and not the parent).
To implement this:
(1) If the BIO_PAGE_REFFED flag is set, this causes attached pages to be passed to put_page() during cleanup.
(2) A BIO_PAGE_PINNED flag is provided. If set, this causes attached pages to be passed to unpin_user_page() during cleanup.
(3) BIO_PAGE_REFFED is set by default and BIO_PAGE_PINNED is cleared by default when the bio is (re-)initialised.
(4) If iov_iter_extract_pages() indicates FOLL_GET, this causes BIO_PAGE_REFFED to be set and if FOLL_PIN is indicated, this causes BIO_PAGE_PINNED to be set. If it returns neither FOLL_* flag, then both BIO_PAGE_* flags will be cleared.
Mixing sets of pages with different clean up modes is not supported.
(5) Cloned bio structs have both flags cleared.
(6) bio_release_pages() will do the release if either BIO_PAGE_* flag is set.
[!] Note that this is tested a bit with ext4, but nothing else.
---
Notes:
ver #7)
- Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN.
ver #5)
- Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode.
- Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch.
block/bio.c | 34 +++++++++++++++++++--------------- block/blk-map.c | 22 +++++++++++----------- block/blk.h | 29 +++++++++++++++++++++++++++++ include/linux/bio.h | 3 ++- 4 files changed, 61 insertions(+), 27 deletions(-)
block/bio.c +19 -15
@@ -245,8 +245,9 @@ static void bio_free(struct bio *bio)* when IO has completed, or when the bio is released. * * We set the initial assumption that pages attached to the bio will be -* released with put_page() by setting BIO_PAGE_REFFED; if the pages -* should not be put, this flag should be cleared. +* released with put_page() by setting BIO_PAGE_REFFED, but this should be set +* to BIO_PAGE_PINNED if the page should be unpinned instead; if the pages +* should not be put or unpinned, these flags should be cleared. */ void bio_init (struct bio *bio,struct block_device *bdev,struct bio_vec *table,unsigned short max_vecs,blk_opf_t opf)
@@ -819,6 +820,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) {bio_set_flag (bio,BIO_CLONED );bio_clear_flag (bio,BIO_PAGE_REFFED ); +bio_clear_flag (bio,BIO_PAGE_PINNED ); bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_iter = bio_src->bi_iter;
@@ -1183,7 +1185,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty)bio_for_each_segment_all (bvec, bio, iter_all) {if (mark_dirty && !PageCompound (bvec->bv_page))set_page_dirty_lock (bvec->bv_page); -put_page (bvec->bv_page); +bio_release_page (bio, bvec->bv_page); } }EXPORT_SYMBOL_GPL (__bio_release_pages);
@@ -1220,7 +1222,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, }if (same_page) -put_page (page); +bio_release_page (bio, page);return 0 ; }
@@ -1234,7 +1236,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len)return -EINVAL ;if (same_page) -put_page (page); +bio_release_page (bio, page);return 0 ; }
@@ -1245,10 +1247,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page,* @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * -* Pins pages from *iter and appends them to @bio's bvec array. The -* pages will have to be released using put_page() when done. -* For multi-segment *iter, this function only adds pages from the -* next non-empty segment of the iov iterator. +* Extracts pages from *iter and appends them to @bio's bvec array. The pages +* will have to be cleaned up in the way indicated by the BIO_PAGE_REFFED and +* BIO_PAGE_PINNED flags. For a multi-segment *iter, this function only adds +* pages from the next non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter) {
@@ -1280,9 +1282,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size =iov_iter_get_pages (iter, pages, -UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, extract_flags); + size =iov_iter_extract_pages (iter, &pages, +UINT_MAX - bio->bi_iter.bi_size, + nr_pages, extract_flags, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;
@@ -1315,7 +1317,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)iov_iter_revert (iter, left); out:while (i < nr_pages) -put_page (pages[i++]); +bio_release_page (bio, pages[i++]);return ret; }
@@ -1344,6 +1346,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) {int ret =0 ; +bio_set_cleanup_mode (bio, iter,0 ); +if (iov_iter_is_bvec (iter)) {bio_iov_bvec_set (bio, iter);iov_iter_advance (iter, bio->bi_iter.bi_size);
@@ -1496,8 +1500,8 @@ void bio_set_pages_dirty(struct bio *bio)* the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from -* here on. It will run one put_page() against each page and will run one -* bio_put() against the BIO. +* here on. It will run one put_page() or unpin_user_page() against each page +* and will run one bio_put() against the BIO. */ static void bio_dirty_fn (struct work_struct *work);
block/blk-map.c +11 -11
@@ -282,20 +282,20 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (blk_queue_pci_p2pdma (rq->q)) extract_flags |=ITER_ALLOW_P2PDMA ; +bio_set_cleanup_mode (bio, iter, extract_flags); +while (iov_iter_count (iter)) { -struct page **pages, *stack_pages[UIO_FASTIOV ]; +struct page *stack_pages[UIO_FASTIOV ]; +struct page **pages = stack_pages;ssize_t bytes;size_t offs;int npages; -if (nr_vecs <=ARRAY_SIZE (stack_pages)) { - pages = stack_pages; - bytes =iov_iter_get_pages (iter, pages,LONG_MAX , - nr_vecs, &offs, extract_flags); - }else { - bytes =iov_iter_get_pages_alloc (iter, &pages, -LONG_MAX , &offs, extract_flags); - } +if (nr_vecs >ARRAY_SIZE (stack_pages)) + pages =NULL ; + + bytes =iov_iter_extract_pages (iter, &pages,LONG_MAX , + nr_vecs, extract_flags, &offs);if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;
@@ -317,7 +317,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (!bio_add_hw_page (rq->q, bio, page, n, offs, max_sectors, &same_page)) {if (same_page) -put_page (page); +bio_release_page (bio, page);break ; }
@@ -329,7 +329,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,* release the pages we didn't map into the bio, if any */ while (j < npages) -put_page (pages[j++]); +bio_release_page (bio, pages[j++]);if (pages != stack_pages)kvfree (pages);/* couldn't stuff something into bio? */
block/blk.h +29 -0
@@ -425,6 +425,35 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* +* Set the cleanup mode for a bio from an iterator and the extraction flags. +*/ +static inline void bio_set_cleanup_mode (struct bio *bio,struct iov_iter *iter, +unsigned int extract_flags) +{ +unsigned int cleanup_mode; + +bio_clear_flag (bio,BIO_PAGE_REFFED ); + cleanup_mode =iov_iter_extract_mode (iter, extract_flags); +if (cleanup_mode &FOLL_GET ) +bio_set_flag (bio,BIO_PAGE_REFFED ); +if (cleanup_mode &FOLL_PIN ) +bio_set_flag (bio,BIO_PAGE_PINNED ); +} + +/* +* Clean up a page appropriately, where the page may be pinned, may have a +* ref taken on it or neither. +*/ +static inline void bio_release_page (struct bio *bio,struct page *page) +{ +unsigned int gup_flags =0 ; + + gup_flags |=bio_flagged (bio,BIO_PAGE_REFFED ) ?FOLL_GET :0 ; + gup_flags |=bio_flagged (bio,BIO_PAGE_PINNED ) ?FOLL_PIN :0 ; +page_put_unpin (page, gup_flags); +} +struct request_queue *blk_alloc_queue (int node_id);int disk_scan_partitions (struct gendisk *disk,fmode_t mode,void *owner);
include/linux/bio.h +2 -1
@@ -482,7 +482,8 @@ void zero_fill_bio(struct bio *bio);static inline void bio_release_pages (struct bio *bio,bool mark_dirty) { -if (bio_flagged (bio,BIO_PAGE_REFFED )) +if (bio_flagged (bio,BIO_PAGE_REFFED ) || +bio_flagged (bio,BIO_PAGE_PINNED ))__bio_release_pages (bio, mark_dirty); }
Re: [PATCH v7 6/8] block: Make bio structs pin pages rather than ref'ing if appropriate
This really does three things:
1) add the BIO_PAGE_PINNED infrastructure
2) convert bio_iov_iter_get_pages to use iov_iter_extract_pages
3) convert bio_map_user_iov to use iov_iter_extract_pages
so it should be three pages, with the first one also containing what is in your previous patch.
block/bio.c void __bio_release_pages(struct bio *bio, bool mark_dirty) · line 1188
Quoted 6 lines
block/bio.c +1
-1@@ -1183,7 +1185,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty)bio_for_each_segment_all (bvec, bio, iter_all) {if (mark_dirty && !PageCompound (bvec->bv_page))set_page_dirty_lock (bvec->bv_page); -put_page (bvec->bv_page); +bio_release_page (bio, bvec->bv_page);
This can be micro-optimized a bit by only doing the flags conversion once.
Re: [PATCH v7 6/8] block: Make bio structs pin pages rather than ref'ing if appropriate
In this function:
void __bio_release_pages(struct bio *bio, bool mark_dirty)
{
unsigned int gup_flags = bio_to_gup_flags(bio);
struct bvec_iter_all iter_all;
struct bio_vec *bvec;
bio_for_each_segment_all(bvec, bio, iter_all) {
if (mark_dirty && !PageCompound(bvec->bv_page))
set_page_dirty_lock(bvec->bv_page);
>>>> page_put_unpin(bvec->bv_page, gup_flags);
}
}that ought to be a call to bio_release_page(), but the optimiser doesn't want to inline it:-/
I found the only way I can get the compiler to properly inline it without it repeating the calculations is to renumber the FOLL_* constants down and then make bio_release_page() something like:
static inline __attribute__((always_inline))
void bio_release_page(struct bio *bio, struct page *page)
{
page_put_unpin(page,
((bio->bi_flags & (1 << BIO_PAGE_REFFED)) ? FOLL_GET : 0) |
((bio->bi_flags & (1 << BIO_PAGE_PINNED)) ? FOLL_PIN : 0));
}I guess the compiler optimiser isn't perfect yet:-)
David
Re: [PATCH v7 6/8] block: Make bio structs pin pages rather than ref'ing if appropriate
Quoted 13 lines · David Howells
void __bio_release_pages(struct bio *bio, bool mark_dirty)
{
unsigned int gup_flags = bio_to_gup_flags(bio);
struct bvec_iter_all iter_all;
struct bio_vec *bvec;
bio_for_each_segment_all(bvec, bio, iter_all) {
if (mark_dirty && !PageCompound(bvec->bv_page))
set_page_dirty_lock(bvec->bv_page);
>>>> page_put_unpin(bvec->bv_page, gup_flags);
}
}Quoted 2 lines
that ought to be a call to
bio_release_page(), but the optimiser doesn't want to inline it:-/
Why? __bio_release_pages is the fast path, no need to force using bio_relese_page which is otherwise only used for error cleanup.
PATCH v7 7/8 block: Fix bio_flagged() so that gcc can better optimise it
In reply to David HowellsFix bio_flagged() so that multiple instances of it, such as:
if (bio_flagged(bio, BIO_PAGE_REFFED) || bio_flagged(bio, BIO_PAGE_PINNED))
can be combined by the gcc optimiser into a single test in assembly (arguably, this is a compiler optimisation issue[1]).
The missed optimisation stems from bio_flagged() comparing the result of the bitwise-AND to zero. This results in an out-of-line bio_release_page() being compiled to something like:
<+0>: mov 0x14(%rdi),%eax <+3>: test $0x1,%al <+5>: jne 0xffffffff816dac53 <bio_release_pages+11> <+7>: test $0x2,%al <+9>: je 0xffffffff816dac5c <bio_release_pages+20> <+11>: movzbl %sil,%esi <+15>: jmp 0xffffffff816daba1 <__bio_release_pages> <+20>: jmp 0xffffffff81d0b800 <__x86_return_thunk>
However, the test is superfluous as the return type is bool. Removing it results in:
<+0>: testb $0x3,0x14(%rdi) <+4>: je 0xffffffff816e4af4 <bio_release_pages+15> <+6>: movzbl %sil,%esi <+10>: jmp 0xffffffff816dab7c <__bio_release_pages> <+15>: jmp 0xffffffff81d0b7c0 <__x86_return_thunk>
instead.
Also, the MOVZBL instruction looks unnecessary[2] - I think it's just 're-booling' the mark_dirty parameter.
--- include/linux/bio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/bio.h +1 -1
@@ -227,7 +227,7 @@ static inline void bio_cnt_set(struct bio *bio, unsigned int count)static inline bool bio_flagged (struct bio *bio,unsigned int bit) { -return (bio->bi_flags & (1U << bit)) !=0 ; +return bio->bi_flags & (1U << bit); }static inline void bio_set_flag (struct bio *bio,unsigned int bit)
PATCH v7 8/8 mm: Renumber FOLL_GET and FOLL_PIN down
In reply to David HowellsRenumber FOLL_GET and FOLL_PIN down to bit 0 and 1 respectively so that they are coincidentally the same as BIO_PAGE_REFFED and BIO_PAGE_PINNED and also so that they can be stored in the bottom two bits of a page pointer (something I'm looking at for zerocopy socket fragments).
--- include/linux/mm.h | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
include/linux/mm.h +7 -7
@@ -3074,12 +3074,13 @@ static inline vm_fault_t vmf_error(int err)struct page *follow_page (struct vm_area_struct *vma,unsigned long address,unsigned int foll_flags); -#define FOLL_WRITE 0x01/* check pte is writable */ -#define FOLL_TOUCH 0x02/* mark page accessed */ -#define FOLL_GET 0x04/* do get_page on page */ -#define FOLL_DUMP 0x08/* give error on hole if it would be zero */ -#define FOLL_FORCE 0x10/* get_user_pages read/write w/o permission */ -#define FOLL_NOWAIT 0x20/* if a disk transfer is needed, start the IO +#define FOLL_GET 0x01/* do get_page on page (equivalent to BIO_FOLL_GET) */ +#define FOLL_PIN 0x02/* pages must be released via unpin_user_page */ +#define FOLL_WRITE 0x04/* check pte is writable */ +#define FOLL_TOUCH 0x08/* mark page accessed */ +#define FOLL_DUMP 0x10/* give error on hole if it would be zero */ +#define FOLL_FORCE 0x20/* get_user_pages read/write w/o permission */ +#define FOLL_NOWAIT 0x40/* if a disk transfer is needed, start the IO * and return without waiting upon it */ #define FOLL_NOFAULT 0x80/* do not fault in pages */ #define FOLL_HWPOISON 0x100/* check page is hwpoisoned */
@@ -3088,7 +3089,6 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_ANON 0x8000/* don't do file mappings */ #define FOLL_LONGTERM 0x10000/* mapping lifetime is indefinite: see below */ #define FOLL_SPLIT_PMD 0x20000/* split huge pmd before returning */ -#define FOLL_PIN 0x40000/* pages must be released via unpin_user_page */ #define FOLL_FAST_ONLY 0x80000/* gup_fast: prevent fall-back to slow gup */ #define FOLL_PCI_P2PDMA 0x100000/* allow returning PCI P2PDMA pages */ #define FOLL_INTERRUPTIBLE 0x200000/* allow interrupts from generic signals */
Re: [PATCH v7 8/8] mm: Renumber FOLL_GET and FOLL_PIN down
Quoted 6 lines · David Howells
Renumber
FOLL_GETandFOLL_PINdown to bit 0 and 1 respectively so that they are coincidentally the same asBIO_PAGE_REFFEDandBIO_PAGE_PINNEDand also so that they can be stored in the bottom two bits of a page pointer (something I'm looking at for zerocopy socket fragments).Signed-off-by David Howells <[email protected]>
Although I was hoping you'd renumber some of the others since we currently have gaps at 0x200, 0x400, 0x1000, and 0x4000 as well as the 0x40 you're using here.
Re: [PATCH v7 8/8] mm: Renumber FOLL_GET and FOLL_PIN down
Quoted 3 lines · Matthew Wilcox
Although I was hoping you'd renumber some of the others since we currently have gaps at
0x200,0x400,0x1000, and0x4000as well as the0x40you're using here.
Since the other patches don't require this one, I can renumber them all and send it directly to Andrew.
David
Quoted 4 lines · David Howells
(3) Make the bio struct carry a pair of flags to indicate the cleanup mode.
BIO_NO_PAGE_REFis replaced withBIO_PAGE_REFFED(equivalent toFOLL_GET) andBIO_PAGE_PINNED(equivalent toBIO_PAGE_PINNED) is added.
I think there's a simpler solution than all of this.
As I understand the fundamental problem here, the question is when to copy a page on fork. We have the optimisation of COW, but O_DIRECT/RDMA/... breaks it. So all this page pinning is to indicate to the fork code "You can't do COW to this page".
Why do we want to track that information on a per-page basis? Wouldn't it be easier to have a VM_NOCOW flag in vma->vm_flags? Set it the first time somebody does an O_DIRECT read or RDMA pin. That's it. Pages in that VMA will now never be COWed, regardless of their refcount/mapcount. And the whole "did we pin or get this page" problem goes away. Along with folio->pincount.
Quoted 6 lines · Matthew Wilcox
Why do we want to track that information on a per-page basis? Wouldn't it be easier to have a
VM_NOCOWflag invma->vm_flags? Set it the first time somebody does anO_DIRECTread or RDMA pin. That's it. Pages in that VMA will now never be COWed, regardless of their refcount/mapcount. And the whole "did we pin or get this page" problem goes away. Along withfolio->pincount.
Wouldn't that potentially make someone's entire malloc() heap entirely NOCOW if they did a single DIO to/from it.
Also you only mention DIO read - but what about "start DIO write; fork(); touch buffer" in the parent - now the write buffer belongs to the child and they can affect the parent's write.
David
Quoted 11 lines · David Howells
Quoted 6 lines · Matthew Wilcox
Why do we want to track that information on a per-page basis? Wouldn't it be easier to have a
VM_NOCOWflag invma->vm_flags? Set it the first time somebody does anO_DIRECTread or RDMA pin. That's it. Pages in that VMA will now never be COWed, regardless of their refcount/mapcount. And the whole "did we pin or get this page" problem goes away. Along withfolio->pincount.Wouldn't that potentially make someone's entire
malloc()heap entirely NOCOW if they did a single DIO to/from it.
Yes. Would that be an actual problem for any real application?
We could do this with a vm_pincount if it's essential to be able to count how often it's happened and be able to fork() without COW if it's something that happened in the past and is now not happening.
Quoted 3 lines
Also you only mention DIO read - but what about "start DIO write;
fork(); touch buffer" in the parent - now the write buffer belongs to the child and they can affect the parent's write.
I'm struggling to see the problem here. If the child hasn't exec'd, the parent and child are still in the same security domain. The parent could have modified the buffer before calling fork().
Quoted 4 lines · Matthew Wilcox
Quoted 2 lines
Wouldn't that potentially make someone's entire
malloc()heap entirely NOCOW if they did a single DIO to/from it.Yes. Would that be an actual problem for any real application?
Without auditing all applications that do direct I/O writes, it's hard to say - but a big database engine, Oracle for example, forking off a process, say, could cause a massive slow down as fork suddenly has to copy a huge amount of malloc'd data unnecessarily[*].
[*] I'm making wild assumptions about how Oracle's DB engine works.
Quoted 7 lines
Quoted 3 lines
Also you only mention DIO read - but what about "start DIO write;
fork(); touch buffer" in the parent - now the write buffer belongs to the child and they can affect the parent's write.I'm struggling to see the problem here. If the child hasn't exec'd, the parent and child are still in the same security domain. The parent could have modified the buffer before calling
fork().
It could still inadvertently change the data its parent set to write out. The child *shouldn't* be able to change the parent's in-progress write. The most obvious problem would be in something that does DIO from a stack buffer, I think.
David
Quoted 5 lines · David Howells
Quoted 3 lines · Matthew Wilcox
Quoted 2 lines
Wouldn't that potentially make someone's entire
malloc()heap entirely NOCOW if they did a single DIO to/from it.
Quoted 8 lines
Quoted 1 line
Yes. Would that be an actual problem for any real application?
Without auditing all applications that do direct I/O writes, it's hard to say - but a big database engine, Oracle for example, forking off a process, say, could cause a massive slow down as fork suddenly has to copy a huge amount of malloc'd data unnecessarily[*].
[*] I'm making wild assumptions about how Oracle's DB engine works.
Yes. The cache is shared between all Oracle processes, so it's not COWed. Indeed (as the mshare patches show), what Oracle wants is _more_ sharing between the processes, not _less_.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Also you only mention DIO read - but what about "start DIO write;
fork(); touch buffer" in the parent - now the write buffer belongs to the child and they can affect the parent's write.
Quoted 8 lines
Quoted 3 lines
I'm struggling to see the problem here. If the child hasn't exec'd, the parent and child are still in the same security domain. The parent could have modified the buffer before calling
fork().It could still inadvertently change the data its parent set to write out. The child *shouldn't* be able to change the parent's in-progress write. The most obvious problem would be in something that does DIO from a stack buffer, I think.
If it's a problem then O_DIRECT writes can also set the NOCOW flag.
Jan Kara
reviewerQuoted 9 lines · Matthew Wilcox
Quoted 4 lines · David Howells
Matthew Wilcox <[email protected]> wrote:
Also you only mention DIO read - but what about "start DIO write;fork(); touch buffer" in the parent - now the write buffer belongs to the child and they can affect the parent's write.I'm struggling to see the problem here. If the child hasn't exec'd, the parent and child are still in the same security domain. The parent could have modified the buffer before calling
fork().
Sadly they are not. Android in particular starts applications by forking one big binary (zygote) that has multiple apps linked together and relies on the fact the child cannot influence the parent after the fork. We've already had CVEs with GUP & COW & fork due to this. David Hildebrand has a lot of memories regarding this I believe ;)
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 11 lines · Jan Kara
Quoted 9 lines · Matthew Wilcox
Quoted 4 lines · David Howells
Matthew Wilcox <[email protected]> wrote:
Also you only mention DIO read - but what about "start DIO write;fork(); touch buffer" in the parent - now the write buffer belongs to the child and they can affect the parent's write.I'm struggling to see the problem here. If the child hasn't exec'd, the parent and child are still in the same security domain. The parent could have modified the buffer before calling
fork().
Quoted 5 lines
Sadly they are not. Android in particular starts applications by forking one big binary (zygote) that has multiple apps linked together and relies on the fact the child cannot influence the parent after the fork. We've already had CVEs with GUP & COW & fork due to this. David Hildebrand has a lot of memories regarding this I believe ;)
:)
Once FOLL_PIN is used most of the issues go away and we don't have to play any games with VM flags or similar ...
With FOLL_PIN, I consider anon a solved problem and not worth any new fancy ideas.
Signature · 3 lines
Thanks, David / dhildenb
Jan Kara
reviewerQuoted 13 lines · Matthew Wilcox
Quoted 4 lines · David Howells
(3) Make the bio struct carry a pair of flags to indicate the cleanup mode.
BIO_NO_PAGE_REFis replaced withBIO_PAGE_REFFED(equivalent toFOLL_GET) andBIO_PAGE_PINNED(equivalent toBIO_PAGE_PINNED) is added.I think there's a simpler solution than all of this.
As I understand the fundamental problem here, the question is when to copy a page on fork. We have the optimisation of COW, but O_DIRECT/RDMA/... breaks it. So all this page pinning is to indicate to the fork code "You can't do COW to this page".
Quoted 6 lines
Why do we want to track that information on a per-page basis? Wouldn't it be easier to have a
VM_NOCOWflag invma->vm_flags? Set it the first time somebody does anO_DIRECTread or RDMA pin. That's it. Pages in that VMA will now never be COWed, regardless of their refcount/mapcount. And the whole "did we pin or get this page" problem goes away. Along withfolio->pincount.
Well, but anon COW code is not the only (planned) consumer of the pincount. Filesystems also need to know whether a (shared pagecache) page is pinned and can thus be modified behind their backs. And for that VMA tracking isn't really an option.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 21 lines · Jan Kara
Quoted 13 lines · Matthew Wilcox
Quoted 4 lines · David Howells
(3) Make the bio struct carry a pair of flags to indicate the cleanup mode.
BIO_NO_PAGE_REFis replaced withBIO_PAGE_REFFED(equivalent toFOLL_GET) andBIO_PAGE_PINNED(equivalent toBIO_PAGE_PINNED) is added.I think there's a simpler solution than all of this.
As I understand the fundamental problem here, the question is when to copy a page on fork. We have the optimisation of COW, but O_DIRECT/RDMA/... breaks it. So all this page pinning is to indicate to the fork code "You can't do COW to this page".
Quoted 6 lines
Why do we want to track that information on a per-page basis? Wouldn't it be easier to have a
VM_NOCOWflag invma->vm_flags? Set it the first time somebody does anO_DIRECTread or RDMA pin. That's it. Pages in that VMA will now never be COWed, regardless of their refcount/mapcount. And the whole "did we pin or get this page" problem goes away. Along withfolio->pincount.
Quoted 4 lines
Well, but anon COW code is not the only (planned) consumer of the pincount. Filesystems also need to know whether a (shared pagecache) page is pinned and can thus be modified behind their backs. And for that VMA tracking isn't really an option.
Bleh, I'd forgotten about that problem. We really do need to keep track of which pages are under I/O for this case, because we need to tell the filesystem that they are now available for writeback.
That said, I don't know that we need to keep track of it in the pages themselves. Can't we have something similar to rmap which keeps track of a range of pinned pages, and have it taken care of at a higher level (ie unpin the pages in the dio_iodone_t rather than in the BIO completion handler)?
I'm not even sure why pinned pagecache pages remain on the LRU. They should probably go onto the unevictable list with the mlocked pages, then on unpin get marked dirty and placed on the active list. There's no point in writing back a pinned page since it can be written to at any instant without any part of the kernel knowing.
John Hubbard
reviewerOn 1/23/23 09:33, Matthew Wilcox wrote:
...
Quoted 4 lines
Bleh, I'd forgotten about that problem. We really do need to keep track of which pages are under I/O for this case, because we need to tell the filesystem that they are now available for writeback.
Quoted 8 lines
That said, I don't know that we need to keep track of it in the pages themselves. Can't we have something similar to rmap which keeps track of a range of pinned pages, and have it taken care of at a higher level (ie unpin the pages in the
dio_iodone_trather than in the BIO completion handler)?I'm not even sure why pinned pagecache pages remain on the LRU. They should probably go onto the unevictable list with the mlocked
This is an intriguing idea, but...
Quoted 4 lines
pages, then on unpin get marked dirty and placed on the active list. There's no point in writing back a pinned page since it can be written to at any instant without any part of the kernel knowing.
There have been filesystems discussions about this: if a page goes unwritten for "too long", it's not good. To address that, bounce buffers were proposed for periodic writeback of pinned pages. The idea with bounce buffers is: even though the page is never guaranteed up to date (because DMA can instantly make it effectively dirty), it is at least much less out of date after a periodic writeback, then it was before.
And that is important for some file system use cases.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Jan Kara
reviewerQuoted 31 lines · Matthew Wilcox
Quoted 21 lines · Jan Kara
Quoted 13 lines · Matthew Wilcox
Quoted 4 lines · David Howells
(3) Make the bio struct carry a pair of flags to indicate the cleanup mode.
BIO_NO_PAGE_REFis replaced withBIO_PAGE_REFFED(equivalent toFOLL_GET) andBIO_PAGE_PINNED(equivalent toBIO_PAGE_PINNED) is added.I think there's a simpler solution than all of this.
As I understand the fundamental problem here, the question is when to copy a page on fork. We have the optimisation of COW, but O_DIRECT/RDMA/... breaks it. So all this page pinning is to indicate to the fork code "You can't do COW to this page".
Quoted 6 lines
Why do we want to track that information on a per-page basis? Wouldn't it be easier to have a
VM_NOCOWflag invma->vm_flags? Set it the first time somebody does anO_DIRECTread or RDMA pin. That's it. Pages in that VMA will now never be COWed, regardless of their refcount/mapcount. And the whole "did we pin or get this page" problem goes away. Along withfolio->pincount.Quoted 4 lines
Well, but anon COW code is not the only (planned) consumer of the pincount. Filesystems also need to know whether a (shared pagecache) page is pinned and can thus be modified behind their backs. And for that VMA tracking isn't really an option.
Bleh, I'd forgotten about that problem. We really do need to keep track of which pages are under I/O for this case, because we need to tell the filesystem that they are now available for writeback.
Quoted 5 lines
That said, I don't know that we need to keep track of it in the pages themselves. Can't we have something similar to rmap which keeps track of a range of pinned pages, and have it taken care of at a higher level (ie unpin the pages in the
dio_iodone_trather than in the BIO completion handler)?
We could but bear in mind that there are more places than just (the two) direct IO paths. There are many GUP users that can be operating on mmapped pagecache and that can modify page data. Also note that e.g. direct IO is rather performance sensitive so any CPU overhead that is added to that path gets noticed. That was actually the reason why we keep pinned pages on the LRU - John tried to remove them while pinning but the overhead was not acceptable.
Finally, since we need to handle pinning for anon pages as well, just (ab)using the page refcount looks like the least painful solution.
Quoted 5 lines
I'm not even sure why pinned pagecache pages remain on the LRU. They should probably go onto the unevictable list with the mlocked pages, then on unpin get marked dirty and placed on the active list. There's no point in writing back a pinned page since it can be written to at any instant without any part of the kernel knowing.
True but as John said sometimes we need to writeout even pinned page - e.g. on fsync(2). For some RDMA users which keep pages pinned for days or months, this is actually crutial...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 3 lines · Jan Kara
True but as John said sometimes we need to writeout even pinned page - e.g. on fsync(2). For some RDMA users which keep pages pinned for days or months, this is actually crutial...
I think we have to distinguish between short term (just FOLL_PIN) and long term (FOLL_PIN | FOLL_LONGERM) pins.
For short term ones the proper thing to do in data integrity writeback is to simply wait for the unpin. For the long term pins that obviously can't work. The right answer for that is complicated and I don't have a good answer yet. The best one would probably to probibit them on MAP_SHARED file backed mappings - this might break some existing software, but that software is already so broken that this might be best.