iov_iter: Improve page extraction (pin or just list)
Subsystems (6)
v10 cover iov_iter: Improve page extraction (pin or just list)
Hi Al, Christoph,
Here are patches to provide support for extracting pages from an iov_iter and to use this in the extraction functions in the block layer bio code.
The patches make the following changes:
(1) Add a function, iov_iter_extract_pages() to replace iov_iter_get_pages*() that gets refs, pins or just lists the pages as appropriate to the iterator type.
Add a function, iov_iter_extract_will_pin() that will indicate from the iterator type how the cleanup is to be performed, returning true if the pages will need unpinning, false otherwise.
(2) Make the bio struct carry a pair of flags to indicate the cleanup mode. BIO_NO_PAGE_REF is replaced with BIO_PAGE_REFFED (indicating FOLL_GET was used) and BIO_PAGE_PINNED (indicating FOLL_PIN was used) is added.
BIO_PAGE_REFFED will go away, but at the moment fs/direct-io.c sets it and this series does not fully address that file.
(4) Add a function, bio_release_page(), to release a page appropriately to the cleanup mode indicated by the BIO_PAGE_* flags.
(5) Make the iter-to-bio code use iov_iter_extract_pages() to retain the pages appropriately and clean them up later.
(6) Fix bio_flagged() so that it doesn't prevent a gcc optimisation.
I've pushed the patches here also:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=iov-extract
David
Changes:
========
ver #10)
- Fix use of i->kvec in iov_iter_extract_bvec_pages() to be i->bvec.
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #9)
- It's now not permitted to use FOLL_PIN outside of mm/, so:
- Change iov_iter_extract_mode() into iov_iter_extract_will_pin() and return true/false instead of FOLL_PIN/0.
- Drop of folio_put_unpin() and page_put_unpin() and instead call unpin_user_page() (and put_page()) directly as necessary.
- Make __bio_release_pages() call bio_release_page() instead of unpin_user_page() as there's no BIO_* -> FOLL_* translation to do.
- Drop the FOLL_* renumbering patch.
- Change extract_flags to extraction_flags.
ver #8)
- Import Christoph Hellwig's changes.
- Split the conversion-to-extraction patch.
- Drop the extract_flags arg from iov_iter_extract_mode().
- Don't default bios to BIO_PAGE_REFFED, but set explicitly.
- Switch FOLL_PIN and FOLL_GET when renumbering so PIN is at bit 0.
- Switch BIO_PAGE_PINNED and BIO_PAGE_REFFED so PINNED is at bit 0.
- We should always be using FOLL_PIN (not FOLL_GET) for DIO, so adjust the patches for that.
ver #7)
- For now, drop the parts to pass the I/O direction to iov_iter_*pages*() as it turned out to be a lot more complicated, with places not setting IOCB_WRITE when they should, for example.
- Drop all the patches that changed things other then the block layer's bio handling. The netfslib and cifs changes can go into a separate patchset.
- Add support for extracting pages from KVEC-type iterators.
- When extracting from BVEC/KVEC, skip over empty vecs at the front.
ver #6)
- Fix write() syscall and co. not setting IOCB_WRITE.
- Added iocb_is_read() and iocb_is_write() to check IOCB_WRITE.
- Use op_is_write() in bio_copy_user_iov().
- Drop the iterator direction checks from smbd_recv().
- Define FOLL_SOURCE_BUF and FOLL_DEST_BUF and pass them in as part of gup_flags to iov_iter_get/extract_pages*().
- Replace iov_iter_get_pages*2() with iov_iter_get_pages*() and remove.
- Add back the function to indicate the cleanup mode.
- Drop the cleanup_mode return arg to iov_iter_extract_pages().
- Provide a helper to clean up a page.
- Renumbered FOLL_GET and FOLL_PIN and made BIO_PAGE_REFFED/PINNED have the same numerical values, enforced with an assertion.
- Converted AF_ALG, SCSI vhost, generic DIO, FUSE, splice to pipe, 9P and NFS.
- Added in the patches to make CIFS do top-to-bottom iterators and use various of the added extraction functions.
- Added a pair of work-in-progess patches to make sk_buff fragments store FOLL_GET and FOLL_PIN.
ver #5)
- Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED and split into own patch.
- Transcribe FOLL_GET/PIN into BIO_PAGE_REFFED/PINNED flags.
- Add patch to allow bio_flagged() to be combined by gcc.
ver #4)
- Drop the patch to move the FOLL_* flags to linux/mm_types.h as they're no longer referenced by linux/uio.h.
- Add ITER_SOURCE/DEST cleanup patches.
- Make iov_iter/netfslib iter extraction patches use ITER_SOURCE/DEST.
- Allow additional gup_flags to be passed into iov_iter_extract_pages().
- Add struct bio patch.
ver #3)
- Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1].
ver #2)
- Rolled the extraction cleanup mode query function into the extraction function, returning the indication through the argument list.
- Fixed patch 4 (extract to scatterlist) to actually use the new extraction API.
Christoph Hellwig (1):
block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted
logic
David Howells (7):
iov_iter: Define flags to qualify page extraction.
iov_iter: Add a function to extract a page list from an iterator
iomap: Don't get an reference on ZERO_PAGE for direct I/O block
zeroing
block: Fix bio_flagged() so that gcc can better optimise it
block: Add BIO_PAGE_PINNED and associated infrastructure
block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages
block: convert bio_map_user_iov to use iov_iter_extract_pages
block/bio.c | 33 ++-- block/blk-map.c | 26 +-- block/blk.h | 12 ++ fs/direct-io.c | 2 + fs/iomap/direct-io.c | 1 - include/linux/bio.h | 5 +- include/linux/blk_types.h | 3 +- include/linux/uio.h | 32 +++- lib/iov_iter.c | 335 +++++++++++++++++++++++++++++++++++++- 9 files changed, 408 insertions(+), 41 deletions(-)
PATCH v10 1/8 iov_iter: Define flags to qualify page extraction.
Define flags to qualify page extraction to pass into iov_iter_*_pages*() rather than passing in FOLL_* flags.
For now only a flag to allow peer-to-peer DMA is supported.
---
Notes:
ver #9)
- Change extract_flags to extraction_flags.
ver #7)
- Don't use FOLL_* as a parameter, but rather define constants specifically to use with iov_iter_*_pages*().
- Drop the I/O direction constants for now.
block/bio.c | 6 +++--- block/blk-map.c | 8 ++++---- include/linux/uio.h | 7 +++++-- lib/iov_iter.c | 14 ++++++++------ 4 files changed, 20 insertions(+), 15 deletions(-)
block/bio.c +3 -3
@@ -1249,7 +1249,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;struct page **pages = (struct page **)bv; -unsigned int gup_flags =0 ; +unsigned int extraction_flags =0 ;ssize_t size, left;unsigned len, i =0 ;size_t offset, trim;
@@ -1264,7 +1264,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) pages += entries_left * (PAGE_PTRS_PER_BVEC -1 );if (bio->bi_bdev &&blk_queue_pci_p2pdma (bio->bi_bdev->bd_disk->queue)) - gup_flags |=FOLL_PCI_P2PDMA ; + extraction_flags |=ITER_ALLOW_P2PDMA ;/* * Each segment in the iov is required to be a block size multiple.
@@ -1275,7 +1275,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)*/ size =iov_iter_get_pages (iter, pages,UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, gup_flags); + nr_pages, &offset, extraction_flags);if (unlikely (size <=0 ))return size ? size : -EFAULT ;
block/blk-map.c +4 -4
@@ -267,7 +267,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, {unsigned int max_sectors =queue_max_hw_sectors (rq->q);unsigned int nr_vecs =iov_iter_npages (iter,BIO_MAX_VECS ); -unsigned int gup_flags =0 ; +unsigned int extraction_flags =0 ;struct bio *bio;int ret;int j;
@@ -280,7 +280,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,return -ENOMEM ;if (blk_queue_pci_p2pdma (rq->q)) - gup_flags |=FOLL_PCI_P2PDMA ; + extraction_flags |=ITER_ALLOW_P2PDMA ;while (iov_iter_count (iter)) {struct page **pages, *stack_pages[UIO_FASTIOV ];
@@ -291,10 +291,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (nr_vecs <=ARRAY_SIZE (stack_pages)) { pages = stack_pages; bytes =iov_iter_get_pages (iter, pages,LONG_MAX , - nr_vecs, &offs, gup_flags); + nr_vecs, &offs, extraction_flags); }else { bytes =iov_iter_get_pages_alloc (iter, &pages, -LONG_MAX , &offs, gup_flags); +LONG_MAX , &offs, extraction_flags); }if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ;
include/linux/uio.h +5 -2
@@ -252,12 +252,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count);ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start, -unsigned gup_flags); +unsigned extraction_flags);ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start);ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start, -unsigned gup_flags); +unsigned extraction_flags);ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages);
@@ -360,4 +360,7 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +/* Flags for iov_iter_get/extract_pages*() */ +#define ITER_ALLOW_P2PDMA 0x01/* Allow P2PDMA on the extracted pages */ +#endif
lib/iov_iter.c +8 -6
@@ -1432,9 +1432,9 @@ static struct page *first_bvec_segment(const struct iov_iter *i,static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages,size_t *start, -unsigned int gup_flags) +unsigned int extraction_flags) { -unsigned int n; +unsigned int n, gup_flags =0 ;if (maxsize > i->count) maxsize = i->count;
@@ -1442,6 +1442,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return 0 ;if (maxsize >MAX_RW_COUNT ) maxsize =MAX_RW_COUNT ; +if (extraction_flags &ITER_ALLOW_P2PDMA ) + gup_flags |=FOLL_PCI_P2PDMA ;if (likely (user_backed_iter (i))) {unsigned long addr;
@@ -1495,14 +1497,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages, -size_t *start,unsigned gup_flags) +size_t *start,unsigned extraction_flags) {if (!maxpages)return 0 ;BUG_ON (!pages);return __iov_iter_get_pages_alloc (i, &pages, maxsize, maxpages, - start, gup_flags); + start, extraction_flags); }EXPORT_SYMBOL_GPL (iov_iter_get_pages);
@@ -1515,14 +1517,14 @@ EXPORT_SYMBOL(iov_iter_get_pages2);ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize, -size_t *start,unsigned gup_flags) +size_t *start,unsigned extraction_flags) {ssize_t len; *pages =NULL ; len =__iov_iter_get_pages_alloc (i, pages, maxsize, ~0U , start, - gup_flags); + extraction_flags);if (len <=0 ) {kvfree (*pages); *pages =NULL ;
Re: [PATCH v10 1/8] iov_iter: Define flags to qualify page extraction.
Quoted 15 lines · David Howells
Define flags to qualify page extraction to pass into iov_iter_*_pages*() rather than passing in FOLL_* flags.
For now only a flag to allow peer-to-peer DMA is supported.
Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by John Hubbard <[email protected]>cc Al Viro <[email protected]>cc Jens Axboe <[email protected]>cc Logan Gunthorpe <[email protected]>---
[...]
Quoted 17 lines
+++
b/include/linux/uio.hinclude/linux/uio.h +2
-2@@ -252,12 +252,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count);ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start, -unsigned gup_flags); +unsigned extraction_flags);ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start);ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start, -unsigned gup_flags); +unsigned extraction_flags);ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages);
include/linux/uio.h static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, · line 364
Quoted 5 lines
include/linux/uio.h +2
-0@@ -360,4 +360,7 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +/* Flags for iov_iter_get/extract_pages*() */ +#define ITER_ALLOW_P2PDMA 0x01/* Allow P2PDMA on the extracted pages */
Just a note that the usage of new __bitwise types instead of "unsigned" is encouraged for flags.
See rmap_t in include/linux/rmap.h as an example.
Apart from that LGTM.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v10 1/8] iov_iter: Define flags to qualify page extraction.
Quoted 4 lines · David Hildenbrand
Just a note that the usage of new
__bitwisetypes instead of "unsigned" is encouraged for flags.See
rmap_tininclude/linux/rmap.has an example.
Hmmm... rmap_t really ought to be unsigned int.
David
Re: [PATCH v10 1/8] iov_iter: Define flags to qualify page extraction.
Quoted 8 lines · David Howells
Quoted 4 lines · David Hildenbrand
Just a note that the usage of new
__bitwisetypes instead of "unsigned" is encouraged for flags.See
rmap_tininclude/linux/rmap.has an example.Hmmm...
rmap_treally ought to be unsigned int.
Not sure if that particularly matters here in practice, ... anyhow
$ git grep "typedef int" | grep __bitwise | wc -l 27 $ git grep "typedef unsigned" | grep __bitwise | wc -l 23
So chose what you prefer ;)
Signature · 3 lines
Thanks, David / dhildenb
PATCH 1/1 iov_iter: Use __bitwise with the extraction_flags
X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 28.2
--------
Quoted 2 lines · David Hildenbrand
Quoted 2 lines
Quoted 2 lines
Just a note that the usage of new
__bitwisetypes instead of "unsigned" is encouraged for flags.
Something like the attached?
Quoted 4 lines
$ git grep "typedef int" | grep __bitwise | wc -l 27 $ git grep "typedef unsigned" | grep __bitwise | wc -l 23
git grep __bitwise | grep typedef | grep __u | wc -l 62
*shrug*
Interestingly, things like __be32 are __bitwise. I wonder if that actually makes sense or if it was just convenient so stop people doing arithmetic on them. I guess doing AND/OR/XOR on them isn't a problem provided both arguments are appropriately byte-swapped.
David
--- block/bio.c | 2 +- block/blk-map.c | 2 +- include/linux/uio.h | 13 ++++++++----- lib/iov_iter.c | 14 +++++++------- 4 files changed, 17 insertions(+), 14 deletions(-)
block/bio.c +1 -1
@@ -1244,11 +1244,11 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page,*/ static int __bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter) { +iov_iter_extraction_t extraction_flags =0 ;unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt;unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;struct page **pages = (struct page **)bv; -unsigned int extraction_flags =0 ;ssize_t size, left;unsigned len, i =0 ;size_t offset, trim;
block/blk-map.c +1 -1
@@ -265,9 +265,9 @@ static struct bio *blk_rq_map_bio_alloc(struct request *rq,static int bio_map_user_iov (struct request *rq,struct iov_iter *iter,gfp_t gfp_mask) { +iov_iter_extraction_t extraction_flags =0 ;unsigned int max_sectors =queue_max_hw_sectors (rq->q);unsigned int nr_vecs =iov_iter_npages (iter,BIO_MAX_VECS ); -unsigned int extraction_flags =0 ;struct bio *bio;int ret;int j;
include/linux/uio.h +8 -5
@@ -13,6 +13,8 @@struct page ;struct pipe_inode_info ; +typedef unsigned int iov_iter_extraction_t ; +struct kvec {void *iov_base;/* and that should *never* hold a userland pointer */ size_t iov_len;
@@ -252,12 +254,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count);ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start, -unsigned extraction_flags); +iov_iter_extraction_t extraction_flags);ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start);ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start, -unsigned extraction_flags); +iov_iter_extraction_t extraction_flags);ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages);
@@ -359,13 +361,14 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, .count = count }; } -/* Flags for iov_iter_get/extract_pages*() */ -#define ITER_ALLOW_P2PDMA 0x01/* Allow P2PDMA on the extracted pages */ +/* Allow P2PDMA on the extracted pages */ +#define ITER_ALLOW_P2PDMA ((__force iov_iter_extraction_t)0x01)ssize_t iov_iter_extract_pages (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages, -unsigned int extraction_flags,size_t *offset0); +iov_iter_extraction_t extraction_flags, +size_t *offset0);/** * iov_iter_extract_will_pin - Indicate how pages from the iterator will be retained
lib/iov_iter.c +7 -7
@@ -1432,7 +1432,7 @@ static struct page *first_bvec_segment(const struct iov_iter *i,static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages,size_t *start, -unsigned int extraction_flags) +iov_iter_extraction_t extraction_flags) {unsigned int n, gup_flags =0 ;
@@ -1929,7 +1929,7 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state)static ssize_t iov_iter_extract_pipe_pages (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages, -unsigned int extraction_flags, +iov_iter_extraction_t extraction_flags,size_t *offset0) {unsigned int nr, offset, chunk, j;
@@ -1971,7 +1971,7 @@ static ssize_t iov_iter_extract_pipe_pages(struct iov_iter *i,static ssize_t iov_iter_extract_xarray_pages (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages, -unsigned int extraction_flags, +iov_iter_extraction_t extraction_flags,size_t *offset0) {struct page *page, **p;
@@ -2017,7 +2017,7 @@ static ssize_t iov_iter_extract_xarray_pages(struct iov_iter *i,static ssize_t iov_iter_extract_bvec_pages (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages, -unsigned int extraction_flags, +iov_iter_extraction_t extraction_flags,size_t *offset0) {struct page **p, *page;
@@ -2060,7 +2060,7 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i,static ssize_t iov_iter_extract_kvec_pages (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned int maxpages, -unsigned int extraction_flags, +iov_iter_extraction_t extraction_flags,size_t *offset0) {struct page **p, *page;
@@ -2125,7 +2125,7 @@ static ssize_t iov_iter_extract_user_pages(struct iov_iter *i,struct page ***pages, size_t maxsize, unsigned int maxpages, - unsigned int extraction_flags, + iov_iter_extraction_t extraction_flags, size_t *offset0) { unsigned long addr;
@@ -2207,7 +2207,7 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i,struct page ***pages, size_t maxsize, unsigned int maxpages, - unsigned int extraction_flags, + iov_iter_extraction_t extraction_flags, size_t *offset0) { maxsize =min_t (size_t,min_t (size_t, maxsize, i->count),MAX_RW_COUNT );
Re: [PATCH] iov_iter: Use __bitwise with the extraction_flags
Quoted 4 lines · David Howells
Interestingly, things like
__be32are__bitwise. I wonder if that actually makes sense or if it was just convenient so stop people doing arithmetic on them. I guess doing AND/OR/XOR on them isn't a problem provided both arguments are appropriately byte-swapped.
I recall that __be32 and friends were one of the early users of __bitwise in the kernel. And the reason IIRC was exactly that: detect when no proper conversion was performed using static code analysis (Sparse). While some operations might make sense, the abuse is much more likely.
LGTM, thanks!
Signature · 3 lines
Thanks, David / dhildenb
Al Viro
maintainerRe: [PATCH] iov_iter: Use __bitwise with the extraction_flags
In reply to David HowellsQuoted 17 lines · David Howells
X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 28.2
--------Quoted 2 lines · David Hildenbrand
Quoted 2 lines
Quoted 2 lines
Just a note that the usage of new
__bitwisetypes instead of "unsigned" is encouraged for flags.Something like the attached?
Quoted 4 lines
$ git grep "typedef int" | grep __bitwise | wc -l 27 $ git grep "typedef unsigned" | grep __bitwise | wc -l 23git grep
__bitwise| grep typedef | grep__u| wc -l 62
Quoted 6 lines
*shrug*
Interestingly, things like
__be32are__bitwise. I wonder if that actually makes sense or if it was just convenient so stop people doing arithmetic on them. I guess doing AND/OR/XOR on them isn't a problem provided both arguments are appropriately byte-swapped.
Forget the words "byte-swapped". There are several data types. With different memory representations. Bitwise operations are valid between the values of the same type and yield the result of that same type.
The fact that mapping between those representations happens to be an involution is an accident; keeping track of the number of times you've done a byteswap to the value currently in this variable is asking for trouble. It's really easy to fuck up.
"Am I trying to store the value of type X in variable of type Y (presumably having forgotten that I need to use X_to_Y(...) to convert)" is much easier to keep track of.
PATCH v10 2/8 iov_iter: Add a function to extract a page list from an iterator
In reply to David HowellsAdd a function, iov_iter_extract_pages(), to extract a list of pages from an iterator. The pages may be returned with a pin added or nothing, depending on the type of iterator.
Add a second function, iov_iter_extract_will_pin(), to determine how the cleanup should be done.
There are two cases:
(1) ITER_IOVEC or ITER_UBUF iterator.
Extracted pages will have pins (FOLL_PIN) obtained on them so that a concurrent fork() will forcibly copy the page so that DMA is done to/from the parent's buffer and is unavailable to/unaffected by the child process.
iov_iter_extract_will_pin() will return true for this case. The caller should use something like unpin_user_page() to dispose of the page.
(2) Any other sort of iterator.
No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention.
iov_iter_extract_will_pin() will return false. The pages don't need additional disposal.
---
Notes:
ver #10)
- Fix use of i->kvec in iov_iter_extract_bvec_pages() to be i->bvec.
ver #9)
- Rename iov_iter_extract_mode() to iov_iter_extract_will_pin() and make it return true/false not FOLL_PIN/0 as FOLL_PIN is going to be made private to mm/.
- Change extract_flags to extraction_flags.
ver #8)
- It seems that all DIO is supposed to be done under FOLL_PIN now, and not FOLL_GET, so switch to only using pin_user_pages() for user-backed iters.
- Wrap an argument in brackets in the iov_iter_extract_mode() macro.
- Drop the extract_flags argument to iov_iter_extract_mode() for now [hch].
ver #7)
- Switch to passing in iter-specific flags rather than FOLL_* flags.
- Drop the direction flags for now.
- Use ITER_ALLOW_P2PDMA to request FOLL_PCI_P2PDMA.
- Disallow use of ITER_ALLOW_P2PDMA with non-user-backed iter.
- Add support for extraction from KVEC-type iters.
- Use iov_iter_advance() rather than open-coding it.
- Make BVEC- and KVEC-type skip over initial empty vectors.
ver #6)
- Add back the function to indicate the cleanup mode.
- Drop the cleanup_mode return arg to iov_iter_extract_pages().
- Pass FOLL_SOURCE/DEST_BUF in gup_flags. Check this against the iter data_source.
ver #4)
- Use ITER_SOURCE/DEST instead of WRITE/READ.
- Allow additional FOLL_* flags, such as FOLL_PCI_P2PDMA to be passed in.
ver #3)
- Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1].
include/linux/uio.h | 25 ++++ lib/iov_iter.c | 321 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 346 insertions(+)
include/linux/uio.h +25 -0
@@ -363,4 +363,29 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction,/* Flags for iov_iter_get/extract_pages*() */ #define ITER_ALLOW_P2PDMA 0x01/* Allow P2PDMA on the extracted pages */ +ssize_t iov_iter_extract_pages (struct iov_iter *i,struct page ***pages, +size_t maxsize,unsigned int maxpages, +unsigned int extraction_flags,size_t *offset0); + +/** +* iov_iter_extract_will_pin - Indicate how pages from the iterator will be retained +* @iter: The iterator +* +* Examine the iterator and indicate by returning true or false as to how, if +* at all, pages extracted from the iterator will be retained by the extraction +* function. +* +* %true indicates that the pages will have a pin placed in them that the +* caller must unpin. This is must be done for DMA/async DIO to force fork() +* to forcibly copy a page for the child (the parent must retain the original +* page). +* +* %false indicates that no measures are taken and that it's up to the caller +* to retain the pages. +*/ +static inline bool iov_iter_extract_will_pin (const struct iov_iter *iter) +{ +return user_backed_iter (iter); +} +#endif
lib/iov_iter.c +321 -0
324 lines not on this page · raw message
Re: [PATCH v10 2/8] iov_iter: Add a function to extract a page list from an iterator
iov_iter_extract_kvec_pages() isn't quite right. The problem is that there's not currently any path by which it can be tested, as currently iov_iter_get_pages*() balk at it, but I managed to do that by piggybacking some testing code on my cifs patches.
The attached change fixes the problem.
David
lib/iov_iter.c +2 -2
@@ -2080,9 +2080,9 @@ static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, skip =0 ; } - offset = skip %PAGE_SIZE ; + kaddr = i->kvec->iov_base + skip; + offset = (unsigned long )kaddr & ~PAGE_MASK ; *offset0 = offset; - kaddr = i->kvec->iov_base; maxpages =want_pages_array (pages, maxsize, offset, maxpages);if (!maxpages)
PATCH v10 3/8 iomap: Don't get an reference on ZERO_PAGE for direct I/O block zeroing
In reply to David HowellsZERO_PAGE can't go away, no need to hold an extra reference.
--- fs/iomap/direct-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/iomap/direct-io.c +1 -1
@@ -202,7 +202,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; -get_page (page); +bio_set_flag (bio,BIO_NO_PAGE_REF );__bio_add_page (bio, page, len,0 );iomap_dio_submit_bio (iter, dio, bio, pos); }
PATCH v10 4/8 block: Fix bio_flagged() so that gcc can better optimise it
In reply to David HowellsFix bio_flagged() so that multiple instances of it, such as:
if (bio_flagged(bio, BIO_PAGE_REFFED) || bio_flagged(bio, BIO_PAGE_PINNED))
can be combined by the gcc optimiser into a single test in assembly (arguably, this is a compiler optimisation issue[1]).
The missed optimisation stems from bio_flagged() comparing the result of the bitwise-AND to zero. This results in an out-of-line bio_release_page() being compiled to something like:
<+0>: mov 0x14(%rdi),%eax <+3>: test $0x1,%al <+5>: jne 0xffffffff816dac53 <bio_release_pages+11> <+7>: test $0x2,%al <+9>: je 0xffffffff816dac5c <bio_release_pages+20> <+11>: movzbl %sil,%esi <+15>: jmp 0xffffffff816daba1 <__bio_release_pages> <+20>: jmp 0xffffffff81d0b800 <__x86_return_thunk>
However, the test is superfluous as the return type is bool. Removing it results in:
<+0>: testb $0x3,0x14(%rdi) <+4>: je 0xffffffff816e4af4 <bio_release_pages+15> <+6>: movzbl %sil,%esi <+10>: jmp 0xffffffff816dab7c <__bio_release_pages> <+15>: jmp 0xffffffff81d0b7c0 <__x86_return_thunk>
instead.
Also, the MOVZBL instruction looks unnecessary[2] - I think it's just 're-booling' the mark_dirty parameter.
--- include/linux/bio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/bio.h +1 -1
@@ -227,7 +227,7 @@ static inline void bio_cnt_set(struct bio *bio, unsigned int count)static inline bool bio_flagged (struct bio *bio,unsigned int bit) { -return (bio->bi_flags & (1U << bit)) !=0 ; +return bio->bi_flags & (1U << bit); }static inline void bio_set_flag (struct bio *bio,unsigned int bit)
PATCH v10 5/8 block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic
In reply to David HowellsFrom: Christoph Hellwig <[email protected]>
Replace BIO_NO_PAGE_REF with a BIO_PAGE_REFFED flag that has the inverted meaning is only set when a page reference has been acquired that needs to be released by bio_release_pages().
---
Notes:
ver #8)
- Don't default to BIO_PAGE_REFFED [hch].
ver #5)
- Split from patch that uses iov_iter_extract_pages().
block/bio.c | 2 +- block/blk-map.c | 1 + fs/direct-io.c | 2 ++ fs/iomap/direct-io.c | 1 - include/linux/bio.h | 2 +- include/linux/blk_types.h | 2 +- 6 files changed, 6 insertions(+), 4 deletions(-)
block/bio.c +1 -1
@@ -1198,7 +1198,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; bio->bi_iter.bi_size = size; -bio_set_flag (bio,BIO_NO_PAGE_REF );bio_set_flag (bio,BIO_CLONED ); }
@@ -1343,6 +1342,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)return 0 ; } +bio_set_flag (bio,BIO_PAGE_REFFED );do { ret =__bio_iov_iter_get_pages (bio, iter); }while (!ret &&iov_iter_count (iter) && !bio_full (bio,0 ));
block/blk-map.c +1 -0
@@ -282,6 +282,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (blk_queue_pci_p2pdma (rq->q)) extraction_flags |=ITER_ALLOW_P2PDMA ; +bio_set_flag (bio,BIO_PAGE_REFFED );while (iov_iter_count (iter)) {struct page **pages, *stack_pages[UIO_FASTIOV ];ssize_t bytes;
fs/direct-io.c +2 -0
@@ -403,6 +403,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_end_io = dio_bio_end_aio; else bio->bi_end_io = dio_bio_end_io; +/* for now require references for all pages */ +bio_set_flag (bio,BIO_PAGE_REFFED ); sdio->bio = bio; sdio->logical_offset_in_bio = sdio->cur_page_fs_offset; }
fs/iomap/direct-io.c +0 -1
@@ -202,7 +202,6 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; -bio_set_flag (bio,BIO_NO_PAGE_REF );__bio_add_page (bio, page, len,0 );iomap_dio_submit_bio (iter, dio, bio, pos); }
include/linux/bio.h +1 -1
@@ -484,7 +484,7 @@ void zero_fill_bio(struct bio *bio);static inline void bio_release_pages (struct bio *bio,bool mark_dirty) { -if (!bio_flagged (bio,BIO_NO_PAGE_REF )) +if (bio_flagged (bio,BIO_PAGE_REFFED ))__bio_release_pages (bio, mark_dirty); }
include/linux/blk_types.h +1 -1
@@ -318,7 +318,7 @@ struct bio {* bio flags */ enum { -BIO_NO_PAGE_REF ,/* don't put release vec pages */ +BIO_PAGE_REFFED ,/* put pages in bio_release_pages() */ BIO_CLONED ,/* doesn't own data */ BIO_BOUNCED ,/* bio is a bounce bio */ BIO_QUIET ,/* Make BIO Quiet */
Re: [PATCH v10 5/8] block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic
Quoted 15 lines · David Howells
From: Christoph Hellwig <[email protected]>
Replace
BIO_NO_PAGE_REFwith aBIO_PAGE_REFFEDflag that has the inverted meaning is only set when a page reference has been acquired that needs to be released bybio_release_pages().Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by John Hubbard <[email protected]>cc Al Viro <[email protected]>cc Jens Axboe <[email protected]>cc Jan Kara <[email protected]>cc Matthew Wilcox <[email protected]>cc Logan Gunthorpe <[email protected]>
Quoted 1 line
---
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v10 5/8] block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic
In reply to David HowellsQuoted 15 lines · David Howells
From: Christoph Hellwig <[email protected]>
Replace
BIO_NO_PAGE_REFwith aBIO_PAGE_REFFEDflag that has the inverted meaning is only set when a page reference has been acquired that needs to be released bybio_release_pages().Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by John Hubbard <[email protected]>cc Al Viro <[email protected]>cc Jens Axboe <[email protected]>cc Jan Kara <[email protected]>cc Matthew Wilcox <[email protected]>cc Logan Gunthorpe <[email protected]>
Quoted 1 line
---
Oh, and I agree with a previous comment that this patch should also hold a Signed-off-by from Christoph above your Signed-off-by, if he is mentioned as the author via "From:".
I remember stumbling over that in submitting-patches.rst ...
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v10 5/8] block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic
Quoted 11 lines · David Hildenbrand
Quoted 10 lines
Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by John Hubbard <[email protected]>cc Al Viro <[email protected]>cc Jens Axboe <[email protected]>cc Jan Kara <[email protected]>cc Matthew Wilcox <[email protected]>cc Logan Gunthorpe <[email protected]>---
Quoted 3 lines
Oh, and I agree with a previous comment that this patch should also hold a Signed-off-by from Christoph above your Signed-off-by, if he is mentioned as the author via "From:".
I think that was actually in reference to patch 3, but yes - and possibly a Co-developed-by tag. Christoph?
David
Re: [PATCH v10 5/8] block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic
Quoted 13 lines · David Howells
Quoted 11 lines · David Hildenbrand
Quoted 10 lines
Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by John Hubbard <[email protected]>cc Al Viro <[email protected]>cc Jens Axboe <[email protected]>cc Jan Kara <[email protected]>cc Matthew Wilcox <[email protected]>cc Logan Gunthorpe <[email protected]>---
Quoted 6 lines
Quoted 3 lines
Oh, and I agree with a previous comment that this patch should also hold a Signed-off-by from Christoph above your Signed-off-by, if he is mentioned as the author via "From:".
I think that was actually in reference to patch 3, but yes - and possibly a Co-developed-by tag. Christoph?
As I said feel free to add a co-developed by if you want. But more importantly restore my signoff from here:
http://git.infradead.org/users/hch/misc.git/commitdiff/37ea178d5d95166196112905aa75d85cb0416d49
PATCH v10 6/8 block: Add BIO_PAGE_PINNED and associated infrastructure
In reply to David HowellsAdd BIO_PAGE_PINNED to indicate that the pages in a bio are pinned (FOLL_PIN) and that the pin will need removing.
---
Notes:
ver #10)
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #9)
- Only consider pinning in bio_set_cleanup_mode(). Ref'ing pages in struct bio is going away.
- page_put_unpin() is removed; call unpin_user_page() and put_page() directly.
- Use bio_release_page() in __bio_release_pages().
- BIO_PAGE_PINNED and BIO_PAGE_REFFED can't both be set, so use if-else when testing both of them.
ver #8)
- Move the infrastructure to clean up pinned pages to this patch [hch].
- Put BIO_PAGE_PINNED before BIO_PAGE_REFFED as the latter should probably be removed at some point. FOLL_PIN can then be renumbered first.
block/bio.c | 6 +++--- block/blk.h | 12 ++++++++++++ include/linux/bio.h | 3 ++- include/linux/blk_types.h | 1 + 4 files changed, 18 insertions(+), 4 deletions(-)
block/bio.c +3 -3
@@ -1176,7 +1176,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty)bio_for_each_segment_all (bvec, bio, iter_all) {if (mark_dirty && !PageCompound (bvec->bv_page))set_page_dirty_lock (bvec->bv_page); -put_page (bvec->bv_page); +bio_release_page (bio, bvec->bv_page); } }EXPORT_SYMBOL_GPL (__bio_release_pages);
@@ -1496,8 +1496,8 @@ void bio_set_pages_dirty(struct bio *bio)* the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from -* here on. It will run one put_page() against each page and will run one -* bio_put() against the BIO. +* here on. It will unpin each page and will run one bio_put() against the +* BIO. */ static void bio_dirty_fn (struct work_struct *work);
block/blk.h +12 -0
@@ -425,6 +425,18 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* +* Clean up a page appropriately, where the page may be pinned, may have a +* ref taken on it or neither. +*/ +static inline void bio_release_page (struct bio *bio,struct page *page) +{ +if (bio_flagged (bio,BIO_PAGE_PINNED )) +unpin_user_page (page); +else if (bio_flagged (bio,BIO_PAGE_REFFED )) +put_page (page); +} +struct request_queue *blk_alloc_queue (int node_id);int disk_scan_partitions (struct gendisk *disk,fmode_t mode,void *owner);
include/linux/bio.h +2 -1
@@ -484,7 +484,8 @@ void zero_fill_bio(struct bio *bio);static inline void bio_release_pages (struct bio *bio,bool mark_dirty) { -if (bio_flagged (bio,BIO_PAGE_REFFED )) +if (bio_flagged (bio,BIO_PAGE_REFFED ) || +bio_flagged (bio,BIO_PAGE_PINNED ))__bio_release_pages (bio, mark_dirty); }
include/linux/blk_types.h +1 -0
@@ -318,6 +318,7 @@ struct bio {* bio flags */ enum { +BIO_PAGE_PINNED ,/* Unpin pages in bio_release_pages() */ BIO_PAGE_REFFED ,/* put pages in bio_release_pages() */ BIO_CLONED ,/* doesn't own data */ BIO_BOUNCED ,/* bio is a bounce bio */
Re: [PATCH v10 6/8] block: Add BIO_PAGE_PINNED and associated infrastructure
Quoted 12 lines · David Howells
Add
BIO_PAGE_PINNEDto indicate that the pages in a bio are pinned (FOLL_PIN) and that the pin will need removing.Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by John Hubbard <[email protected]>cc Al Viro <[email protected]>cc Jens Axboe <[email protected]>cc Jan Kara <[email protected]>cc Matthew Wilcox <[email protected]>cc Logan Gunthorpe <[email protected]>
Quoted 1 line
---
Signature · 3 lines
Thanks, David / dhildenb
PATCH v10 7/8 block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages
In reply to David HowellsThis will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator.
The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being affected by/visible to the child process).
---
Notes:
ver #10)
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #8)
- Split the patch up a bit [hch].
- We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately.
ver #7)
- Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN.
ver #5)
- Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode.
- Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch.
block/bio.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
block/bio.c +12 -11
@@ -1212,7 +1212,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, }if (same_page) -put_page (page); +bio_release_page (bio, page);return 0 ; }
@@ -1226,7 +1226,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len)return -EINVAL ;if (same_page) -put_page (page); +bio_release_page (bio, page);return 0 ; }
@@ -1237,10 +1237,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page,* @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * -* Pins pages from *iter and appends them to @bio's bvec array. The -* pages will have to be released using put_page() when done. -* For multi-segment *iter, this function only adds pages from the -* next non-empty segment of the iov iterator. +* Extracts pages from *iter and appends them to @bio's bvec array. The pages +* will have to be cleaned up in the way indicated by the BIO_PAGE_PINNED flag. +* For a multi-segment *iter, this function only adds pages from the next +* non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter) {
@@ -1272,9 +1272,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size =iov_iter_get_pages (iter, pages, -UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, extraction_flags); + size =iov_iter_extract_pages (iter, &pages, +UINT_MAX - bio->bi_iter.bi_size, + nr_pages, extraction_flags, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;
@@ -1307,7 +1307,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)iov_iter_revert (iter, left); out:while (i < nr_pages) -put_page (pages[i++]); +bio_release_page (bio, pages[i++]);return ret; }
@@ -1342,7 +1342,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)return 0 ; } -bio_set_flag (bio,BIO_PAGE_REFFED ); +if (iov_iter_extract_will_pin (iter)) +bio_set_flag (bio,BIO_PAGE_PINNED );do { ret =__bio_iov_iter_get_pages (bio, iter); }while (!ret &&iov_iter_count (iter) && !bio_full (bio,0 ));
PATCH v10 8/8 block: convert bio_map_user_iov to use iov_iter_extract_pages
In reply to David HowellsThis will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator.
The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being visible to/affected by the child process).
---
Notes:
ver #10)
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #8)
- Split the patch up a bit [hch].
- We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately.
ver #7)
- Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN.
ver #5)
- Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode.
- Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch.
block/blk-map.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
block/blk-map.c +11 -12
@@ -281,22 +281,21 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (blk_queue_pci_p2pdma (rq->q)) extraction_flags |=ITER_ALLOW_P2PDMA ; +if (iov_iter_extract_will_pin (iter)) +bio_set_flag (bio,BIO_PAGE_PINNED ); -bio_set_flag (bio,BIO_PAGE_REFFED );while (iov_iter_count (iter)) { -struct page **pages, *stack_pages[UIO_FASTIOV ]; +struct page *stack_pages[UIO_FASTIOV ]; +struct page **pages = stack_pages;ssize_t bytes;size_t offs;int npages; -if (nr_vecs <=ARRAY_SIZE (stack_pages)) { - pages = stack_pages; - bytes =iov_iter_get_pages (iter, pages,LONG_MAX , - nr_vecs, &offs, extraction_flags); - }else { - bytes =iov_iter_get_pages_alloc (iter, &pages, -LONG_MAX , &offs, extraction_flags); - } +if (nr_vecs >ARRAY_SIZE (stack_pages)) + pages =NULL ; + + bytes =iov_iter_extract_pages (iter, &pages,LONG_MAX , + nr_vecs, extraction_flags, &offs);if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;
@@ -318,7 +317,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (!bio_add_hw_page (rq->q, bio, page, n, offs, max_sectors, &same_page)) {if (same_page) -put_page (page); +bio_release_page (bio, page);break ; }
@@ -330,7 +329,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,* release the pages we didn't map into the bio, if any */ while (j < npages) -put_page (pages[j++]); +bio_release_page (bio, pages[j++]);if (pages != stack_pages)kvfree (pages);/* couldn't stuff something into bio? */