switch the block layer to use kmap_local_page v2
Subsystems (9)
Hi all,
this series switches the core block layer code and all users of the existing bvec kmap helpers to use kmap_local_page. Drivers that currently use open coded kmap_atomic calls will converted in a follow on series.
To do so a new kunmap variant is added that calls flush_kernel_dcache_page. I'm not entirely sure where to call flush_dcache_page vs flush_kernel_dcache_page, so I've tried to follow the documentation here, but additional feedback would be welcome.
Changes since v1:
- add more/better comments
- add a new kunmap_local_dirty helper to feal with flush(_kernel)_dcache_page
Diffstat:
arch/mips/include/asm/mach-rc32434/rb.h | 2 - block/bio-integrity.c | 14 +++----- block/bio.c | 37 ++++++---------------- block/blk-map.c | 2 - block/bounce.c | 39 +++++------------------- block/t10-pi.c | 16 +++------ drivers/block/ps3disk.c | 19 +---------- drivers/block/rbd.c | 15 +-------- drivers/md/dm-writecache.c | 5 +-- include/linux/bio.h | 42 ------------------------- include/linux/bvec.h | 52 ++++++++++++++++++++++++++++++-- include/linux/highmem-internal.h | 7 ++++ include/linux/highmem.h | 10 ++++-- 13 files changed, 102 insertions(+), 158 deletions(-)
PATCH 1/18 mm: add a kunmap_local_dirty helper
Add a helper that calls flush_kernel_dcache_page before unmapping the local mapping. flush_kernel_dcache_page is required for all pages potentially mapped into userspace that were written to using kmap*, so having a helper that does the right thing can be very convenient.
--- include/linux/highmem-internal.h | 7 +++++++ include/linux/highmem.h | 4 ++++ 2 files changed, 11 insertions(+)
include/linux/highmem-internal.h +7 -0
@@ -224,4 +224,11 @@ do { \__kunmap_local (__addr); \ } while (0) +#define kunmap_local_dirty (__page, __addr) \ +do { \ + if (!PageSlab(__page)) \ + flush_kernel_dcache_page(__page); \ + kunmap_local(__addr); \ +} while (0) +#endif
include/linux/highmem.h +4 -0
@@ -93,6 +93,10 @@ static inline void kmap_flush_unused(void);* On HIGHMEM enabled systems mapping a highmem page has the side effect of * disabling migration in order to keep the virtual address stable across * preemption. No caller of kmap_local_page() can rely on this side effect. +* +* If data is written to the returned kernel mapping, the callers needs to +* unmap the mapping using kunmap_local_dirty(), else kunmap_local() should +* be used. */ static inline void *kmap_local_page (struct page *page);
Signature · 1 line
2.30.2
Re: [PATCH 01/18] mm: add a kunmap_local_dirty helper
Quoted 16 lines · Christoph Hellwig
Add a helper that calls
flush_kernel_dcache_pagebefore unmapping the local mapping.flush_kernel_dcache_pageis required for all pages potentially mapped into userspace that were written to using kmap*, so having a helper that does the right thing can be very convenient.Signed-off-by Christoph Hellwig <[email protected]>--- include/linux/highmem-internal.h | 7 +++++++ include/linux/highmem.h | 4 ++++ 2 files changed, 11 insertions(+)include/linux/highmem-internal.h +0
-0
include/linux/highmem-internal.h do { \ · line 227
Quoted 4 lines
include/linux/highmem-internal.h +1
-0@@ -224,4 +224,11 @@ do { \__kunmap_local (__addr); \ }while (0 ) +#define kunmap_local_dirty(__page, __addr) \
I think having to store the page and addr to return to kunmap_local_dirty() is going to be a pain in some code paths. Not a show stopper but see below...
include/linux/highmem-internal.h do { \ · line 229
Quoted 2 lines
include/linux/highmem-internal.h +2
-0+do { \ + if (!PageSlab(__page)) \
Was there some clarification why the page can't be a Slab page? Or is this just an optimization?
include/linux/highmem-internal.h do { \ · line 230
Quoted 1 line
include/linux/highmem-internal.h +1
-0+flush_kernel_dcache_page (__page); \
Is this required on 32bit systems? Why is kunmap_flush_on_unmap() not sufficient on 64bit systems? The normal kunmap_local() path does that.
I'm sorry but I did not see a conclusion to my query on V1. Herbert implied the he just copied from the crypto code.[1] I'm concerned that this _dirty() call is just going to confuse the users of kmap even more. So why can't we get to the bottom of why flush_kernel_dcache_page() needs so much logic around it before complicating the general kernel users.
I would like to see it go away if possible.
Ira
Quoted 22 lines
include/linux/highmem-internal.h +3
-0+kunmap_local (__addr); \ +} while (0) +#endif include/linux/highmem.h +4
-0@@ -93,6 +93,10 @@ static inline void kmap_flush_unused(void);* On HIGHMEM enabled systems mapping a highmem page has the side effect of * disabling migration in order to keep the virtual address stable across * preemption. No caller of kmap_local_page() can rely on this side effect. +* +* If data is written to the returned kernel mapping, the callers needs to +* unmap the mapping using kunmap_local_dirty(), else kunmap_local() should +* be used. */ static inline void *kmap_local_page (struct page *page);Signature · 1 line
2.30.2
Herbert Xu
maintainerRe: [PATCH 01/18] mm: add a kunmap_local_dirty helper
Quoted 6 lines · Ira Weiny
include/linux/highmem-internal.h do { \ · line 230
Quoted 2 lines
include/linux/highmem-internal.h +1
-0+flush_kernel_dcache_page (__page); \Is this required on 32bit systems? Why is
kunmap_flush_on_unmap()not sufficient on 64bit systems? The normalkunmap_local()path does that.
Quoted 7 lines
I'm sorry but I did not see a conclusion to my query on V1. Herbert implied the he just copied from the crypto code.[1] I'm concerned that this
_dirty()call is just going to confuse the users of kmap even more. So why can't we get to the bottom of whyflush_kernel_dcache_page()needs so much logic around it before complicating the general kernel users.I would like to see it go away if possible.
This thread may be related:
https://lwn.net/Articles/240249/
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [PATCH 01/18] mm: add a kunmap_local_dirty helper
Quoted 13 lines · Herbert Xu
Quoted 12 lines · Ira Weiny
include/linux/highmem-internal.h do { \ · line 230
Quoted 2 lines
include/linux/highmem-internal.h +1
-0+flush_kernel_dcache_page (__page); \Is this required on 32bit systems? Why is
kunmap_flush_on_unmap()not sufficient on 64bit systems? The normalkunmap_local()path does that.I'm sorry but I did not see a conclusion to my query on V1. Herbert implied the he just copied from the crypto code.[1] I'm concerned that this
_dirty()call is just going to confuse the users of kmap even more. So why can't we get to the bottom of whyflush_kernel_dcache_page()needs so much logic around it before complicating the general kernel users.
Quoted 5 lines
Quoted 1 line
I would like to see it go away if possible.
This thread may be related:
Interesting! Thanks!
Digging around a bit more I found:
https://lore.kernel.org/patchwork/patch/439637/
Auditing all the flush_dcache_page() arch code reveals that the mapping field is either unused, or is checked for NULL. Furthermore, all the implementations call page_mapping_file() which further limits the page to not be a swap page.
All flush_kernel_dcache_page() implementations appears to operate the same way in all arch's which define that call.
So I'm confident now that additional !PageSlab(__page) checks are not needed
and this patch is unnecessary. Christoph, can we leave this out of the kmap
API and just fold the flush_kernel_dcache_page() calls back into the bvec code?
Unfortunately, I'm not convinced this can be handled completely by kunmap_local() nor the mem*_page() calls because there is a difference between flush_dcache_page() and flush_kernel_dcache_page() in most archs... [parisc being an exception which falls back to flush_kernel_dcache_page()]...
It seems like the generic unmap path _should_ be able to determine which call to make based on the page but I'd have to look at that more.
Ira
Herbert Xu
maintainerPATCH 1/1 crypto: scatterwalk - Remove obsolete PageSlab check
Quoted 6 lines · Ira Weiny
Interesting! Thanks!
Digging around a bit more I found:
Nice find. So we can at least get rid of the PageSlab call from the Crypto API.
---8<---
As it is now legal to call flush_dcache_page on slab pages we no longer need to do the check in the Crypto API.
include/crypto/scatterwalk.h +1 -6
@@ -81,12 +81,7 @@ static inline void scatterwalk_pagedone(struct scatter_walk *walk, int out,struct page *page; page =sg_page (walk->sg) + ((walk->offset -1 ) >>PAGE_SHIFT ); -/* Test ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE first as -* PageSlab cannot be optimised away per se due to -* use of volatile pointer. -*/ -if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE && !PageSlab (page)) -flush_dcache_page (page); +flush_dcache_page (page); }if (more && walk->offset >= walk->sg->offset + walk->sg->length)
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
PATCH 2/18 mm: use kunmap_local_dirty in memcpy_to_page
In reply to Christoph Hellwigmemcpy_to_page can write to potentially mapped page cache pages, so use kunmap_local_dirty to make sure flush_kernel_dcache_pages is called.
--- include/linux/highmem.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/highmem.h +1 -1
@@ -333,7 +333,7 @@ static inline void memcpy_to_page(struct page *page, size_t offset,VM_BUG_ON (offset + len >PAGE_SIZE );memcpy (to + offset, from, len); -kunmap_local (to); +kunmap_local_dirty (page, to); }static inline void memzero_page (struct page *page,size_t offset,size_t len)
Signature · 1 line
2.30.2
PATCH 3/18 mm: use kmap_local_page in memzero_page
In reply to Christoph HellwigNo need for kmap_atomic here.
--- include/linux/highmem.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/highmem.h +2 -2
@@ -338,9 +338,9 @@ static inline void memcpy_to_page(struct page *page, size_t offset,static inline void memzero_page (struct page *page,size_t offset,size_t len) { -char *addr =kmap_atomic (page); +char *addr =kmap_local_page (page);memset (addr + offset,0 , len); -kunmap_atomic (addr); +kunmap_local_dirty (page, addr); }#endif /* _LINUX_HIGHMEM_H */
Signature · 1 line
2.30.2
PATCH 4/18 MIPS: don't include <linux/genhd.h> in <asm/mach-rc32434/rb.h>
In reply to Christoph HellwigThere is no need to include genhd.h from a random arch header, and not doing so prevents the possibility for nasty include loops.
--- arch/mips/include/asm/mach-rc32434/rb.h | 2 -- 1 file changed, 2 deletions(-)
arch/mips/include/asm/mach-rc32434/rb.h +0 -2
@@ -7,8 +7,6 @@#ifndef __ASM_RC32434_RB_H#define __ASM_RC32434_RB_H -#include <linux/genhd.h> -#define REGBASE 0x18000000#define IDT434_REG_BASE ((volatile void *) KSEG1ADDR(REGBASE))#define UART0BASE 0x58000
Signature · 1 line
2.30.2
PATCH 5/18 bvec: fix the include guards for bvec.h
In reply to Christoph HellwigFix the include guards to match the file naming.
--- include/linux/bvec.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/bvec.h +3 -3
@@ -4,8 +4,8 @@* * Copyright (C) 2001 Ming Lei <[email protected]> */ -#ifndef __LINUX_BVEC_ITER_H -#define __LINUX_BVEC_ITER_H +#ifndef __LINUX_BVEC_H +#define __LINUX_BVEC_H#include <linux/bug.h> #include <linux/errno.h>
@@ -183,4 +183,4 @@ static inline void bvec_advance(const struct bio_vec *bvec, } } -#endif /* __LINUX_BVEC_ITER_H */ +#endif /* __LINUX_BVEC_H */
Signature · 1 line
2.30.2
PATCH 6/18 bvec: add a bvec_kmap_local helper
In reply to Christoph HellwigAdd a helper to call kmap_local_page on a bvec. There is no need for an unmap helper given that kunmap_local accept any address in the mapped page.
--- include/linux/bvec.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
include/linux/bvec.h +13 -0
@@ -7,6 +7,7 @@#ifndef __LINUX_BVEC_H#define __LINUX_BVEC_H +#include <linux/highmem.h> #include <linux/bug.h> #include <linux/errno.h> #include <linux/limits.h>
@@ -183,4 +184,16 @@ static inline void bvec_advance(const struct bio_vec *bvec, } } +/** +* bvec_kmap_local - map a bvec into the kernel virtual address space +* @bvec: bvec to map +* +* Must be called on single-page bvecs only. Call kunmap_local on the returned +* address to unmap. +*/ +static inline void *bvec_kmap_local (struct bio_vec *bvec) +{ +return kmap_local_page (bvec->bv_page) + bvec->bv_offset; +} +#endif /* __LINUX_BVEC_H */
Signature · 1 line
2.30.2
Re: [dm-devel] [PATCH 06/18] bvec: add a bvec_kmap_local helper
include/linux/bvec.h static inline void bvec_advance(const struct bio_vec *bvec, · line 197
Quoted 11 lines · Christoph Hellwig
include/linux/bvec.h +11
-0+/** +* bvec_kmap_local - map a bvec into the kernel virtual address space +* @bvec: bvec to map +* +* Must be called on single-page bvecs only. Call kunmap_local on the returned +* address to unmap. +*/ +static inline void *bvec_kmap_local (struct bio_vec *bvec) +{ +return kmap_local_page (bvec->bv_page) + bvec->bv_offset; +}
Hi Christoph,
Would it be appropriate to add WARN_ON_ONCE(bvec->bv_offset >= PAGE_SIZE) in this function?
Thanks,
Bart.
PATCH 7/18 bvec: add memcpy_{from,to}_bvec and memzero_bvec helper
In reply to Christoph HellwigAdd helpers to perform common memory operation on a bvec.
--- include/linux/bvec.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
include/linux/bvec.h +33 -0
@@ -196,4 +196,37 @@ static inline void *bvec_kmap_local(struct bio_vec *bvec) return kmap_local_page(bvec->bv_page) + bvec->bv_offset; } +/** +* memcpy_from_bvec - copy data from a bvec +* @bvec: bvec to copy from +* +* Must be called on single-page bvecs only. +*/ +static inline void memcpy_from_bvec (char *to,struct bio_vec *bvec) +{ +memcpy_from_page (to, bvec->bv_page, bvec->bv_offset, bvec->bv_len); +} + +/** +* memcpy_to_bvec - copy data to a bvec +* @bvec: bvec to copy to +* +* Must be called on single-page bvecs only. +*/ +static inline void memcpy_to_bvec (struct bio_vec *bvec,const char *from) +{ +memcpy_to_page (bvec->bv_page, bvec->bv_offset, from, bvec->bv_len); +} + +/** +* memzero_bvec - zero all data in a bvec +* @bvec: bvec to zero +* +* Must be called on single-page bvecs only. +*/ +static inline void memzero_bvec (struct bio_vec *bvec) +{ +memzero_page (bvec->bv_page, bvec->bv_offset, bvec->bv_len); +} +#endif /* __LINUX_BVEC_H */
Signature · 6 lines
2.30.2 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 8/18 block: use memzero_page in zero_fill_bio
In reply to Christoph HellwigUse memzero_bvec to zero each segment in the bio instead of manually mapping and zeroing the data.
--- block/bio.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
block/bio.c +2 -7
@@ -495,16 +495,11 @@ EXPORT_SYMBOL(bio_kmalloc);void zero_fill_bio (struct bio *bio) { -unsigned long flags;struct bio_vec bv;struct bvec_iter iter; -bio_for_each_segment (bv, bio, iter) { -char *data =bvec_kmap_irq (&bv, &flags); -memset (data,0 , bv.bv_len); -flush_dcache_page (bv.bv_page); -bvec_kunmap_irq (data, &flags); - } +bio_for_each_segment (bv, bio, iter) +memzero_bvec (&bv); }EXPORT_SYMBOL (zero_fill_bio);
Signature · 6 lines
2.30.2 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 9/18 rbd: use memzero_bvec
In reply to Christoph HellwigUse memzero_bvec instead of reimplementing it.
--- drivers/block/rbd.c | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-)
drivers/block/rbd.c +2 -13
@@ -1219,24 +1219,13 @@ static void rbd_dev_mapping_clear(struct rbd_device *rbd_dev) rbd_dev->mapping.size =0 ; } -static void zero_bvec (struct bio_vec *bv) -{ -void *buf; -unsigned long flags; - - buf =bvec_kmap_irq (bv, &flags); -memset (buf,0 , bv->bv_len); -flush_dcache_page (bv->bv_page); -bvec_kunmap_irq (buf, &flags); -} -static void zero_bios (struct ceph_bio_iter *bio_pos,u32 off,u32 bytes) {struct ceph_bio_iter it = *bio_pos;ceph_bio_iter_advance (&it, off);ceph_bio_iter_advance_step (&it, bytes, ({ -zero_bvec (&bv); +memzero_bvec (&bv); })); }
@@ -1246,7 +1235,7 @@ static void zero_bvecs(struct ceph_bvec_iter *bvec_pos, u32 off, u32 bytes)ceph_bvec_iter_advance (&it, off);ceph_bvec_iter_advance_step (&it, bytes, ({ -zero_bvec (&bv); +memzero_bvec (&bv); })); }
Signature · 6 lines
2.30.2 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 10/18 dm-writecache: use bvec_kmap_local instead of bvec_kmap_irq
In reply to Christoph HellwigThere is no need to disable interrupts in bio_copy_block, and the local only mappings helps to avoid any sort of problems with stray writes into the bio data.
--- drivers/md/dm-writecache.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/md/dm-writecache.c +2 -3
@@ -1205,14 +1205,13 @@ static void memcpy_flushcache_optimized(void *dest, void *source, size_t size)static void bio_copy_block (struct dm_writecache *wc,struct bio *bio,void *data) {void *buf; -unsigned long flags;unsigned size;int rw =bio_data_dir (bio);unsigned remaining_size = wc->block_size; do { struct bio_vec bv = bio_iter_iovec(bio, bio->bi_iter); - buf = bvec_kmap_irq(&bv, &flags); + buf = bvec_kmap_local(&bv); size = bv.bv_len; if (unlikely(size > remaining_size)) size = remaining_size;
@@ -1230,7 +1229,7 @@ static void bio_copy_block(struct dm_writecache *wc, struct bio *bio, void *datamemcpy_flushcache_optimized (data, buf, size); } -bvec_kunmap_irq (buf, &flags); +kunmap_local (buf); data = (char *)data + size; remaining_size -= size;
Signature · 6 lines
2.30.2 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 11/18 ps3disk: use memcpy_{from,to}_bvec
In reply to Christoph HellwigUse the bvec helpers instead of open coding the copy.
--- drivers/block/ps3disk.c | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-)
drivers/block/ps3disk.c +3 -16
@@ -83,26 +83,13 @@ static void ps3disk_scatter_gather(struct ps3_storage_device *dev,unsigned int offset =0 ;struct req_iterator iter;struct bio_vec bvec; -unsigned int i =0 ; -size_t size; -void *buf;rq_for_each_segment (bvec, req, iter) { -unsigned long flags; -dev_dbg (&dev->sbd.core,"%s:%u: bio %u: %u sectors from %llu\n" , - __func__, __LINE__, i,bio_sectors (iter.bio), - iter.bio->bi_iter.bi_sector); - - size = bvec.bv_len; - buf =bvec_kmap_irq (&bvec, &flags);if (gather) -memcpy (dev->bounce_buf+offset, buf, size); +memcpy_from_bvec (dev->bounce_buf + offset, &bvec);else -memcpy (buf, dev->bounce_buf+offset, size); - offset += size; -flush_kernel_dcache_page (bvec.bv_page); -bvec_kunmap_irq (buf, &flags); - i++; +memcpy_to_bvec (&bvec, dev->bounce_buf + offset); + offset += bvec.bv_len; } }
Signature · 1 line
2.30.2
Re: [PATCH 11/18] ps3disk: use memcpy_{from,to}_bvec
Hi Christoph,
Quoted 6 lines · Christoph Hellwig
Use the bvec helpers instead of open coding the copy.
Signed-off-by Christoph Hellwig <[email protected]>--- drivers/block/ps3disk.c | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-)
I tested your patch set applied to v5.13-rc6 on PS3 and it seemed to be working OK.
I did some rsync's, some dd's, some fsck's, etc. If you have anything you could suggest that you think would exercise your changes I could try that also.
PATCH 12/18 block: remove bvec_kmap_irq and bvec_kunmap_irq
In reply to Christoph HellwigThese two helpers are entirely unused now.
--- include/linux/bio.h | 42 ------------------------------------------ 1 file changed, 42 deletions(-)
include/linux/bio.h +0 -42
@@ -5,7 +5,6 @@#ifndef __LINUX_BIO_H#define __LINUX_BIO_H -#include <linux/highmem.h> #include <linux/mempool.h> #include <linux/ioprio.h> /* struct bio, bio_vec and BIO_* flags are defined in blk_types.h */
@@ -519,47 +518,6 @@ static inline void bio_clone_blkg_association(struct bio *dst, struct bio *src) { }#endif /* CONFIG_BLK_CGROUP */ -#ifdef CONFIG_HIGHMEM -/* -* remember never ever reenable interrupts between a bvec_kmap_irq and -* bvec_kunmap_irq! -*/ -static inline char *bvec_kmap_irq (struct bio_vec *bvec,unsigned long *flags) -{ -unsigned long addr; - -/* -* might not be a highmem page, but the preempt/irq count -* balancing is a lot nicer this way -*/ -local_irq_save (*flags); - addr = (unsigned long )kmap_atomic (bvec->bv_page); - -BUG_ON (addr & ~PAGE_MASK ); - -return (char *) addr + bvec->bv_offset; -} - -static inline void bvec_kunmap_irq (char *buffer,unsigned long *flags) -{ -unsigned long ptr = (unsigned long ) buffer &PAGE_MASK ; - -kunmap_atomic ((void *) ptr); -local_irq_restore (*flags); -} - -#else -static inline char *bvec_kmap_irq (struct bio_vec *bvec,unsigned long *flags) -{ -return page_address (bvec->bv_page) + bvec->bv_offset; -} - -static inline void bvec_kunmap_irq (char *buffer,unsigned long *flags) -{ - *flags =0 ; -} -#endif -/* * BIO list management for use by remapping drivers (e.g. DM or MD) and loop. *
Signature · 1 line
2.30.2
PATCH 13/18 block: rewrite bio_copy_data_iter to use bvec_kmap_local and memcpy_to_bvec
In reply to Christoph HellwigUse the proper helpers instead of open coding the copy.
--- block/bio.c | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-)
block/bio.c +8 -20
@@ -1186,27 +1186,15 @@ EXPORT_SYMBOL(bio_advance);void bio_copy_data_iter (struct bio *dst,struct bvec_iter *dst_iter,struct bio *src,struct bvec_iter *src_iter) { -struct bio_vec src_bv, dst_bv; -void *src_p, *dst_p; -unsigned bytes; -while (src_iter->bi_size && dst_iter->bi_size) { - src_bv =bio_iter_iovec (src, *src_iter); - dst_bv =bio_iter_iovec (dst, *dst_iter); - - bytes =min (src_bv.bv_len, dst_bv.bv_len); - - src_p =kmap_atomic (src_bv.bv_page); - dst_p =kmap_atomic (dst_bv.bv_page); - -memcpy (dst_p + dst_bv.bv_offset, - src_p + src_bv.bv_offset, - bytes); - -kunmap_atomic (dst_p); -kunmap_atomic (src_p); - -flush_dcache_page (dst_bv.bv_page); +struct bio_vec src_bv =bio_iter_iovec (src, *src_iter); +struct bio_vec dst_bv =bio_iter_iovec (dst, *dst_iter); +unsigned int bytes =min (src_bv.bv_len, dst_bv.bv_len); +void *src_buf; + + src_buf =bvec_kmap_local (&src_bv); +memcpy_to_bvec (&dst_bv, src_buf); +kunmap_local (src_buf);bio_advance_iter_single (src, src_iter, bytes);bio_advance_iter_single (dst, dst_iter, bytes);
Signature · 6 lines
2.30.2 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 14/18 block: use memcpy_to_bvec in copy_to_high_bio_irq
In reply to Christoph HellwigUse memcpy_to_bvec instead of opencoding the logic.
--- block/bounce.c | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-)
block/bounce.c +2 -18
@@ -67,18 +67,6 @@ static __init int init_emergency_pool(void)__initcall (init_emergency_pool); -/* -* highmem version, map in to vec -*/ -static void bounce_copy_vec (struct bio_vec *to,unsigned char *vfrom) -{ -unsigned char *vto; - - vto =kmap_atomic (to->bv_page); -memcpy (vto + to->bv_offset, vfrom, to->bv_len); -kunmap_atomic (vto); -} -/* * Simple bounce buffer support for highmem pages. Depending on the * queue gfp mask set, *to may or may not be a highmem page. kmap it
@@ -86,7 +74,6 @@ static void bounce_copy_vec(struct bio_vec *to, unsigned char *vfrom)*/ static void copy_to_high_bio_irq (struct bio *to,struct bio *from) { -unsigned char *vfrom;struct bio_vec tovec, fromvec;struct bvec_iter iter;/*
@@ -104,11 +91,8 @@ static void copy_to_high_bio_irq(struct bio *to, struct bio *from)* been modified by the block layer, so use the original * copy, bounce_copy_vec already uses tovec->bv_len */ - vfrom =page_address (fromvec.bv_page) + - tovec.bv_offset; - -bounce_copy_vec (&tovec, vfrom); -flush_dcache_page (tovec.bv_page); +memcpy_to_bvec (&tovec,page_address (fromvec.bv_page) + + tovec.bv_offset); }bio_advance_iter (from, &from_iter, tovec.bv_len); }
Signature · 1 line
2.30.2
PATCH 15/18 block: use memcpy_from_bvec in bio_copy_kern_endio_read
In reply to Christoph HellwigUse memcpy_from_bvec instead of open coding the logic.
--- block/blk-map.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-map.c +1 -1
@@ -400,7 +400,7 @@ static void bio_copy_kern_endio_read(struct bio *bio)struct bvec_iter_all iter_all;bio_for_each_segment_all (bvec, bio, iter_all) { -memcpy (p,page_address (bvec->bv_page), bvec->bv_len); +memcpy_from_bvec (p, bvec); p += bvec->bv_len; }
Signature · 1 line
2.30.2
PATCH 16/18 block: use memcpy_from_bvec in __blk_queue_bounce
In reply to Christoph HellwigRewrite the actual bounce buffering loop in __blk_queue_bounce to that the memcpy_to_bvec helper can be used to perform the data copies.
--- block/bounce.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
block/bounce.c +7 -12
@@ -239,24 +239,19 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig)* because the 'bio' is single-page bvec. */ for (i =0 , to = bio->bi_io_vec; i < bio->bi_vcnt; to++, i++) { -struct page *page = to->bv_page; +struct page *bounce_page; -if (!PageHighMem (page)) +if (!PageHighMem (to->bv_page))continue ; - to->bv_page =mempool_alloc (&page_pool,GFP_NOIO ); -inc_zone_page_state (to->bv_page,NR_BOUNCE ); + bounce_page =mempool_alloc (&page_pool,GFP_NOIO ); +inc_zone_page_state (bounce_page,NR_BOUNCE );if (rw ==WRITE ) { -char *vto, *vfrom; - -flush_dcache_page (page); - - vto =page_address (to->bv_page) + to->bv_offset; - vfrom =kmap_atomic (page) + to->bv_offset; -memcpy (vto, vfrom, to->bv_len); -kunmap_atomic (vfrom); +flush_dcache_page (to->bv_page); +memcpy_from_bvec (page_address (bounce_page), to); } + to->bv_page = bounce_page; }trace_block_bio_bounce (*bio_orig);
Signature · 1 line
2.30.2
PATCH 17/18 block: use bvec_kmap_local in t10_pi_type1_{prepare,complete}
In reply to Christoph HellwigUsing local kmaps slightly reduces the chances to stray writes, and the bvec interface cleans up the code a little bit.
--- block/t10-pi.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
block/t10-pi.c +6 -10
@@ -147,11 +147,10 @@ static void t10_pi_type1_prepare(struct request *rq)break ;bip_for_each_vec (iv, bip, iter) { -void *p, *pmap;unsigned int j; +void *p; - pmap =kmap_atomic (iv.bv_page); - p = pmap + iv.bv_offset; + p =bvec_kmap_local (&iv);for (j =0 ; j < iv.bv_len; j += tuple_sz) {struct t10_pi_tuple *pi = p;
@@ -161,8 +160,7 @@ static void t10_pi_type1_prepare(struct request *rq) ref_tag++; p += tuple_sz; } - -kunmap_atomic (pmap); +kunmap_local (p); } bip->bip_flags |=BIP_MAPPED_INTEGRITY ;
@@ -195,11 +193,10 @@ static void t10_pi_type1_complete(struct request *rq, unsigned int nr_bytes)struct bvec_iter iter;bip_for_each_vec (iv, bip, iter) { -void *p, *pmap;unsigned int j; +void *p; - pmap =kmap_atomic (iv.bv_page); - p = pmap + iv.bv_offset; + p =bvec_kmap_local (&iv);for (j =0 ; j < iv.bv_len && intervals; j += tuple_sz) {struct t10_pi_tuple *pi = p;
@@ -210,8 +207,7 @@ static void t10_pi_type1_complete(struct request *rq, unsigned int nr_bytes) intervals--; p += tuple_sz; } - -kunmap_atomic (pmap); +kunmap_local (p); } } }
Signature · 6 lines
2.30.2 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 18/18 block: use bvec_kmap_local in bio_integrity_process
In reply to Christoph HellwigUsing local kmaps slightly reduces the chances to stray writes, and the bvec interface cleans up the code a little bit.
--- block/bio-integrity.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
block/bio-integrity.c +6 -8
@@ -172,18 +172,16 @@ static blk_status_t bio_integrity_process(struct bio *bio, iter.prot_buf = prot_buf;__bio_for_each_segment (bv, bio, bviter, *proc_iter) { -void *kaddr =kmap_atomic (bv.bv_page); +void *kaddr =bvec_kmap_local (&bv); - iter.data_buf = kaddr + bv.bv_offset; + iter.data_buf = kaddr; iter.data_size = bv.bv_len; - ret =proc_fn (&iter); -if (ret) { -kunmap_atomic (kaddr); -return ret; - } +kunmap_local (kaddr); + +if (ret) +break ; -kunmap_atomic (kaddr); }return ret; }
Signature · 6 lines
2.30.2 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Martin K. Petersen
Reviewed-byChristoph,
Quoted 4 lines
this series switches the core block layer code and all users of the existing bvec kmap helpers to use
kmap_local_page. Drivers that currently use open codedkmap_atomiccalls will converted in a follow on series.
Looks OK to me.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering