switch the block layer to use kmap_local_page
Subsystems (20)
Hi all,
this series switches the core block layer code and all users of the existing bvec kmap helpers to use kmap_local_page. Drivers that currently use open coded kmap_atomic calls will converted in a follow on series.
Diffstat:
arch/mips/include/asm/mach-rc32434/rb.h | 2 - block/bio-integrity.c | 14 ++++------ block/bio.c | 37 +++++++--------------------- block/blk-map.c | 2 - block/bounce.c | 35 ++++++-------------------- block/t10-pi.c | 16 ++++-------- drivers/block/ps3disk.c | 19 ++------------ drivers/block/rbd.c | 15 +---------- drivers/md/dm-writecache.c | 5 +-- include/linux/bio.h | 42 -------------------------------- include/linux/bvec.h | 27 ++++++++++++++++++-- include/linux/highmem.h | 4 +-- 12 files changed, 64 insertions(+), 154 deletions(-)
PATCH 1/16 mm: use kmap_local_page in memzero_page
No need for kmap_atomic here.
--- include/linux/highmem.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/highmem.h +2 -2
@@ -334,9 +334,9 @@ static inline void memcpy_to_page(struct page *page, size_t offset,static inline void memzero_page (struct page *page,size_t offset,size_t len) { -char *addr =kmap_atomic (page); +char *addr =kmap_local_page (page);memset (addr + offset,0 , len); -kunmap_atomic (addr); +kunmap_local (addr); }#endif /* _LINUX_HIGHMEM_H */
Signature · 1 line
2.30.2
Re: [PATCH 01/16] mm: use kmap_local_page in memzero_page
Quoted 3 lines · Christoph Hellwig
No need for
kmap_atomichere.Signed-off-by Christoph Hellwig <[email protected]>
Looks good.
PATCH 2/16 MIPS: don't include <linux/genhd.h> in <asm/mach-rc32434/rb.h>
In reply to Christoph HellwigThere is no need to include genhd.h from a random arch header, and not doing so prevents the possibility for nasty include loops.
--- arch/mips/include/asm/mach-rc32434/rb.h | 2 -- 1 file changed, 2 deletions(-)
arch/mips/include/asm/mach-rc32434/rb.h +0 -2
@@ -7,8 +7,6 @@#ifndef __ASM_RC32434_RB_H#define __ASM_RC32434_RB_H -#include <linux/genhd.h> -#define REGBASE 0x18000000#define IDT434_REG_BASE ((volatile void *) KSEG1ADDR(REGBASE))#define UART0BASE 0x58000
Signature · 1 line
2.30.2
Re: [PATCH 02/16] MIPS: don't include <linux/genhd.h> in <asm/mach-rc32434/rb.h>
Quoted 13 lines · Christoph Hellwig
There is no need to include
genhd.hfrom a random arch header, and not doing so prevents the possibility for nasty include loops.Signed-off-by Christoph Hellwig <[email protected]>--- arch/mips/include/asm/mach-rc32434/rb.h | 2 -- 1 file changed, 2 deletions(-)arch/mips/include/asm/mach-rc32434/rb.h +0
-0
arch/mips/include/asm/mach-rc32434/rb.h line 12
Quoted 8 lines
arch/mips/include/asm/mach-rc32434/rb.h +0
-2@@ -7,8 +7,6 @@#ifndef __ASM_RC32434_RB_H#define __ASM_RC32434_RB_H -#include <linux/genhd.h> -#define REGBASE 0x18000000#define IDT434_REG_BASE ((volatile void *) KSEG1ADDR(REGBASE))#define UART0BASE 0x58000
PATCH 3/16 bvec: fix the include guards for bvec.h
In reply to Christoph HellwigFix the include guards to match the file naming.
--- include/linux/bvec.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/bvec.h +3 -3
@@ -4,8 +4,8 @@* * Copyright (C) 2001 Ming Lei <[email protected]> */ -#ifndef __LINUX_BVEC_ITER_H -#define __LINUX_BVEC_ITER_H +#ifndef __LINUX_BVEC_H +#define __LINUX_BVEC_H#include <linux/bug.h> #include <linux/errno.h>
@@ -183,4 +183,4 @@ static inline void bvec_advance(const struct bio_vec *bvec, } } -#endif /* __LINUX_BVEC_ITER_H */ +#endif /* __LINUX_BVEC_H */
Signature · 1 line
2.30.2
Re: [PATCH 03/16] bvec: fix the include guards for bvec.h
Quoted 1 line · Christoph Hellwig
Fix the include guards to match the file naming.
Re: [PATCH 03/16] bvec: fix the include guards for bvec.h
In reply to Christoph HellwigQuoted 3 lines · Christoph Hellwig
Fix the include guards to match the file naming.
Signed-off-by Christoph Hellwig <[email protected]>
Looks good.
PATCH 4/16 bvec: add a bvec_kmap_local helper
In reply to Christoph HellwigAdd a helper to call kmap_local_page on a bvec. There is no need for an unmap helper given that kunmap_local accept any address in the mapped page.
--- include/linux/bvec.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/bvec.h +6 -0
@@ -7,6 +7,7 @@#ifndef __LINUX_BVEC_H#define __LINUX_BVEC_H +#include <linux/highmem.h> #include <linux/bug.h> #include <linux/errno.h> #include <linux/limits.h>
@@ -183,4 +184,9 @@ static inline void bvec_advance(const struct bio_vec *bvec, } } +static inline void *bvec_kmap_local (struct bio_vec *bvec) +{ +return kmap_local_page (bvec->bv_page) + bvec->bv_offset; +} +#endif /* __LINUX_BVEC_H */
Signature · 1 line
2.30.2
Re: [PATCH 04/16] bvec: add a bvec_kmap_local helper
Quoted 5 lines · Christoph Hellwig
Add a helper to call
kmap_local_pageon a bvec. There is no need for an unmap helper given thatkunmap_localaccept any address in the mapped page.Signed-off-by Christoph Hellwig <[email protected]>
Looks good.
Ilya Dryomov
maintainerRe: [PATCH 04/16] bvec: add a bvec_kmap_local helper
In reply to Christoph HellwigQuoted 23 lines · Christoph Hellwig
Add a helper to call
kmap_local_pageon a bvec. There is no need for an unmap helper given thatkunmap_localaccept any address in the mapped page.Signed-off-by Christoph Hellwig <[email protected]>--- include/linux/bvec.h | 6 ++++++ 1 file changed, 6 insertions(+)include/linux/bvec.h +1
-0@@ -7,6 +7,7 @@#ifndef __LINUX_BVEC_H#define __LINUX_BVEC_H +#include <linux/highmem.h> #include <linux/bug.h> #include <linux/errno.h> #include <linux/limits.h>
include/linux/bvec.h static inline void bvec_advance(const struct bio_vec *bvec, · line 192
Quoted 9 lines
include/linux/bvec.h +5
-0@@ -183,4 +184,9 @@ static inline void bvec_advance(const struct bio_vec *bvec, } } +static inline void *bvec_kmap_local (struct bio_vec *bvec) +{ +return kmap_local_page (bvec->bv_page) + bvec->bv_offset; +} +#endif /* __LINUX_BVEC_H */
Might be useful to add the second sentence of the commit message as a comment for bvec_kmap_local(). It could be expanded to mention the single-page bvec caveat too.
Thanks,
Ilya
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 5/16 bvec: add memcpy_{from,to}_bvec and memzero_bvec helper
In reply to Christoph HellwigAdd helpers to perform common memory operation on a bvec.
--- include/linux/bvec.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
include/linux/bvec.h +15 -0
@@ -189,4 +189,19 @@ static inline void *bvec_kmap_local(struct bio_vec *bvec) return kmap_local_page(bvec->bv_page) + bvec->bv_offset; } +static inline void memcpy_from_bvec (char *to,struct bio_vec *bvec) +{ +memcpy_from_page (to, bvec->bv_page, bvec->bv_offset, bvec->bv_len); +} + +static inline void memcpy_to_bvec (struct bio_vec *bvec,const char *from) +{ +memcpy_to_page (bvec->bv_page, bvec->bv_offset, from, bvec->bv_len); +} + +static inline void memzero_bvec (struct bio_vec *bvec) +{ +memzero_page (bvec->bv_page, bvec->bv_offset, bvec->bv_len); +} +#endif /* __LINUX_BVEC_H */
Signature · 1 line
2.30.2
Re: [PATCH 05/16] bvec: add memcpy_{from,to}_bvec and memzero_bvec helper
Quoted 3 lines · Christoph Hellwig
Add helpers to perform common memory operation on a bvec.
Signed-off-by Christoph Hellwig <[email protected]>
Looks good.
PATCH 6/16 block: use memzero_page in zero_fill_bio
In reply to Christoph HellwigUse memzero_bvec to zero each segment in the bio instead of manually mapping and zeroing the data.
--- block/bio.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
block/bio.c +2 -7
@@ -495,16 +495,11 @@ EXPORT_SYMBOL(bio_kmalloc);void zero_fill_bio (struct bio *bio) { -unsigned long flags;struct bio_vec bv;struct bvec_iter iter; -bio_for_each_segment (bv, bio, iter) { -char *data =bvec_kmap_irq (&bv, &flags); -memset (data,0 , bv.bv_len); -flush_dcache_page (bv.bv_page); -bvec_kunmap_irq (data, &flags); - } +bio_for_each_segment (bv, bio, iter) +memzero_bvec (&bv); }EXPORT_SYMBOL (zero_fill_bio);
Signature · 1 line
2.30.2
Re: [PATCH 06/16] block: use memzero_page in zero_fill_bio
Quoted 4 lines · Christoph Hellwig
Use
memzero_bvecto zero each segment in the bio instead of manually mapping and zeroing the data.Signed-off-by Christoph Hellwig <[email protected]>
Looks good.
PATCH 7/16 rbd: use memzero_bvec
In reply to Christoph HellwigUse memzero_bvec instead of reimplementing it.
--- drivers/block/rbd.c | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-)
drivers/block/rbd.c +2 -13
@@ -1219,24 +1219,13 @@ static void rbd_dev_mapping_clear(struct rbd_device *rbd_dev) rbd_dev->mapping.size =0 ; } -static void zero_bvec (struct bio_vec *bv) -{ -void *buf; -unsigned long flags; - - buf =bvec_kmap_irq (bv, &flags); -memset (buf,0 , bv->bv_len); -flush_dcache_page (bv->bv_page); -bvec_kunmap_irq (buf, &flags); -} -static void zero_bios (struct ceph_bio_iter *bio_pos,u32 off,u32 bytes) {struct ceph_bio_iter it = *bio_pos;ceph_bio_iter_advance (&it, off);ceph_bio_iter_advance_step (&it, bytes, ({ -zero_bvec (&bv); +memzero_bvec (&bv); })); }
@@ -1246,7 +1235,7 @@ static void zero_bvecs(struct ceph_bvec_iter *bvec_pos, u32 off, u32 bytes)ceph_bvec_iter_advance (&it, off);ceph_bvec_iter_advance_step (&it, bytes, ({ -zero_bvec (&bv); +memzero_bvec (&bv); })); }
Signature · 1 line
2.30.2
Re: [PATCH 07/16] rbd: use memzero_bvec
Quoted 39 lines · Christoph Hellwig
Use
memzero_bvecinstead of reimplementing it.Signed-off-by Christoph Hellwig <[email protected]>--- drivers/block/rbd.c | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-)drivers/block/rbd.c +1
-12@@ -1219,24 +1219,13 @@ static void rbd_dev_mapping_clear(struct rbd_device *rbd_dev) rbd_dev->mapping.size =0 ; } -static void zero_bvec (struct bio_vec *bv) -{ -void *buf; -unsigned long flags; - - buf =bvec_kmap_irq (bv, &flags); -memset (buf,0 , bv->bv_len); -flush_dcache_page (bv->bv_page); -bvec_kunmap_irq (buf, &flags); -} -static void zero_bios (struct ceph_bio_iter *bio_pos,u32 off,u32 bytes) {struct ceph_bio_iter it = *bio_pos;ceph_bio_iter_advance (&it, off);ceph_bio_iter_advance_step (&it, bytes, ({ -zero_bvec (&bv); +memzero_bvec (&bv); })); }
drivers/block/rbd.c static void zero_bvecs(struct ceph_bvec_iter *bvec_pos, u32 off, u32 bytes) · line 1241
Quoted 8 lines
drivers/block/rbd.c +1
-1@@ -1246,7 +1235,7 @@ static void zero_bvecs(struct ceph_bvec_iter *bvec_pos, u32 off, u32 bytes)ceph_bvec_iter_advance (&it, off);ceph_bvec_iter_advance_step (&it, bytes, ({ -zero_bvec (&bv); +memzero_bvec (&bv); })); }
Ira already brought up the fact that this conversion drops flush_dcache_page() calls throughout. Other than that:
Thanks,
Ilya
PATCH 8/16 dm-writecache: use bvec_kmap_local instead of bvec_kmap_irq
In reply to Christoph HellwigThere is no need to disable interrupts in bio_copy_block, and the local only mappings helps to avoid any sort of problems with stray writes into the bio data.
--- drivers/md/dm-writecache.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/md/dm-writecache.c +2 -3
@@ -1205,14 +1205,13 @@ static void memcpy_flushcache_optimized(void *dest, void *source, size_t size)static void bio_copy_block (struct dm_writecache *wc,struct bio *bio,void *data) {void *buf; -unsigned long flags;unsigned size;int rw =bio_data_dir (bio);unsigned remaining_size = wc->block_size; do { struct bio_vec bv = bio_iter_iovec(bio, bio->bi_iter); - buf = bvec_kmap_irq(&bv, &flags); + buf = bvec_kmap_local(&bv); size = bv.bv_len; if (unlikely(size > remaining_size)) size = remaining_size;
@@ -1230,7 +1229,7 @@ static void bio_copy_block(struct dm_writecache *wc, struct bio *bio, void *datamemcpy_flushcache_optimized (data, buf, size); } -bvec_kunmap_irq (buf, &flags); +kunmap_local (buf); data = (char *)data + size; remaining_size -= size;
Signature · 1 line
2.30.2
Re: [PATCH 08/16] dm-writecache: use bvec_kmap_local instead of bvec_kmap_irq
Quoted 21 lines · Christoph Hellwig
drivers/md/dm-writecache.c +1
-2@@ -1205,14 +1205,13 @@ static void memcpy_flushcache_optimized(void *dest, void *source, size_t size)static void bio_copy_block (struct dm_writecache *wc,struct bio *bio,void *data) {void *buf; -unsigned long flags;unsigned size;int rw =bio_data_dir (bio);unsigned remaining_size = wc->block_size; do { struct bio_vec bv = bio_iter_iovec(bio, bio->bi_iter); - buf = bvec_kmap_irq(&bv, &flags); + buf = bvec_kmap_local(&bv); size = bv.bv_len; if (unlikely(size > remaining_size)) size = remaining_size;
drivers/md/dm-writecache.c static void bio_copy_block(struct dm_writecache *wc, struct bio *bio, void *data · line 1235
Quoted 8 lines
drivers/md/dm-writecache.c +1
-1@@ -1230,7 +1229,7 @@ static void bio_copy_block(struct dm_writecache *wc, struct bio *bio, void *datamemcpy_flushcache_optimized (data, buf, size); } -bvec_kunmap_irq (buf, &flags); +kunmap_local (buf); data = (char *)data + size; remaining_size -= size;
From one of the functions called by kunmap_local():
unsigned long addr = (unsigned long) vaddr & PAGE_MASK;
This won't work well if bvec->bv_offset >= PAGE_SIZE I assume?
Thanks,
Bart.
Re: [PATCH 08/16] dm-writecache: use bvec_kmap_local instead of bvec_kmap_irq
Quoted 5 lines · Bart Van Assche
Quoted 1 line
From one of the functions called by
kunmap_local():unsigned long addr = (unsigned long) vaddr &
PAGE_MASK;This won't work well if
bvec->bv_offset>=PAGE_SIZEI assume?
It won't indeed. Both the existing and new helpers operate on single page bvecs only, and all callers only use those. I should have probably mentioned that in the cover letter and documented the assumptions in the code, though.
PATCH 9/16 ps3disk: use memcpy_{from,to}_bvec
In reply to Christoph HellwigUse the bvec helpers instead of open coding the copy.
--- drivers/block/ps3disk.c | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-)
drivers/block/ps3disk.c +3 -16
@@ -84,26 +84,13 @@ static void ps3disk_scatter_gather(struct ps3_storage_device *dev,unsigned int offset =0 ;struct req_iterator iter;struct bio_vec bvec; -unsigned int i =0 ; -size_t size; -void *buf;rq_for_each_segment (bvec, req, iter) { -unsigned long flags; -dev_dbg (&dev->sbd.core,"%s:%u: bio %u: %u sectors from %llu\n" , - __func__, __LINE__, i,bio_sectors (iter.bio), - iter.bio->bi_iter.bi_sector); - - size = bvec.bv_len; - buf =bvec_kmap_irq (&bvec, &flags);if (gather) -memcpy (dev->bounce_buf+offset, buf, size); +memcpy_from_bvec (dev->bounce_buf + offset, &bvec);else -memcpy (buf, dev->bounce_buf+offset, size); - offset += size; -flush_kernel_dcache_page (bvec.bv_page); -bvec_kunmap_irq (buf, &flags); - i++; +memcpy_to_bvec (&bvec, dev->bounce_buf + offset); + offset += bvec.bv_len; } }
Signature · 1 line
2.30.2
Re: [PATCH 09/16] ps3disk: use memcpy_{from,to}_bvec
Quoted 6 lines · Christoph Hellwig
drivers/block/ps3disk.c +0
-4rq_for_each_segment (bvec, req, iter) { -unsigned long flags; -dev_dbg (&dev->sbd.core,"%s:%u: bio %u: %u sectors from %llu\n" , - __func__, __LINE__, i,bio_sectors (iter.bio), - iter.bio->bi_iter.bi_sector);
drivers/block/ps3disk.c static void ps3disk_scatter_gather(struct ps3_storage_device *dev, · line 104
Quoted 10 lines
drivers/block/ps3disk.c +1
-7- - size = bvec.bv_len; - buf =bvec_kmap_irq (&bvec, &flags);if (gather) -memcpy (dev->bounce_buf+offset, buf, size); +memcpy_from_bvec (dev->bounce_buf + offset, &bvec);else -memcpy (buf, dev->bounce_buf+offset, size); - offset += size; -flush_kernel_dcache_page (bvec.bv_page);
I'm still not 100% sure that these flushes are needed but the are not no-ops on every arch. Would it be best to preserve them after the memcpy_to/from_bvec()?
Same thing in patch 11 and 14.
Ira
Re: [PATCH 09/16] ps3disk: use memcpy_{from,to}_bvec
Quoted 4 lines · Ira Weiny
I'm still not 100% sure that these flushes are needed but the are not no-ops on every arch. Would it be best to preserve them after the memcpy_to/from_bvec()?
Same thing in patch 11 and 14.
To me it seems kunmap_local should basically always call the equivalent of flush_kernel_dcache_page. parisc does this through kunmap_flush_on_unmap, but none of the other architectures with VIVT caches or other coherency issues does.
Does anyone have a history or other insights here?
Re: [PATCH 09/16] ps3disk: use memcpy_{from,to}_bvec
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Ira Weiny
I'm still not 100% sure that these flushes are needed but the are not no-ops on every arch. Would it be best to preserve them after the memcpy_to/from_bvec()?
Quoted 8 lines
Quoted 1 line
Same thing in patch 11 and 14.
To me it seems
kunmap_localshould basically always call the equivalent offlush_kernel_dcache_page. parisc does this throughkunmap_flush_on_unmap, but none of the other architectures with VIVT caches or other coherency issues does.Does anyone have a history or other insights here?
I went digging into the current callers of flush_kernel_dcache_page() other than this one. To see if adding kunmap_flush_on_unmap() to the other arch's would cause any problems.
In particular this call site stood out because it is not always called?!?!?!?
void sg_miter_stop(struct sg_mapping_iter *miter)
{
...
if ((miter->__flags & SG_MITER_TO_SG) &&
!PageSlab(miter->page))
flush_kernel_dcache_page(miter->page);...
}
Looking at
3d77b50c5874 lib/scatterlist.c: don't flush_kernel_dcache_page on slab page[1]
It seems the restrictions they are quoting for the page are completely out of date. I don't see any current way for a VM_BUG_ON() to be triggered. So is this code really necessary?
More recently this was added:
7e34e0bbc644 crypto: omap-crypto - fix userspace copied buffer access
I'm CC'ing Tero and Herbert to see why they added the SLAB check.
Then we have interesting comments like this...
...
/* This can go away once MIPS implements * flush_kernel_dcache_page */ flush_dcache_page(miter->page);
...
And some users optimizing.
...
/* discard mappings */ if (direction == DMA_FROM_DEVICE) flush_kernel_dcache_page(sg_page(sg));
...
The uses in fs/exec.c are the most straight forward and can simply rely on the kunmap() code to replace the call.
In conclusion I don't see a lot of reason to not define kunmap_flush_on_unmap() on arm, csky, mips, nds32, and sh... Then remove all the flush_kernel_dcache_page() call sites and the documentation...
Something like [2] below... Completely untested of course...
Ira
[1] commit 3d77b50c5874b7e923be946ba793644f82336b75
Author: Ming Lei <[email protected]> Date: Thu Oct 31 16:34:17 2013 -0700
lib/scatterlist.c: don't flush_kernel_dcache_page on slab page
Commit b1adaf65ba03 ("[SCSI] block: add sg buffer copy helper functions") introduces two sg buffer copy helpers, and calls flush_kernel_dcache_page() on pages in SG list after these pages are written to.
Unfortunately, the commit may introduce a potential bug:
- Before sending some SCSI commands, kmalloc() buffer may be passed to block layper, so flush_kernel_dcache_page() can see a slab page finally
- According to cachetlb.txt, flush_kernel_dcache_page() is only called on "a user page", which surely can't be a slab page.
- ARCH's implementation of flush_kernel_dcache_page() may use page mapping information to do optimization so page_mapping() will see the slab page, then VM_BUG_ON() is triggered.
Aaro Koskinen reported the bug on ARM/kirkwood when DEBUG_VM is enabled, and this patch fixes the bug by adding test of '!PageSlab(miter->page)' before calling flush_kernel_dcache_page().
[2]
From 70b537c31d16c2a5e4e92c35895e8c59303bcbef Mon Sep 17 00:00:00 2001
From: Ira Weiny <[email protected]>
Date: Fri, 11 Jun 2021 18:24:27 -0700
Subject: [PATCH] COMPLETELY UNTESTED: highmem: Remove direct calls to flush_kernel_dcache_page
When to call flush_kernel_dcache_page() is confusing and inconsistent. For architectures which may need to do something the core kmap code should be leveraged to handle this when direct kernel access is needed.
Like parisc define kunmap_flush_on_unmap() to be called when pages are unmapped on arm, csky, mpis, nds32, and sh.
Remove all direct calls to flush_kernel_dcache_page() and let the kunmap() code do this for the users.
--- Documentation/core-api/cachetlb.rst | 13 ------------- arch/arm/include/asm/cacheflush.h | 6 ++++++ arch/csky/abiv1/inc/abi/cacheflush.h | 6 ++++++ arch/mips/include/asm/cacheflush.h | 6 ++++++ arch/nds32/include/asm/cacheflush.h | 6 ++++++ arch/sh/include/asm/cacheflush.h | 6 ++++++ drivers/crypto/omap-crypto.c | 3 --- drivers/mmc/host/mmc_spi.c | 3 --- drivers/scsi/aacraid/aachba.c | 1 - fs/exec.c | 3 --- include/linux/highmem.h | 3 --- lib/scatterlist.c | 4 ---- 12 files changed, 30 insertions(+), 30 deletions(-)
Documentation/core-api/cachetlb.rst +0 -13
@@ -351,19 +351,6 @@ maps this page at its virtual address. architectures). For incoherent architectures, it should flush the cache of the page at vmaddr. - ``void flush_kernel_dcache_page(struct page *page)`` - - When the kernel needs to modify a user page is has obtained - with kmap, it calls this function after all modifications are - complete (but before kunmapping it) to bring the underlying - page up to date. It is assumed here that the user has no - incoherent cached copies (i.e. the original page was obtained - from a mechanism like get_user_pages()). The default - implementation is a nop and should remain so on all coherent - architectures. On incoherent architectures, this should flush - the kernel cache for page (using page_address(page)). - - ``void flush_icache_range(unsigned long start, unsigned long end)`` When the kernel stores into addresses that it will execute
arch/arm/include/asm/cacheflush.h +6 -0
@@ -315,6 +315,12 @@ static inline void flush_anon_page(struct vm_area_struct *vma,#define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE extern void flush_kernel_dcache_page (struct page *); +#define ARCH_HAS_FLUSH_ON_KUNMAP +static inline void kunmap_flush_on_unmap (void *addr) +{ +flush_kernel_dcache_page_addr (addr); +} +#define flush_dcache_mmap_lock (mapping) xa_lock_irq(&mapping->i_pages)#define flush_dcache_mmap_unlock (mapping) xa_unlock_irq(&mapping->i_pages)
arch/csky/abiv1/inc/abi/cacheflush.h +6 -0
@@ -17,6 +17,12 @@ extern void flush_dcache_page(struct page *);#define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE extern void flush_kernel_dcache_page (struct page *); +#define ARCH_HAS_FLUSH_ON_KUNMAP +static inline void kunmap_flush_on_unmap (void *addr) +{ +flush_kernel_dcache_page_addr (addr); +} +#define flush_dcache_mmap_lock (mapping) xa_lock_irq(&mapping->i_pages)#define flush_dcache_mmap_unlock (mapping) xa_unlock_irq(&mapping->i_pages)
arch/mips/include/asm/cacheflush.h +6 -0
@@ -132,6 +132,12 @@ static inline void flush_kernel_dcache_page(struct page *page)flush_dcache_page (page); } +#define ARCH_HAS_FLUSH_ON_KUNMAP +static inline void kunmap_flush_on_unmap (void *addr) +{ +flush_kernel_dcache_page_addr (addr); +} +/* * For now flush_kernel_vmap_range and invalidate_kernel_vmap_range both do a * cache writeback and invalidate operation.
arch/nds32/include/asm/cacheflush.h +6 -0
@@ -43,6 +43,12 @@ void invalidate_kernel_vmap_range(void *addr, int size);#define flush_dcache_mmap_lock (mapping) xa_lock_irq(&(mapping)->i_pages)#define flush_dcache_mmap_unlock (mapping) xa_unlock_irq(&(mapping)->i_pages) +#define ARCH_HAS_FLUSH_ON_KUNMAP +static inline void kunmap_flush_on_unmap (void *addr) +{ +flush_kernel_dcache_page_addr (addr); +} +#else void flush_icache_user_page (struct vm_area_struct *vma,struct page *page,unsigned long addr,int len);
arch/sh/include/asm/cacheflush.h +6 -0
@@ -78,6 +78,12 @@ static inline void flush_kernel_dcache_page(struct page *page)flush_dcache_page (page); } +#define ARCH_HAS_FLUSH_ON_KUNMAP +static inline void kunmap_flush_on_unmap (void *addr) +{ +flush_kernel_dcache_page_addr (addr); +} +extern void copy_to_user_page (struct vm_area_struct *vma,struct page *page,unsigned long vaddr,void *dst,const void *src,unsigned long len);
drivers/crypto/omap-crypto.c +0 -3
@@ -183,9 +183,6 @@ static void omap_crypto_copy_data(struct scatterlist *src,memcpy (dstb, srcb, amt); -if (!PageSlab (sg_page (dst))) -flush_kernel_dcache_page (sg_page (dst)); -kunmap_atomic (srcb);kunmap_atomic (dstb);
drivers/mmc/host/mmc_spi.c +0 -3
@@ -947,9 +947,6 @@ mmc_spi_data_do(struct mmc_spi_host *host, struct mmc_command *cmd,break ; } -/* discard mappings */ -if (direction ==DMA_FROM_DEVICE ) -flush_kernel_dcache_page (sg_page (sg));kunmap (sg_page (sg));if (dma_dev)dma_unmap_page (dma_dev, dma_addr,PAGE_SIZE , dir);
drivers/scsi/aacraid/aachba.c +0 -1
@@ -25,7 +25,6 @@#include <linux/completion.h> #include <linux/blkdev.h> #include <linux/uaccess.h> -#include <linux/highmem.h> /* For flush_kernel_dcache_page */ #include <linux/module.h> #include <asm/unaligned.h>
fs/exec.c +0 -3
@@ -577,7 +577,6 @@ static int copy_strings(int argc, struct user_arg_ptr argv, }if (kmapped_page) { -flush_kernel_dcache_page (kmapped_page);kunmap (kmapped_page);put_arg_page (kmapped_page); }
@@ -595,7 +594,6 @@ static int copy_strings(int argc, struct user_arg_ptr argv, ret =0 ; out:if (kmapped_page) { -flush_kernel_dcache_page (kmapped_page);kunmap (kmapped_page);put_arg_page (kmapped_page); }
@@ -637,7 +635,6 @@ int copy_string_kernel(const char *arg, struct linux_binprm *bprm) kaddr =kmap_atomic (page);flush_arg_page (bprm, pos &PAGE_MASK , page);memcpy (kaddr +offset_in_page (pos), arg, bytes_to_copy); -flush_kernel_dcache_page (page);kunmap_atomic (kaddr);put_arg_page (page); }
include/linux/highmem.h +0 -3
@@ -131,9 +131,6 @@ static inline void flush_anon_page(struct vm_area_struct *vma, struct page *page#endif #ifndef ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE -static inline void flush_kernel_dcache_page (struct page *page) -{ -}static inline void flush_kernel_vmap_range (void *vaddr,int size) { }
lib/scatterlist.c +0 -4
@@ -887,10 +887,6 @@ void sg_miter_stop(struct sg_mapping_iter *miter) miter->__offset += miter->consumed; miter->__remaining -= miter->consumed; -if ((miter->__flags &SG_MITER_TO_SG ) && - !PageSlab (miter->page)) -flush_kernel_dcache_page (miter->page); -if (miter->__flags &SG_MITER_ATOMIC ) {WARN_ON_ONCE (preemptible ());kunmap_atomic (miter->addr);
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Herbert Xu
maintainerRe: [PATCH 09/16] ps3disk: use memcpy_{from,to}_bvec
Quoted 6 lines · Ira Weiny
More recently this was added:
7e34e0bbc644crypto: omap-crypto - fix userspace copied buffer accessI'm CC'ing Tero and Herbert to see why they added the SLAB check.
Probably because the generic Crypto API has the same check. This all goes back to
commit 4f3e797ad07d52d34983354a77b365dfcd48c1b4
Author: Herbert Xu <[email protected]> Date: Mon Feb 9 14:22:14 2009 +1100
crypto: scatterwalk - Avoid flush_dcache_page on slab pages
It's illegal to call flush_dcache_page on slab pages on a number of architectures. So this patch avoids doing so if PageSlab is true.
In future we can move the flush_dcache_page call to those page cache users that actually need it.
But I can't find any emails discussing this so let me ask Dave directly and see if he can tell us what the issue was or might have been.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
PATCH 10/16 block: remove bvec_kmap_irq and bvec_kunmap_irq
In reply to Christoph HellwigThese two helpers are entirely unused now.
--- include/linux/bio.h | 42 ------------------------------------------ 1 file changed, 42 deletions(-)
include/linux/bio.h +0 -42
@@ -5,7 +5,6 @@#ifndef __LINUX_BIO_H#define __LINUX_BIO_H -#include <linux/highmem.h> #include <linux/mempool.h> #include <linux/ioprio.h> /* struct bio, bio_vec and BIO_* flags are defined in blk_types.h */
@@ -523,47 +522,6 @@ static inline void bio_clone_blkg_association(struct bio *dst, struct bio *src) { }#endif /* CONFIG_BLK_CGROUP */ -#ifdef CONFIG_HIGHMEM -/* -* remember never ever reenable interrupts between a bvec_kmap_irq and -* bvec_kunmap_irq! -*/ -static inline char *bvec_kmap_irq (struct bio_vec *bvec,unsigned long *flags) -{ -unsigned long addr; - -/* -* might not be a highmem page, but the preempt/irq count -* balancing is a lot nicer this way -*/ -local_irq_save (*flags); - addr = (unsigned long )kmap_atomic (bvec->bv_page); - -BUG_ON (addr & ~PAGE_MASK ); - -return (char *) addr + bvec->bv_offset; -} - -static inline void bvec_kunmap_irq (char *buffer,unsigned long *flags) -{ -unsigned long ptr = (unsigned long ) buffer &PAGE_MASK ; - -kunmap_atomic ((void *) ptr); -local_irq_restore (*flags); -} - -#else -static inline char *bvec_kmap_irq (struct bio_vec *bvec,unsigned long *flags) -{ -return page_address (bvec->bv_page) + bvec->bv_offset; -} - -static inline void bvec_kunmap_irq (char *buffer,unsigned long *flags) -{ - *flags =0 ; -} -#endif -/* * BIO list management for use by remapping drivers (e.g. DM or MD) and loop. *
Signature · 1 line
2.30.2
PATCH 11/16 block: rewrite bio_copy_data_iter to use bvec_kmap_local and memcpy_to_bvec
In reply to Christoph HellwigUse the proper helpers instead of open coding the copy.
--- block/bio.c | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-)
block/bio.c +8 -20
@@ -1186,27 +1186,15 @@ EXPORT_SYMBOL(bio_advance);void bio_copy_data_iter (struct bio *dst,struct bvec_iter *dst_iter,struct bio *src,struct bvec_iter *src_iter) { -struct bio_vec src_bv, dst_bv; -void *src_p, *dst_p; -unsigned bytes; -while (src_iter->bi_size && dst_iter->bi_size) { - src_bv =bio_iter_iovec (src, *src_iter); - dst_bv =bio_iter_iovec (dst, *dst_iter); - - bytes =min (src_bv.bv_len, dst_bv.bv_len); - - src_p =kmap_atomic (src_bv.bv_page); - dst_p =kmap_atomic (dst_bv.bv_page); - -memcpy (dst_p + dst_bv.bv_offset, - src_p + src_bv.bv_offset, - bytes); - -kunmap_atomic (dst_p); -kunmap_atomic (src_p); - -flush_dcache_page (dst_bv.bv_page); +struct bio_vec src_bv =bio_iter_iovec (src, *src_iter); +struct bio_vec dst_bv =bio_iter_iovec (dst, *dst_iter); +unsigned int bytes =min (src_bv.bv_len, dst_bv.bv_len); +void *src_buf; + + src_buf =bvec_kmap_local (&src_bv); +memcpy_to_bvec (&dst_bv, src_buf); +kunmap_local (src_buf);bio_advance_iter_single (src, src_iter, bytes);bio_advance_iter_single (dst, dst_iter, bytes);
Signature · 1 line
2.30.2
PATCH 12/16 block: use memcpy_to_bvec in copy_to_high_bio_irq
In reply to Christoph HellwigUse memcpy_to_bvec instead of opencoding the logic.
--- block/bounce.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-)
block/bounce.c +1 -13
@@ -67,18 +67,6 @@ static __init int init_emergency_pool(void)__initcall (init_emergency_pool); -/* -* highmem version, map in to vec -*/ -static void bounce_copy_vec (struct bio_vec *to,unsigned char *vfrom) -{ -unsigned char *vto; - - vto =kmap_atomic (to->bv_page); -memcpy (vto + to->bv_offset, vfrom, to->bv_len); -kunmap_atomic (vto); -} -/* * Simple bounce buffer support for highmem pages. Depending on the * queue gfp mask set, *to may or may not be a highmem page. kmap it
@@ -107,7 +95,7 @@ static void copy_to_high_bio_irq(struct bio *to, struct bio *from) vfrom =page_address (fromvec.bv_page) + tovec.bv_offset; -bounce_copy_vec (&tovec, vfrom); +memcpy_to_bvec (&tovec, vfrom);flush_dcache_page (tovec.bv_page); }bio_advance_iter (from, &from_iter, tovec.bv_len);
Signature · 1 line
2.30.2
Re: [PATCH 12/16] block: use memcpy_to_bvec in copy_to_high_bio_irq
Quoted 3 lines · Christoph Hellwig
Use
memcpy_to_bvecinstead of opencoding the logic.Signed-off-by Christoph Hellwig <[email protected]>
Looks good.
PATCH 13/16 block: use memcpy_from_bvec in bio_copy_kern_endio_read
In reply to Christoph HellwigUse memcpy_from_bvec instead of open coding the logic.
--- block/blk-map.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-map.c +1 -1
@@ -400,7 +400,7 @@ static void bio_copy_kern_endio_read(struct bio *bio)struct bvec_iter_all iter_all;bio_for_each_segment_all (bvec, bio, iter_all) { -memcpy (p,page_address (bvec->bv_page), bvec->bv_len); +memcpy_from_bvec (p, bvec); p += bvec->bv_len; }
Signature · 1 line
2.30.2
Re: [PATCH 13/16] block: use memcpy_from_bvec in bio_copy_kern_endio_read
Quoted 3 lines · Christoph Hellwig
Use
memcpy_from_bvecinstead of open coding the logic.Signed-off-by Christoph Hellwig <[email protected]>
Looks good.
PATCH 14/16 block: use memcpy_from_bvec in __blk_queue_bounce
In reply to Christoph HellwigRewrite the actual bounce buffering loop in __blk_queue_bounce to that the memcpy_to_bvec helper can be used to perform the data copies.
--- block/bounce.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)
block/bounce.c +7 -14
@@ -243,24 +243,17 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig)* because the 'bio' is single-page bvec. */ for (i =0 , to = bio->bi_io_vec; i < bio->bi_vcnt; to++, i++) { -struct page *page = to->bv_page; +struct page *bounce_page; -if (!PageHighMem (page)) +if (!PageHighMem (to->bv_page))continue ; - to->bv_page =mempool_alloc (&page_pool,GFP_NOIO ); -inc_zone_page_state (to->bv_page,NR_BOUNCE ); + bounce_page =mempool_alloc (&page_pool,GFP_NOIO ); +inc_zone_page_state (bounce_page,NR_BOUNCE ); -if (rw ==WRITE ) { -char *vto, *vfrom; - -flush_dcache_page (page); - - vto =page_address (to->bv_page) + to->bv_offset; - vfrom =kmap_atomic (page) + to->bv_offset; -memcpy (vto, vfrom, to->bv_len); -kunmap_atomic (vfrom); - } +if (rw ==WRITE ) +memcpy_from_bvec (page_address (bounce_page), to); + to->bv_page = bounce_page; }trace_block_bio_bounce (*bio_orig);
Signature · 1 line
2.30.2
Re: [PATCH 14/16] block: use memcpy_from_bvec in __blk_queue_bounce
Quoted 30 lines · Christoph Hellwig
Rewrite the actual bounce buffering loop in
__blk_queue_bounceto that thememcpy_to_bvechelper can be used to perform the data copies.Signed-off-by Christoph Hellwig <[email protected]>--- block/bounce.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)block/bounce.c +4
-6@@ -243,24 +243,17 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig)* because the 'bio' is single-page bvec. */ for (i =0 , to = bio->bi_io_vec; i < bio->bi_vcnt; to++, i++) { -struct page *page = to->bv_page; +struct page *bounce_page; -if (!PageHighMem (page)) +if (!PageHighMem (to->bv_page))continue ; - to->bv_page =mempool_alloc (&page_pool,GFP_NOIO ); -inc_zone_page_state (to->bv_page,NR_BOUNCE ); + bounce_page =mempool_alloc (&page_pool,GFP_NOIO ); +inc_zone_page_state (bounce_page,NR_BOUNCE ); -if (rw ==WRITE ) { -char *vto, *vfrom;
block/bounce.c void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig) · line 255
Quoted 10 lines
block/bounce.c +2
-8- -flush_dcache_page (page); - - vto =page_address (to->bv_page) + to->bv_offset; - vfrom =kmap_atomic (page) + to->bv_offset; -memcpy (vto, vfrom, to->bv_len); -kunmap_atomic (vfrom); - } +if (rw ==WRITE ) +memcpy_from_bvec (page_address (bounce_page), to);
NIT: the fact that the copy is from 'to' makes my head hurt... But I don't see a good way to change that without declaring unnecessary variables... :-(
The logic seems right.
Ira
Quoted 7 lines
block/bounce.c +1
-0+ to->bv_page = bounce_page; }trace_block_bio_bounce (*bio_orig);Signature · 1 line
2.30.2
PATCH 15/16 block: use bvec_kmap_local in t10_pi_type1_{prepare,complete}
In reply to Christoph HellwigUsing local kmaps slightly reduces the chances to stray writes, and the bvec interface cleans up the code a little bit.
--- block/t10-pi.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
block/t10-pi.c +6 -10
@@ -147,11 +147,10 @@ static void t10_pi_type1_prepare(struct request *rq)break ;bip_for_each_vec (iv, bip, iter) { -void *p, *pmap;unsigned int j; +void *p; - pmap =kmap_atomic (iv.bv_page); - p = pmap + iv.bv_offset; + p =bvec_kmap_local (&iv);for (j =0 ; j < iv.bv_len; j += tuple_sz) {struct t10_pi_tuple *pi = p;
@@ -161,8 +160,7 @@ static void t10_pi_type1_prepare(struct request *rq) ref_tag++; p += tuple_sz; } - -kunmap_atomic (pmap); +kunmap_local (p); } bip->bip_flags |=BIP_MAPPED_INTEGRITY ;
@@ -195,11 +193,10 @@ static void t10_pi_type1_complete(struct request *rq, unsigned int nr_bytes)struct bvec_iter iter;bip_for_each_vec (iv, bip, iter) { -void *p, *pmap;unsigned int j; +void *p; - pmap =kmap_atomic (iv.bv_page); - p = pmap + iv.bv_offset; + p =bvec_kmap_local (&iv);for (j =0 ; j < iv.bv_len && intervals; j += tuple_sz) {struct t10_pi_tuple *pi = p;
@@ -210,8 +207,7 @@ static void t10_pi_type1_complete(struct request *rq, unsigned int nr_bytes) intervals--; p += tuple_sz; } - -kunmap_atomic (pmap); +kunmap_local (p); } } }
Signature · 1 line
2.30.2
PATCH 16/16 block: use bvec_kmap_local in bio_integrity_process
In reply to Christoph HellwigUsing local kmaps slightly reduces the chances to stray writes, and the bvec interface cleans up the code a little bit.
--- block/bio-integrity.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
block/bio-integrity.c +6 -8
@@ -172,18 +172,16 @@ static blk_status_t bio_integrity_process(struct bio *bio, iter.prot_buf = prot_buf;__bio_for_each_segment (bv, bio, bviter, *proc_iter) { -void *kaddr =kmap_atomic (bv.bv_page); +void *kaddr =bvec_kmap_local (&bv); - iter.data_buf = kaddr + bv.bv_offset; + iter.data_buf = kaddr; iter.data_size = bv.bv_len; - ret =proc_fn (&iter); -if (ret) { -kunmap_atomic (kaddr); -return ret; - } +kunmap_local (kaddr); + +if (ret) +break ; -kunmap_atomic (kaddr); }return ret; }
Signature · 1 line
2.30.2
Ira Weiny
Reviewed-byQuoted 6 lines · Christoph Hellwig
Hi all,
this series switches the core block layer code and all users of the existing bvec kmap helpers to use
kmap_local_page. Drivers that currently use open codedkmap_atomiccalls will converted in a follow on series.
Other than the missing flush_dcache's.
For the series.
Quoted 15 lines
Diffstat:
arch/mips/include/asm/mach-rc32434/rb.h | 2 - block/bio-integrity.c | 14 ++++------ block/bio.c | 37 +++++++--------------------- block/blk-map.c | 2 - block/bounce.c | 35 ++++++-------------------- block/t10-pi.c | 16 ++++-------- drivers/block/ps3disk.c | 19 ++------------ drivers/block/rbd.c | 15 +---------- drivers/md/dm-writecache.c | 5 +-- include/linux/bio.h | 42 -------------------------------- include/linux/bvec.h | 27 ++++++++++++++++++-- include/linux/highmem.h | 4 +-- 12 files changed, 64 insertions(+), 154 deletions(-)