New page table range API
Subsystems (51)
Matthew Wilcox (Oracle)
maintainerv6 cover New page table range API
For some reason, I didn't get an email when v5 was dropped from linux-next (I got an email from Andrew saying he was going to, but when I didn't get the automated emails, I assumed he'd changed his mind). So here's v6, far later than I would have resent it.
This patchset changes the API used by the MM to set up page table entries. The four APIs are:
set_ptes(mm, addr, ptep, pte, nr) update_mmu_cache_range(vma, addr, ptep, nr) flush_dcache_folio(folio)
flush_icache_pages(vma, page, nr)
flush_dcache_folio() isn't technically new, but no architecture implemented it, so I've done that for them. The old APIs remain around but are mostly implemented by calling the new interfaces.
The new APIs are based around setting up N page table entries at once. The N entries belong to the same PMD, the same folio and the same VMA, so ptep++ is a legitimate operation, and locking is taken care of for you. Some architectures can do a better job of it than just a loop, but I have hesitated to make too deep a change to architectures I don't understand well.
One thing I have changed in every architecture is that PG_arch_1 is now a per-folio bit instead of a per-page bit when used for dcache clean/dirty tracking. This was something that would have to happen eventually, and it makes sense to do it now rather than iterate over every page involved in a cache flush and figure out if it needs to happen.
The point of all this is better performance, and Fengwei Yin has measured improvement on x86. I suspect you'll see improvement on your architecture too. Try the new will-it-scale test mentioned here:
https://lore.kernel.org/linux-mm/[email protected]/
You'll need to run it on an XFS filesystem and have CONFIG_TRANSPARENT_HUGEPAGE set.
This patchset is the basis for much of the anonymous large folio work being done by Ryan, so it's received quite a lot of testing over the last few months.
v6:
- Fix other definitions of in_range(); either renaming them to not conflict or removing them and using the new common definition.
- Use arch_enter_lazy_mmu_mode() and arch_leave_lazy_mmu_mode()
- Fix a compiler warning in the riscv implementation
- Update to next-20230731
v5:
- Add in_range() macro
- Fix numerous compilation problems on minority architectures (thanks LKP!)
- Add the 'vmf' argument to update_mmu_cache_range() to help MIPS and other architectures that insert TLB entries in software, rather than using a hardware page table walker
- Get rid of first_map_page() and next_map_page(); use next_uptodate_folio() directly
- Actually move the mmap_miss accounting in filemap.c
- Add kernel-doc for set_pte_range()
- Correct determination of prefaulting in set_pte_range()
- More Acked & Reviewed tags
v4:
- Fix a few compile errors (mostly Mike Rapoport)
- Incorporate Mike's suggestion to avoid having to define set_ptes() or set_pte_at() on the majority of architectures
- Optimise m68k's __flush_pages_to_ram (Geert Uytterhoeven)
- Fix sun3 (me)
- Fix sparc32 (me)
- Pick up a few more Ack/Reviewed tags
v3:
- Reinstate flush_dcache_icache_phys() on PowerPC
- Fix folio_flush_mapping(). The documentation was correct and the implementation was completely wrong
- Change the flush_dcache_page() documentation to describe flush_dcache_folio() instead
- Split ARM from ARC. I messed up my git commands
- Remove page_mapping_file()
- Rationalise how flush_icache_pages() and flush_icache_page() are defined
- Use flush_icache_pages() in do_set_pmd()
- Pick up Guo Ren's Ack for csky
Matthew Wilcox (Oracle) (34):
minmax: Add in_range() macro
mm: Convert page_table_check_pte_set() to page_table_check_ptes_set()
mm: Add generic flush_icache_pages() and documentation
mm: Add folio_flush_mapping()
mm: Remove ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO
mm: Add default definition of set_ptes()
alpha: Implement the new page table range API
arc: Implement the new page table range API
arm: Implement the new page table range API
arm64: Implement the new page table range API
csky: Implement the new page table range API
hexagon: Implement the new page table range API
ia64: Implement the new page table range API
loongarch: Implement the new page table range API
m68k: Implement the new page table range API
microblaze: Implement the new page table range API
mips: Implement the new page table range API
nios2: Implement the new page table range API
openrisc: Implement the new page table range API
parisc: Implement the new page table range API
powerpc: Implement the new page table range API
riscv: Implement the new page table range API
s390: Implement the new page table range API
sh: Implement the new page table range API
sparc32: Implement the new page table range API
sparc64: Implement the new page table range API
um: Implement the new page table range API
x86: Implement the new page table range API
xtensa: Implement the new page table range API
mm: Remove page_mapping_file()
mm: Rationalise flush_icache_pages() and flush_icache_page()
mm: Tidy up set_ptes definition
mm: Use flush_icache_pages() in do_set_pmd()
mm: Call update_mmu_cache_range() in more page fault handling paths
Yin Fengwei (4):
filemap: Add filemap_map_folio_range()
rmap: add folio_add_file_rmap_range()
mm: Convert do_set_pte() to set_pte_range()
filemap: Batch PTE mappings
Documentation/core-api/cachetlb.rst | 55 ++++---- Documentation/filesystems/locking.rst | 2 +- arch/alpha/include/asm/cacheflush.h | 13 +- arch/alpha/include/asm/pgtable.h | 10 +- arch/arc/include/asm/cacheflush.h | 14 +- arch/arc/include/asm/pgtable-bits-arcv2.h | 12 +- arch/arc/include/asm/pgtable-levels.h | 1 + arch/arc/mm/cache.c | 61 +++++---- arch/arc/mm/tlb.c | 18 ++- arch/arm/include/asm/cacheflush.h | 29 ++--- arch/arm/include/asm/pgtable.h | 5 +- arch/arm/include/asm/tlbflush.h | 14 +- arch/arm/mm/copypage-v4mc.c | 5 +- arch/arm/mm/copypage-v6.c | 5 +- arch/arm/mm/copypage-xscale.c | 5 +- arch/arm/mm/dma-mapping.c | 24 ++-- arch/arm/mm/fault-armv.c | 16 +-- arch/arm/mm/flush.c | 99 ++++++++------ arch/arm/mm/mm.h | 2 +- arch/arm/mm/mmu.c | 14 +- arch/arm/mm/nommu.c | 6 + arch/arm/mm/pageattr.c | 6 +- arch/arm64/include/asm/cacheflush.h | 4 +- arch/arm64/include/asm/pgtable.h | 26 +++- arch/arm64/mm/flush.c | 36 ++--- arch/csky/abiv1/cacheflush.c | 32 +++-- arch/csky/abiv1/inc/abi/cacheflush.h | 3 +- arch/csky/abiv2/cacheflush.c | 32 ++--- arch/csky/abiv2/inc/abi/cacheflush.h | 11 +- arch/csky/include/asm/pgtable.h | 8 +- arch/hexagon/include/asm/cacheflush.h | 10 +- arch/hexagon/include/asm/pgtable.h | 9 +- arch/ia64/hp/common/sba_iommu.c | 26 ++-- arch/ia64/include/asm/cacheflush.h | 14 +- arch/ia64/include/asm/pgtable.h | 4 +- arch/ia64/mm/init.c | 28 ++-- arch/loongarch/include/asm/cacheflush.h | 1 - arch/loongarch/include/asm/pgtable-bits.h | 4 +- arch/loongarch/include/asm/pgtable.h | 33 ++--- arch/loongarch/mm/pgtable.c | 2 +- arch/loongarch/mm/tlb.c | 2 +- arch/m68k/include/asm/cacheflush_mm.h | 26 ++-- arch/m68k/include/asm/mcf_pgtable.h | 1 + arch/m68k/include/asm/motorola_pgtable.h | 1 + arch/m68k/include/asm/pgtable_mm.h | 10 +- arch/m68k/include/asm/sun3_pgtable.h | 1 + arch/m68k/mm/motorola.c | 2 +- arch/microblaze/include/asm/cacheflush.h | 8 ++ arch/microblaze/include/asm/pgtable.h | 15 +-- arch/microblaze/include/asm/tlbflush.h | 4 +- arch/mips/bcm47xx/prom.c | 2 +- arch/mips/include/asm/cacheflush.h | 32 +++-- arch/mips/include/asm/pgtable-32.h | 10 +- arch/mips/include/asm/pgtable-64.h | 6 +- arch/mips/include/asm/pgtable-bits.h | 6 +- arch/mips/include/asm/pgtable.h | 63 +++++---- arch/mips/mm/c-r4k.c | 5 +- arch/mips/mm/cache.c | 56 ++++---- arch/mips/mm/init.c | 21 +-- arch/mips/mm/pgtable-32.c | 2 +- arch/mips/mm/pgtable-64.c | 2 +- arch/mips/mm/tlbex.c | 2 +- arch/nios2/include/asm/cacheflush.h | 6 +- arch/nios2/include/asm/pgtable.h | 28 ++-- arch/nios2/mm/cacheflush.c | 79 ++++++----- arch/openrisc/include/asm/cacheflush.h | 8 +- arch/openrisc/include/asm/pgtable.h | 15 ++- arch/openrisc/mm/cache.c | 12 +- arch/parisc/include/asm/cacheflush.h | 14 +- arch/parisc/include/asm/pgtable.h | 37 ++++-- arch/parisc/kernel/cache.c | 107 ++++++++++----- arch/powerpc/include/asm/book3s/32/pgtable.h | 5 - arch/powerpc/include/asm/book3s/64/pgtable.h | 6 +- arch/powerpc/include/asm/book3s/pgtable.h | 11 +- arch/powerpc/include/asm/cacheflush.h | 14 +- arch/powerpc/include/asm/kvm_ppc.h | 10 +- arch/powerpc/include/asm/nohash/pgtable.h | 16 +-- arch/powerpc/include/asm/pgtable.h | 12 ++ arch/powerpc/mm/book3s64/hash_utils.c | 11 +- arch/powerpc/mm/cacheflush.c | 40 ++---- arch/powerpc/mm/nohash/e500_hugetlbpage.c | 3 +- arch/powerpc/mm/pgtable.c | 53 ++++---- arch/riscv/include/asm/cacheflush.h | 19 ++- arch/riscv/include/asm/pgtable.h | 37 ++++-- arch/riscv/mm/cacheflush.c | 13 +- arch/s390/include/asm/pgtable.h | 33 +++-- arch/sh/include/asm/cacheflush.h | 21 ++- arch/sh/include/asm/pgtable.h | 7 +- arch/sh/include/asm/pgtable_32.h | 5 +- arch/sh/mm/cache-j2.c | 4 +- arch/sh/mm/cache-sh4.c | 26 ++-- arch/sh/mm/cache-sh7705.c | 26 ++-- arch/sh/mm/cache.c | 52 ++++---- arch/sh/mm/kmap.c | 3 +- arch/sparc/include/asm/cacheflush_32.h | 10 +- arch/sparc/include/asm/cacheflush_64.h | 19 +-- arch/sparc/include/asm/pgtable_32.h | 8 +- arch/sparc/include/asm/pgtable_64.h | 29 ++++- arch/sparc/kernel/smp_64.c | 56 +++++--- arch/sparc/mm/init_32.c | 13 +- arch/sparc/mm/init_64.c | 78 ++++++----- arch/sparc/mm/tlb.c | 5 +- arch/um/include/asm/pgtable.h | 7 +- arch/x86/include/asm/pgtable.h | 14 +- arch/xtensa/include/asm/cacheflush.h | 11 +- arch/xtensa/include/asm/pgtable.h | 18 ++- arch/xtensa/mm/cache.c | 83 +++++++----- .../drm/arm/display/include/malidp_utils.h | 2 +- .../display/komeda/komeda_pipeline_state.c | 24 ++-- drivers/gpu/drm/msm/adreno/a6xx_gmu.c | 6 - .../net/ethernet/chelsio/cxgb3/cxgb3_main.c | 18 +-- drivers/virt/acrn/ioreq.c | 4 +- fs/btrfs/misc.h | 2 - fs/ext2/balloc.c | 2 - fs/ext4/ext4.h | 2 - fs/ufs/util.h | 6 - include/asm-generic/cacheflush.h | 7 - include/linux/cacheflush.h | 13 +- include/linux/minmax.h | 27 ++++ include/linux/mm.h | 3 +- include/linux/page_table_check.h | 13 +- include/linux/pagemap.h | 28 ++-- include/linux/pgtable.h | 75 ++++++++--- include/linux/rmap.h | 2 + lib/logic_pio.c | 3 - mm/filemap.c | 123 ++++++++++-------- mm/memory.c | 56 ++++---- mm/page_table_check.c | 16 ++- mm/rmap.c | 60 +++++++-- mm/util.c | 2 +- net/netfilter/nf_nat_core.c | 6 +- net/tipc/core.h | 2 +- net/tipc/link.c | 10 +- .../selftests/bpf/progs/get_branch_snapshot.c | 4 +- 134 files changed, 1520 insertions(+), 1056 deletions(-)
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 1/38 minmax: Add in_range() macro
Determine if a value lies within a range more efficiently (subtraction + comparison vs two comparisons and an AND). It also has useful (under some circumstances) behaviour if the range exceeds the maximum value of the type. Convert all the conflicting definitions of in_range() within the kernel; some can use the generic definition while others need their own definition.
--- arch/arm/mm/pageattr.c | 6 ++--- .../drm/arm/display/include/malidp_utils.h | 2 +- .../display/komeda/komeda_pipeline_state.c | 24 ++++++++--------- drivers/gpu/drm/msm/adreno/a6xx_gmu.c | 6 ----- .../net/ethernet/chelsio/cxgb3/cxgb3_main.c | 18 ++++++------- drivers/virt/acrn/ioreq.c | 4 +-- fs/btrfs/misc.h | 2 -- fs/ext2/balloc.c | 2 -- fs/ext4/ext4.h | 2 -- fs/ufs/util.h | 6 ----- include/linux/minmax.h | 27 +++++++++++++++++++ lib/logic_pio.c | 3 --- net/netfilter/nf_nat_core.c | 6 ++--- net/tipc/core.h | 2 +- net/tipc/link.c | 10 +++---- .../selftests/bpf/progs/get_branch_snapshot.c | 4 +-- 16 files changed, 65 insertions(+), 59 deletions(-)
arch/arm/mm/pageattr.c +3 -3
@@ -25,7 +25,7 @@ static int change_page_range(pte_t *ptep, unsigned long addr, void *data)return 0 ; } -static bool in_range (unsigned long start,unsigned long size, +static bool range_in_range (unsigned long start,unsigned long size,unsigned long range_start,unsigned long range_end) { return start >= range_start && start < range_end &&
@@ -63,8 +63,8 @@ static int change_memory_common(unsigned long addr, int numpages,if (!size)return 0 ; -if (!in_range (start, size,MODULES_VADDR ,MODULES_END ) && - !in_range (start, size,VMALLOC_START ,VMALLOC_END )) +if (!range_in_range (start, size,MODULES_VADDR ,MODULES_END ) && + !range_in_range (start, size,VMALLOC_START ,VMALLOC_END ))return -EINVAL ;return __change_memory_common (start, size, set_mask, clear_mask);
drivers/gpu/drm/arm/display/include/malidp_utils.h +1 -1
@@ -35,7 +35,7 @@ static inline void set_range(struct malidp_range *rg, u32 start, u32 end) rg->end = end; } -static inline bool in_range (struct malidp_range *rg,u32 v) +static inline bool malidp_in_range (struct malidp_range *rg,u32 v) {return (v >= rg->start) && (v <= rg->end); }
drivers/gpu/drm/arm/display/komeda/komeda_pipeline_state.c +12 -12
@@ -305,12 +305,12 @@ komeda_layer_check_cfg(struct komeda_layer *layer,if (komeda_fb_check_src_coords (kfb, src_x, src_y, src_w, src_h))return -EINVAL ; -if (!in_range (&layer->hsize_in, src_w)) { +if (!malidp_in_range (&layer->hsize_in, src_w)) {DRM_DEBUG_ATOMIC ("invalidate src_w %d.\n" , src_w);return -EINVAL ; } -if (!in_range (&layer->vsize_in, src_h)) { +if (!malidp_in_range (&layer->vsize_in, src_h)) {DRM_DEBUG_ATOMIC ("invalidate src_h %d.\n" , src_h);return -EINVAL ; }
@@ -452,14 +452,14 @@ komeda_scaler_check_cfg(struct komeda_scaler *scaler, hsize_out = dflow->out_w; vsize_out = dflow->out_h; -if (!in_range (&scaler->hsize, hsize_in) || - !in_range (&scaler->hsize, hsize_out)) { +if (!malidp_in_range (&scaler->hsize, hsize_in) || + !malidp_in_range (&scaler->hsize, hsize_out)) {DRM_DEBUG_ATOMIC ("Invalid horizontal sizes" );return -EINVAL ; } -if (!in_range (&scaler->vsize, vsize_in) || - !in_range (&scaler->vsize, vsize_out)) { +if (!malidp_in_range (&scaler->vsize, vsize_in) || + !malidp_in_range (&scaler->vsize, vsize_out)) {DRM_DEBUG_ATOMIC ("Invalid vertical sizes" );return -EINVAL ; }
@@ -574,13 +574,13 @@ komeda_splitter_validate(struct komeda_splitter *splitter,return -EINVAL ; } -if (!in_range (&splitter->hsize, dflow->in_w)) { +if (!malidp_in_range (&splitter->hsize, dflow->in_w)) {DRM_DEBUG_ATOMIC ("split in_w:%d is out of the acceptable range.\n" , dflow->in_w);return -EINVAL ; } -if (!in_range (&splitter->vsize, dflow->in_h)) { +if (!malidp_in_range (&splitter->vsize, dflow->in_h)) {DRM_DEBUG_ATOMIC ("split in_h: %d exceeds the acceptable range.\n" , dflow->in_h);return -EINVAL ;
@@ -624,13 +624,13 @@ komeda_merger_validate(struct komeda_merger *merger,return -EINVAL ; } -if (!in_range (&merger->hsize_merged, output->out_w)) { +if (!malidp_in_range (&merger->hsize_merged, output->out_w)) {DRM_DEBUG_ATOMIC ("merged_w: %d is out of the accepted range.\n" , output->out_w);return -EINVAL ; } -if (!in_range (&merger->vsize_merged, output->out_h)) { +if (!malidp_in_range (&merger->vsize_merged, output->out_h)) {DRM_DEBUG_ATOMIC ("merged_h: %d is out of the accepted range.\n" , output->out_h);return -EINVAL ;
@@ -866,8 +866,8 @@ void komeda_complete_data_flow_cfg(struct komeda_layer *layer,* input/output range. */ if (dflow->en_scaling && scaler) - dflow->en_split = !in_range (&scaler->hsize, dflow->in_w) || - !in_range (&scaler->hsize, dflow->out_w); + dflow->en_split = !malidp_in_range (&scaler->hsize, dflow->in_w) || + !malidp_in_range (&scaler->hsize, dflow->out_w); } static bool merger_is_available(struct komeda_pipeline *pipe,
drivers/gpu/drm/msm/adreno/a6xx_gmu.c +0 -6
@@ -678,12 +678,6 @@ struct block_header {u32 data[]; }; -/* this should be a general kernel helper */ -static int in_range (u32 addr,u32 start,u32 size) -{ -return addr >= start && addr < start + size; -} -static bool fw_block_mem (struct a6xx_gmu_bo *bo,const struct block_header *blk) {if (!in_range (blk->addr, bo->iova, bo->size))
drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c +9 -9
@@ -2126,7 +2126,7 @@ static const struct ethtool_ops cxgb_ethtool_ops = { .set_link_ksettings = set_link_ksettings, }; -static int in_range (int val,int lo,int hi) +static int cxgb_in_range (int val,int lo,int hi) {return val <0 || (val <= hi && val >= lo); }
@@ -2162,19 +2162,19 @@ static int cxgb_siocdevprivate(struct net_device *dev,return -EINVAL ;if (t.qset_idx >=SGE_QSETS )return -EINVAL ; -if (!in_range (t.intr_lat,0 ,M_NEWTIMER ) || - !in_range (t.cong_thres,0 ,255 ) || - !in_range (t.txq_size[0 ],MIN_TXQ_ENTRIES , +if (!cxgb_in_range (t.intr_lat,0 ,M_NEWTIMER ) || + !cxgb_in_range (t.cong_thres,0 ,255 ) || + !cxgb_in_range (t.txq_size[0 ],MIN_TXQ_ENTRIES ,MAX_TXQ_ENTRIES ) || - !in_range (t.txq_size[1 ],MIN_TXQ_ENTRIES , + !cxgb_in_range (t.txq_size[1 ],MIN_TXQ_ENTRIES ,MAX_TXQ_ENTRIES ) || - !in_range (t.txq_size[2 ],MIN_CTRL_TXQ_ENTRIES , + !cxgb_in_range (t.txq_size[2 ],MIN_CTRL_TXQ_ENTRIES ,MAX_CTRL_TXQ_ENTRIES ) || - !in_range (t.fl_size[0 ],MIN_FL_ENTRIES , + !cxgb_in_range (t.fl_size[0 ],MIN_FL_ENTRIES ,MAX_RX_BUFFERS ) || - !in_range (t.fl_size[1 ],MIN_FL_ENTRIES , + !cxgb_in_range (t.fl_size[1 ],MIN_FL_ENTRIES ,MAX_RX_JUMBO_BUFFERS ) || - !in_range (t.rspq_size,MIN_RSPQ_ENTRIES , + !cxgb_in_range (t.rspq_size,MIN_RSPQ_ENTRIES ,MAX_RSPQ_ENTRIES ))return -EINVAL ;
drivers/virt/acrn/ioreq.c +2 -2
@@ -351,7 +351,7 @@ static bool handle_cf8cfc(struct acrn_vm *vm,return is_handled; } -static bool in_range (struct acrn_ioreq_range *range, +static bool acrn_in_range (struct acrn_ioreq_range *range,struct acrn_io_request *req) {bool ret = false;
@@ -389,7 +389,7 @@ static struct acrn_ioreq_client *find_ioreq_client(struct acrn_vm *vm,list_for_each_entry (client, &vm->ioreq_clients, list) {read_lock_bh (&client->range_lock);list_for_each_entry (range, &client->range_list, list) { -if (in_range (range, req)) { +if (acrn_in_range (range, req)) { found = client;break ; }
fs/btrfs/misc.h +0 -2
@@ -8,8 +8,6 @@#include <linux/math64.h> #include <linux/rbtree.h> -#define in_range (b, first, len) ((b) >= (first) && (b) < (first) + (len)) -/* * Enumerate bits using enum autoincrement. Define the @name as the n-th bit. */
fs/ext2/balloc.c +0 -2
@@ -36,8 +36,6 @@*/ -#define in_range (b, first, len) ((b) >= (first) && (b) <= (first) + (len) - 1) -struct ext2_group_desc *ext2_get_group_desc (struct super_block * sb,unsigned int block_group,struct buffer_head ** bh)
fs/ext4/ext4.h +0 -2
@@ -3780,8 +3780,6 @@ static inline void set_bitmap_uptodate(struct buffer_head *bh)set_bit (BH_BITMAP_UPTODATE , &(bh)->b_state); } -#define in_range (b, first, len) ((b) >= (first) && (b) <= (first) + (len) - 1) -/* For ioend & aio unwritten conversion wait queues */ #define EXT4_WQ_HASH_SZ 37#define ext4_ioend_wq (v) (&ext4__ioend_wq[((unsigned long)(v)) %\
fs/ufs/util.h +0 -6
@@ -11,12 +11,6 @@#include <linux/fs.h> #include "swab.h" - -/* -* some useful macros -*/ -#define in_range (b,first,len) ((b)>=(first)&&(b)<(first)+(len)) -/* * functions used for retyping */
include/linux/minmax.h +27 -0
@@ -3,6 +3,7 @@#define _LINUX_MINMAX_H#include <linux/const.h> +#include <linux/types.h> /* * min()/max()/clamp() macros must accomplish three things:
@@ -222,6 +223,32 @@*/ #define clamp_val (val, lo, hi) clamp_t(typeof(val), val, lo, hi) +static inline bool in_range64 (u64 val,u64 start,u64 len) +{ +return (val - start) < len; +} + +static inline bool in_range32 (u32 val,u32 start,u32 len) +{ +return (val - start) < len; +} + +/** +* in_range - Determine if a value lies within a range. +* @val: Value to test. +* @start: First value in range. +* @len: Number of values in range. +* +* This is more efficient than "if (start <= val && val < (start + len))". +* It also gives a different answer if @start + @len overflows the size of +* the type by a sufficient amount to encompass @val. Decide for yourself +* which behaviour you want, or prove that start + len never overflow. +* Do not blindly replace one form with the other. +*/ +#define in_range (val, start, len) \ + ((sizeof(start) | sizeof(len) | sizeof(val)) <= sizeof(u32) ? \ + in_range32(val, start, len) : in_range64(val, start, len)) +/** * swap - swap values of @a and @b * @a: first value
lib/logic_pio.c +0 -3
@@ -20,9 +20,6 @@static LIST_HEAD (io_range_list);static DEFINE_MUTEX (io_range_mutex); -/* Consider a kernel general helper for this */ -#define in_range (b, first, len) ((b) >= (first) && (b) < (first) + (len)) -/** * logic_pio_register_range - register logical PIO range for a host * @new_range: pointer to the IO range to be registered.
net/netfilter/nf_nat_core.c +3 -3
@@ -327,7 +327,7 @@ static bool l4proto_in_range(const struct nf_conntrack_tuple *tuple,/* If we source map this tuple so reply looks like reply_tuple, will * that meet the constraints of range. */ -static int in_range (const struct nf_conntrack_tuple *tuple, +static int nf_in_range (const struct nf_conntrack_tuple *tuple,const struct nf_nat_range2 *range) {/* If we are supposed to map IPs, then we must be in the
@@ -376,7 +376,7 @@ find_appropriate_src(struct net *net, &ct->tuplehash[IP_CT_DIR_REPLY].tuple); result->dst = tuple->dst; -if (in_range (result, range)) +if (nf_in_range (result, range))return 1 ; } }
@@ -607,7 +607,7 @@ get_unique_tuple(struct nf_conntrack_tuple *tuple,if (maniptype ==NF_NAT_MANIP_SRC && !(range->flags &NF_NAT_RANGE_PROTO_RANDOM_ALL )) {/* try the original tuple first */ -if (in_range (orig_tuple, range)) { +if (nf_in_range (orig_tuple, range)) {if (!nf_nat_used_tuple (orig_tuple, ct)) { *tuple = *orig_tuple;return ;
net/tipc/core.h +1 -1
@@ -197,7 +197,7 @@ static inline int less(u16 left, u16 right)return less_eq (left, right) && (mod (right) !=mod (left)); } -static inline int in_range (u16 val,u16 min,u16 max) +static inline int tipc_in_range (u16 val,u16 min,u16 max) {return !less (val, min) && !more (val, max); }
net/tipc/link.c +5 -5
@@ -1623,7 +1623,7 @@ static int tipc_link_advance_transmq(struct tipc_link *l, struct tipc_link *r, last_ga->bgack_cnt); }/* Check against the last Gap ACK block */ -if (in_range (seqno, start, end)) +if (tipc_in_range (seqno, start, end))continue ;/* Update/release the packet peer is acking */ bc_has_acked = true;
@@ -2251,12 +2251,12 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb,strncpy (if_name, data,TIPC_MAX_IF_NAME );/* Update own tolerance if peer indicates a non-zero value */ -if (in_range (peers_tol,TIPC_MIN_LINK_TOL ,TIPC_MAX_LINK_TOL )) { +if (tipc_in_range (peers_tol,TIPC_MIN_LINK_TOL ,TIPC_MAX_LINK_TOL )) { l->tolerance = peers_tol; l->bc_rcvlink->tolerance = peers_tol; }/* Update own priority if peer's priority is higher */ -if (in_range (peers_prio, l->priority +1 ,TIPC_MAX_LINK_PRI )) +if (tipc_in_range (peers_prio, l->priority +1 ,TIPC_MAX_LINK_PRI )) l->priority = peers_prio;/* If peer is going down we want full re-establish cycle */
@@ -2299,13 +2299,13 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, l->rcv_nxt_state =msg_seqno (hdr) +1 ;/* Update own tolerance if peer indicates a non-zero value */ -if (in_range (peers_tol,TIPC_MIN_LINK_TOL ,TIPC_MAX_LINK_TOL )) { +if (tipc_in_range (peers_tol,TIPC_MIN_LINK_TOL ,TIPC_MAX_LINK_TOL )) { l->tolerance = peers_tol; l->bc_rcvlink->tolerance = peers_tol; }/* Update own prio if peer indicates a different value */ if ((peers_prio != l->priority) && -in_range (peers_prio,1 ,TIPC_MAX_LINK_PRI )) { +tipc_in_range (peers_prio,1 ,TIPC_MAX_LINK_PRI )) { l->priority = peers_prio; rc =tipc_link_fsm_evt (l,LINK_FAILURE_EVT ); }
tools/testing/selftests/bpf/progs/get_branch_snapshot.c +2 -2
@@ -15,7 +15,7 @@ long total_entries = 0;#define ENTRY_CNT 32struct perf_branch_entry entries[ENTRY_CNT ] = {}; -static inline bool in_range (__u64 val) +static inline bool gbs_in_range (__u64 val) {return (val >= address_low) && (val < address_high); }
@@ -31,7 +31,7 @@ int BPF_PROG(test1, int n, int ret)for (i =0 ; i <ENTRY_CNT ; i++) {if (i >= total_entries)break ; -if (in_range (entries[i].from) &&in_range (entries[i].to)) +if (gbs_in_range (entries[i].from) &&gbs_in_range (entries[i].to)) test1_hits++;else if (!test1_hits) wasted_entries++;
Signature · 1 line
2.40.1
Re: [PATCH v6 01/38] minmax: Add in_range() macro
Quoted 16 lines · Matthew Wilcox (Oracle)
include/linux/minmax.h +1
-0@@ -3,6 +3,7 @@#define _LINUX_MINMAX_H#include <linux/const.h> +#include <linux/types.h> /* * min()/max()/clamp() macros must accomplish three things: @@ -222,6 +223,32 @@*/ #define clamp_val (val, lo, hi) clamp_t(typeof(val), val, lo, hi)
include/linux/minmax.h line 235
Quoted 10 lines
include/linux/minmax.h +10
-0+static inline bool in_range64 (u64 val,u64 start,u64 len) +{ +return (val - start) < len; +} + +static inline bool in_range32 (u32 val,u32 start,u32 len) +{ +return (val - start) < len; +} +
I think these two functions return wrong result if val is smaller than start and len is big enough.
Matthew Wilcox
maintainerRe: [PATCH v6 01/38] minmax: Add in_range() macro
Quoted 14 lines · Phi Nguyen
include/linux/minmax.h line 235
Quoted 10 lines · Matthew Wilcox (Oracle)
include/linux/minmax.h +10
-0+static inline bool in_range64 (u64 val,u64 start,u64 len) +{ +return (val - start) < len; +} + +static inline bool in_range32 (u32 val,u32 start,u32 len) +{ +return (val - start) < len; +} +I think these two functions return wrong result if val is smaller than start and len is big enough.
How is it that you stopped reading at exactly the point where I explained that this is intentional?
+/**
+ * in_range - Determine if a value lies within a range.
+ * @val: Value to test.
+ * @start: First value in range.
+ * @len: Number of values in range.
+ *
+ * This is more efficient than "if (start <= val && val < (start + len))".
+ * It also gives a different answer if @start + @len overflows the size of
+ * the type by a sufficient amount to encompass @val. Decide for yourself
+ * which behaviour you want, or prove that start + len never overflow.
+ * Do not blindly replace one form with the other.
+ */
Re: [PATCH v6 01/38] minmax: Add in_range() macro
Quoted 19 lines · Matthew Wilcox
Quoted 14 lines · Phi Nguyen
include/linux/minmax.h line 235
Quoted 10 lines · Matthew Wilcox (Oracle)
include/linux/minmax.h +10
-0+static inline bool in_range64 (u64 val,u64 start,u64 len) +{ +return (val - start) < len; +} + +static inline bool in_range32 (u32 val,u32 start,u32 len) +{ +return (val - start) < len; +} +I think these two functions return wrong result if val is smaller than start and len is big enough.
How is it that you stopped reading at exactly the point where I explained that this is intentional?
Quoted 13 lines
+/**
+ *in_range- Determine if a value lies within a range.
+ * @val: Value to test.
+ * @start: First value in range.
+ * @len: Number of values in range.
+ *
+ * This is more efficient than "if (start <= val && val < (start + len))".
+ * It also gives a different answer if @start + @len overflows the size of
+ * the type by a sufficient amount to encompass @val. Decide for yourself
+ * which behaviour you want, or prove that start + len never overflow.
+ * Do not blindly replace one form with the other.
+ */
Oh, sorry, I see, my bad.
Matthew Wilcox (Oracle)
maintainerPATCH v6 2/38 mm: Convert page_table_check_pte_set() to page_table_check_ptes_set()
In reply to Matthew Wilcox (Oracle)Tell the page table check how many PTEs & PFNs we want it to check.
--- arch/arm64/include/asm/pgtable.h | 2 +- arch/riscv/include/asm/pgtable.h | 2 +- arch/x86/include/asm/pgtable.h | 2 +- include/linux/page_table_check.h | 13 +++++++------ mm/page_table_check.c | 16 +++++++++------- 5 files changed, 19 insertions(+), 16 deletions(-)
arch/arm64/include/asm/pgtable.h +1 -1
@@ -348,7 +348,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,static inline void set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte) { -page_table_check_pte_set (mm, ptep, pte); +page_table_check_ptes_set (mm, ptep, pte,1 );return __set_pte_at (mm, addr, ptep, pte); }
arch/riscv/include/asm/pgtable.h +1 -1
@@ -499,7 +499,7 @@ static inline void __set_pte_at(struct mm_struct *mm,static inline void set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval) { -page_table_check_pte_set (mm, ptep, pteval); +page_table_check_ptes_set (mm, ptep, pteval,1 );__set_pte_at (mm, addr, ptep, pteval); }
arch/x86/include/asm/pgtable.h +1 -1
@@ -1023,7 +1023,7 @@ static inline pud_t native_local_pudp_get_and_clear(pud_t *pudp)static inline void set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte) { -page_table_check_pte_set (mm, ptep, pte); +page_table_check_ptes_set (mm, ptep, pte,1 );set_pte (ptep, pte); }
include/linux/page_table_check.h +7 -6
@@ -17,7 +17,8 @@ void __page_table_check_zero(struct page *page, unsigned int order);void __page_table_check_pte_clear (struct mm_struct *mm,pte_t pte);void __page_table_check_pmd_clear (struct mm_struct *mm,pmd_t pmd);void __page_table_check_pud_clear (struct mm_struct *mm,pud_t pud); -void __page_table_check_pte_set (struct mm_struct *mm,pte_t *ptep,pte_t pte); +void __page_table_check_ptes_set (struct mm_struct *mm,pte_t *ptep,pte_t pte, +unsigned int nr);void __page_table_check_pmd_set (struct mm_struct *mm,pmd_t *pmdp,pmd_t pmd);void __page_table_check_pud_set (struct mm_struct *mm,pud_t *pudp,pud_t pud); void __page_table_check_pte_clear_range(struct mm_struct *mm,
@@ -64,13 +65,13 @@ static inline void page_table_check_pud_clear(struct mm_struct *mm, pud_t pud)__page_table_check_pud_clear (mm, pud); } -static inline void page_table_check_pte_set (struct mm_struct *mm,pte_t *ptep, -pte_t pte) +static inline void page_table_check_ptes_set (struct mm_struct *mm, +pte_t *ptep,pte_t pte,unsigned int nr) {if (static_branch_likely (&page_table_check_disabled))return ; -__page_table_check_pte_set (mm, ptep, pte); +__page_table_check_ptes_set (mm, ptep, pte, nr); } static inline void page_table_check_pmd_set(struct mm_struct *mm, pmd_t *pmdp,
@@ -123,8 +124,8 @@ static inline void page_table_check_pud_clear(struct mm_struct *mm, pud_t pud) { } -static inline void page_table_check_pte_set (struct mm_struct *mm,pte_t *ptep, -pte_t pte) +static inline void page_table_check_ptes_set (struct mm_struct *mm, +pte_t *ptep,pte_t pte,unsigned int nr) { }
mm/page_table_check.c +9 -7
@@ -182,18 +182,20 @@ void __page_table_check_pud_clear(struct mm_struct *mm, pud_t pud) }EXPORT_SYMBOL (__page_table_check_pud_clear); -void __page_table_check_pte_set (struct mm_struct *mm,pte_t *ptep,pte_t pte) +void __page_table_check_ptes_set (struct mm_struct *mm,pte_t *ptep,pte_t pte, +unsigned int nr) { +unsigned int i; +if (&init_mm == mm)return ; -__page_table_check_pte_clear (mm,ptep_get (ptep)); -if (pte_user_accessible_page (pte)) { -page_table_check_set (pte_pfn (pte),PAGE_SIZE >>PAGE_SHIFT , -pte_write (pte)); - } +for (i =0 ; i < nr; i++) +__page_table_check_pte_clear (mm,ptep_get (ptep + i)); +if (pte_user_accessible_page (pte)) +page_table_check_set (pte_pfn (pte), nr,pte_write (pte)); } -EXPORT_SYMBOL (__page_table_check_pte_set); +EXPORT_SYMBOL (__page_table_check_ptes_set);void __page_table_check_pmd_set (struct mm_struct *mm,pmd_t *pmdp,pmd_t pmd) {
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 3/38 mm: Add generic flush_icache_pages() and documentation
In reply to Matthew Wilcox (Oracle)flush_icache_page() is deprecated but not yet removed, so add a range version of it. Change the documentation to refer to update_mmu_cache_range() instead of update_mmu_cache().
--- Documentation/core-api/cachetlb.rst | 39 ++++++++++++++++------------- include/asm-generic/cacheflush.h | 5 ++++ 2 files changed, 27 insertions(+), 17 deletions(-)
Documentation/core-api/cachetlb.rst +22 -17
@@ -88,13 +88,17 @@ changes occur: This is used primarily during fault processing. -5) ``void update_mmu_cache(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep)`` +5) ``void update_mmu_cache_range(struct vm_fault *vmf, + struct vm_area_struct *vma, unsigned long address, pte_t *ptep, + unsigned int nr)`` - At the end of every page fault, this routine is invoked to - tell the architecture specific code that a translation - now exists at virtual address "address" for address space - "vma->vm_mm", in the software page tables. + At the end of every page fault, this routine is invoked to tell + the architecture specific code that translations now exists + in the software page tables for address space "vma->vm_mm" + at virtual address "address" for "nr" consecutive pages. + + This routine is also invoked in various other places which pass + a NULL "vmf". A port may use this information in any way it so chooses. For example, it could use this event to pre-load TLB
@@ -306,17 +310,18 @@ maps this page at its virtual address. private". The kernel guarantees that, for pagecache pages, it will clear this bit when such a page first enters the pagecache. - This allows these interfaces to be implemented much more efficiently. - It allows one to "defer" (perhaps indefinitely) the actual flush if - there are currently no user processes mapping this page. See sparc64's - flush_dcache_page and update_mmu_cache implementations for an example - of how to go about doing this. + This allows these interfaces to be implemented much more + efficiently. It allows one to "defer" (perhaps indefinitely) the + actual flush if there are currently no user processes mapping this + page. See sparc64's flush_dcache_page and update_mmu_cache_range + implementations for an example of how to go about doing this. - The idea is, first at flush_dcache_page() time, if page_file_mapping() - returns a mapping, and mapping_mapped on that mapping returns %false, - just mark the architecture private page flag bit. Later, in - update_mmu_cache(), a check is made of this flag bit, and if set the - flush is done and the flag bit is cleared. + The idea is, first at flush_dcache_page() time, if + page_file_mapping() returns a mapping, and mapping_mapped on that + mapping returns %false, just mark the architecture private page + flag bit. Later, in update_mmu_cache_range(), a check is made + of this flag bit, and if set the flush is done and the flag bit + is cleared. .. important::
@@ -369,7 +374,7 @@ maps this page at its virtual address. ``void flush_icache_page(struct vm_area_struct *vma, struct page *page)`` All the functionality of flush_icache_page can be implemented in - flush_dcache_page and update_mmu_cache. In the future, the hope + flush_dcache_page and update_mmu_cache_range. In the future, the hope is to remove this interface completely. The final category of APIs is for I/O to deliberately aliased address
include/asm-generic/cacheflush.h +5 -0
@@ -78,6 +78,11 @@ static inline void flush_icache_range(unsigned long start, unsigned long end)#endif #ifndef flush_icache_page +static inline void flush_icache_pages (struct vm_area_struct *vma, +struct page *page,unsigned int nr) +{ +} +static inline void flush_icache_page (struct vm_area_struct *vma,struct page *page) {
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 4/38 mm: Add folio_flush_mapping()
In reply to Matthew Wilcox (Oracle)This is the folio equivalent of page_mapping_file(), but rename it to make it clear that it's very different from page_file_mapping(). Theoretically, there's nothing flush-only about it, but there are no other users today, and I doubt there will be; it's almost always more useful to know the swapfile's mapping or the swapcache's mapping.
--- include/linux/pagemap.h | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)
include/linux/pagemap.h +21 -5
@@ -389,6 +389,26 @@ static inline struct address_space *folio_file_mapping(struct folio *folio)return folio->mapping; } +/** +* folio_flush_mapping - Find the file mapping this folio belongs to. +* @folio: The folio. +* +* For folios which are in the page cache, return the mapping that this +* page belongs to. Anonymous folios return NULL, even if they're in +* the swap cache. Other kinds of folio also return NULL. +* +* This is ONLY used by architecture cache flushing code. If you aren't +* writing cache flushing code, you want either folio_mapping() or +* folio_file_mapping(). +*/ +static inline struct address_space *folio_flush_mapping (struct folio *folio) +{ +if (unlikely (folio_test_swapcache (folio))) +return NULL ; + +return folio_mapping (folio); +} +static inline struct address_space *page_file_mapping (struct page *page) {return folio_file_mapping (page_folio (page));
@@ -399,11 +419,7 @@ static inline struct address_space *page_file_mapping(struct page *page)*/ static inline struct address_space *page_mapping_file (struct page *page) { -struct folio *folio =page_folio (page); - -if (unlikely (folio_test_swapcache (folio))) -return NULL ; -return folio_mapping (folio); +return folio_flush_mapping (page_folio (page)); }/**
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 5/38 mm: Remove ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO
In reply to Matthew Wilcox (Oracle)Current best practice is to reuse the name of the function as a define to indicate that the function is implemented by the architecture.
--- Documentation/core-api/cachetlb.rst | 24 +++++++++--------------- include/linux/cacheflush.h | 4 ++-- mm/util.c | 2 +- 3 files changed, 12 insertions(+), 18 deletions(-)
Documentation/core-api/cachetlb.rst +9 -15
@@ -273,7 +273,7 @@ maps this page at its virtual address. If D-cache aliasing is not an issue, these two routines may simply call memcpy/memset directly and do nothing more. - ``void flush_dcache_page(struct page *page)`` + ``void flush_dcache_folio(struct folio *folio)`` This routines must be called when:
@@ -281,7 +281,7 @@ maps this page at its virtual address. and / or in high memory b) the kernel is about to read from a page cache page and user space shared/writable mappings of this page potentially exist. Note - that {get,pin}_user_pages{_fast} already call flush_dcache_page + that {get,pin}_user_pages{_fast} already call flush_dcache_folio on any page found in the user address space and thus driver code rarely needs to take this into account.
@@ -295,7 +295,7 @@ maps this page at its virtual address. The phrase "kernel writes to a page cache page" means, specifically, that the kernel executes store instructions that dirty data in that - page at the page->virtual mapping of that page. It is important to + page at the kernel virtual mapping of that page. It is important to flush here to handle D-cache aliasing, to make sure these kernel stores are visible to user space mappings of that page.
@@ -306,18 +306,18 @@ maps this page at its virtual address. If D-cache aliasing is not an issue, this routine may simply be defined as a nop on that architecture. - There is a bit set aside in page->flags (PG_arch_1) as "architecture + There is a bit set aside in folio->flags (PG_arch_1) as "architecture private". The kernel guarantees that, for pagecache pages, it will clear this bit when such a page first enters the pagecache. This allows these interfaces to be implemented much more efficiently. It allows one to "defer" (perhaps indefinitely) the actual flush if there are currently no user processes mapping this - page. See sparc64's flush_dcache_page and update_mmu_cache_range + page. See sparc64's flush_dcache_folio and update_mmu_cache_range implementations for an example of how to go about doing this. - The idea is, first at flush_dcache_page() time, if - page_file_mapping() returns a mapping, and mapping_mapped on that + The idea is, first at flush_dcache_folio() time, if + folio_flush_mapping() returns a mapping, and mapping_mapped() on that mapping returns %false, just mark the architecture private page flag bit. Later, in update_mmu_cache_range(), a check is made of this flag bit, and if set the flush is done and the flag bit
@@ -331,12 +331,6 @@ maps this page at its virtual address. dirty. Again, see sparc64 for examples of how to deal with this. - ``void flush_dcache_folio(struct folio *folio)`` - This function is called under the same circumstances as - flush_dcache_page(). It allows the architecture to - optimise for flushing the entire folio of pages instead - of flushing one page at a time. - ``void copy_to_user_page(struct vm_area_struct *vma, struct page *page, unsigned long user_vaddr, void *dst, void *src, int len)`` ``void copy_from_user_page(struct vm_area_struct *vma, struct page *page,
@@ -357,7 +351,7 @@ maps this page at its virtual address. When the kernel needs to access the contents of an anonymous page, it calls this function (currently only - get_user_pages()). Note: flush_dcache_page() deliberately + get_user_pages()). Note: flush_dcache_folio() deliberately doesn't work for an anonymous page. The default implementation is a nop (and should remain so for all coherent architectures). For incoherent architectures, it should flush
@@ -374,7 +368,7 @@ maps this page at its virtual address. ``void flush_icache_page(struct vm_area_struct *vma, struct page *page)`` All the functionality of flush_icache_page can be implemented in - flush_dcache_page and update_mmu_cache_range. In the future, the hope + flush_dcache_folio and update_mmu_cache_range. In the future, the hope is to remove this interface completely. The final category of APIs is for I/O to deliberately aliased address
include/linux/cacheflush.h +2 -2
@@ -7,14 +7,14 @@struct folio ;#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE -#ifndef ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO +#ifndef flush_dcache_foliovoid flush_dcache_folio (struct folio *folio);#endif #else static inline void flush_dcache_folio (struct folio *folio) { } -#define ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO 0 +#define flush_dcache_folio flush_dcache_folio#endif /* ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE */ #endif /* _LINUX_CACHEFLUSH_H */
mm/util.c +1 -1
@@ -1119,7 +1119,7 @@ void page_offline_end(void) }EXPORT_SYMBOL (page_offline_end); -#ifndef ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO +#ifndef flush_dcache_foliovoid flush_dcache_folio (struct folio *folio) {long i, nr =folio_nr_pages (folio);
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 6/38 mm: Add default definition of set_ptes()
In reply to Matthew Wilcox (Oracle)Most architectures can just define set_pte() and PFN_PTE_SHIFT to use this definition. It's also a handy spot to document the guarantees provided by the MM.
--- include/linux/pgtable.h | 81 ++++++++++++++++++++++++++++++----------- 1 file changed, 60 insertions(+), 21 deletions(-)
include/linux/pgtable.h +60 -21
@@ -182,6 +182,66 @@ static inline int pmd_young(pmd_t pmd) }#endif +/* +* A facility to provide lazy MMU batching. This allows PTE updates and +* page invalidations to be delayed until a call to leave lazy MMU mode +* is issued. Some architectures may benefit from doing this, and it is +* beneficial for both shadow and direct mode hypervisors, which may batch +* the PTE updates which happen during this window. Note that using this +* interface requires that read hazards be removed from the code. A read +* hazard could result in the direct mode hypervisor case, since the actual +* write to the page tables may not yet have taken place, so reads though +* a raw PTE pointer after it has been modified are not guaranteed to be +* up to date. This mode can only be entered and left under the protection of +* the page table locks for all page tables which may be modified. In the UP +* case, this is required so that preemption is disabled, and in the SMP case, +* it must synchronize the delayed page table writes properly on other CPUs. +*/ +#ifndef __HAVE_ARCH_ENTER_LAZY_MMU_MODE +#define arch_enter_lazy_mmu_mode () do {} while (0) +#define arch_leave_lazy_mmu_mode () do {} while (0) +#define arch_flush_lazy_mmu_mode () do {} while (0) +#endif + +#ifndef set_ptes +#ifdef PFN_PTE_SHIFT +/** +* set_ptes - Map consecutive pages to a contiguous range of addresses. +* @mm: Address space to map the pages into. +* @addr: Address to map the first page at. +* @ptep: Page table pointer for the first entry. +* @pte: Page table entry for the first page. +* @nr: Number of pages to map. +* +* May be overridden by the architecture, or the architecture can define +* set_pte() and PFN_PTE_SHIFT. +* +* Context: The caller holds the page table lock. The pages all belong +* to the same folio. The PTEs are all in the same PMD. +*/ +static inline void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pte,unsigned int nr) +{ +page_table_check_ptes_set (mm, ptep, pte, nr); + +arch_enter_lazy_mmu_mode (); +for (;;) { +set_pte (ptep, pte); +if (--nr ==0 ) +break ; + ptep++; + pte =__pte (pte_val (pte) + (1UL <<PFN_PTE_SHIFT )); + } +arch_leave_lazy_mmu_mode (); +} +#ifndef set_pte_at +#define set_pte_at (mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1) +#endif +#endif +#else +#define set_pte_at (mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1) +#endif +#ifndef __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep,
@@ -1051,27 +1111,6 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)#define pgprot_decrypted (prot) (prot)#endif -/* -* A facility to provide lazy MMU batching. This allows PTE updates and -* page invalidations to be delayed until a call to leave lazy MMU mode -* is issued. Some architectures may benefit from doing this, and it is -* beneficial for both shadow and direct mode hypervisors, which may batch -* the PTE updates which happen during this window. Note that using this -* interface requires that read hazards be removed from the code. A read -* hazard could result in the direct mode hypervisor case, since the actual -* write to the page tables may not yet have taken place, so reads though -* a raw PTE pointer after it has been modified are not guaranteed to be -* up to date. This mode can only be entered and left under the protection of -* the page table locks for all page tables which may be modified. In the UP -* case, this is required so that preemption is disabled, and in the SMP case, -* it must synchronize the delayed page table writes properly on other CPUs. -*/ -#ifndef __HAVE_ARCH_ENTER_LAZY_MMU_MODE -#define arch_enter_lazy_mmu_mode () do {} while (0) -#define arch_leave_lazy_mmu_mode () do {} while (0) -#define arch_flush_lazy_mmu_mode () do {} while (0) -#endif -/* * A facility to provide batching of the reload of page tables and * other process state with the actual context switch code for
Signature · 1 line
2.40.1
Re: [PATCH v6 06/38] mm: Add default definition of set_ptes()
Quoted 61 lines · Matthew Wilcox (Oracle)
Most architectures can just define
set_pte()andPFN_PTE_SHIFTto use this definition. It's also a handy spot to document the guarantees provided by the MM.Suggested-by Mike Rapoport (IBM) <[email protected]>Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Reviewed-by Mike Rapoport (IBM) <[email protected]>---
include/linux/pgtable.h | 81 ++++++++++++++++++++++++++++++-----------1 file changed, 60 insertions(+), 21 deletions(-)include/linux/pgtable.h +0
-0@@ -182,6 +182,66 @@ static inline int pmd_young(pmd_t pmd) } #endifinclude/linux/pgtable.h +42
-0+/* +* A facility to provide lazy MMU batching. This allows PTE updates and +* page invalidations to be delayed until a call to leave lazy MMU mode +* is issued. Some architectures may benefit from doing this, and it is +* beneficial for both shadow and direct mode hypervisors, which may batch +* the PTE updates which happen during this window. Note that using this +* interface requires that read hazards be removed from the code. A read +* hazard could result in the direct mode hypervisor case, since the actual +* write to the page tables may not yet have taken place, so reads though +* a raw PTE pointer after it has been modified are not guaranteed to be +* up to date. This mode can only be entered and left under the protection of +* the page table locks for all page tables which may be modified. In the UP +* case, this is required so that preemption is disabled, and in the SMP case, +* it must synchronize the delayed page table writes properly on other CPUs. +*/ +#ifndef __HAVE_ARCH_ENTER_LAZY_MMU_MODE +#define arch_enter_lazy_mmu_mode () do {} while (0) +#define arch_leave_lazy_mmu_mode () do {} while (0) +#define arch_flush_lazy_mmu_mode () do {} while (0) +#endif + +#ifndef set_ptes +#ifdef PFN_PTE_SHIFT +/** + * set_ptes - Map consecutive pages to a contiguous range of addresses. + * @mm: Address space to map the pages into. + * @addr: Address to map the first page at. + * @ptep: Page table pointer for the first entry. + * @pte: Page table entry for the first page. + * @nr: Number of pages to map. + * + * May be overridden by the architecture, or the architecture can define + * set_pte() and PFN_PTE_SHIFT. + * + * Context: The caller holds the page table lock. The pages all belong + * to the same folio. The PTEs are all in the same PMD. + */ +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + page_table_check_ptes_set(mm, ptep, pte, nr); +
include/linux/pgtable.h static inline int pmd_young(pmd_t pmd) · line 236
Quoted 10 lines
include/linux/pgtable.h +10
-0+ arch_enter_lazy_mmu_mode(); + for (;;) { + set_pte(ptep, pte); + if (--nr == 0) + break; + ptep++; + pte = __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + } + arch_leave_lazy_mmu_mode(); +}
This breaks the Xen PV guest.
In move_ptes() in mm/mremap.c we arch_enter_lazy_mmu_mode() and then loop calling set_pte_at(). Which now (or at least in a few commits time when you wire it up for x86 in commit a3e1c9372c9b959) ends up in your implementation of set_ptes(), calls arch_enter_lazy_mmu_mode() again, and:
[ 0.628700] ------------[ cut here ]------------ [ 0.628718] kernel BUG at arch/x86/kernel/paravirt.c:144! [ 0.628743] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 0.628769] CPU: 0 PID: 1 Comm: init Not tainted 6.5.0-rc4+ #1295 [ 0.628818] RIP: e030:paravirt_enter_lazy_mmu+0x24/0x30 [ 0.628839] Code: 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 65 8b 05 90 28 f9 7e 85 c0 75 10 65 c7 05 81 28 f9 7e 01 00 00 00 c3 cc cc cc cc <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 [ 0.628875] RSP: e02b:ffffc9004000ba48 EFLAGS: 00010202 [ 0.628891] RAX: 0000000000000001 RBX: ffff8880051b7100 RCX: 000ffffffffff000 [ 0.628908] RDX: 80000000763ff967 RSI: 80000000763ff967 RDI: ffff8880051b7100 [ 0.628925] RBP: 80000000763ff967 R08: ffff8880051b6868 R09: 00007ffce1a20000 [ 0.628943] R10: deadbeefdeadf00d R11: 0000000000000000 R12: 00007ffffffff000 [ 0.628964] R13: ffff8880050b7000 R14: 0000000000000001 R15: 00007fffffffe000 [ 0.628988] FS: 0000000000000000(0000) GS:ffff88807b800000(0000) knlGS:0000000000000000 [ 0.629007] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.629024] CR2: ffffc900003f5000 CR3: 0000000003904000 CR4: 0000000000050660 [ 0.629046] Call Trace: [ 0.629055] <TASK> [ 0.629066] ? die+0x36/0x90 [ 0.629081] ? do_trap+0xda/0x100 [ 0.629093] ? paravirt_enter_lazy_mmu+0x24/0x30 [ 0.629112] ? do_error_trap+0x6a/0x90 [ 0.629123] ? paravirt_enter_lazy_mmu+0x24/0x30 [ 0.629138] ? exc_invalid_op+0x50/0x70 [ 0.629155] ? paravirt_enter_lazy_mmu+0x24/0x30 [ 0.629169] ? asm_exc_invalid_op+0x1a/0x20 [ 0.629185] ? paravirt_enter_lazy_mmu+0x24/0x30 [ 0.629212] ? pte_offset_map_nolock+0x48/0xc0 [ 0.629226] set_ptes.constprop.0+0xd/0x30 [ 0.629240] move_ptes.isra.0+0xdd/0x290 [ 0.629253] ? pmd_install+0xab/0xd0 [ 0.629267] move_page_tables+0x3a0/0x850 [ 0.629294] shift_arg_pages+0xf4/0x1d0 [ 0.629317] setup_arg_pages+0x205/0x380 [ 0.629330] load_elf_binary+0x398/0xe00
I'm working on making PV kernels testable in qemu. With...
• some qemu fixes and a nasty hackish Xen console implementation:
https://git.infradead.org/users/dwmw2/qemu.git/shortlog/refs/heads/xenfv-console
• a CONFIG_PV_SHIM_EXCLUSIVE build of Xen itself to run in the guest,
• some suitable disk image lying around, in ${GUEST_IMAGE}, and
• CONFIG_KVM_XEN enabled in your host kernel,
...you should be able to do something like:
$ ./qemu-system-x86_64 --accel kvm,xen-version=0x40011,kernel-irqchip=split -drive file=${GUEST_IMAGE},if=none,id=disk -device xen-disk,drive=disk,vdev=xvda -m 1G -kernel ~/git/xen/xen/xen -initrd ~/git/linux/arch/x86/boot/bzImage -append "loglvl=all -- console=hvc0 root=/dev/xvda1" -display noneMatthew Wilcox
maintainerRe: [PATCH v6 06/38] mm: Add default definition of set_ptes()
Quoted 12 lines · David Woodhouse
Quoted 9 lines
+ arch_enter_lazy_mmu_mode();
+ for (;;) {
+ set_pte(ptep, pte);
+ if (--nr == 0)
+ break;
+ ptep++;
+ pte =__pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + }
+ arch_leave_lazy_mmu_mode();This breaks the Xen PV guest.
Quoted 8 lines
In
move_ptes()inmm/mremap.cwearch_enter_lazy_mmu_mode()and then loop callingset_pte_at(). Which now (or at least in a few commits time when you wire it up for x86 in commita3e1c9372c9b959) ends up in your implementation ofset_ptes(), callsarch_enter_lazy_mmu_mode()again, and:[ 0.628700] ------------[ cut here ]------------ [ 0.628718] kernel BUG at arch/x86/kernel/paravirt.c:144!
Easy fix ... don't do that ;-)
include/linux/pgtable.h +5 -2
@@ -231,9 +231,11 @@ static inline pte_t pte_next_pfn(pte_t pte)static inline void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned int nr) { +bool multiple = nr >1 ;page_table_check_ptes_set (mm, ptep, pte, nr); -arch_enter_lazy_mmu_mode (); +if (multiple) +arch_enter_lazy_mmu_mode ();for (;;) {set_pte (ptep, pte);if (--nr ==0 )
@@ -241,7 +243,8 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, ptep++; pte = pte_next_pfn(pte); } -arch_leave_lazy_mmu_mode (); +if (multiple) +arch_leave_lazy_mmu_mode (); }#endif #define set_pte_at (mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1)
I think long-term, we should make lazy_mmu_mode nestable. But this is a reasonable quick fix.
David Woodhouse
Tested-byRe: [PATCH v6 06/38] mm: Add default definition of set_ptes()
Quoted 52 lines · Matthew Wilcox
Quoted 12 lines · David Woodhouse
Quoted 9 lines
+ arch_enter_lazy_mmu_mode();
+ for (;;) {
+ set_pte(ptep, pte);
+ if (--nr == 0)
+ break;
+ ptep++;
+ pte =__pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + }
+ arch_leave_lazy_mmu_mode();This breaks the Xen PV guest.
Quoted 8 lines
In
move_ptes()inmm/mremap.cwearch_enter_lazy_mmu_mode()and then loop callingset_pte_at(). Which now (or at least in a few commits time when you wire it up for x86 in commita3e1c9372c9b959) ends up in your implementation ofset_ptes(), callsarch_enter_lazy_mmu_mode()again, and:[ 0.628700] ------------[ cut here ]------------ [ 0.628718] kernel BUG at
arch/x86/kernel/paravirt.c:144!Easy fix ... don't do that ;-)
include/linux/pgtable.h +1
-0@@ -231,9 +231,11 @@ static inline pte_t pte_next_pfn(pte_t pte) static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr) { + bool multiple = nr > 1; page_table_check_ptes_set(mm, ptep, pte, nr);
- arch_enter_lazy_mmu_mode();
+ if (multiple)
+ arch_enter_lazy_mmu_mode();
for (;;) {
set_pte(ptep, pte);
if (--nr == 0)include/linux/pgtable.h +2
-1@@ -241,7 +243,8 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, ptep++; pte = pte_next_pfn(pte); } - arch_leave_lazy_mmu_mode(); + if (multiple) + arch_leave_lazy_mmu_mode(); } #endif #define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1)
Quoted 2 lines
I think long-term, we should make
lazy_mmu_modenestable. But this is a reasonable quick fix.
I don't much like doing it implicitly based on (nr==1) but sure, as a quick fix that works. The 64-bit PV guest now boots again.
Thanks.
Matthew Wilcox (Oracle)
maintainerPATCH v6 7/38 alpha: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range() and flush_icache_pages().
--- arch/alpha/include/asm/cacheflush.h | 10 ++++++++++ arch/alpha/include/asm/pgtable.h | 10 ++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-)
arch/alpha/include/asm/cacheflush.h +10 -0
@@ -57,6 +57,16 @@ extern void flush_icache_user_page(struct vm_area_struct *vma,#define flush_icache_page (vma, page) \ flush_icache_user_page((vma), (page), 0, 0) +/* +* Both implementations of flush_icache_user_page flush the entire +* address space, so one call, no matter how many pages. +*/ +static inline void flush_icache_pages (struct vm_area_struct *vma, +struct page *page,unsigned int nr) +{ +flush_icache_user_page (vma, page,0 ,0 ); +} +#include <asm-generic/cacheflush.h> #endif /* _ALPHA_CACHEFLUSH_H */
arch/alpha/include/asm/pgtable.h +8 -2
@@ -26,7 +26,6 @@ struct vm_area_struct;* hook is made available. */ #define set_pte (pteptr, pteval) ((*(pteptr)) = (pteval)) -#define set_pte_at (mm,addr,ptep,pteval) set_pte(ptep,pteval)/* PMD_SHIFT determines the size of the area a second-level page table can map */ #define PMD_SHIFT (PAGE_SHIFT + (PAGE_SHIFT-3))
@@ -189,7 +188,8 @@ extern unsigned long __zero_page(void);* and a page entry and page directory to the page they refer to. */ #define page_to_pa (page) (page_to_pfn(page) << PAGE_SHIFT) -#define pte_pfn (pte) (pte_val(pte) >> 32) +#define PFN_PTE_SHIFT 32 +#define pte_pfn (pte) (pte_val(pte) >> PFN_PTE_SHIFT)#define pte_page (pte) pfn_to_page(pte_pfn(pte))#define mk_pte (page, pgprot) \
@@ -303,6 +303,12 @@ extern inline void update_mmu_cache(struct vm_area_struct * vma, { } +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) +{ +} +/* * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that * are !pte_none() && !pte_present().
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 8/38 arc: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages().
Change the PG_dc_clean flag from being per-page to per-folio (which means it cannot always be set as we don't know that all pages in this folio were cleaned). Enhance the internal flush routines to take the number of pages to flush.
--- arch/arc/include/asm/cacheflush.h | 7 ++- arch/arc/include/asm/pgtable-bits-arcv2.h | 12 ++--- arch/arc/include/asm/pgtable-levels.h | 1 + arch/arc/mm/cache.c | 61 ++++++++++++++--------- arch/arc/mm/tlb.c | 18 ++++--- 5 files changed, 59 insertions(+), 40 deletions(-)
arch/arc/include/asm/cacheflush.h +5 -2
@@ -25,17 +25,20 @@* in update_mmu_cache() */ #define flush_icache_page (vma, page) +#define flush_icache_pages (vma, page, nr)void flush_cache_all (void );void flush_icache_range (unsigned long kstart,unsigned long kend);void __sync_icache_dcache (phys_addr_t paddr,unsigned long vaddr,int len); -void __inv_icache_page (phys_addr_t paddr,unsigned long vaddr); -void __flush_dcache_page (phys_addr_t paddr,unsigned long vaddr); +void __inv_icache_pages (phys_addr_t paddr,unsigned long vaddr,unsigned nr); +void __flush_dcache_pages (phys_addr_t paddr,unsigned long vaddr,unsigned nr);#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1void flush_dcache_page (struct page *page); +void flush_dcache_folio (struct folio *folio); +#define flush_dcache_folio flush_dcache_foliovoid dma_cache_wback_inv (phys_addr_t start,unsigned long sz);void dma_cache_inv (phys_addr_t start,unsigned long sz);
arch/arc/include/asm/pgtable-bits-arcv2.h +5 -7
@@ -100,14 +100,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)return __pte ((pte_val (pte) & _PAGE_CHG_MASK) |pgprot_val (newprot)); } -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pteval) -{ -set_pte (ptep, pteval); -} +struct vm_fault ; +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long address,pte_t *ptep,unsigned int nr); -void update_mmu_cache (struct vm_area_struct *vma,unsigned long address, -pte_t *ptep); +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)/* * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
arch/arc/include/asm/pgtable-levels.h +1 -0
@@ -169,6 +169,7 @@#define pte_ERROR (e) \ pr_crit("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e)) +#define PFN_PTE_SHIFT PAGE_SHIFT#define pte_none (x) (!pte_val(x))#define pte_present (x) (pte_val(x) & _PAGE_PRESENT)#define pte_clear (mm,addr,ptep) set_pte_at(mm, addr, ptep, __pte(0))
arch/arc/mm/cache.c +37 -24
146 lines not on this page · raw message
arch/arc/mm/tlb.c +11 -7
@@ -467,8 +467,8 @@ void create_tlb(struct vm_area_struct *vma, unsigned long vaddr, pte_t *ptep)* Note that flush (when done) involves both WBACK - so physical page is * in sync as well as INV - so any non-congruent aliases don't remain */ -void update_mmu_cache (struct vm_area_struct *vma,unsigned long vaddr_unaligned, -pte_t *ptep) +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long vaddr_unaligned,pte_t *ptep,unsigned int nr) {unsigned long vaddr = vaddr_unaligned &PAGE_MASK ;phys_addr_t paddr =pte_val (*ptep) &PAGE_MASK_PHYS ;
@@ -491,15 +491,19 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long vaddr_unaligned,*/ if ((vma->vm_flags &VM_EXEC ) ||addr_not_cache_congruent (paddr, vaddr)) { - -int dirty = !test_and_set_bit (PG_dc_clean, &page->flags); +struct folio *folio =page_folio (page); +int dirty = !test_and_set_bit (PG_dc_clean, &folio->flags);if (dirty) { +unsigned long offset =offset_in_folio (folio, paddr); + nr =folio_nr_pages (folio); + paddr -= offset; + vaddr -= offset;/* wback + inv dcache lines (K-mapping) */ -__flush_dcache_page (paddr, paddr); +__flush_dcache_pages (paddr, paddr, nr);/* invalidate any existing icache lines (U-mapping) */ if (vma->vm_flags &VM_EXEC ) -__inv_icache_page (paddr, vaddr); +__inv_icache_pages (paddr, vaddr, nr); } } }
@@ -531,7 +535,7 @@ void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) {pte_t pte =__pte (pmd_val (*pmd)); -update_mmu_cache (vma, addr, &pte); +update_mmu_cache_range (NULL , vma, addr, &pte,HPAGE_PMD_NR ); } void local_flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
Signature · 7 lines
2.40.1 _______________________________________________ linux-snps-arc mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Matthew Wilcox (Oracle)
maintainerPATCH v6 9/38 arm: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes(), update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages(). Change the PG_dcache_clear flag from being per-page to per-folio which makes __dma_page_dev_to_cpu() a bit more exciting. Also add flush_cache_pages(), even though this isn't used by generic code (yet?)
--- arch/arm/include/asm/cacheflush.h | 24 +++++--- arch/arm/include/asm/pgtable.h | 5 +- arch/arm/include/asm/tlbflush.h | 14 +++-- arch/arm/mm/copypage-v4mc.c | 5 +- arch/arm/mm/copypage-v6.c | 5 +- arch/arm/mm/copypage-xscale.c | 5 +- arch/arm/mm/dma-mapping.c | 24 ++++---- arch/arm/mm/fault-armv.c | 16 ++--- arch/arm/mm/flush.c | 99 +++++++++++++++++++------------ arch/arm/mm/mm.h | 2 +- arch/arm/mm/mmu.c | 14 +++-- arch/arm/mm/nommu.c | 6 ++ 12 files changed, 133 insertions(+), 86 deletions(-)
arch/arm/include/asm/cacheflush.h +15 -9
@@ -231,14 +231,15 @@ vivt_flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned vma->vm_flags); } -static inline void -vivt_flush_cache_page (struct vm_area_struct *vma,unsigned long user_addr,unsigned long pfn) +static inline void vivt_flush_cache_pages (struct vm_area_struct *vma, +unsigned long user_addr,unsigned long pfn,unsigned int nr) {struct mm_struct *mm = vma->vm_mm;if (!mm ||cpumask_test_cpu (smp_processor_id (),mm_cpumask (mm))) {unsigned long addr = user_addr &PAGE_MASK ; -__cpuc_flush_user_range (addr, addr +PAGE_SIZE , vma->vm_flags); +__cpuc_flush_user_range (addr, addr + nr *PAGE_SIZE , + vma->vm_flags); } }
@@ -247,15 +248,17 @@ vivt_flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsig vivt_flush_cache_mm(mm)#define flush_cache_range (vma,start,end) \ vivt_flush_cache_range(vma,start,end) -#define flush_cache_page (vma,addr,pfn) \ - vivt_flush_cache_page(vma,addr,pfn) +#define flush_cache_pages (vma, addr, pfn, nr) \ + vivt_flush_cache_pages(vma, addr, pfn, nr)#else -extern void flush_cache_mm (struct mm_struct *mm); -extern void flush_cache_range (struct vm_area_struct *vma,unsigned long start,unsigned long end); -extern void flush_cache_page (struct vm_area_struct *vma,unsigned long user_addr,unsigned long pfn); +void flush_cache_mm (struct mm_struct *mm); +void flush_cache_range (struct vm_area_struct *vma,unsigned long start,unsigned long end); +void flush_cache_pages (struct vm_area_struct *vma,unsigned long user_addr, +unsigned long pfn,unsigned int nr);#endif #define flush_cache_dup_mm (mm) flush_cache_mm(mm) +#define flush_cache_page (vma, addr, pfn) flush_cache_pages(vma, addr, pfn, 1)/* * flush_icache_user_range is used when we want to ensure that the
@@ -289,7 +292,9 @@ extern void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr* See update_mmu_cache for the user space part. */ #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -extern void flush_dcache_page (struct page *); +void flush_dcache_page (struct page *); +void flush_dcache_folio (struct folio *folio); +#define flush_dcache_folio flush_dcache_folio#define ARCH_IMPLEMENTS_FLUSH_KERNEL_VMAP_RANGE 1static inline void flush_kernel_vmap_range (void *addr,int size)
@@ -321,6 +326,7 @@ static inline void flush_anon_page(struct vm_area_struct *vma,* duplicate cache flushing elsewhere performed by flush_dcache_page(). */ #define flush_icache_page (vma,page) do { } while (0) +#define flush_icache_pages (vma, page, nr) do { } while (0)/* * flush_cache_vmap() is used when creating mappings (eg, via vmap,
arch/arm/include/asm/pgtable.h +3 -2
@@ -207,8 +207,9 @@ static inline void __sync_icache_dcache(pte_t pteval)extern void __sync_icache_dcache (pte_t pteval);#endif -void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pteval); +void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pteval,unsigned int nr); +#define set_ptes set_ptesstatic inline pte_t clear_pte_bit (pte_t pte,pgprot_t prot) {
arch/arm/include/asm/tlbflush.h +9 -5
@@ -619,18 +619,22 @@ extern void flush_bp_all(void);* If PG_dcache_clean is not set for the page, we need to ensure that any * cache entries for the kernels virtual memory range are written * back to the page. On ARMv6 and later, the cache coherency is handled via -* the set_pte_at() function. +* the set_ptes() function. */ #if __LINUX_ARM_ARCH__ <6 -extern void update_mmu_cache (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep); +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr);#else -static inline void update_mmu_cache (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long addr,pte_t *ptep, +unsigned int nr) { }#endif +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1) +#define update_mmu_cache_pmd (vma, address, pmd) do { } while (0)#endif
arch/arm/mm/copypage-v4mc.c +3 -2
@@ -64,10 +64,11 @@ static void mc_copy_user_page(void *from, void *to)void v4_mc_copy_user_highpage (struct page *to,struct page *from,unsigned long vaddr,struct vm_area_struct *vma) { +struct folio *src =page_folio (from);void *kto =kmap_atomic (to); -if (!test_and_set_bit (PG_dcache_clean, &from->flags)) -__flush_dcache_page (page_mapping_file (from), from); +if (!test_and_set_bit (PG_dcache_clean, &src->flags)) +__flush_dcache_folio (folio_flush_mapping (src), src);raw_spin_lock (&minicache_lock);
arch/arm/mm/copypage-v6.c +3 -2
@@ -69,11 +69,12 @@ static void discard_old_kernel_data(void *kto)static void v6_copy_user_highpage_aliasing (struct page *to,struct page *from,unsigned long vaddr,struct vm_area_struct *vma) { +struct folio *src =page_folio (from);unsigned int offset =CACHE_COLOUR (vaddr);unsigned long kfrom, kto; -if (!test_and_set_bit (PG_dcache_clean, &from->flags)) -__flush_dcache_page (page_mapping_file (from), from); +if (!test_and_set_bit (PG_dcache_clean, &src->flags)) +__flush_dcache_folio (folio_flush_mapping (src), src);/* FIXME: not highmem safe */ discard_old_kernel_data (page_address (to));
arch/arm/mm/copypage-xscale.c +3 -2
@@ -84,10 +84,11 @@ static void mc_copy_user_page(void *from, void *to)void xscale_mc_copy_user_highpage (struct page *to,struct page *from,unsigned long vaddr,struct vm_area_struct *vma) { +struct folio *src =page_folio (from);void *kto =kmap_atomic (to); -if (!test_and_set_bit (PG_dcache_clean, &from->flags)) -__flush_dcache_page (page_mapping_file (from), from); +if (!test_and_set_bit (PG_dcache_clean, &src->flags)) +__flush_dcache_folio (folio_flush_mapping (src), src);raw_spin_lock (&minicache_lock);
arch/arm/mm/dma-mapping.c +12 -12
@@ -695,6 +695,7 @@ static void __dma_page_cpu_to_dev(struct page *page, unsigned long off,static void __dma_page_dev_to_cpu (struct page *page,unsigned long off,size_t size,enum dma_data_direction dir) { +struct folio *folio =page_folio (page);phys_addr_t paddr =page_to_phys (page) + off;/* FIXME: non-speculating: not required */
@@ -709,19 +710,18 @@ static void __dma_page_dev_to_cpu(struct page *page, unsigned long off,* Mark the D-cache clean for these pages to avoid extra flushing. */ if (dir !=DMA_TO_DEVICE && size >=PAGE_SIZE ) { -unsigned long pfn; -size_t left = size; - - pfn =page_to_pfn (page) + off /PAGE_SIZE ; - off %=PAGE_SIZE ; -if (off) { - pfn++; - left -=PAGE_SIZE - off; +ssize_t left = size; +size_t offset =offset_in_folio (folio, paddr); + +if (offset) { + left -=folio_size (folio) - offset; + folio =folio_next (folio); } -while (left >=PAGE_SIZE ) { - page =pfn_to_page (pfn++); -set_bit (PG_dcache_clean, &page->flags); - left -=PAGE_SIZE ; + +while (left >= (ssize_t )folio_size (folio)) { +set_bit (PG_dcache_clean, &folio->flags); + left -=folio_size (folio); + folio =folio_next (folio); } } }
arch/arm/mm/fault-armv.c +8 -8
@@ -180,12 +180,12 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma,* * Note that the pte lock will be held. */ -void update_mmu_cache (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep) +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) {unsigned long pfn =pte_pfn (*ptep);struct address_space *mapping; -struct page *page; +struct folio *folio;if (!pfn_valid (pfn))return ;
@@ -194,13 +194,13 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long addr,* The zero page is never written to, so never has any dirty * cache lines, and therefore never needs to be flushed. */ - page =pfn_to_page (pfn); -if (page ==ZERO_PAGE (0 )) +if (is_zero_pfn (pfn))return ; - mapping =page_mapping_file (page); -if (!test_and_set_bit (PG_dcache_clean, &page->flags)) -__flush_dcache_page (mapping, page); + folio =page_folio (pfn_to_page (pfn)); + mapping =folio_flush_mapping (folio); +if (!test_and_set_bit (PG_dcache_clean, &folio->flags)) +__flush_dcache_folio (mapping, folio);if (mapping) {if (cache_is_vivt ())make_coherent (mapping, vma, addr, ptep, pfn);
arch/arm/mm/flush.c +60 -39
194 lines not on this page · raw message
arch/arm/mm/mm.h +1 -1
@@ -45,7 +45,7 @@ struct mem_type {const struct mem_type *get_mem_type (unsigned int type); -extern void __flush_dcache_page (struct address_space *mapping,struct page *page); +void __flush_dcache_folio (struct address_space *mapping,struct folio *folio);/* * ARM specific vm_struct->flags bits.
arch/arm/mm/mmu.c +10 -4
@@ -1788,7 +1788,7 @@ void __init paging_init(const struct machine_desc *mdesc)bootmem_init (); empty_zero_page =virt_to_page (zero_page); -__flush_dcache_page (NULL , empty_zero_page); +__flush_dcache_folio (NULL ,page_folio (empty_zero_page)); }void __init early_mm_init (const struct machine_desc *mdesc)
@@ -1797,8 +1797,8 @@ void __init early_mm_init(const struct machine_desc *mdesc)early_paging_init (mdesc); } -void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pteval) +void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pteval,unsigned int nr) {unsigned long ext =0 ;
@@ -1808,5 +1808,11 @@ void set_pte_at(struct mm_struct *mm, unsigned long addr, ext |=PTE_EXT_NG ; } -set_pte_ext (ptep, pteval, ext); +for (;;) { +set_pte_ext (ptep, pteval, ext); +if (--nr ==0 ) +break ; + ptep++; +pte_val (pteval) +=PAGE_SIZE ; + } }
arch/arm/mm/nommu.c +6 -0
@@ -180,6 +180,12 @@ void setup_mm_for_reboot(void) { } +void flush_dcache_folio (struct folio *folio) +{ +__cpuc_flush_dcache_area (folio_address (folio),folio_size (folio)); +} +EXPORT_SYMBOL (flush_dcache_folio); +void flush_dcache_page (struct page *page) {__cpuc_flush_dcache_area (page_address (page),PAGE_SIZE );
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 10/38 arm64: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes(), update_mmu_cache_range() and flush_dcache_folio(). Change the PG_dcache_clean flag from being per-page to per-folio.
--- arch/arm64/include/asm/cacheflush.h | 4 +++- arch/arm64/include/asm/pgtable.h | 26 +++++++++++++++------ arch/arm64/mm/flush.c | 36 +++++++++++------------------ 3 files changed, 36 insertions(+), 30 deletions(-)
arch/arm64/include/asm/cacheflush.h +3 -1
@@ -114,7 +114,7 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *,#define copy_to_user_page copy_to_user_page/* -* flush_dcache_page is used when the kernel has written to the page +* flush_dcache_folio is used when the kernel has written to the page * cache page at virtual address page->virtual. * * If this page isn't mapped (ie, page_mapping == NULL), or it might
@@ -127,6 +127,8 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *,*/ #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1extern void flush_dcache_page (struct page *); +void flush_dcache_folio (struct folio *); +#define flush_dcache_folio flush_dcache_folio static __always_inline void icache_inval_all_pou(void) {
arch/arm64/include/asm/pgtable.h +19 -7
@@ -345,12 +345,21 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,set_pte (ptep, pte); } -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte) -{ -page_table_check_ptes_set (mm, ptep, pte,1 ); -return __set_pte_at (mm, addr, ptep, pte); +static inline void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pte,unsigned int nr) +{ +page_table_check_ptes_set (mm, ptep, pte, nr); + +for (;;) { +__set_pte_at (mm, addr, ptep, pte); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; +pte_val (pte) +=PAGE_SIZE ; + } } +#define set_ptes set_ptes/* * Huge pte definitions.
@@ -1049,8 +1058,9 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)/* * On AArch64, the cache coherency is handled via the set_pte_at() function. */ -static inline void update_mmu_cache (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long addr,pte_t *ptep, +unsigned int nr) {/* * We don't do anything here, so there's a very small chance of
@@ -1059,6 +1069,8 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,*/ } +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)#define update_mmu_cache_pmd (vma, address, pmd) do { } while (0)#ifdef CONFIG_ARM64_PA_BITS_52
arch/arm64/mm/flush.c +14 -22
@@ -51,20 +51,13 @@ void copy_to_user_page(struct vm_area_struct *vma, struct page *page,void __sync_icache_dcache (pte_t pte) { -struct page *page =pte_page (pte); +struct folio *folio =page_folio (pte_page (pte)); -/* -* HugeTLB pages are always fully mapped, so only setting head page's -* PG_dcache_clean flag is enough. -*/ -if (PageHuge (page)) - page =compound_head (page); - -if (!test_bit (PG_dcache_clean, &page->flags)) { -sync_icache_aliases ((unsigned long )page_address (page), - (unsigned long )page_address (page) + -page_size (page)); -set_bit (PG_dcache_clean, &page->flags); +if (!test_bit (PG_dcache_clean, &folio->flags)) { +sync_icache_aliases ((unsigned long )folio_address (folio), + (unsigned long )folio_address (folio) + +folio_size (folio)); +set_bit (PG_dcache_clean, &folio->flags); } }EXPORT_SYMBOL_GPL (__sync_icache_dcache);
@@ -74,17 +67,16 @@ EXPORT_SYMBOL_GPL(__sync_icache_dcache);* it as dirty for later flushing when mapped in user space (if executable, * see __sync_icache_dcache). */ -void flush_dcache_page (struct page *page) +void flush_dcache_folio (struct folio *folio) { -/* -* HugeTLB pages are always fully mapped and only head page will be -* set PG_dcache_clean (see comments in __sync_icache_dcache()). -*/ -if (PageHuge (page)) - page =compound_head (page); +if (test_bit (PG_dcache_clean, &folio->flags)) +clear_bit (PG_dcache_clean, &folio->flags); +} +EXPORT_SYMBOL (flush_dcache_folio); -if (test_bit (PG_dcache_clean, &page->flags)) -clear_bit (PG_dcache_clean, &page->flags); +void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); }EXPORT_SYMBOL (flush_dcache_page);
Signature · 7 lines
2.40.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Matthew Wilcox (Oracle)
maintainerPATCH v6 11/38 csky: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range() and flush_dcache_folio(). Change the PG_dcache_clean flag from being per-page to per-folio.
--- arch/csky/abiv1/cacheflush.c | 32 +++++++++++++++++----------- arch/csky/abiv1/inc/abi/cacheflush.h | 2 ++ arch/csky/abiv2/cacheflush.c | 32 ++++++++++++++-------------- arch/csky/abiv2/inc/abi/cacheflush.h | 10 +++++++-- arch/csky/include/asm/pgtable.h | 8 ++++--- 5 files changed, 50 insertions(+), 34 deletions(-)
arch/csky/abiv1/cacheflush.c +19 -13
@@ -15,45 +15,51 @@#define PG_dcache_clean PG_arch_1 -void flush_dcache_page (struct page *page) +void flush_dcache_folio (struct folio *folio) {struct address_space *mapping; -if (page ==ZERO_PAGE (0 )) +if (is_zero_pfn (folio_pfn (folio)))return ; - mapping =page_mapping_file (page); + mapping =folio_flush_mapping (folio); -if (mapping && !page_mapcount (page)) -clear_bit (PG_dcache_clean, &page->flags); +if (mapping && !folio_mapped (folio)) +clear_bit (PG_dcache_clean, &folio->flags);else {dcache_wbinv_all ();if (mapping)icache_inv_all (); -set_bit (PG_dcache_clean, &page->flags); +set_bit (PG_dcache_clean, &folio->flags); } } +EXPORT_SYMBOL (flush_dcache_folio); + +void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +}EXPORT_SYMBOL (flush_dcache_page); -void update_mmu_cache (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep) +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) {unsigned long pfn =pte_pfn (*ptep); -struct page *page; +struct folio *folio;flush_tlb_page (vma, addr);if (!pfn_valid (pfn))return ; - page =pfn_to_page (pfn); -if (page ==ZERO_PAGE (0 )) +if (is_zero_pfn (pfn))return ; -if (!test_and_set_bit (PG_dcache_clean, &page->flags)) + folio =page_folio (pfn_to_page (pfn)); +if (!test_and_set_bit (PG_dcache_clean, &folio->flags))dcache_wbinv_all (); -if (page_mapping_file (page)) { +if (folio_flush_mapping (folio)) {if (vma->vm_flags &VM_EXEC )icache_inv_all (); }
arch/csky/abiv1/inc/abi/cacheflush.h +2 -0
@@ -9,6 +9,8 @@#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1extern void flush_dcache_page (struct page *); +void flush_dcache_folio (struct folio *); +#define flush_dcache_folio flush_dcache_folio#define flush_cache_mm (mm) dcache_wbinv_all()#define flush_cache_page (vma, page, pfn) cache_wbinv_all()
arch/csky/abiv2/cacheflush.c +16 -16
@@ -7,32 +7,32 @@#include <asm/cache.h> #include <asm/tlbflush.h> -void update_mmu_cache (struct vm_area_struct *vma,unsigned long address, -pte_t *pte) +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long address,pte_t *pte,unsigned int nr) { -unsigned long addr; -struct page *page; +unsigned long pfn =pte_pfn (*pte); +struct folio *folio; +unsigned int i;flush_tlb_page (vma, address); -if (!pfn_valid (pte_pfn (*pte))) +if (!pfn_valid (pfn))return ; - page =pfn_to_page (pte_pfn (*pte)); -if (page ==ZERO_PAGE (0 )) -return ; + folio =page_folio (pfn_to_page (pfn)); -if (test_and_set_bit (PG_dcache_clean, &page->flags)) +if (test_and_set_bit (PG_dcache_clean, &folio->flags))return ; - addr = (unsigned long )kmap_atomic (page); - -dcache_wb_range (addr, addr +PAGE_SIZE ); +for (i =0 ; i <folio_nr_pages (folio); i++) { +unsigned long addr = (unsigned long )kmap_local_folio (folio, + i *PAGE_SIZE ); -if (vma->vm_flags &VM_EXEC ) -icache_inv_range (addr, addr +PAGE_SIZE ); - -kunmap_atomic ((void *) addr); +dcache_wb_range (addr, addr +PAGE_SIZE ); +if (vma->vm_flags &VM_EXEC ) +icache_inv_range (addr, addr +PAGE_SIZE ); +kunmap_local ((void *) addr); + } }void flush_icache_deferred (struct mm_struct *mm)
arch/csky/abiv2/inc/abi/cacheflush.h +8 -2
@@ -18,11 +18,17 @@#define PG_dcache_clean PG_arch_1 +static inline void flush_dcache_folio (struct folio *folio) +{ +if (test_bit (PG_dcache_clean, &folio->flags)) +clear_bit (PG_dcache_clean, &folio->flags); +} +#define flush_dcache_folio flush_dcache_folio +#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1static inline void flush_dcache_page (struct page *page) { -if (test_bit (PG_dcache_clean, &page->flags)) -clear_bit (PG_dcache_clean, &page->flags); +flush_dcache_folio (page_folio (page)); }#define flush_dcache_mmap_lock (mapping) do { } while (0)
arch/csky/include/asm/pgtable.h +5 -3
@@ -28,6 +28,7 @@#define pgd_ERROR (e) \ pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e)) +#define PFN_PTE_SHIFT PAGE_SHIFT#define pmd_pfn (pmd) (pmd_phys(pmd) >> PAGE_SHIFT)#define pmd_page (pmd) (pfn_to_page(pmd_phys(pmd) >> PAGE_SHIFT))#define pte_clear (mm, addr, ptep) set_pte((ptep), \
@@ -90,7 +91,6 @@ static inline void set_pte(pte_t *p, pte_t pte)/* prevent out of order excution */ smp_mb (); } -#define set_pte_at (mm, addr, ptep, pteval) set_pte(ptep, pteval)static inline pte_t *pmd_page_vaddr (pmd_t pmd) {
@@ -263,8 +263,10 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)extern pgd_t swapper_pg_dir[PTRS_PER_PGD ];extern void paging_init (void ); -void update_mmu_cache (struct vm_area_struct *vma,unsigned long address, -pte_t *pte); +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long address,pte_t *pte,unsigned int nr); +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)#define io_remap_pfn_range (vma, vaddr, pfn, size, prot) \ remap_pfn_range(vma, vaddr, pfn, size, prot)
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 12/38 hexagon: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT and update_mmu_cache_range().
--- arch/hexagon/include/asm/cacheflush.h | 8 ++++++-- arch/hexagon/include/asm/pgtable.h | 9 +-------- 2 files changed, 7 insertions(+), 10 deletions(-)
arch/hexagon/include/asm/cacheflush.h +6 -2
@@ -58,12 +58,16 @@ extern void flush_cache_all_hexagon(void);* clean the cache when the PTE is set. * */ -static inline void update_mmu_cache (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {/* generic_ptrace_pokedata doesn't wind up here, does it? */ } +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1) +void copy_to_user_page (struct vm_area_struct *vma,struct page *page,unsigned long vaddr,void *dst,void *src,int len);#define copy_to_user_page copy_to_user_page
arch/hexagon/include/asm/pgtable.h +1 -8
@@ -338,6 +338,7 @@ static inline int pte_exec(pte_t pte)/* __swp_entry_to_pte - extract PTE from swap entry */ #define __swp_entry_to_pte (x) ((pte_t) { (x).val }) +#define PFN_PTE_SHIFT PAGE_SHIFT/* pfn_pte - convert page number and protection value to page table entry */ #define pfn_pte (pfn, pgprot) __pte((pfn << PAGE_SHIFT) | pgprot_val(pgprot))
@@ -345,14 +346,6 @@ static inline int pte_exec(pte_t pte)#define pte_pfn (pte) (pte_val(pte) >> PAGE_SHIFT)#define set_pmd (pmdptr, pmdval) (*(pmdptr) = (pmdval)) -/* -* set_pte_at - update page table and do whatever magic may be -* necessary to make the underlying hardware/firmware take note. -* -* VM may require a virtual instruction to alert the MMU. -*/ -#define set_pte_at (mm, addr, ptep, pte) set_pte(ptep, pte) -static inline unsigned long pmd_page_vaddr (pmd_t pmd) {return (unsigned long )__va (pmd_val (pmd) &PAGE_MASK );
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 13/38 ia64: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range() and flush_dcache_folio(). Change the PG_arch_1 (aka PG_dcache_clean) flag from being per-page to per-folio, which makes arch_dma_mark_clean() and mark_clean() a little more exciting.
--- arch/ia64/hp/common/sba_iommu.c | 26 +++++++++++++++----------- arch/ia64/include/asm/cacheflush.h | 14 ++++++++++---- arch/ia64/include/asm/pgtable.h | 4 ++-- arch/ia64/mm/init.c | 28 +++++++++++++++++++--------- 4 files changed, 46 insertions(+), 26 deletions(-)
arch/ia64/hp/common/sba_iommu.c +15 -11
@@ -798,22 +798,26 @@ sba_io_pdir_entry(u64 *pdir_ptr, unsigned long vba)#endif #ifdef ENABLE_MARK_CLEAN -/** +/* * Since DMA is i-cache coherent, any (complete) pages that were written via * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't have to * flush them when they get mapped into an executable vm-area. */ -static void -mark_clean (void *addr,size_t size) +static void mark_clean (void *addr,size_t size) { -unsigned long pg_addr, end; - - pg_addr =PAGE_ALIGN ((unsigned long ) addr); - end = (unsigned long ) addr + size; -while (pg_addr +PAGE_SIZE <= end) { -struct page *page =virt_to_page ((void *)pg_addr); -set_bit (PG_arch_1, &page->flags); - pg_addr +=PAGE_SIZE ; +struct folio *folio =virt_to_folio (addr); +ssize_t left = size; +size_t offset =offset_in_folio (folio, addr); + +if (offset) { + left -=folio_size (folio) - offset; + folio =folio_next (folio); + } + +while (left >=folio_size (folio)) { +set_bit (PG_arch_1, &folio->flags); + left -=folio_size (folio); + folio =folio_next (folio); } }#endif
arch/ia64/include/asm/cacheflush.h +10 -4
@@ -13,10 +13,16 @@#include <asm/page.h> #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -#define flush_dcache_page (page) \ -do { \ - clear_bit(PG_arch_1, &(page)->flags); \ -} while (0) +static inline void flush_dcache_folio (struct folio *folio) +{ +clear_bit (PG_arch_1, &folio->flags); +} +#define flush_dcache_folio flush_dcache_folio + +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +}extern void flush_icache_range (unsigned long start,unsigned long end);#define flush_icache_range flush_icache_range
arch/ia64/include/asm/pgtable.h +2 -2
@@ -206,6 +206,7 @@ ia64_phys_addr_valid (unsigned long addr)#define RGN_MAP_SHIFT (PGDIR_SHIFT + PTRS_PER_PGD_SHIFT - 3)#define RGN_MAP_LIMIT ((1UL << RGN_MAP_SHIFT) - PAGE_SIZE)/* per region addr limit */ +#define PFN_PTE_SHIFT PAGE_SHIFT/* * Conversion functions: convert page frame number (pfn) and a protection value to a page * table entry (pte).
@@ -303,8 +304,6 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) *ptep = pteval; } -#define set_pte_at (mm,addr,ptep,pteval) set_pte(ptep,pteval) -/* * Make page protection values cacheable, uncacheable, or write- * combining. Note that "protection" is really a misnomer here as the
@@ -396,6 +395,7 @@ pte_same (pte_t a, pte_t b)return pte_val (a) ==pte_val (b); } +#define update_mmu_cache_range (vmf, vma, address, ptep, nr) do { } while (0)#define update_mmu_cache (vma, address, ptep) do { } while (0)extern pgd_t swapper_pg_dir[PTRS_PER_PGD ];
arch/ia64/mm/init.c +19 -9
@@ -50,30 +50,40 @@ void __ia64_sync_icache_dcache (pte_t pte) {unsigned long addr; -struct page *page; +struct folio *folio; - page =pte_page (pte); - addr = (unsigned long )page_address (page); + folio =page_folio (pte_page (pte)); + addr = (unsigned long )folio_address (folio); -if (test_bit (PG_arch_1, &page->flags)) +if (test_bit (PG_arch_1, &folio->flags))return ;/* i-cache is already coherent with d-cache */ -flush_icache_range (addr, addr +page_size (page)); -set_bit (PG_arch_1, &page->flags);/* mark page as clean */ +flush_icache_range (addr, addr +folio_size (folio)); +set_bit (PG_arch_1, &folio->flags);/* mark page as clean */ }/* -* Since DMA is i-cache coherent, any (complete) pages that were written via +* Since DMA is i-cache coherent, any (complete) folios that were written via * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't have to * flush them when they get mapped into an executable vm-area. */ void arch_dma_mark_clean (phys_addr_t paddr,size_t size) {unsigned long pfn =PHYS_PFN (paddr); +struct folio *folio =page_folio (pfn_to_page (pfn)); +ssize_t left = size; +size_t offset =offset_in_folio (folio, paddr); -do { +if (offset) { + left -=folio_size (folio) - offset; + folio =folio_next (folio); + } + +while (left >= (ssize_t )folio_size (folio)) {set_bit (PG_arch_1, &pfn_to_page (pfn)->flags); - }while (++pfn <=PHYS_PFN (paddr + size -1 )); + left -=folio_size (folio); + folio =folio_next (folio); + } } inline void
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 14/38 loongarch: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add update_mmu_cache_range() and change _PFN_SHIFT to PFN_PTE_SHIFT. It would probably be more efficient to implement __update_tlb() by flushing the entire folio instead of calling __update_tlb() N times, but I'll leave that for someone who understands the architecture better.
--- arch/loongarch/include/asm/cacheflush.h | 1 + arch/loongarch/include/asm/pgtable-bits.h | 4 +-- arch/loongarch/include/asm/pgtable.h | 33 ++++++++++++----------- arch/loongarch/mm/pgtable.c | 2 +- arch/loongarch/mm/tlb.c | 2 +- 5 files changed, 23 insertions(+), 19 deletions(-)
arch/loongarch/include/asm/cacheflush.h +1 -0
@@ -47,6 +47,7 @@ void local_flush_icache_range(unsigned long start, unsigned long end);#define flush_cache_vmap (start, end) do { } while (0)#define flush_cache_vunmap (start, end) do { } while (0)#define flush_icache_page (vma, page) do { } while (0) +#define flush_icache_pages (vma, page) do { } while (0)#define flush_icache_user_page (vma, page, addr, len) do { } while (0)#define flush_dcache_page (page) do { } while (0)#define flush_dcache_mmap_lock (mapping) do { } while (0)
arch/loongarch/include/asm/pgtable-bits.h +2 -2
@@ -50,12 +50,12 @@#define _PAGE_NO_EXEC (_ULCAST_(1) << _PAGE_NO_EXEC_SHIFT)#define _PAGE_RPLV (_ULCAST_(1) << _PAGE_RPLV_SHIFT)#define _CACHE_MASK (_ULCAST_(3) << _CACHE_SHIFT) -#define _PFN_SHIFT (PAGE_SHIFT - 12 + _PAGE_PFN_SHIFT) +#define PFN_PTE_SHIFT (PAGE_SHIFT - 12 + _PAGE_PFN_SHIFT)#define _PAGE_USER (PLV_USER << _PAGE_PLV_SHIFT)#define _PAGE_KERN (PLV_KERN << _PAGE_PLV_SHIFT) -#define _PFN_MASK (~((_ULCAST_(1) << (_PFN_SHIFT)) - 1) & \ +#define _PFN_MASK (~((_ULCAST_(1) << (PFN_PTE_SHIFT)) - 1) & \ ((_ULCAST_(1) << (_PAGE_PFN_END_SHIFT)) - 1))/*
arch/loongarch/include/asm/pgtable.h +18 -15
@@ -237,9 +237,9 @@ extern pmd_t mk_pmd(struct page *page, pgprot_t prot);extern void set_pmd_at (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,pmd_t pmd);#define pte_page (x) pfn_to_page(pte_pfn(x)) -#define pte_pfn (x) ((unsigned long)(((x).pte & _PFN_MASK) >> _PFN_SHIFT)) -#define pfn_pte (pfn, prot) __pte(((pfn) << _PFN_SHIFT) | pgprot_val(prot)) -#define pfn_pmd (pfn, prot) __pmd(((pfn) << _PFN_SHIFT) | pgprot_val(prot)) +#define pte_pfn (x) ((unsigned long)(((x).pte & _PFN_MASK) >> PFN_PTE_SHIFT)) +#define pfn_pte (pfn, prot) __pte(((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot)) +#define pfn_pmd (pfn, prot) __pmd(((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot))/* * Initialize a new pgd / pud / pmd table with invalid pointers.
@@ -334,19 +334,13 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) } } -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pteval) -{ -set_pte (ptep, pteval); -} -static inline void pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {/* Preserve global status for the pair */ if (pte_val (*ptep_buddy (ptep)) & _PAGE_GLOBAL) -set_pte_at (mm, addr, ptep,__pte (_PAGE_GLOBAL)); +set_pte (ptep,__pte (_PAGE_GLOBAL));else -set_pte_at (mm, addr, ptep,__pte (0 )); +set_pte (ptep,__pte (0 )); }#define PGD_T_LOG2 (__builtin_ffs(sizeof(pgd_t)) - 1)
@@ -445,11 +439,20 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)extern void __update_tlb (struct vm_area_struct *vma,unsigned long address,pte_t *ptep); -static inline void update_mmu_cache (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) { -__update_tlb (vma, address, ptep); +for (;;) { +__update_tlb (vma, address, ptep); +if (--nr ==0 ) +break ; + address +=PAGE_SIZE ; + ptep++; + } } +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)#define __HAVE_ARCH_UPDATE_MMU_TLB#define update_mmu_tlb update_mmu_cache
@@ -462,7 +465,7 @@ static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,static inline unsigned long pmd_pfn (pmd_t pmd) { -return (pmd_val (pmd) & _PFN_MASK) >> _PFN_SHIFT; +return (pmd_val (pmd) & _PFN_MASK) >>PFN_PTE_SHIFT ; }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
arch/loongarch/mm/pgtable.c +1 -1
@@ -107,7 +107,7 @@ pmd_t mk_pmd(struct page *page, pgprot_t prot) {pmd_t pmd; -pmd_val (pmd) = (page_to_pfn (page) << _PFN_SHIFT) |pgprot_val (prot); +pmd_val (pmd) = (page_to_pfn (page) <<PFN_PTE_SHIFT ) |pgprot_val (prot);return pmd; }
arch/loongarch/mm/tlb.c +1 -1
@@ -252,7 +252,7 @@ static void output_pgtable_bits_defines(void)pr_define ("_PAGE_WRITE_SHIFT %d\n" , _PAGE_WRITE_SHIFT);pr_define ("_PAGE_NO_READ_SHIFT %d\n" , _PAGE_NO_READ_SHIFT);pr_define ("_PAGE_NO_EXEC_SHIFT %d\n" , _PAGE_NO_EXEC_SHIFT); -pr_define ("_PFN_SHIFT %d\n" , _PFN_SHIFT); +pr_define ("PFN_PTE_SHIFT %d\n" ,PFN_PTE_SHIFT );pr_debug ("\n" ); }
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 15/38 m68k: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio().
--- arch/m68k/include/asm/cacheflush_mm.h | 27 ++++++++++++++++-------- arch/m68k/include/asm/mcf_pgtable.h | 1 + arch/m68k/include/asm/motorola_pgtable.h | 1 + arch/m68k/include/asm/pgtable_mm.h | 10 +++++---- arch/m68k/include/asm/sun3_pgtable.h | 1 + arch/m68k/mm/motorola.c | 2 +- 6 files changed, 28 insertions(+), 14 deletions(-)
arch/m68k/include/asm/cacheflush_mm.h +18 -9
@@ -220,24 +220,29 @@ static inline void flush_cache_page(struct vm_area_struct *vma, unsigned long vm/* Push the page at kernel virtual address and clear the icache */ /* RZ: use cpush %bc instead of cpush %dc, cinv %ic */ -static inline void __flush_page_to_ram (void *vaddr) +static inline void __flush_pages_to_ram (void *vaddr,unsigned int nr) {if (CPU_IS_COLDFIRE ) {unsigned long addr, start, end; addr = ((unsigned long ) vaddr) & ~(PAGE_SIZE -1 ); start = addr &ICACHE_SET_MASK ; - end = (addr +PAGE_SIZE -1 ) &ICACHE_SET_MASK ; + end = (addr + nr *PAGE_SIZE -1 ) &ICACHE_SET_MASK ;if (start > end) {flush_cf_bcache (0 , end); end =ICACHE_MAX_ADDR ; }flush_cf_bcache (start, end); }else if (CPU_IS_040_OR_060 ) { - __asm__ __volatile__("nop\n\t" -".chip 68040\n\t" -"cpushp %%bc,(%0)\n\t" -".chip 68k" - : :"a" (__pa (vaddr))); +unsigned long paddr =__pa (vaddr); + +do { + __asm__ __volatile__("nop\n\t" +".chip 68040\n\t" +"cpushp %%bc,(%0)\n\t" +".chip 68k" + : :"a" (paddr)); + paddr +=PAGE_SIZE ; + }while (--nr); }else {unsigned long _tmp; __asm__ __volatile__("movec %%cacr,%0\n\t"
@@ -249,10 +254,14 @@ static inline void __flush_page_to_ram(void *vaddr) }#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -#define flush_dcache_page (page) __flush_page_to_ram(page_address(page)) +#define flush_dcache_page (page) __flush_pages_to_ram(page_address(page), 1) +#define flush_dcache_folio (folio) \ + __flush_pages_to_ram(folio_address(folio), folio_nr_pages(folio))#define flush_dcache_mmap_lock (mapping) do { } while (0)#define flush_dcache_mmap_unlock (mapping) do { } while (0) -#define flush_icache_page (vma, page) __flush_page_to_ram(page_address(page)) +#define flush_icache_pages (vma, page, nr) \ + __flush_pages_to_ram(page_address(page), nr) +#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1)extern void flush_icache_user_page (struct vm_area_struct *vma,struct page *page,unsigned long addr,int len);
arch/m68k/include/asm/mcf_pgtable.h +1 -0
@@ -291,6 +291,7 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte; } +#define PFN_PTE_SHIFT PAGE_SHIFT#define pmd_pfn (pmd) (pmd_val(pmd) >> PAGE_SHIFT)#define pmd_page (pmd) (pfn_to_page(pmd_val(pmd) >> PAGE_SHIFT))
arch/m68k/include/asm/motorola_pgtable.h +1 -0
@@ -112,6 +112,7 @@ static inline void pud_set(pud_t *pudp, pmd_t *pmdp)#define pte_present (pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE))#define pte_clear (mm,addr,ptep) ({ pte_val(*(ptep)) = 0; }) +#define PFN_PTE_SHIFT PAGE_SHIFT#define pte_page (pte) virt_to_page(__va(pte_val(pte)))#define pte_pfn (pte) (pte_val(pte) >> PAGE_SHIFT)#define pfn_pte (pfn, prot) __pte(((pfn) << PAGE_SHIFT) | pgprot_val(prot))
arch/m68k/include/asm/pgtable_mm.h +6 -4
@@ -31,8 +31,6 @@do { \ *(pteptr) = (pteval); \ }while (0 ) -#define set_pte_at (mm,addr,ptep,pteval) set_pte(ptep,pteval) - /* PMD_SHIFT determines the size of the area a second-level page table can map */#if CONFIG_PGTABLE_LEVELS ==3
@@ -138,11 +136,15 @@ extern void kernel_set_cachemode(void *addr, unsigned long size, int cmode);* tables contain all the necessary information. The Sun3 does, but * they are updated on demand. */ -static inline void update_mmu_cache (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) { } +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1) +#endif /* !__ASSEMBLY__ */ /* MMU-specific headers */
arch/m68k/include/asm/sun3_pgtable.h +1 -0
@@ -105,6 +105,7 @@ static inline void pte_clear (struct mm_struct *mm, unsigned long addr, pte_t *ppte_val (*ptep) =0 ; } +#define PFN_PTE_SHIFT 0#define pte_pfn (pte) (pte_val(pte) & SUN3_PAGE_PGNUM_MASK)#define pfn_pte (pfn, pgprot) \ ({ pte_t __pte; pte_val(__pte) = pfn | pgprot_val(pgprot); __pte; })
arch/m68k/mm/motorola.c +1 -1
@@ -81,7 +81,7 @@ static inline void cache_page(void *vaddr)void mmu_page_ctor (void *page) { -__flush_page_to_ram (page); +__flush_pages_to_ram (page,1 );flush_tlb_kernel_page (page);nocache_page (page); }
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 16/38 microblaze: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Rename PFN_SHIFT_OFFSET to PTE_PFN_SHIFT. Change the calling convention for set_pte() to be the same as other architectures. Add update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio().
--- arch/microblaze/include/asm/cacheflush.h | 8 ++++++++ arch/microblaze/include/asm/pgtable.h | 15 ++++----------- arch/microblaze/include/asm/tlbflush.h | 4 +++- 3 files changed, 15 insertions(+), 12 deletions(-)
arch/microblaze/include/asm/cacheflush.h +8 -0
@@ -74,6 +74,14 @@ do { \flush_dcache_range ((unsigned ) (addr), (unsigned ) (addr) +PAGE_SIZE ); \ }while (0 ); +static void flush_dcache_folio (struct folio *folio) +{ +unsigned long addr =folio_pfn (folio) <<PAGE_SHIFT ; + +flush_dcache_range (addr, addr +folio_size (folio)); +} +#define flush_dcache_folio flush_dcache_folio +#define flush_cache_page (vma, vmaddr, pfn) \ flush_dcache_range(pfn << PAGE_SHIFT, (pfn << PAGE_SHIFT) + PAGE_SIZE);
arch/microblaze/include/asm/pgtable.h +4 -11
@@ -230,12 +230,12 @@ extern unsigned long empty_zero_page[1024];#define pte_page (x) (mem_map + (unsigned long) \ ((pte_val(x) - memory_start) >> PAGE_SHIFT)) -#define PFN_SHIFT_OFFSET (PAGE_SHIFT) +#define PFN_PTE_SHIFT PAGE_SHIFT -#define pte_pfn (x) (pte_val(x) >> PFN_SHIFT_OFFSET) +#define pte_pfn (x) (pte_val(x) >> PFN_PTE_SHIFT)#define pfn_pte (pfn, prot) \ - __pte(((pte_basic_t)(pfn) << PFN_SHIFT_OFFSET) | pgprot_val(prot)) + __pte(((pte_basic_t)(pfn) << PFN_PTE_SHIFT) | pgprot_val(prot))#ifndef __ASSEMBLY__/*
@@ -330,14 +330,7 @@ static inline unsigned long pte_update(pte_t *p, unsigned long clr,/* * set_pte stores a linux PTE into the linux page table. */ -static inline void set_pte (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte) -{ - *ptep = pte; -} - -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte) +static inline void set_pte (pte_t *ptep,pte_t pte) { *ptep = pte; }
arch/microblaze/include/asm/tlbflush.h +3 -1
@@ -33,7 +33,9 @@ static inline void local_flush_tlb_range(struct vm_area_struct *vma,#define flush_tlb_kernel_range (start, end) do { } while (0) -#define update_mmu_cache (vma, addr, ptep) do { } while (0) +#define update_mmu_cache_range (vmf, vma, addr, ptep, nr) do { } while (0) +#define update_mmu_cache (vma, addr, pte) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)#define flush_tlb_all local_flush_tlb_all#define flush_tlb_mm local_flush_tlb_mm
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 17/38 mips: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Rename _PFN_SHIFT to PFN_PTE_SHIFT. Convert a few places
to call set_pte() instead of set_pte_at(). Add set_ptes(), update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio(). Change the PG_arch_1 (aka PG_dcache_dirty) flag from being per-page to per-folio.
--- arch/mips/bcm47xx/prom.c | 2 +- arch/mips/include/asm/cacheflush.h | 32 +++++++++----- arch/mips/include/asm/pgtable-32.h | 10 ++--- arch/mips/include/asm/pgtable-64.h | 6 +-- arch/mips/include/asm/pgtable-bits.h | 6 +-- arch/mips/include/asm/pgtable.h | 63 ++++++++++++++++++---------- arch/mips/mm/c-r4k.c | 5 ++- arch/mips/mm/cache.c | 56 ++++++++++++------------- arch/mips/mm/init.c | 21 ++++++---- arch/mips/mm/pgtable-32.c | 2 +- arch/mips/mm/pgtable-64.c | 2 +- arch/mips/mm/tlbex.c | 2 +- 12 files changed, 121 insertions(+), 86 deletions(-)
arch/mips/bcm47xx/prom.c +1 -1
@@ -116,7 +116,7 @@ void __init prom_init(void)#if defined(CONFIG_BCM47XX_BCMA ) && defined(CONFIG_HIGHMEM )#define EXTVBASE 0xc0000000 -#define ENTRYLO (x) ((pte_val(pfn_pte((x) >> _PFN_SHIFT, PAGE_KERNEL_UNCACHED)) >> 6) | 1) +#define ENTRYLO (x) ((pte_val(pfn_pte((x) >> PFN_PTE_SHIFT, PAGE_KERNEL_UNCACHED)) >> 6) | 1)#include <asm/tlbflush.h>
arch/mips/include/asm/cacheflush.h +21 -11
@@ -36,12 +36,12 @@*/ #define PG_dcache_dirty PG_arch_1 -#define Page_dcache_dirty (page) \ - test_bit(PG_dcache_dirty, &(page)->flags) -#define SetPageDcacheDirty (page) \ - set_bit(PG_dcache_dirty, &(page)->flags) -#define ClearPageDcacheDirty (page) \ - clear_bit(PG_dcache_dirty, &(page)->flags) +#define folio_test_dcache_dirty (folio) \ + test_bit(PG_dcache_dirty, &(folio)->flags) +#define folio_set_dcache_dirty (folio) \ + set_bit(PG_dcache_dirty, &(folio)->flags) +#define folio_clear_dcache_dirty (folio) \ + clear_bit(PG_dcache_dirty, &(folio)->flags)extern void (*flush_cache_all)(void );extern void (*__flush_cache_all)(void );
@@ -50,15 +50,24 @@ extern void (*flush_cache_mm)(struct mm_struct *mm);extern void (*flush_cache_range)(struct vm_area_struct *vma,unsigned long start,unsigned long end);extern void (*flush_cache_page)(struct vm_area_struct *vma,unsigned long page,unsigned long pfn); -extern void __flush_dcache_page (struct page *page); +extern void __flush_dcache_pages (struct page *page,unsigned int nr);#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +static inline void flush_dcache_folio (struct folio *folio) +{ +if (cpu_has_dc_aliases) +__flush_dcache_pages (&folio->page,folio_nr_pages (folio)); +else if (!cpu_has_ic_fills_f_dc) +folio_set_dcache_dirty (folio); +} +#define flush_dcache_folio flush_dcache_folio +static inline void flush_dcache_page (struct page *page) {if (cpu_has_dc_aliases) -__flush_dcache_page (page); +__flush_dcache_pages (page,1 );else if (!cpu_has_ic_fills_f_dc) -SetPageDcacheDirty (page); +folio_set_dcache_dirty (page_folio (page)); }#define flush_dcache_mmap_lock (mapping) do { } while (0)
@@ -73,10 +82,11 @@ static inline void flush_anon_page(struct vm_area_struct *vma,__flush_anon_page (page, vmaddr); } -static inline void flush_icache_page (struct vm_area_struct *vma, -struct page *page) +static inline void flush_icache_pages (struct vm_area_struct *vma, +struct page *page,unsigned int nr) { } +#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1)extern void (*flush_icache_range)(unsigned long start,unsigned long end);extern void (*local_flush_icache_range)(unsigned long start,unsigned long end);
arch/mips/include/asm/pgtable-32.h +5 -5
@@ -153,7 +153,7 @@ static inline void pmd_clear(pmd_t *pmdp)#if defined(CONFIG_XPA )#define MAX_POSSIBLE_PHYSMEM_BITS 40 -#define pte_pfn (x) (((unsigned long)((x).pte_high >> _PFN_SHIFT)) | (unsigned long)((x).pte_low << _PAGE_PRESENT_SHIFT)) +#define pte_pfn (x) (((unsigned long)((x).pte_high >> PFN_PTE_SHIFT)) | (unsigned long)((x).pte_low << _PAGE_PRESENT_SHIFT))static inline pte_t pfn_pte (unsigned long pfn,pgprot_t prot) {
@@ -161,7 +161,7 @@ pfn_pte(unsigned long pfn, pgprot_t prot) pte.pte_low = (pfn >> _PAGE_PRESENT_SHIFT) | (pgprot_val (prot) & ~_PFNX_MASK); - pte.pte_high = (pfn << _PFN_SHIFT) | + pte.pte_high = (pfn <<PFN_PTE_SHIFT ) | (pgprot_val (prot) & ~_PFN_MASK);return pte; }
@@ -184,9 +184,9 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t prot)#else #define MAX_POSSIBLE_PHYSMEM_BITS 32 -#define pte_pfn (x) ((unsigned long)((x).pte >> _PFN_SHIFT)) -#define pfn_pte (pfn, prot) __pte(((unsigned long long)(pfn) << _PFN_SHIFT) | pgprot_val(prot)) -#define pfn_pmd (pfn, prot) __pmd(((unsigned long long)(pfn) << _PFN_SHIFT) | pgprot_val(prot)) +#define pte_pfn (x) ((unsigned long)((x).pte >> PFN_PTE_SHIFT)) +#define pfn_pte (pfn, prot) __pte(((unsigned long long)(pfn) << PFN_PTE_SHIFT) | pgprot_val(prot)) +#define pfn_pmd (pfn, prot) __pmd(((unsigned long long)(pfn) << PFN_PTE_SHIFT) | pgprot_val(prot))#endif /* defined(CONFIG_PHYS_ADDR_T_64BIT) && defined(CONFIG_CPU_MIPS32) */ #define pte_page (x) pfn_to_page(pte_pfn(x))
arch/mips/include/asm/pgtable-64.h +3 -3
@@ -298,9 +298,9 @@ static inline void pud_clear(pud_t *pudp)#define pte_page (x) pfn_to_page(pte_pfn(x)) -#define pte_pfn (x) ((unsigned long)((x).pte >> _PFN_SHIFT)) -#define pfn_pte (pfn, prot) __pte(((pfn) << _PFN_SHIFT) | pgprot_val(prot)) -#define pfn_pmd (pfn, prot) __pmd(((pfn) << _PFN_SHIFT) | pgprot_val(prot)) +#define pte_pfn (x) ((unsigned long)((x).pte >> PFN_PTE_SHIFT)) +#define pfn_pte (pfn, prot) __pte(((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot)) +#define pfn_pmd (pfn, prot) __pmd(((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot))#ifndef __PAGETABLE_PMD_FOLDEDstatic inline pmd_t *pud_pgtable (pud_t pud)
arch/mips/include/asm/pgtable-bits.h +3 -3
@@ -182,10 +182,10 @@ enum pgtable_bits {#if defined(CONFIG_CPU_R3K_TLB )# define _CACHE_UNCACHED (1 << _CACHE_UNCACHED_SHIFT)# define _CACHE_MASK _CACHE_UNCACHED -# define _PFN_SHIFT PAGE_SHIFT +# define PFN_PTE_SHIFT PAGE_SHIFT#else # define _CACHE_MASK (7 << _CACHE_SHIFT) -# define _PFN_SHIFT (PAGE_SHIFT - 12 + _CACHE_SHIFT + 3) +# define PFN_PTE_SHIFT (PAGE_SHIFT - 12 + _CACHE_SHIFT + 3)#endif #ifndef _PAGE_NO_EXEC
@@ -195,7 +195,7 @@ enum pgtable_bits {#define _PAGE_SILENT_READ _PAGE_VALID#define _PAGE_SILENT_WRITE _PAGE_DIRTY -#define _PFN_MASK (~((1 << (_PFN_SHIFT)) - 1)) +#define _PFN_MASK (~((1 << (PFN_PTE_SHIFT)) - 1))/* * The final layouts of the PTE bits are:
arch/mips/include/asm/pgtable.h +41 -22
111 lines not on this page · raw message
arch/mips/mm/c-r4k.c +3 -2
@@ -568,13 +568,14 @@ static inline void local_r4k_flush_cache_page(void *args)if ((mm == current->active_mm) && (pte_val (*ptep) & _PAGE_VALID)) vaddr =NULL ;else { +struct folio *folio =page_folio (page);/* * Use kmap_coherent or kmap_atomic to do flushes for * another ASID than the current one. */ map_coherent = (cpu_has_dc_aliases && -page_mapcount (page) && - !Page_dcache_dirty (page)); +folio_mapped (folio) && + !folio_test_dcache_dirty (folio));if (map_coherent) vaddr =kmap_coherent (page, addr);else
arch/mips/mm/cache.c +28 -28
93 lines not on this page · raw message
arch/mips/mm/init.c +13 -8
@@ -88,7 +88,7 @@ static void *__kmap_pgprot(struct page *page, unsigned long addr, pgprot_t prot)pte_t pte;int tlbidx; -BUG_ON (Page_dcache_dirty (page)); +BUG_ON (folio_test_dcache_dirty (page_folio (page)));preempt_disable ();pagefault_disable ();
@@ -169,11 +169,12 @@ void kunmap_coherent(void)void copy_user_highpage (struct page *to,struct page *from,unsigned long vaddr,struct vm_area_struct *vma) { +struct folio *src =page_folio (from);void *vfrom, *vto; vto =kmap_atomic (to);if (cpu_has_dc_aliases && -page_mapcount (from) && !Page_dcache_dirty (from)) { +folio_mapped (src) && !folio_test_dcache_dirty (src)) { vfrom =kmap_coherent (from, vaddr);copy_page (vto, vfrom);kunmap_coherent ();
@@ -194,15 +195,17 @@ void copy_to_user_page(struct vm_area_struct *vma, struct page *page, unsigned long vaddr, void *dst, const void *src, unsigned long len) { + struct folio *folio = page_folio(page); +if (cpu_has_dc_aliases && - page_mapcount(page) && !Page_dcache_dirty(page)) { +folio_mapped (folio) && !folio_test_dcache_dirty (folio)) {void *vto =kmap_coherent (page, vaddr) + (vaddr & ~PAGE_MASK );memcpy (vto, src, len);kunmap_coherent (); }else {memcpy (dst, src, len);if (cpu_has_dc_aliases) -SetPageDcacheDirty (page); +folio_set_dcache_dirty (folio); }if (vma->vm_flags &VM_EXEC )flush_cache_page (vma, vaddr,page_to_pfn (page));
@@ -212,15 +215,17 @@ void copy_from_user_page(struct vm_area_struct *vma, struct page *page, unsigned long vaddr, void *dst, const void *src, unsigned long len) { + struct folio *folio = page_folio(page); +if (cpu_has_dc_aliases && - page_mapcount(page) && !Page_dcache_dirty(page)) { +folio_mapped (folio) && !folio_test_dcache_dirty (folio)) {void *vfrom =kmap_coherent (page, vaddr) + (vaddr & ~PAGE_MASK );memcpy (dst, vfrom, len);kunmap_coherent (); }else {memcpy (dst, src, len);if (cpu_has_dc_aliases) -SetPageDcacheDirty (page); +folio_set_dcache_dirty (folio); } }EXPORT_SYMBOL_GPL (copy_from_user_page);
@@ -448,10 +453,10 @@ static inline void __init mem_init_free_highmem(void)void __init mem_init (void ) {/* -* When _PFN_SHIFT is greater than PAGE_SHIFT we won't have enough PTE +* When PFN_PTE_SHIFT is greater than PAGE_SHIFT we won't have enough PTE * bits to hold a full 32b physical address on MIPS32 systems. */ -BUILD_BUG_ON (IS_ENABLED (CONFIG_32BIT ) && (_PFN_SHIFT >PAGE_SHIFT )); +BUILD_BUG_ON (IS_ENABLED (CONFIG_32BIT ) && (PFN_PTE_SHIFT >PAGE_SHIFT ));#ifdef CONFIG_HIGHMEM max_mapnr = highend_pfn ? highend_pfn : max_low_pfn;
arch/mips/mm/pgtable-32.c +1 -1
@@ -35,7 +35,7 @@ pmd_t mk_pmd(struct page *page, pgprot_t prot) {pmd_t pmd; -pmd_val (pmd) = (page_to_pfn (page) << _PFN_SHIFT) |pgprot_val (prot); +pmd_val (pmd) = (page_to_pfn (page) <<PFN_PTE_SHIFT ) |pgprot_val (prot);return pmd; }
arch/mips/mm/pgtable-64.c +1 -1
@@ -93,7 +93,7 @@ pmd_t mk_pmd(struct page *page, pgprot_t prot) {pmd_t pmd; -pmd_val (pmd) = (page_to_pfn (page) << _PFN_SHIFT) |pgprot_val (prot); +pmd_val (pmd) = (page_to_pfn (page) <<PFN_PTE_SHIFT ) |pgprot_val (prot);return pmd; }
arch/mips/mm/tlbex.c +1 -1
@@ -253,7 +253,7 @@ static void output_pgtable_bits_defines(void)pr_define ("_PAGE_GLOBAL_SHIFT %d\n" , _PAGE_GLOBAL_SHIFT);pr_define ("_PAGE_VALID_SHIFT %d\n" , _PAGE_VALID_SHIFT);pr_define ("_PAGE_DIRTY_SHIFT %d\n" , _PAGE_DIRTY_SHIFT); -pr_define ("_PFN_SHIFT %d\n" , _PFN_SHIFT); +pr_define ("PFN_PTE_SHIFT %d\n" ,PFN_PTE_SHIFT );pr_debug ("\n" ); }
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 18/38 nios2: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes(), update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio(). Change the PG_arch_1 (aka PG_dcache_dirty) flag from being per-page to per-folio.
--- arch/nios2/include/asm/cacheflush.h | 6 ++- arch/nios2/include/asm/pgtable.h | 28 ++++++---- arch/nios2/mm/cacheflush.c | 79 ++++++++++++++++------------- 3 files changed, 67 insertions(+), 46 deletions(-)
arch/nios2/include/asm/cacheflush.h +5 -1
@@ -29,9 +29,13 @@ extern void flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long pfn);#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1void flush_dcache_page (struct page *page); +void flush_dcache_folio (struct folio *folio); +#define flush_dcache_folio flush_dcache_folioextern void flush_icache_range (unsigned long start,unsigned long end); -extern void flush_icache_page (struct vm_area_struct *vma,struct page *page); +void flush_icache_pages (struct vm_area_struct *vma,struct page *page, +unsigned int nr); +#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1);#define flush_cache_vmap (start, end) flush_dcache_range(start, end)#define flush_cache_vunmap (start, end) flush_dcache_range(start, end)
arch/nios2/include/asm/pgtable.h +19 -9
@@ -178,14 +178,21 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) *ptep = pteval; } -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pteval) +static inline void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pte,unsigned int nr) { -unsigned long paddr = (unsigned long )page_to_virt (pte_page (pteval)); - -flush_dcache_range (paddr, paddr +PAGE_SIZE ); -set_pte (ptep, pteval); +unsigned long paddr = (unsigned long )page_to_virt (pte_page (pte)); + +flush_dcache_range (paddr, paddr + nr *PAGE_SIZE ); +for (;;) { +set_pte (ptep, pte); +if (--nr ==0 ) +break ; + ptep++; +pte_val (pte) +=1 ; + } } +#define set_ptes set_ptesstatic inline int pmd_none (pmd_t pmd) {
@@ -202,7 +209,7 @@ static inline void pte_clear(struct mm_struct *mm,pte_val (null) = (addr >>PAGE_SHIFT ) &0xf ; -set_pte_at (mm, addr, ptep, null); +set_pte (ptep, null); }/*
@@ -273,7 +280,10 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)extern void __init paging_init (void );extern void __init mmu_init (void ); -extern void update_mmu_cache (struct vm_area_struct *vma, -unsigned long address,pte_t *pte); +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long address,pte_t *ptep,unsigned int nr); + +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)#endif /* _ASM_NIOS2_PGTABLE_H */
arch/nios2/mm/cacheflush.c +43 -36
149 lines not on this page · raw message
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 19/38 openrisc: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range() and flush_dcache_folio(). Change the PG_arch_1 (aka PG_dcache_dirty) flag from being per-page to per-folio.
--- arch/openrisc/include/asm/cacheflush.h | 8 +++++++- arch/openrisc/include/asm/pgtable.h | 15 ++++++++++----- arch/openrisc/mm/cache.c | 12 ++++++++---- 3 files changed, 25 insertions(+), 10 deletions(-)
arch/openrisc/include/asm/cacheflush.h +7 -1
@@ -56,10 +56,16 @@ static inline void sync_icache_dcache(struct page *page)*/ #define PG_dc_clean PG_arch_1 +static inline void flush_dcache_folio (struct folio *folio) +{ +clear_bit (PG_dc_clean, &folio->flags); +} +#define flush_dcache_folio flush_dcache_folio +#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1static inline void flush_dcache_page (struct page *page) { -clear_bit (PG_dc_clean, &page->flags); +flush_dcache_folio (page_folio (page)); }#define flush_icache_user_page (vma, page, addr, len) \
arch/openrisc/include/asm/pgtable.h +10 -5
@@ -46,7 +46,7 @@ extern void paging_init(void);* hook is made available. */ #define set_pte (pteptr, pteval) ((*(pteptr)) = (pteval)) -#define set_pte_at (mm, addr, ptep, pteval) set_pte(ptep, pteval) +/* * (pmds are folded into pgds so this doesn't get actually called, * but the define is needed for a generic inline function.)
@@ -357,6 +357,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)#define __pmd_offset (address) \ (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1)) +#define PFN_PTE_SHIFT PAGE_SHIFT#define pte_pfn (x) ((unsigned long)(((x).pte)) >> PAGE_SHIFT)#define pfn_pte (pfn, prot) __pte((((pfn) << PAGE_SHIFT)) | pgprot_val(prot))
@@ -379,13 +380,17 @@ static inline void update_tlb(struct vm_area_struct *vma,extern void update_cache (struct vm_area_struct *vma,unsigned long address,pte_t *pte); -static inline void update_mmu_cache (struct vm_area_struct *vma, -unsigned long address,pte_t *pte) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) { -update_tlb (vma, address, pte); -update_cache (vma, address, pte); +update_tlb (vma, address, ptep); +update_cache (vma, address, ptep); } +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1) +/* __PHX__ FIXME, SWAP, this probably doesn't work */ /*
arch/openrisc/mm/cache.c +8 -4
@@ -43,15 +43,19 @@ void update_cache(struct vm_area_struct *vma, unsigned long address, pte_t *pte) {unsigned long pfn =pte_val (*pte) >>PAGE_SHIFT ; -struct page *page =pfn_to_page (pfn); -int dirty = !test_and_set_bit (PG_dc_clean, &page->flags); +struct folio *folio =page_folio (pfn_to_page (pfn)); +int dirty = !test_and_set_bit (PG_dc_clean, &folio->flags);/* * Since icaches do not snoop for updated data on OpenRISC, we * must write back and invalidate any dirty pages manually. We * can skip data pages, since they will not end up in icaches. */ -if ((vma->vm_flags &VM_EXEC ) && dirty) -sync_icache_dcache (page); +if ((vma->vm_flags &VM_EXEC ) && dirty) { +unsigned int nr =folio_nr_pages (folio); + +while (nr--) +sync_icache_dcache (folio_page (folio, nr)); + } }
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 20/38 parisc: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes(), update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages(). Change the PG_arch_1 (aka PG_dcache_dirty) flag from being per-page to per-folio.
--- arch/parisc/include/asm/cacheflush.h | 14 ++-- arch/parisc/include/asm/pgtable.h | 37 +++++---- arch/parisc/kernel/cache.c | 107 ++++++++++++++++++--------- 3 files changed, 105 insertions(+), 53 deletions(-)
arch/parisc/include/asm/cacheflush.h +9 -5
@@ -43,8 +43,13 @@ void invalidate_kernel_vmap_range(void *vaddr, int size);#define flush_cache_vmap (start, end) flush_cache_all()#define flush_cache_vunmap (start, end) flush_cache_all() +void flush_dcache_folio (struct folio *folio); +#define flush_dcache_folio flush_dcache_folio#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -void flush_dcache_page (struct page *page); +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +}#define flush_dcache_mmap_lock (mapping) xa_lock_irq(&mapping->i_pages)#define flush_dcache_mmap_unlock (mapping) xa_unlock_irq(&mapping->i_pages)
@@ -53,10 +58,9 @@ void flush_dcache_page(struct page *page);#define flush_dcache_mmap_unlock_irqrestore (mapping, flags) \ xa_unlock_irqrestore(&mapping->i_pages, flags) -#define flush_icache_page (vma,page) do { \ - flush_kernel_dcache_page_addr(page_address(page)); \ - flush_kernel_icache_page(page_address(page)); \ -} while (0) +void flush_icache_pages (struct vm_area_struct *vma,struct page *page, +unsigned int nr); +#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1)#define flush_icache_range (s,e) do { \ flush_kernel_dcache_range_asm(s,e); \
arch/parisc/include/asm/pgtable.h +23 -14
@@ -73,15 +73,6 @@ extern void __update_cache(pte_t pte);mb (); \ } while(0) -#define set_pte_at (mm, addr, pteptr, pteval) \ - do { \ - if (pte_present(pteval) && \ - pte_user(pteval)) \ - __update_cache(pteval); \ - *(pteptr) = (pteval); \ - purge_tlb_entries(mm, addr); \ - } while (0) -#endif /* !__ASSEMBLY__ */ #define pte_ERROR (e) \
@@ -285,7 +276,7 @@ extern unsigned long *empty_zero_page;#define pte_none (x) (pte_val(x) == 0)#define pte_present (x) (pte_val(x) & _PAGE_PRESENT)#define pte_user (x) (pte_val(x) & _PAGE_USER) -#define pte_clear (mm, addr, xp) set_pte_at(mm, addr, xp, __pte(0)) +#define pte_clear (mm, addr, xp) set_pte(xp, __pte(0))#define pmd_flag (x) (pmd_val(x) & PxD_FLAG_MASK)#define pmd_address (x) ((unsigned long)(pmd_val(x) &~ PxD_FLAG_MASK) << PxD_VALUE_SHIFT)
@@ -391,11 +382,29 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)extern void paging_init (void ); +static inline void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pte,unsigned int nr) +{ +if (pte_present (pte) &&pte_user (pte)) +__update_cache (pte); +for (;;) { + *ptep = pte; +purge_tlb_entries (mm, addr); +if (--nr ==0 ) +break ; + ptep++; +pte_val (pte) +=1 <<PFN_PTE_SHIFT ; + addr +=PAGE_SIZE ; + } +} +#define set_ptes set_ptes +/* Used for deferring calls to flush_dcache_page() */ #define PG_dcache_dirty PG_arch_1 -#define update_mmu_cache (vms,addr,ptep) __update_cache(*ptep) +#define update_mmu_cache_range (vmf, vma, addr, ptep, nr) __update_cache(*ptep) +#define update_mmu_cache (vma, addr, ptep) __update_cache(*ptep)/* * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
@@ -450,7 +459,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsignedif (!pte_young (pte)) {return 0 ; } -set_pte_at (vma->vm_mm, addr, ptep,pte_mkold (pte)); +set_pte (ptep,pte_mkold (pte));return 1 ; }
@@ -460,14 +469,14 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr,pte_t old_pte; old_pte = *ptep; -set_pte_at (mm, addr, ptep,__pte (0 )); +set_pte (ptep,__pte (0 ));return old_pte; }static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { -set_pte_at (mm, addr, ptep,pte_wrprotect (*ptep)); +set_pte (ptep,pte_wrprotect (*ptep)); }#define pte_same (A ,B ) (pte_val(A) == pte_val(B))
arch/parisc/kernel/cache.c +73 -34
177 lines not on this page · raw message
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 21/38 powerpc: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes(), update_mmu_cache_range() and flush_dcache_folio(). Change the PG_arch_1 (aka PG_dcache_dirty) flag from being per-page to per-folio.
--- arch/powerpc/include/asm/book3s/32/pgtable.h | 5 -- arch/powerpc/include/asm/book3s/64/pgtable.h | 6 +-- arch/powerpc/include/asm/book3s/pgtable.h | 11 ++-- arch/powerpc/include/asm/cacheflush.h | 14 ++++-- arch/powerpc/include/asm/kvm_ppc.h | 10 ++-- arch/powerpc/include/asm/nohash/pgtable.h | 16 ++---- arch/powerpc/include/asm/pgtable.h | 12 +++++ arch/powerpc/mm/book3s64/hash_utils.c | 11 ++-- arch/powerpc/mm/cacheflush.c | 40 +++++---------- arch/powerpc/mm/nohash/e500_hugetlbpage.c | 3 +- arch/powerpc/mm/pgtable.c | 53 ++++++++++++-------- 11 files changed, 88 insertions(+), 93 deletions(-)
arch/powerpc/include/asm/book3s/32/pgtable.h +0 -5
@@ -462,11 +462,6 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t pgprot) pgprot_val(pgprot)); } -static inline unsigned long pte_pfn (pte_t pte) -{ -return pte_val (pte) >>PTE_RPN_SHIFT ; -} -/* Generic modifiers for PTE bits */ static inline pte_t pte_wrprotect (pte_t pte) {
arch/powerpc/include/asm/book3s/64/pgtable.h +1 -5
@@ -104,6 +104,7 @@* and every thing below PAGE_SHIFT; */ #define PTE_RPN_MASK (((1UL << _PAGE_PA_MAX) - 1) & (PAGE_MASK)) +#define PTE_RPN_SHIFT PAGE_SHIFT/* * set of bits not changed in pmd_modify. Even though we have hash specific bits * in here, on radix we expect them to be zero.
@@ -569,11 +570,6 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t pgprot)return __pte (((pte_basic_t )pfn <<PAGE_SHIFT ) |pgprot_val (pgprot) | _PAGE_PTE); } -static inline unsigned long pte_pfn (pte_t pte) -{ -return (pte_val (pte) &PTE_RPN_MASK ) >>PAGE_SHIFT ; -} -/* Generic modifiers for PTE bits */ static inline pte_t pte_wrprotect (pte_t pte) {
arch/powerpc/include/asm/book3s/pgtable.h +3 -8
@@ -9,13 +9,6 @@#endif #ifndef __ASSEMBLY__ -/* Insert a PTE, top-level function is out of line. It uses an inline -* low level function in the respective pgtable-* files -*/ -extern void set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, -pte_t pte); - -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGSextern int ptep_set_access_flags (struct vm_area_struct *vma,unsigned long address,pte_t *ptep,pte_t entry,int dirty);
@@ -36,7 +29,9 @@ void __update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t* corresponding HPTE into the hash table ahead of time, instead of * waiting for the inevitable extra hash-table miss exception. */ -static inline void update_mmu_cache (struct vm_area_struct *vma,unsigned long address,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {if (IS_ENABLED (CONFIG_PPC32 ) && !mmu_has_feature (MMU_FTR_HPTE_TABLE ))return ;
arch/powerpc/include/asm/cacheflush.h +10 -4
@@ -35,13 +35,19 @@ static inline void flush_cache_vmap(unsigned long start, unsigned long end)* It just marks the page as not i-cache clean. We do the i-cache * flush later when the page is given to a user process, if necessary. */ -static inline void flush_dcache_page (struct page *page) +static inline void flush_dcache_folio (struct folio *folio) {if (cpu_has_feature (CPU_FTR_COHERENT_ICACHE ))return ;/* avoid an atomic op if possible */ -if (test_bit (PG_dcache_clean, &page->flags)) -clear_bit (PG_dcache_clean, &page->flags); +if (test_bit (PG_dcache_clean, &folio->flags)) +clear_bit (PG_dcache_clean, &folio->flags); +} +#define flush_dcache_folio flush_dcache_folio + +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); }void flush_icache_range (unsigned long start,unsigned long stop);
@@ -51,7 +57,7 @@ void flush_icache_user_page(struct vm_area_struct *vma, struct page *page, unsigned long addr, int len);#define flush_icache_user_page flush_icache_user_page -void flush_dcache_icache_page (struct page *page); +void flush_dcache_icache_folio (struct folio *folio);/** * flush_dcache_range(): Write any modified data cache blocks out to memory and
arch/powerpc/include/asm/kvm_ppc.h +5 -5
@@ -894,7 +894,7 @@ void kvmppc_init_lpid(unsigned long nr_lpids);static inline void kvmppc_mmu_flush_icache (kvm_pfn_t pfn) { -struct page *page; +struct folio *folio;/* * We can only access pages that the kernel maps * as memory. Bail out for unmapped ones.
@@ -903,10 +903,10 @@ static inline void kvmppc_mmu_flush_icache(kvm_pfn_t pfn)return ;/* Clear i-cache for new pages */ - page =pfn_to_page (pfn); -if (!test_bit (PG_dcache_clean, &page->flags)) { -flush_dcache_icache_page (page); -set_bit (PG_dcache_clean, &page->flags); + folio =page_folio (pfn_to_page (pfn)); +if (!test_bit (PG_dcache_clean, &folio->flags)) { +flush_dcache_icache_folio (folio); +set_bit (PG_dcache_clean, &folio->flags); } }
arch/powerpc/include/asm/nohash/pgtable.h +5 -11
@@ -101,8 +101,6 @@ static inline bool pte_access_permitted(pte_t pte, bool write)static inline pte_t pfn_pte (unsigned long pfn,pgprot_t pgprot) {return __pte (((pte_basic_t)(pfn) <<PTE_RPN_SHIFT ) |pgprot_val (pgprot)); } -static inline unsigned long pte_pfn (pte_t pte) { -return pte_val (pte) >>PTE_RPN_SHIFT ; }/* Generic modifiers for PTE bits */ static inline pte_t pte_exprotect (pte_t pte)
@@ -166,12 +164,6 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return __pte (pte_val (pte) & ~_PAGE_SWP_EXCLUSIVE); } -/* Insert a PTE, top-level function is out of line. It uses an inline -* low level function in the respective pgtable-* files -*/ -extern void set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, -pte_t pte); -/* This low level function performs the actual PTE insertion * Setting the PTE depends on the MMU type and other factors. It's * an horrible mess that I'm not going to try to clean up now but
@@ -282,10 +274,12 @@ static inline int pud_huge(pud_t pud)* for the page which has just been mapped in. */ #if defined(CONFIG_PPC_E500 ) && defined(CONFIG_HUGETLB_PAGE ) -void update_mmu_cache (struct vm_area_struct *vma,unsigned long address,pte_t *ptep); +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long address,pte_t *ptep,unsigned int nr);#else -static inline -void update_mmu_cache (struct vm_area_struct *vma,unsigned long address,pte_t *ptep) {} +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {}#endif #endif /* __ASSEMBLY__ */
arch/powerpc/include/asm/pgtable.h +12 -0
@@ -41,6 +41,12 @@ struct mm_struct;#ifndef __ASSEMBLY__ +void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned int nr); +#define set_ptes set_ptes +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1) +#ifndef MAX_PTRS_PER_PGD #define MAX_PTRS_PER_PGD PTRS_PER_PGD#endif
@@ -48,6 +54,12 @@ struct mm_struct;/* Keep these as a macros to avoid include dependency mess */ #define pte_page (x) pfn_to_page(pte_pfn(x))#define mk_pte (page, pgprot) pfn_pte(page_to_pfn(page), (pgprot)) + +static inline unsigned long pte_pfn (pte_t pte) +{ +return (pte_val (pte) &PTE_RPN_MASK ) >>PTE_RPN_SHIFT ; +} +/* * Select all bits except the pfn */
arch/powerpc/mm/book3s64/hash_utils.c +6 -5
@@ -1307,18 +1307,19 @@ void hash__early_init_mmu_secondary(void)*/ unsigned int hash_page_do_lazy_icache (unsigned int pp,pte_t pte,int trap) { -struct page *page; +struct folio *folio;if (!pfn_valid (pte_pfn (pte)))return pp; - page =pte_page (pte); + folio =page_folio (pte_page (pte));/* page is dirty */ -if (!test_bit (PG_dcache_clean, &page->flags) && !PageReserved (page)) { +if (!test_bit (PG_dcache_clean, &folio->flags) && + !folio_test_reserved (folio)) {if (trap ==INTERRUPT_INST_STORAGE ) { -flush_dcache_icache_page (page); -set_bit (PG_dcache_clean, &page->flags); +flush_dcache_icache_folio (folio); +set_bit (PG_dcache_clean, &folio->flags); }else pp |=HPTE_R_N ; }
arch/powerpc/mm/cacheflush.c +13 -27
@@ -148,44 +148,30 @@ static void __flush_dcache_icache(void *p)invalidate_icache_range (addr, addr +PAGE_SIZE ); } -static void flush_dcache_icache_hugepage (struct page *page) +void flush_dcache_icache_folio (struct folio *folio) { -int i; -int nr =compound_nr (page); +unsigned int i, nr =folio_nr_pages (folio); -if (!PageHighMem (page)) { +if (flush_coherent_icache ()) +return ; + +if (!folio_test_highmem (folio)) { +void *addr =folio_address (folio);for (i =0 ; i < nr; i++) -__flush_dcache_icache (lowmem_page_address (page + i)); - }else { +__flush_dcache_icache (addr + i *PAGE_SIZE ); + }else if (IS_ENABLED (CONFIG_BOOKE ) ||sizeof (phys_addr_t) >sizeof (void *)) {for (i =0 ; i < nr; i++) { -void *start =kmap_local_page (page + i); +void *start =kmap_local_folio (folio, i *PAGE_SIZE );__flush_dcache_icache (start);kunmap_local (start); } - } -} - -void flush_dcache_icache_page (struct page *page) -{ -if (flush_coherent_icache ()) -return ; - -if (PageCompound (page)) -return flush_dcache_icache_hugepage (page); - -if (!PageHighMem (page)) { -__flush_dcache_icache (lowmem_page_address (page)); - }else if (IS_ENABLED (CONFIG_BOOKE ) ||sizeof (phys_addr_t) >sizeof (void *)) { -void *start =kmap_local_page (page); - -__flush_dcache_icache (start); -kunmap_local (start); }else { -flush_dcache_icache_phys (page_to_phys (page)); +unsigned long pfn =folio_pfn (folio); +for (i =0 ; i < nr; i++) +flush_dcache_icache_phys ((pfn + i) *PAGE_SIZE ); } } -EXPORT_SYMBOL (flush_dcache_icache_page);void clear_user_page (void *page,unsigned long vaddr,struct page *pg) {
arch/powerpc/mm/nohash/e500_hugetlbpage.c +2 -1
@@ -178,7 +178,8 @@ book3e_hugetlb_preload(struct vm_area_struct *vma, unsigned long ea, pte_t pte)* * This must always be called with the pte lock held. */ -void update_mmu_cache (struct vm_area_struct *vma,unsigned long address,pte_t *ptep) +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long address,pte_t *ptep,unsigned int nr) {if (is_vm_hugetlb_page (vma))book3e_hugetlb_preload (vma, address, *ptep);
arch/powerpc/mm/pgtable.c +31 -22
121 lines not on this page · raw message
Signature · 1 line
2.40.1
Re: [PATCH v6 21/38] powerpc: Implement the new page table range API
Hi Matthew,
Quoted 10 lines · Matthew Wilcox (Oracle)
Add
set_ptes(),update_mmu_cache_range()andflush_dcache_folio(). Change thePG_arch_1(akaPG_dcache_dirty) flag from being per-page to per-folio.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Cc Michael Ellerman <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Christophe Leroy <[email protected]>
...
Quoted 27 lines
arch/powerpc/include/asm/kvm_ppc.h +5
-5@@ -894,7 +894,7 @@ void kvmppc_init_lpid(unsigned long nr_lpids);static inline void kvmppc_mmu_flush_icache (kvm_pfn_t pfn) { -struct page *page; +struct folio *folio;/* * We can only access pages that the kernel maps * as memory. Bail out for unmapped ones. @@ -903,10 +903,10 @@ static inline void kvmppc_mmu_flush_icache(kvm_pfn_t pfn)return ;/* Clear i-cache for new pages */ - page =pfn_to_page (pfn); -if (!test_bit (PG_dcache_clean, &page->flags)) { -flush_dcache_icache_page (page); -set_bit (PG_dcache_clean, &page->flags); + folio =page_folio (pfn_to_page (pfn)); +if (!test_bit (PG_dcache_clean, &folio->flags)) { +flush_dcache_icache_folio (folio); +set_bit (PG_dcache_clean, &folio->flags); } }
...
Quoted 49 lines
arch/powerpc/mm/cacheflush.c +10
-23@@ -148,44 +148,30 @@ static void __flush_dcache_icache(void *p)invalidate_icache_range (addr, addr +PAGE_SIZE ); } -static void flush_dcache_icache_hugepage (struct page *page) +void flush_dcache_icache_folio (struct folio *folio) { -int i; -int nr =compound_nr (page); +unsigned int i, nr =folio_nr_pages (folio); -if (!PageHighMem (page)) { +if (flush_coherent_icache ()) +return ; + +if (!folio_test_highmem (folio)) { +void *addr =folio_address (folio);for (i =0 ; i < nr; i++) -__flush_dcache_icache (lowmem_page_address (page + i)); - }else { +__flush_dcache_icache (addr + i *PAGE_SIZE ); + }else if (IS_ENABLED (CONFIG_BOOKE ) ||sizeof (phys_addr_t) >sizeof (void *)) {for (i =0 ; i < nr; i++) { -void *start =kmap_local_page (page + i); +void *start =kmap_local_folio (folio, i *PAGE_SIZE );__flush_dcache_icache (start);kunmap_local (start); } - } -} - -void flush_dcache_icache_page (struct page *page) -{ -if (flush_coherent_icache ()) -return ; - -if (PageCompound (page)) -return flush_dcache_icache_hugepage (page); - -if (!PageHighMem (page)) { -__flush_dcache_icache (lowmem_page_address (page)); - }else if (IS_ENABLED (CONFIG_BOOKE ) ||sizeof (phys_addr_t) >sizeof (void *)) { -void *start =kmap_local_page (page); -
arch/powerpc/mm/cacheflush.c static void __flush_dcache_icache(void *p) · line 188
Quoted 10 lines
arch/powerpc/mm/cacheflush.c +3
-4-__flush_dcache_icache (start); -kunmap_local (start); }else { -flush_dcache_icache_phys (page_to_phys (page)); +unsigned long pfn =folio_pfn (folio); +for (i =0 ; i < nr; i++) +flush_dcache_icache_phys ((pfn + i) *PAGE_SIZE ); } } -EXPORT_SYMBOL (flush_dcache_icache_page);
Apologies if this has already been fixed or reported, I searched lore and did not find anything. The dropping of this export in combination with the conversion above appears to cause ARCH=powerpc allmodconfig to fail with:
ERROR: modpost: "flush_dcache_icache_folio" [arch/powerpc/kvm/kvm-pr.ko] undefined!
I don't know if this should be re-exported or not but that does obviously resolve the issue.
Cheers,
Nathan
Matthew Wilcox
maintainerRe: [PATCH v6 21/38] powerpc: Implement the new page table range API
Quoted 11 lines · Nathan Chancellor
Quoted 1 line
-EXPORT_SYMBOL(flush_dcache_icache_page);
Apologies if this has already been fixed or reported, I searched lore and did not find anything. The dropping of this export in combination with the conversion above appears to cause ARCH=powerpc allmodconfig to fail with:
ERROR: modpost: "
flush_dcache_icache_folio" [arch/powerpc/kvm/kvm-pr.ko] undefined!I don't know if this should be re-exported or not but that does obviously resolve the issue.
Well, that was clumsy of me. I (and the Intel build bot) did test several build combos, but clearly didn't manage to find a config that showed this problem. Andrew, a fix patch for you to integrate, if you would:
arch/powerpc/mm/cacheflush.c +1 -0
@@ -172,6 +172,7 @@ void flush_dcache_icache_folio(struct folio *folio)flush_dcache_icache_phys ((pfn + i) *PAGE_SIZE ); } } +EXPORT_SYMBOL (flush_dcache_icache_folio);void clear_user_page (void *page,unsigned long vaddr,struct page *pg) {
Matthew Wilcox (Oracle)
maintainerPATCH v6 22/38 riscv: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes(), update_mmu_cache_range() and flush_dcache_folio(). Change the PG_dcache_clean flag from being per-page to per-folio.
--- arch/riscv/include/asm/cacheflush.h | 19 +++++++-------- arch/riscv/include/asm/pgtable.h | 37 +++++++++++++++++++---------- arch/riscv/mm/cacheflush.c | 13 +++------- 3 files changed, 36 insertions(+), 33 deletions(-)
arch/riscv/include/asm/cacheflush.h +9 -10
@@ -15,20 +15,19 @@ static inline void local_flush_icache_all(void)#define PG_dcache_clean PG_arch_1 -static inline void flush_dcache_page (struct page *page) +static inline void flush_dcache_folio (struct folio *folio) { -/* -* HugeTLB pages are always fully mapped and only head page will be -* set PG_dcache_clean (see comments in flush_icache_pte()). -*/ -if (PageHuge (page)) - page =compound_head (page); - -if (test_bit (PG_dcache_clean, &page->flags)) -clear_bit (PG_dcache_clean, &page->flags); +if (test_bit (PG_dcache_clean, &folio->flags)) +clear_bit (PG_dcache_clean, &folio->flags); } +#define flush_dcache_folio flush_dcache_folio#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +} +/* * RISC-V doesn't have an instruction to flush parts of the instruction cache, * so instead we just flush the whole thing.
arch/riscv/include/asm/pgtable.h +24 -13
@@ -445,8 +445,9 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)/* Commit new configuration to MMU hardware */ -static inline void update_mmu_cache (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {/* * The kernel assumes that TLBs don't cache invalid entries, but
@@ -455,8 +456,11 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,* Relying on flush_tlb_fix_spurious_fault would suffice, but * the extra traps reduce performance. So, eagerly SFENCE.VMA. */ -local_flush_tlb_page (address); +while (nr--) +local_flush_tlb_page (address + nr *PAGE_SIZE ); } +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)#define __HAVE_ARCH_UPDATE_MMU_TLB#define update_mmu_tlb update_mmu_cache
@@ -487,8 +491,7 @@ static inline void set_pte(pte_t *ptep, pte_t pteval)void flush_icache_pte (pte_t pte); -static inline void __set_pte_at (struct mm_struct *mm, -unsigned long addr,pte_t *ptep,pte_t pteval) +static inline void __set_pte_at (pte_t *ptep,pte_t pteval) {if (pte_present (pteval) &&pte_exec (pteval))flush_icache_pte (pteval);
@@ -496,17 +499,25 @@ static inline void __set_pte_at(struct mm_struct *mm,set_pte (ptep, pteval); } -static inline void set_pte_at (struct mm_struct *mm, -unsigned long addr,pte_t *ptep,pte_t pteval) +static inline void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pteval,unsigned int nr) { -page_table_check_ptes_set (mm, ptep, pteval,1 ); -__set_pte_at (mm, addr, ptep, pteval); +page_table_check_ptes_set (mm, ptep, pteval, nr); + +for (;;) { +__set_pte_at (ptep, pteval); +if (--nr ==0 ) +break ; + ptep++; +pte_val (pteval) +=1 << _PAGE_PFN_SHIFT; + } } +#define set_ptes set_ptesstatic inline void pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { -__set_pte_at (mm, addr, ptep,__pte (0 )); +__set_pte_at (ptep,__pte (0 )); }#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
@@ -515,7 +526,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty) {if (!pte_same (*ptep, entry)) -set_pte_at (vma->vm_mm, address, ptep, entry); +__set_pte_at (ptep, entry);/* * update_mmu_cache will unconditionally execute, handling both * the case that the PTE changed and the spurious fault case.
@@ -688,14 +699,14 @@ static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd) {page_table_check_pmd_set (mm ,pmdp ,pmd ); -return __set_pte_at (mm, addr, (pte_t *)pmdp,pmd_pte (pmd)); +return __set_pte_at ((pte_t *)pmdp,pmd_pte (pmd)); }static inline void set_pud_at (struct mm_struct *mm,unsigned long addr,pud_t *pudp,pud_t pud) {page_table_check_pud_set (mm, pudp, pud); -return __set_pte_at (mm, addr, (pte_t *)pudp,pud_pte (pud)); +return __set_pte_at ((pte_t *)pudp,pud_pte (pud)); }#ifdef CONFIG_PAGE_TABLE_CHECK
arch/riscv/mm/cacheflush.c +3 -10
@@ -82,18 +82,11 @@ void flush_icache_mm(struct mm_struct *mm, bool local)#ifdef CONFIG_MMU void flush_icache_pte (pte_t pte) { -struct page *page =pte_page (pte); +struct folio *folio =page_folio (pte_page (pte)); -/* -* HugeTLB pages are always fully mapped, so only setting head page's -* PG_dcache_clean flag is enough. -*/ -if (PageHuge (page)) - page =compound_head (page); - -if (!test_bit (PG_dcache_clean, &page->flags)) { +if (!test_bit (PG_dcache_clean, &folio->flags)) {flush_icache_all (); -set_bit (PG_dcache_clean, &page->flags); +set_bit (PG_dcache_clean, &folio->flags); } }#endif /* CONFIG_MMU */
Signature · 7 lines
2.40.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH v6 22/38] riscv: Implement the new page table range API
This patch was applied to riscv/linux.git (fixes) by Andrew Morton <[email protected]>:
Hello:
This patch was applied to riscv/linux.git (fixes)
by Andrew Morton <[email protected]>:
Quoted 11 lines · On Wed, 2 Aug 2023 16:13:50 +0100 you wrote:
Add
set_ptes(),update_mmu_cache_range()andflush_dcache_folio(). Change thePG_dcache_cleanflag from being per-page to per-folio.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Reviewed-by Alexandre Ghiti <[email protected]>Cc Paul Walmsley <[email protected]>Cc Palmer Dabbelt <[email protected]>Cc Albert Ou <[email protected]>
Quoted 1 line
[...]
Here is the summary with links:
- [v6,22/38] riscv: Implement the new page table range API
https://git.kernel.org/riscv/c/864609c6a0b5
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Matthew Wilcox (Oracle)
maintainerPATCH v6 23/38 s390: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes() and update_mmu_cache_range().
--- arch/s390/include/asm/pgtable.h | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-)
arch/s390/include/asm/pgtable.h +24 -9
@@ -47,6 +47,7 @@ static inline void update_page_count(int level, long count)* tables contain all the necessary information. */ #define update_mmu_cache (vma, address, ptep) do { } while (0) +#define update_mmu_cache_range (vmf, vma, addr, ptep, nr) do { } while (0)#define update_mmu_cache_pmd (vma, address, ptep) do { } while (0)/*
@@ -1314,20 +1315,34 @@ pgprot_t pgprot_writecombine(pgprot_t prot);pgprot_t pgprot_writethrough (pgprot_t prot);/* -* Certain architectures need to do special things when PTEs -* within a page table are directly modified. Thus, the following -* hook is made available. +* Set multiple PTEs to consecutive pages with a single call. All PTEs +* are within the same folio, PMD and VMA. */ -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t entry) +static inline void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t entry,unsigned int nr) {if (pte_present (entry)) entry =clear_pte_bit (entry,__pgprot (_PAGE_UNUSED)); -if (mm_has_pgste (mm)) -ptep_set_pte_at (mm, addr, ptep, entry); -else -set_pte (ptep, entry); +if (mm_has_pgste (mm)) { +for (;;) { +ptep_set_pte_at (mm, addr, ptep, entry); +if (--nr ==0 ) +break ; + ptep++; + entry =__pte (pte_val (entry) +PAGE_SIZE ); + addr +=PAGE_SIZE ; + } + }else { +for (;;) { +set_pte (ptep, entry); +if (--nr ==0 ) +break ; + ptep++; + entry =__pte (pte_val (entry) +PAGE_SIZE ); + } + } } +#define set_ptes set_ptes/* * Conversion functions: convert a page and protection to a page entry,
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 24/38 sh: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages(). Change the PG_dcache_clean flag from being per-page to per-folio. Flush the entire folio containing the pages in flush_icache_pages() for ease of implementation.
--- arch/sh/include/asm/cacheflush.h | 21 ++++++++----- arch/sh/include/asm/pgtable.h | 7 +++-- arch/sh/include/asm/pgtable_32.h | 5 ++- arch/sh/mm/cache-j2.c | 4 +-- arch/sh/mm/cache-sh4.c | 26 +++++++++++----- arch/sh/mm/cache-sh7705.c | 26 ++++++++++------ arch/sh/mm/cache.c | 52 ++++++++++++++++++-------------- arch/sh/mm/kmap.c | 3 +- 8 files changed, 89 insertions(+), 55 deletions(-)
arch/sh/include/asm/cacheflush.h +14 -7
@@ -13,9 +13,9 @@* - flush_cache_page(mm, vmaddr, pfn) flushes a single page * - flush_cache_range(vma, start, end) flushes a range of pages * -* - flush_dcache_page(pg) flushes(wback&invalidates) a page for dcache +* - flush_dcache_folio(folio) flushes(wback&invalidates) a folio for dcache * - flush_icache_range(start, end) flushes(invalidates) a range for icache -* - flush_icache_page(vma, pg) flushes(invalidates) a page for icache +* - flush_icache_pages(vma, pg, nr) flushes(invalidates) pages for icache * - flush_cache_sigtramp(vaddr) flushes the signal trampoline */ extern void (*local_flush_cache_all)(void *args);
@@ -23,9 +23,9 @@ extern void (*local_flush_cache_mm)(void *args);extern void (*local_flush_cache_dup_mm)(void *args);extern void (*local_flush_cache_page)(void *args);extern void (*local_flush_cache_range)(void *args); -extern void (*local_flush_dcache_page)(void *args); +extern void (*local_flush_dcache_folio)(void *args);extern void (*local_flush_icache_range)(void *args); -extern void (*local_flush_icache_page)(void *args); +extern void (*local_flush_icache_folio)(void *args);extern void (*local_flush_cache_sigtramp)(void *args);static inline void cache_noop (void *args) { }
@@ -42,11 +42,18 @@ extern void flush_cache_page(struct vm_area_struct *vma,extern void flush_cache_range (struct vm_area_struct *vma,unsigned long start,unsigned long end);#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -void flush_dcache_page (struct page *page); +void flush_dcache_folio (struct folio *folio); +#define flush_dcache_folio flush_dcache_folio +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +} +extern void flush_icache_range (unsigned long start,unsigned long end);#define flush_icache_user_range flush_icache_range -extern void flush_icache_page (struct vm_area_struct *vma, -struct page *page); +void flush_icache_pages (struct vm_area_struct *vma,struct page *page, +unsigned int nr); +#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1)extern void flush_cache_sigtramp (unsigned long address);struct flusher_data {
arch/sh/include/asm/pgtable.h +5 -2
@@ -102,13 +102,16 @@ extern void __update_cache(struct vm_area_struct *vma,extern void __update_tlb (struct vm_area_struct *vma,unsigned long address,pte_t pte); -static inline void -update_mmu_cache (struct vm_area_struct *vma,unsigned long address,pte_t *ptep) +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {pte_t pte = *ptep;__update_cache (vma, address, pte);__update_tlb (vma, address, pte); } +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)extern pgd_t swapper_pg_dir[PTRS_PER_PGD ];extern void paging_init (void );
arch/sh/include/asm/pgtable_32.h +2 -3
@@ -307,14 +307,13 @@ static inline void set_pte(pte_t *ptep, pte_t pte)#define set_pte (pteptr, pteval) (*(pteptr) = pteval)#endif -#define set_pte_at (mm,addr,ptep,pteval) set_pte(ptep,pteval) -/* * (pmds are folded into pgds so this doesn't get actually called, * but the define is needed for a generic inline function.) */ #define set_pmd (pmdptr, pmdval) (*(pmdptr) = pmdval) +#define PFN_PTE_SHIFT PAGE_SHIFT#define pfn_pte (pfn, prot) \ __pte(((unsigned long long)(pfn) << PAGE_SHIFT) | pgprot_val(prot))#define pfn_pmd (pfn, prot) \
@@ -323,7 +322,7 @@ static inline void set_pte(pte_t *ptep, pte_t pte)#define pte_none (x) (!pte_val(x))#define pte_present (x) ((x).pte_low & (_PAGE_PRESENT | _PAGE_PROTNONE)) -#define pte_clear (mm,addr,xp) do { set_pte_at(mm, addr, xp, __pte(0)); } while (0) +#define pte_clear (mm, addr, ptep) set_pte(ptep, __pte(0))#define pmd_none (x) (!pmd_val(x))#define pmd_present (x) (pmd_val(x))
arch/sh/mm/cache-j2.c +2 -2
@@ -55,9 +55,9 @@ void __init j2_cache_init(void) local_flush_cache_dup_mm = j2_flush_both; local_flush_cache_page = j2_flush_both; local_flush_cache_range = j2_flush_both; - local_flush_dcache_page = j2_flush_dcache; + local_flush_dcache_folio = j2_flush_dcache; local_flush_icache_range = j2_flush_icache; - local_flush_icache_page = j2_flush_icache; + local_flush_icache_folio = j2_flush_icache; local_flush_cache_sigtramp = j2_flush_icache;pr_info ("Initial J2 CCR is %.8x\n" ,__raw_readl (j2_ccr_base));
arch/sh/mm/cache-sh4.c +18 -8
@@ -107,19 +107,29 @@ static inline void flush_cache_one(unsigned long start, unsigned long phys)* Write back & invalidate the D-cache of the page. * (To avoid "alias" issues) */ -static void sh4_flush_dcache_page (void *arg) +static void sh4_flush_dcache_folio (void *arg) { -struct page *page = arg; -unsigned long addr = (unsigned long )page_address (page); +struct folio *folio = arg;#ifndef CONFIG_SMP -struct address_space *mapping =page_mapping_file (page); +struct address_space *mapping =folio_flush_mapping (folio);if (mapping && !mapping_mapped (mapping)) -clear_bit (PG_dcache_clean, &page->flags); +clear_bit (PG_dcache_clean, &folio->flags); else#endif -flush_cache_one (CACHE_OC_ADDRESS_ARRAY | - (addr & shm_align_mask),page_to_phys (page)); + { +unsigned long pfn =folio_pfn (folio); +unsigned long addr = (unsigned long )folio_address (folio); +unsigned int i, nr =folio_nr_pages (folio); + +for (i =0 ; i < nr; i++) { +flush_cache_one (CACHE_OC_ADDRESS_ARRAY | + (addr & shm_align_mask), + pfn *PAGE_SIZE ); + addr +=PAGE_SIZE ; + pfn++; + } + }wmb (); }
@@ -379,7 +389,7 @@ void __init sh4_cache_init(void) __raw_readl(CCN_PRR)); local_flush_icache_range = sh4_flush_icache_range; - local_flush_dcache_page = sh4_flush_dcache_page; + local_flush_dcache_folio = sh4_flush_dcache_folio; local_flush_cache_all = sh4_flush_cache_all; local_flush_cache_mm = sh4_flush_cache_mm; local_flush_cache_dup_mm = sh4_flush_cache_mm;
arch/sh/mm/cache-sh7705.c +16 -10
@@ -132,15 +132,20 @@ static void __flush_dcache_page(unsigned long phys)* Write back & invalidate the D-cache of the page. * (To avoid "alias" issues) */ -static void sh7705_flush_dcache_page (void *arg) +static void sh7705_flush_dcache_folio (void *arg) { -struct page *page = arg; -struct address_space *mapping =page_mapping_file (page); +struct folio *folio = arg; +struct address_space *mapping =folio_flush_mapping (folio);if (mapping && !mapping_mapped (mapping)) -clear_bit (PG_dcache_clean, &page->flags); -else -__flush_dcache_page (__pa (page_address (page))); +clear_bit (PG_dcache_clean, &folio->flags); +else { +unsigned long pfn =folio_pfn (folio); +unsigned int i, nr =folio_nr_pages (folio); + +for (i =0 ; i < nr; i++) +__flush_dcache_page ((pfn + i) *PAGE_SIZE ); + } }static void sh7705_flush_cache_all (void *args)
@@ -176,19 +181,20 @@ static void sh7705_flush_cache_page(void *args)* Not entirely sure why this is necessary on SH3 with 32K cache but * without it we get occasional "Memory fault" when loading a program. */ -static void sh7705_flush_icache_page (void *page) +static void sh7705_flush_icache_folio (void *arg) { -__flush_purge_region (page_address (page),PAGE_SIZE ); +struct folio *folio = arg; +__flush_purge_region (folio_address (folio),folio_size (folio)); }void __init sh7705_cache_init (void ) { local_flush_icache_range = sh7705_flush_icache_range; - local_flush_dcache_page = sh7705_flush_dcache_page; + local_flush_dcache_folio = sh7705_flush_dcache_folio; local_flush_cache_all = sh7705_flush_cache_all; local_flush_cache_mm = sh7705_flush_cache_all; local_flush_cache_dup_mm = sh7705_flush_cache_all; local_flush_cache_range = sh7705_flush_cache_all; local_flush_cache_page = sh7705_flush_cache_page; - local_flush_icache_page = sh7705_flush_icache_page; + local_flush_icache_folio = sh7705_flush_icache_folio; }
arch/sh/mm/cache.c +30 -22
136 lines not on this page · raw message
arch/sh/mm/kmap.c +2 -1
@@ -27,10 +27,11 @@ void __init kmap_coherent_init(void)void *kmap_coherent (struct page *page,unsigned long addr) { +struct folio *folio =page_folio (page);enum fixed_addresses idx;unsigned long vaddr; -BUG_ON (!test_bit (PG_dcache_clean, &page->flags)); +BUG_ON (!test_bit (PG_dcache_clean, &folio->flags));preempt_disable ();pagefault_disable ();
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 25/38 sparc32: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages().
--- arch/sparc/include/asm/cacheflush_32.h | 10 ++++++++-- arch/sparc/include/asm/pgtable_32.h | 8 ++++---- arch/sparc/mm/init_32.c | 13 +++++++++++-- 3 files changed, 23 insertions(+), 8 deletions(-)
arch/sparc/include/asm/cacheflush_32.h +8 -2
@@ -2,6 +2,7 @@#ifndef _SPARC_CACHEFLUSH_H#define _SPARC_CACHEFLUSH_H +#include <linux/page-flags.h> #include <asm/cachetlb_32.h> #define flush_cache_all () \
@@ -16,6 +17,7 @@ sparc32_cachetlb_ops->cache_page(vma, addr)#define flush_icache_range (start, end) do { } while (0)#define flush_icache_page (vma, pg) do { } while (0) +#define flush_icache_pages (vma, pg, nr) do { } while (0)#define copy_to_user_page (vma, page, vaddr, dst, src, len) \ do { \
@@ -35,11 +37,15 @@#define flush_page_for_dma (addr) \ sparc32_cachetlb_ops->page_for_dma(addr) -struct page ;void sparc_flush_page_to_ram (struct page *page); +void sparc_flush_folio_to_ram (struct folio *folio);#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -#define flush_dcache_page (page) sparc_flush_page_to_ram(page) +#define flush_dcache_folio (folio) sparc_flush_folio_to_ram(folio) +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +}#define flush_dcache_mmap_lock (mapping) do { } while (0)#define flush_dcache_mmap_unlock (mapping) do { } while (0)
arch/sparc/include/asm/pgtable_32.h +4 -4
@@ -101,8 +101,6 @@ static inline void set_pte(pte_t *ptep, pte_t pteval)srmmu_swap ((unsigned long *)ptep,pte_val (pteval)); } -#define set_pte_at (mm,addr,ptep,pteval) set_pte(ptep,pteval) -static inline int srmmu_device_memory (unsigned long x) {return ((x &0xF0000000 ) !=0 );
@@ -256,6 +254,7 @@ static inline pte_t pte_mkyoung(pte_t pte)return __pte (pte_val (pte) |SRMMU_REF ); } +#define PFN_PTE_SHIFT (PAGE_SHIFT - 4)#define pfn_pte (pfn, prot) mk_pte(pfn_to_page(pfn), prot)static inline unsigned long pte_pfn (pte_t pte)
@@ -268,7 +267,7 @@ static inline unsigned long pte_pfn(pte_t pte)*/ return ~0UL ; } -return (pte_val (pte) &SRMMU_PTE_PMASK ) >> (PAGE_SHIFT -4 ); +return (pte_val (pte) &SRMMU_PTE_PMASK ) >>PFN_PTE_SHIFT ; }#define pte_page (pte) pfn_to_page(pte_pfn(pte))
@@ -318,6 +317,7 @@ void mmu_info(struct seq_file *m);#define FAULT_CODE_USER 0x4#define update_mmu_cache (vma, address, ptep) do { } while (0) +#define update_mmu_cache_range (vmf, vma, address, ptep, nr) do { } while (0)void srmmu_mapiorange (unsigned int bus,unsigned long xpa,unsigned long xva,unsigned int len);
@@ -422,7 +422,7 @@ static inline int io_remap_pfn_range(struct vm_area_struct *vma, ({ \ int __changed = !pte_same(*(__ptep), __entry); \ if (__changed) { \ - set_pte_at((__vma)->vm_mm, (__address), __ptep, __entry); \ + set_pte(__ptep, __entry); \ flush_tlb_page(__vma, __address); \ } \ __changed; \
arch/sparc/mm/init_32.c +11 -2
@@ -297,11 +297,20 @@ void sparc_flush_page_to_ram(struct page *page) {unsigned long vaddr = (unsigned long )page_address (page); -if (vaddr) -__flush_page_to_ram (vaddr); +__flush_page_to_ram (vaddr); }EXPORT_SYMBOL (sparc_flush_page_to_ram); +void sparc_flush_folio_to_ram (struct folio *folio) +{ +unsigned long vaddr = (unsigned long )folio_address (folio); +unsigned int i, nr =folio_nr_pages (folio); + +for (i =0 ; i < nr; i++) +__flush_page_to_ram (vaddr + i *PAGE_SIZE ); +} +EXPORT_SYMBOL (sparc_flush_folio_to_ram); +static const pgprot_t protection_map[16 ] = { [VM_NONE ] =PAGE_NONE , [VM_READ ] =PAGE_READONLY ,
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 26/38 sparc64: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add set_ptes(), update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages(). Convert the PG_dcache_dirty flag from being per-page to per-folio.
--- arch/sparc/include/asm/cacheflush_64.h | 18 ++++-- arch/sparc/include/asm/pgtable_64.h | 29 +++++++--- arch/sparc/kernel/smp_64.c | 56 +++++++++++------- arch/sparc/mm/init_64.c | 78 +++++++++++++++----------- arch/sparc/mm/tlb.c | 5 +- 5 files changed, 119 insertions(+), 67 deletions(-)
arch/sparc/include/asm/cacheflush_64.h +12 -6
@@ -35,20 +35,26 @@ void flush_icache_range(unsigned long start, unsigned long end);void __flush_icache_page (unsigned long );void __flush_dcache_page (void *addr,int flush_icache); -void flush_dcache_page_impl (struct page *page); +void flush_dcache_folio_impl (struct folio *folio);#ifdef CONFIG_SMP -void smp_flush_dcache_page_impl (struct page *page,int cpu); -void flush_dcache_page_all (struct mm_struct *mm,struct page *page); +void smp_flush_dcache_folio_impl (struct folio *folio,int cpu); +void flush_dcache_folio_all (struct mm_struct *mm,struct folio *folio);#else -#define smp_flush_dcache_page_impl (page,cpu) flush_dcache_page_impl(page) -#define flush_dcache_page_all (mm,page) flush_dcache_page_impl(page) +#define smp_flush_dcache_folio_impl (folio, cpu) flush_dcache_folio_impl(folio) +#define flush_dcache_folio_all (mm, folio) flush_dcache_folio_impl(folio)#endif void __flush_dcache_range (unsigned long start,unsigned long end);#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -void flush_dcache_page (struct page *page); +void flush_dcache_folio (struct folio *folio); +#define flush_dcache_folio flush_dcache_folio +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +}#define flush_icache_page (vma, pg) do { } while(0) +#define flush_icache_pages (vma, pg, nr) do { } while(0) void flush_ptrace_access(struct vm_area_struct *, struct page *, unsigned long uaddr, void *kaddr,
arch/sparc/include/asm/pgtable_64.h +22 -7
@@ -86,6 +86,7 @@ extern unsigned long VMALLOC_END;#define vmemmap ((struct page *)VMEMMAP_BASE)#include <linux/sched.h> +#include <asm/tlbflush.h> bool kern_addr_valid (unsigned long addr);
@@ -927,8 +928,21 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,maybe_tlb_batch_add (mm, addr, ptep, orig, fullmm,PAGE_SHIFT ); } -#define set_pte_at (mm,addr,ptep,pte) \ - __set_pte_at((mm), (addr), (ptep), (pte), 0) +static inline void set_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,pte_t pte,unsigned int nr) +{ +arch_enter_lazy_mmu_mode (); +for (;;) { +__set_pte_at (mm, addr, ptep, pte,0 ); +if (--nr ==0 ) +break ; + ptep++; +pte_val (pte) +=PAGE_SIZE ; + addr +=PAGE_SIZE ; + } +arch_leave_lazy_mmu_mode (); +} +#define set_ptes set_ptes#define pte_clear (mm,addr,ptep) \ set_pte_at((mm), (addr), (ptep), __pte(0UL))
@@ -947,8 +961,8 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, \if (pfn_valid (this_pfn) && \ (((old_addr) ^ (new_addr)) & (1 <<13 ))) \ -flush_dcache_page_all (current->mm, \ -pfn_to_page (this_pfn)); \ +flush_dcache_folio_all (current->mm, \ +page_folio (pfn_to_page (this_pfn))); \ } \ newpte; \ })
@@ -963,7 +977,10 @@ struct seq_file;void mmu_info (struct seq_file *);struct vm_area_struct ; -void update_mmu_cache (struct vm_area_struct *,unsigned long ,pte_t *); +void update_mmu_cache_range (struct vm_fault *,struct vm_area_struct *, +unsigned long addr,pte_t *ptep,unsigned int nr); +#define update_mmu_cache (vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1)#ifdef CONFIG_TRANSPARENT_HUGEPAGE void update_mmu_cache_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd);
@@ -1121,8 +1138,6 @@ static inline bool pte_access_permitted(pte_t pte, bool write) }#define pte_access_permitted pte_access_permitted -#include <asm/tlbflush.h> -/* We provide our own get_unmapped_area to cope with VA holes and * SHM area cache aliasing for userland. */
arch/sparc/kernel/smp_64.c +36 -20
111 lines not on this page · raw message
arch/sparc/mm/init_64.c +46 -32
187 lines not on this page · raw message
arch/sparc/mm/tlb.c +3 -2
@@ -118,6 +118,7 @@ void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr,unsigned long paddr, pfn =pte_pfn (orig);struct address_space *mapping;struct page *page; +struct folio *folio;if (!pfn_valid (pfn)) goto no_cache_flush;
@@ -127,13 +128,13 @@ void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr, goto no_cache_flush;/* A real file page? */ - mapping =page_mapping_file (page); + mapping =folio_flush_mapping (folio);if (!mapping) goto no_cache_flush; paddr = (unsigned long )page_address (page);if ((paddr ^ vaddr) & (1 <<13 )) -flush_dcache_page_all (mm, page); +flush_dcache_folio_all (mm, folio); } no_cache_flush:
Signature · 1 line
2.40.1
Re: [PATCH v6 26/38] sparc64: Implement the new page table range API
Hi,
Quoted 8 lines · Matthew Wilcox (Oracle)
Add
set_ptes(),update_mmu_cache_range(),flush_dcache_folio()andflush_icache_pages(). Convert thePG_dcache_dirtyflag from being per-page to per-folio.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Cc "David S. Miller" <[email protected]>
This patch causes all my sparc64 qemu boot tests to crash.
[ 4.890744] Unable to handle kernel NULL pointer dereference
[ 4.891273] tsk->{mm,active_mm}->context = 0000000000000001
[ 4.891475] tsk->{mm,active_mm}->pgd = fffff80005452000
[ 4.891660] \|/ ____ \|/
[ 4.891660] "@'/ .. \`@"
[ 4.891660] /_| \__/ |_\
[ 4.891660] \__U_/
[ 4.892116] modprobe(45): Oops [#1]
[ 4.892555] CPU: 0 PID: 45 Comm: modprobe Tainted: G N 6.5.0+ #1
[ 4.892949] TSTATE: 0000004411001601 TPC: 00000000004565d8 TNPC: 00000000004565dc Y: 00000008 Tainted: G N
[ 4.893307] TPC: <tlb_batch_add+0xf8/0x2a0>
[ 4.893829] g0: ec3264ae48eeb037 g1: 0000000000000028 g2: 00000000000a010c g3: 0000060000000000
[ 4.894103] g4: fffff80004c8c1a0 g5: fffff8001dc2e000 g6: fffff80005440000 g7: fffff8001fe3f700
[ 4.894373] o0: fffff80005420008 o1: 000007feffffe000 o2: fffff80005443788 o3: fffff80004c8c1a0
[ 4.894642] o4: fffff80004c8cc88 o5: 0000000001a3c000 sp: fffff80005442ee1 ret_pc: 00000000006058c8
[ 4.894918] RPC: <__pte_offset_map_lock+0x68/0x120>
[ 4.895118] l0: 0000000000605908 l1: 00000000011e8998 l2: 00000000011e8968 l3: fffff80005420008
[ 4.895392] l4: 000000000541e000 l5: fffff80005410060 l6: 00000000023c7800 l7: 00000000020fdee8
[ 4.895661] i0: fffff80005410020 i1: 000007feffffe000 i2: 0000060000052600 i3: 8000000002931fb2
[ 4.895931] i4: 0000000000000000 i5: 000000000000000d i6: fffff80005442f91 i7: 0000000000601718
[ 4.896204] I7: <change_protection+0x678/0x9e0>
[ 4.896403] Call Trace:
[ 4.896567] [<0000000000601718>] change_protection+0x678/0x9e0
[ 4.896807] [<0000000000601b80>] mprotect_fixup+0x100/0x2e0
[ 4.896990] [<0000000000652050>] setup_arg_pages+0x130/0x2a0
[ 4.897172] [<00000000006ba758>] load_elf_binary+0x358/0x13c0
[ 4.897367] [<0000000000652a00>] bprm_execve+0x2e0/0x8e0
[ 4.897540] [<0000000000653e24>] kernel_execve+0x144/0x200
[ 4.898023] [<000000000048775c>] call_usermodehelper_exec_async+0xbc/0x140
[ 4.898248] [<00000000004060e8>] ret_from_fork+0x1c/0x2c
[ 4.898422] [<0000000000000000>] 0x0
[ 4.898641] Disabling lock debugging due to kernel taint
[ 4.898860] Caller[0000000000601718]: change_protection+0x678/0x9e0
[ 4.899073] Caller[0000000000601b80]: mprotect_fixup+0x100/0x2e0
[ 4.899254] Caller[0000000000652050]: setup_arg_pages+0x130/0x2a0
[ 4.899435] Caller[00000000006ba758]: load_elf_binary+0x358/0x13c0
[ 4.899618] Caller[0000000000652a00]: bprm_execve+0x2e0/0x8e0
[ 4.899789] Caller[0000000000653e24]: kernel_execve+0x144/0x200
[ 4.899964] Caller[000000000048775c]: call_usermodehelper_exec_async+0xbc/0x140
[ 4.900177] Caller[00000000004060e8]: ret_from_fork+0x1c/0x2c
[ 4.900349] Caller[0000000000000000]: 0x0
[ 4.900487] Instruction DUMP:
[ 4.900517] 80886001
[ 4.900714] 126fffca
[ 4.900797] 01000000
[ 4.900876] <c2582000>
[ 4.900954] 83307013
[ 4.901032] 80886001
[ 4.901109] 02680007
[ 4.901187] 01000000
[ 4.901269] c2582000
[ 4.901353]
[ 4.901535] note: modprobe[45] exited with preempt_count 2Bisect log attached.
Guenter
---
# bad: [708283abf896dd4853e673cc8cba70acaf9bf4ea] Merge tag 'dmaengine-6.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine
# good: [2dde18cd1d8fac735875f2e4987f11817cc0bc2c] Linux 6.5
git bisect start 'HEAD' 'v6.5'
# bad: [53ea7f624fb91074c2f9458832ed74975ee5d64c] Merge tag 'xfs-6.6-merge-1' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
git bisect bad 53ea7f624fb91074c2f9458832ed74975ee5d64c
# good: [c873512ef3a39cc1a605b7a5ff2ad0a33d619aa8] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
git bisect good c873512ef3a39cc1a605b7a5ff2ad0a33d619aa8
# good: [3b6bf5b1f8e3d17d7566027cdc5a8262991eb5bc] Merge tag 'spi-v6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
git bisect good 3b6bf5b1f8e3d17d7566027cdc5a8262991eb5bc
# bad: [b96a3e9142fdf346b05b20e867b4f0dfca119e96] Merge tag 'mm-stable-2023-08-28-18-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
git bisect bad b96a3e9142fdf346b05b20e867b4f0dfca119e96
# bad: [bb7dbaafff3f582d18028a5b99a8faa789842678] mm: remove checks for pte_index
git bisect bad bb7dbaafff3f582d18028a5b99a8faa789842678
# good: [601f006fddc66e369fdac7c572f981eafd159dac] powerpc/book3s64/radix: remove mmu_vmemmap_psize
git bisect good 601f006fddc66e369fdac7c572f981eafd159dac
# good: [7e5f42ae3413785c68c383acb787f9ce8f243096] mm: convert pmd_ptlock_free() to use ptdescs
git bisect good 7e5f42ae3413785c68c383acb787f9ce8f243096
# good: [a644b0abbfe1d7cf775082cafdcc7b5f3c35becf] mm: convert split_huge_pages_pid() to use a folio
git bisect good a644b0abbfe1d7cf775082cafdcc7b5f3c35becf
# good: [994209410919f2b84b7e4ab2e78785d9715308ad] nios2: implement the new page table range API
git bisect good 994209410919f2b84b7e4ab2e78785d9715308ad
# bad: [9f1f5b60e76d44fa85fef6970b7477f72d3999eb] mm: use flush_icache_pages() in do_set_pmd()
git bisect bad 9f1f5b60e76d44fa85fef6970b7477f72d3999eb
# good: [665f640294540a941aabb81ae46dfc671aff5259] sparc32: implement the new page table range API
git bisect good 665f640294540a941aabb81ae46dfc671aff5259
# bad: [4fbb7e7f47dbc631a9f5bad3171ccbca171ed1d3] xtensa: implement the new page table range API
git bisect bad 4fbb7e7f47dbc631a9f5bad3171ccbca171ed1d3
# bad: [fd8132e6e9fdecb9ff7d1db98014d372e03f3c9d] um: implement the new page table range API
git bisect bad fd8132e6e9fdecb9ff7d1db98014d372e03f3c9d
# bad: [1a10a44dfc1d55ba84987da1f8377258a044499c] sparc64: implement the new page table range API
git bisect bad 1a10a44dfc1d55ba84987da1f8377258a044499c
# first bad commit: [1a10a44dfc1d55ba84987da1f8377258a044499c] sparc64: implement the new page table range API
Mike Rapoport
reviewerRe: [PATCH v6 26/38] sparc64: Implement the new page table range API
Quoted 24 lines · Guenter Roeck
Hi,
Quoted 8 lines · Matthew Wilcox (Oracle)
Add
set_ptes(),update_mmu_cache_range(),flush_dcache_folio()andflush_icache_pages(). Convert thePG_dcache_dirtyflag from being per-page to per-folio.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Cc "David S. Miller" <[email protected]>This patch causes all my sparc64 qemu boot tests to crash.
[ 4.890744] Unable to handle kernel NULL pointer dereference [ 4.891273] tsk->{mm,active_mm}->context = 0000000000000001 [ 4.891475] tsk->{mm,active_mm}->pgd = fffff80005452000 [ 4.891660] \|/ ____ \|/ [ 4.891660] "@'/ .. \`@" [ 4.891660] /_| \__/ |_\ [ 4.891660] \__U_/ [ 4.892116] modprobe(45): Oops [#1] [ 4.892555] CPU: 0 PID: 45 Comm: modprobe Tainted: G N 6.5.0+ #1 [ 4.892949] TSTATE: 0000004411001601 TPC: 00000000004565d8 TNPC: 00000000004565dc Y: 00000008 Tainted: G N
...
Quoted 1 line
[ 4.901535] note: modprobe[45] exited with preempt_count 2
This should fix it:
From 8181d1f582a309b51fe4cb02a783628257b91c86 Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (IBM)" <[email protected]>
Date: Mon, 4 Sep 2023 20:37:59 +0300
Subject: [PATCH] sparc64: add missing initialization of folio in
tlb_batch_add()
Commit 1a10a44dfc1d ("sparc64: implement the new page table range API") missed initialization of folio variable in tlb_batch_add() which causes boot tests to crash.
Add missing initialization.
--- arch/sparc/mm/tlb.c | 1 + 1 file changed, 1 insertion(+)
arch/sparc/mm/tlb.c +1 -0
@@ -128,6 +128,7 @@ void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr, goto no_cache_flush;/* A real file page? */ + folio =page_folio (page); mapping =folio_flush_mapping (folio);if (!mapping) goto no_cache_flush;
Signature · 1 line
2.39.2
Quoted 2 lines
Guenter
Signature · 2 lines
Sincerely yours, Mike.
Guenter Roeck
Tested-byRe: [PATCH v6 26/38] sparc64: Implement the new page table range API
Quoted 42 lines · Mike Rapoport
Quoted 24 lines · Guenter Roeck
Hi,
Quoted 8 lines · Matthew Wilcox (Oracle)
Add
set_ptes(),update_mmu_cache_range(),flush_dcache_folio()andflush_icache_pages(). Convert thePG_dcache_dirtyflag from being per-page to per-folio.Signed-off-by Matthew Wilcox (Oracle) <[email protected]>Cc "David S. Miller" <[email protected]>This patch causes all my sparc64 qemu boot tests to crash.
[ 4.890744] Unable to handle kernel NULL pointer dereference [ 4.891273] tsk->{mm,active_mm}->context = 0000000000000001 [ 4.891475] tsk->{mm,active_mm}->pgd = fffff80005452000 [ 4.891660] \|/ ____ \|/ [ 4.891660] "@'/ .. \`@" [ 4.891660] /_| \__/ |_\ [ 4.891660] \__U_/ [ 4.892116] modprobe(45): Oops [#1] [ 4.892555] CPU: 0 PID: 45 Comm: modprobe Tainted: G N 6.5.0+ #1 [ 4.892949] TSTATE: 0000004411001601 TPC: 00000000004565d8 TNPC: 00000000004565dc Y: 00000008 Tainted: G N...
Quoted 1 line
[ 4.901535] note: modprobe[45] exited with preempt_count 2This should fix it:
Quoted 1 line
From
8181d1f582a309b51fe4cb02a783628257b91c86Mon Sep 17 00:00:00 2001From: "Mike Rapoport (IBM)" <[email protected]>
Date: Mon, 4 Sep 2023 20:37:59 +0300
Subject: [PATCH] sparc64: add missing initialization of folio in
tlb_batch_add()Commit
1a10a44dfc1d("sparc64: implement the new page table range API") missed initialization of folio variable intlb_batch_add()which causes boot tests to crash.
Quoted 5 lines
Add missing initialization.
Reported-by Guenter Roeck <[email protected]>Fixes 1a10a44dfc1d ("sparc64: implement the new page table range API")Signed-off-by Mike Rapoport (IBM) <[email protected]>
Yes, it does.
Thanks,
Guenter
Quoted 16 lines
--- arch/sparc/mm/tlb.c | 1 + 1 file changed, 1 insertion(+)arch/sparc/mm/tlb.c +1
-0@@ -128,6 +128,7 @@ void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr, goto no_cache_flush;/* A real file page? */ + folio =page_folio (page); mapping =folio_flush_mapping (folio);if (!mapping) goto no_cache_flush;
Matthew Wilcox (Oracle)
maintainerPATCH v6 27/38 um: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT and update_mmu_cache_range().
--- arch/um/include/asm/pgtable.h | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
arch/um/include/asm/pgtable.h +2 -5
@@ -242,11 +242,7 @@ static inline void set_pte(pte_t *pteptr, pte_t pteval)if (pte_present (*pteptr)) *pteptr =pte_mknewprot (*pteptr); } -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *pteptr,pte_t pteval) -{ -set_pte (pteptr, pteval); -} +#define PFN_PTE_SHIFT PAGE_SHIFT#define __HAVE_ARCH_PTE_SAMEstatic inline int pte_same (pte_t pte_a,pte_t pte_b)
@@ -290,6 +286,7 @@ struct mm_struct;extern pte_t *virt_to_pte (struct mm_struct *mm,unsigned long addr);#define update_mmu_cache (vma,address,ptep) do {} while (0) +#define update_mmu_cache_range (vmf, vma, address, ptep, nr) do {} while (0)/* * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
Signature · 7 lines
2.40.1 _______________________________________________ linux-um mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-um
Matthew Wilcox (Oracle)
maintainerPATCH v6 28/38 x86: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT and a noop update_mmu_cache_range().
--- arch/x86/include/asm/pgtable.h | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
arch/x86/include/asm/pgtable.h +7 -7
@@ -185,6 +185,8 @@ static inline int pte_special(pte_t pte)static inline u64 protnone_mask (u64 val); +#define PFN_PTE_SHIFT PAGE_SHIFT +static inline unsigned long pte_pfn (pte_t pte) {phys_addr_t pfn =pte_val (pte);
@@ -1020,13 +1022,6 @@ static inline pud_t native_local_pudp_get_and_clear(pud_t *pudp)return res; } -static inline void set_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte) -{ -page_table_check_ptes_set (mm, ptep, pte,1 ); -set_pte (ptep, pte); -} -static inline void set_pmd_at (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,pmd_t pmd) {
@@ -1292,6 +1287,11 @@ static inline void update_mmu_cache(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { } +static inline void update_mmu_cache_range (struct vm_fault *vmf, +struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,unsigned int nr) +{ +}static inline void update_mmu_cache_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd) {
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 29/38 xtensa: Implement the new page table range API
In reply to Matthew Wilcox (Oracle)Add PFN_PTE_SHIFT, update_mmu_cache_range(), flush_dcache_folio() and flush_icache_pages().
--- arch/xtensa/include/asm/cacheflush.h | 9 ++- arch/xtensa/include/asm/pgtable.h | 18 +++--- arch/xtensa/mm/cache.c | 83 ++++++++++++++++------------ 3 files changed, 63 insertions(+), 47 deletions(-)
arch/xtensa/include/asm/cacheflush.h +8 -1
@@ -119,8 +119,14 @@ void flush_cache_page(struct vm_area_struct*,#define flush_cache_vmap (start,end) flush_cache_all()#define flush_cache_vunmap (start,end) flush_cache_all() +void flush_dcache_folio (struct folio *folio); +#define flush_dcache_folio flush_dcache_folio +#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -void flush_dcache_page (struct page *); +static inline void flush_dcache_page (struct page *page) +{ +flush_dcache_folio (page_folio (page)); +}void local_flush_cache_range (struct vm_area_struct *vma,unsigned long start,unsigned long end);
@@ -156,6 +162,7 @@ void local_flush_cache_page(struct vm_area_struct *vma,/* This is not required, see Documentation/core-api/cachetlb.rst */ #define flush_icache_page (vma,page) do { } while (0) +#define flush_icache_pages (vma, page, nr) do { } while (0)#define flush_dcache_mmap_lock (mapping) do { } while (0)#define flush_dcache_mmap_unlock (mapping) do { } while (0)
arch/xtensa/include/asm/pgtable.h +8 -10
@@ -274,6 +274,7 @@ static inline pte_t pte_mkwrite(pte_t pte)* and a page entry and page directory to the page they refer to. */ +#define PFN_PTE_SHIFT PAGE_SHIFT#define pte_pfn (pte) (pte_val(pte) >> PAGE_SHIFT)#define pte_same (a,b) (pte_val(a) == pte_val(b))#define pte_page (x) pfn_to_page(pte_pfn(x))
@@ -301,15 +302,9 @@ static inline void update_pte(pte_t *ptep, pte_t pteval)struct mm_struct ; -static inline void -set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval) -{ -update_pte (ptep, pteval); -} - -static inline void set_pte (pte_t *ptep,pte_t pteval) +static inline void set_pte (pte_t *ptep,pte_t pte) { -update_pte (ptep, pteval); +update_pte (ptep, pte); } static inline void
@@ -407,8 +402,11 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)#else -extern void update_mmu_cache (struct vm_area_struct * vma, -unsigned long address,pte_t *ptep); +struct vm_fault ; +void update_mmu_cache_range (struct vm_fault *vmf,struct vm_area_struct *vma, +unsigned long address,pte_t *ptep,unsigned int nr); +#define update_mmu_cache (vma, address, ptep) \ + update_mmu_cache_range(NULL, vma, address, ptep, 1)typedef pte_t *pte_addr_t ;
arch/xtensa/mm/cache.c +47 -36
137 lines not on this page · raw message
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 30/38 mm: Remove page_mapping_file()
In reply to Matthew Wilcox (Oracle)This function has no more users.
--- include/linux/pagemap.h | 8 -------- 1 file changed, 8 deletions(-)
include/linux/pagemap.h +0 -8
@@ -414,14 +414,6 @@ static inline struct address_space *page_file_mapping(struct page *page)return folio_file_mapping (page_folio (page)); } -/* -* For file cache pages, return the address_space, otherwise return NULL -*/ -static inline struct address_space *page_mapping_file (struct page *page) -{ -return folio_flush_mapping (page_folio (page)); -} -/** * folio_inode - Get the host inode for this folio. * @folio: The folio.
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 31/38 mm: Rationalise flush_icache_pages() and flush_icache_page()
In reply to Matthew Wilcox (Oracle)Move the default (no-op) implementation of flush_icache_pages() to <linux/cacheflush.h> from <asm-generic/cacheflush.h>. Remove the flush_icache_page() wrapper from each architecture into <linux/cacheflush.h>.
--- arch/alpha/include/asm/cacheflush.h | 5 +---- arch/arc/include/asm/cacheflush.h | 9 --------- arch/arm/include/asm/cacheflush.h | 7 ------- arch/csky/abiv1/inc/abi/cacheflush.h | 1 - arch/csky/abiv2/inc/abi/cacheflush.h | 1 - arch/hexagon/include/asm/cacheflush.h | 2 +- arch/loongarch/include/asm/cacheflush.h | 2 -- arch/m68k/include/asm/cacheflush_mm.h | 1 - arch/mips/include/asm/cacheflush.h | 6 ------ arch/nios2/include/asm/cacheflush.h | 2 +- arch/parisc/include/asm/cacheflush.h | 2 +- arch/sh/include/asm/cacheflush.h | 2 +- arch/sparc/include/asm/cacheflush_32.h | 2 -- arch/sparc/include/asm/cacheflush_64.h | 3 --- arch/xtensa/include/asm/cacheflush.h | 4 ---- include/asm-generic/cacheflush.h | 12 ------------ include/linux/cacheflush.h | 9 +++++++++ 17 files changed, 14 insertions(+), 56 deletions(-)
arch/alpha/include/asm/cacheflush.h +1 -4
@@ -53,10 +53,6 @@ extern void flush_icache_user_page(struct vm_area_struct *vma,#define flush_icache_user_page flush_icache_user_page#endif /* CONFIG_SMP */ -/* This is used only in __do_fault and do_swap_page. */ -#define flush_icache_page (vma, page) \ - flush_icache_user_page((vma), (page), 0, 0) -/* * Both implementations of flush_icache_user_page flush the entire * address space, so one call, no matter how many pages.
@@ -66,6 +62,7 @@ static inline void flush_icache_pages(struct vm_area_struct *vma, {flush_icache_user_page (vma, page,0 ,0 ); } +#define flush_icache_pages flush_icache_pages#include <asm-generic/cacheflush.h>
arch/arc/include/asm/cacheflush.h +0 -9
@@ -18,15 +18,6 @@#include <linux/mm.h> #include <asm/shmparam.h> -/* -* Semantically we need this because icache doesn't snoop dcache/dma. -* However ARC Cache flush requires paddr as well as vaddr, latter not available -* in the flush_icache_page() API. So we no-op it but do the equivalent work -* in update_mmu_cache() -*/ -#define flush_icache_page (vma, page) -#define flush_icache_pages (vma, page, nr) -void flush_cache_all (void );void flush_icache_range (unsigned long kstart,unsigned long kend);
arch/arm/include/asm/cacheflush.h +0 -7
@@ -321,13 +321,6 @@ static inline void flush_anon_page(struct vm_area_struct *vma,#define flush_dcache_mmap_lock (mapping) xa_lock_irq(&mapping->i_pages)#define flush_dcache_mmap_unlock (mapping) xa_unlock_irq(&mapping->i_pages) -/* -* We don't appear to need to do anything here. In fact, if we did, we'd -* duplicate cache flushing elsewhere performed by flush_dcache_page(). -*/ -#define flush_icache_page (vma,page) do { } while (0) -#define flush_icache_pages (vma, page, nr) do { } while (0) -/* * flush_cache_vmap() is used when creating mappings (eg, via vmap, * vmalloc, ioremap etc) in kernel space for pages. On non-VIPT
arch/csky/abiv1/inc/abi/cacheflush.h +0 -1
@@ -45,7 +45,6 @@ extern void flush_cache_range(struct vm_area_struct *vma, unsigned long start, u#define flush_cache_vmap (start, end) cache_wbinv_all()#define flush_cache_vunmap (start, end) cache_wbinv_all() -#define flush_icache_page (vma, page) do {} while (0);#define flush_icache_range (start, end) cache_wbinv_range(start, end)#define flush_icache_mm_range (mm, start, end) cache_wbinv_range(start, end)#define flush_icache_deferred (mm) do {} while (0);
arch/csky/abiv2/inc/abi/cacheflush.h +0 -1
@@ -33,7 +33,6 @@ static inline void flush_dcache_page(struct page *page)#define flush_dcache_mmap_lock (mapping) do { } while (0)#define flush_dcache_mmap_unlock (mapping) do { } while (0) -#define flush_icache_page (vma, page) do { } while (0)#define flush_icache_range (start, end) cache_wbinv_range(start, end)
arch/hexagon/include/asm/cacheflush.h +1 -1
@@ -18,7 +18,7 @@* - flush_cache_range(vma, start, end) flushes a range of pages * - flush_icache_range(start, end) flush a range of instructions * - flush_dcache_page(pg) flushes(wback&invalidates) a page for dcache -* - flush_icache_page(vma, pg) flushes(invalidates) a page for icache +* - flush_icache_pages(vma, pg, nr) flushes(invalidates) nr pages for icache * * Need to doublecheck which one is really needed for ptrace stuff to work. */
arch/loongarch/include/asm/cacheflush.h +0 -2
@@ -46,8 +46,6 @@ void local_flush_icache_range(unsigned long start, unsigned long end);#define flush_cache_page (vma, vmaddr, pfn) do { } while (0)#define flush_cache_vmap (start, end) do { } while (0)#define flush_cache_vunmap (start, end) do { } while (0) -#define flush_icache_page (vma, page) do { } while (0) -#define flush_icache_pages (vma, page) do { } while (0)#define flush_icache_user_page (vma, page, addr, len) do { } while (0)#define flush_dcache_page (page) do { } while (0)#define flush_dcache_mmap_lock (mapping) do { } while (0)
arch/m68k/include/asm/cacheflush_mm.h +0 -1
@@ -261,7 +261,6 @@ static inline void __flush_pages_to_ram(void *vaddr, unsigned int nr)#define flush_dcache_mmap_unlock (mapping) do { } while (0)#define flush_icache_pages (vma, page, nr) \ __flush_pages_to_ram(page_address(page), nr) -#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1)extern void flush_icache_user_page (struct vm_area_struct *vma,struct page *page,unsigned long addr,int len);
arch/mips/include/asm/cacheflush.h +0 -6
@@ -82,12 +82,6 @@ static inline void flush_anon_page(struct vm_area_struct *vma,__flush_anon_page (page, vmaddr); } -static inline void flush_icache_pages (struct vm_area_struct *vma, -struct page *page,unsigned int nr) -{ -} -#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1) -extern void (*flush_icache_range)(unsigned long start,unsigned long end);extern void (*local_flush_icache_range)(unsigned long start,unsigned long end); extern void (*__flush_icache_user_range)(unsigned long start,
arch/nios2/include/asm/cacheflush.h +1 -1
@@ -35,7 +35,7 @@ void flush_dcache_folio(struct folio *folio);extern void flush_icache_range (unsigned long start,unsigned long end);void flush_icache_pages (struct vm_area_struct *vma,struct page *page,unsigned int nr); -#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1); +#define flush_icache_pages flush_icache_pages#define flush_cache_vmap (start, end) flush_dcache_range(start, end)#define flush_cache_vunmap (start, end) flush_dcache_range(start, end)
arch/parisc/include/asm/cacheflush.h +1 -1
@@ -60,7 +60,7 @@ static inline void flush_dcache_page(struct page *page)void flush_icache_pages (struct vm_area_struct *vma,struct page *page,unsigned int nr); -#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1) +#define flush_icache_pages flush_icache_pages#define flush_icache_range (s,e) do { \ flush_kernel_dcache_range_asm(s,e); \
arch/sh/include/asm/cacheflush.h +1 -1
@@ -53,7 +53,7 @@ extern void flush_icache_range(unsigned long start, unsigned long end);#define flush_icache_user_range flush_icache_rangevoid flush_icache_pages (struct vm_area_struct *vma,struct page *page,unsigned int nr); -#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1) +#define flush_icache_pages flush_icache_pagesextern void flush_cache_sigtramp (unsigned long address);struct flusher_data {
arch/sparc/include/asm/cacheflush_32.h +0 -2
@@ -16,8 +16,6 @@#define flush_cache_page (vma,addr,pfn) \ sparc32_cachetlb_ops->cache_page(vma, addr)#define flush_icache_range (start, end) do { } while (0) -#define flush_icache_page (vma, pg) do { } while (0) -#define flush_icache_pages (vma, pg, nr) do { } while (0)#define copy_to_user_page (vma, page, vaddr, dst, src, len) \ do { \
arch/sparc/include/asm/cacheflush_64.h +0 -3
@@ -53,9 +53,6 @@ static inline void flush_dcache_page(struct page *page)flush_dcache_folio (page_folio (page)); } -#define flush_icache_page (vma, pg) do { } while(0) -#define flush_icache_pages (vma, pg, nr) do { } while(0) -void flush_ptrace_access (struct vm_area_struct *,struct page *,unsigned long uaddr,void *kaddr,unsigned long len,int write);
arch/xtensa/include/asm/cacheflush.h +0 -4
@@ -160,10 +160,6 @@ void local_flush_cache_page(struct vm_area_struct *vma,__invalidate_icache_range (start,(end) - (start)); \ } while (0) -/* This is not required, see Documentation/core-api/cachetlb.rst */ -#define flush_icache_page (vma,page) do { } while (0) -#define flush_icache_pages (vma, page, nr) do { } while (0) -#define flush_dcache_mmap_lock (mapping) do { } while (0)#define flush_dcache_mmap_unlock (mapping) do { } while (0)
include/asm-generic/cacheflush.h +0 -12
@@ -77,18 +77,6 @@ static inline void flush_icache_range(unsigned long start, unsigned long end)#define flush_icache_user_range flush_icache_range#endif -#ifndef flush_icache_page -static inline void flush_icache_pages (struct vm_area_struct *vma, -struct page *page,unsigned int nr) -{ -} - -static inline void flush_icache_page (struct vm_area_struct *vma, -struct page *page) -{ -} -#endif -#ifndef flush_icache_user_page static inline void flush_icache_user_page(struct vm_area_struct *vma, struct page *page,
include/linux/cacheflush.h +9 -0
@@ -17,4 +17,13 @@ static inline void flush_dcache_folio(struct folio *folio)#define flush_dcache_folio flush_dcache_folio#endif /* ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE */ +#ifndef flush_icache_pages +static inline void flush_icache_pages (struct vm_area_struct *vma, +struct page *page,unsigned int nr) +{ +} +#endif + +#define flush_icache_page (vma, page) flush_icache_pages(vma, page, 1) +#endif /* _LINUX_CACHEFLUSH_H */
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 32/38 mm: Tidy up set_ptes definition
In reply to Matthew Wilcox (Oracle)Now that all architectures are converted, we can remove the PFN_PTE_SHIFT ifdef and we can define set_pte_at() unconditionally.
--- include/linux/pgtable.h | 6 ------ 1 file changed, 6 deletions(-)
include/linux/pgtable.h +0 -6
@@ -204,7 +204,6 @@ static inline int pmd_young(pmd_t pmd)#endif #ifndef set_ptes -#ifdef PFN_PTE_SHIFT /** * set_ptes - Map consecutive pages to a contiguous range of addresses. * @mm: Address space to map the pages into.
@@ -234,13 +233,8 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, }arch_leave_lazy_mmu_mode (); } -#ifndef set_pte_at -#define set_pte_at (mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1) -#endif #endif -#else #define set_pte_at (mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1) -#endif #ifndef __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS extern int ptep_set_access_flags(struct vm_area_struct *vma,
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 33/38 mm: Use flush_icache_pages() in do_set_pmd()
In reply to Matthew Wilcox (Oracle)Push the iteration over each page down to the architectures (many can flush the entire THP without iteration).
--- mm/memory.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
mm/memory.c +1 -3
@@ -4400,7 +4400,6 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)bool write = vmf->flags &FAULT_FLAG_WRITE ;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ;pmd_t entry; -int i;vm_fault_t ret =VM_FAULT_FALLBACK ;if (!transhuge_vma_suitable (vma, haddr))
@@ -4433,8 +4432,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)if (unlikely (!pmd_none (*vmf->pmd))) goto out; -for (i =0 ; i <HPAGE_PMD_NR ; i++) -flush_icache_page (vma, page + i); +flush_icache_pages (vma, page,HPAGE_PMD_NR ); entry =mk_huge_pmd (page, vma->vm_page_prot);if (write)
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 34/38 filemap: Add filemap_map_folio_range()
In reply to Matthew Wilcox (Oracle)From: Yin Fengwei <[email protected]>
filemap_map_folio_range() maps partial/full folio. Comparing to original filemap_map_pages(), it updates refcount once per folio instead of per page and gets minor performance improvement for large folio.
With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]), got 2% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs.
[1]: https://github.com/antonblanchard/will-it-scale/pull/37
--- mm/filemap.c | 109 ++++++++++++++++++++++++++------------------------- 1 file changed, 55 insertions(+), 54 deletions(-)
mm/filemap.c +55 -54
@@ -2173,16 +2173,6 @@ unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start, }EXPORT_SYMBOL (filemap_get_folios); -static inline -bool folio_more_pages (struct folio *folio,pgoff_t index,pgoff_t max) -{ -if (!folio_test_large (folio) ||folio_test_hugetlb (folio)) -return false; -if (index >= max) -return false; -return index <folio_next_index (folio) -1 ; -} -/** * filemap_get_folios_contig - Get a batch of contiguous folios * @mapping: The address_space to search
@@ -3441,10 +3431,10 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct folio *folio,return false; } -static struct folio *next_uptodate_page (struct folio *folio, -struct address_space *mapping, -struct xa_state *xas,pgoff_t end_pgoff) +static struct folio *next_uptodate_folio (struct xa_state *xas, +struct address_space *mapping,pgoff_t end_pgoff) { +struct folio *folio =xas_next_entry (xas, end_pgoff);unsigned long max_idx; do {
@@ -3482,20 +3472,51 @@ static struct folio *next_uptodate_page(struct folio *folio,return NULL ; } -static inline struct folio *first_map_page (struct address_space *mapping, -struct xa_state *xas, -pgoff_t end_pgoff) +/* +* Map page range [start_page, start_page + nr_pages) of folio. +* start_page is gotten from start by folio_page(folio, start) +*/ +static vm_fault_t filemap_map_folio_range (struct vm_fault *vmf, +struct folio *folio,unsigned long start, +unsigned long addr,unsigned int nr_pages) { -return next_uptodate_page (xas_find (xas, end_pgoff), - mapping, xas, end_pgoff); -} +vm_fault_t ret =0 ; +struct vm_area_struct *vma = vmf->vma; +struct file *file = vma->vm_file; +struct page *page =folio_page (folio, start); +unsigned int mmap_miss =READ_ONCE (file->f_ra.mmap_miss); +unsigned int ref_count =0 , count =0 ; -static inline struct folio *next_map_page (struct address_space *mapping, -struct xa_state *xas, -pgoff_t end_pgoff) -{ -return next_uptodate_page (xas_next_entry (xas, end_pgoff), - mapping, xas, end_pgoff); +do { +if (PageHWPoison (page)) +continue ; + +if (mmap_miss >0 ) + mmap_miss--; + +/* +* NOTE: If there're PTE markers, we'll leave them to be +* handled in the specific fault path, and it'll prohibit the +* fault-around logic. +*/ +if (!pte_none (*vmf->pte)) +continue ; + +if (vmf->address == addr) + ret =VM_FAULT_NOPAGE ; + + ref_count++; +do_set_pte (vmf, page, addr); +update_mmu_cache (vma, addr, vmf->pte); + }while (vmf->pte++, page++, addr +=PAGE_SIZE , ++count < nr_pages); + +/* Restore the vmf->pte */ + vmf->pte -= nr_pages; + +folio_ref_add (folio, ref_count); +WRITE_ONCE (file->f_ra.mmap_miss, mmap_miss); + +return ret; } vm_fault_t filemap_map_pages(struct vm_fault *vmf,
@@ -3508,12 +3529,11 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,unsigned long addr;XA_STATE (xas, &mapping->i_pages, start_pgoff);struct folio *folio; -struct page *page; -unsigned int mmap_miss =READ_ONCE (file->f_ra.mmap_miss);vm_fault_t ret =0 ; +int nr_pages =0 ;rcu_read_lock (); - folio =first_map_page (mapping, &xas, end_pgoff); + folio =next_uptodate_folio (&xas, mapping, end_pgoff);if (!folio) goto out;
@@ -3530,17 +3550,13 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, goto out; } do { -again: - page = folio_file_page(folio, xas.xa_index); - if (PageHWPoison(page)) - goto unlock; - - if (mmap_miss > 0) - mmap_miss--; + unsigned long end; addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT; vmf->pte += xas.xa_index - last_pgoff; last_pgoff = xas.xa_index; + end = folio->index + folio_nr_pages(folio) - 1; + nr_pages = min(end, end_pgoff) - xas.xa_index + 1; /* * NOTE: If there're PTE markers, we'll leave them to be
@@ -3550,32 +3566,17 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,if (!pte_none (ptep_get (vmf->pte))) goto unlock; -/* We're about to handle the fault */ -if (vmf->address == addr) - ret =VM_FAULT_NOPAGE ; + ret |=filemap_map_folio_range (vmf, folio, + xas.xa_index - folio->index, addr, nr_pages); -do_set_pte (vmf, page, addr); -/* no need to invalidate: a not-present page won't be cached */ -update_mmu_cache (vma, addr, vmf->pte); -if (folio_more_pages (folio, xas.xa_index, end_pgoff)) { - xas.xa_index++; -folio_ref_inc (folio); - goto again; - } -folio_unlock (folio); -continue ; unlock: -if (folio_more_pages (folio, xas.xa_index, end_pgoff)) { - xas.xa_index++; - goto again; - }folio_unlock (folio);folio_put (folio); - }while ((folio =next_map_page (mapping, &xas, end_pgoff)) !=NULL ); + folio =next_uptodate_folio (&xas, mapping, end_pgoff); + }while (folio);pte_unmap_unlock (vmf->pte, vmf->ptl); out:rcu_read_unlock (); -WRITE_ONCE (file->f_ra.mmap_miss, mmap_miss);return ret; }EXPORT_SYMBOL (filemap_map_pages);
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 35/38 rmap: add folio_add_file_rmap_range()
In reply to Matthew Wilcox (Oracle)From: Yin Fengwei <[email protected]>
folio_add_file_rmap_range() allows to add pte mapping to a specific range of file folio. Comparing to page_add_file_rmap(), it batched updates __lruvec_stat for large folio.
--- include/linux/rmap.h | 2 ++ mm/rmap.c | 60 +++++++++++++++++++++++++++++++++----------- 2 files changed, 48 insertions(+), 14 deletions(-)
include/linux/rmap.h +2 -0
@@ -198,6 +198,8 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address);void page_add_file_rmap (struct page *,struct vm_area_struct *,bool compound); +void folio_add_file_rmap_range (struct folio *,struct page *,unsigned int nr, +struct vm_area_struct *,bool compound);void page_remove_rmap (struct page *,struct vm_area_struct *,bool compound); void folio_remove_rmap_range(struct folio *folio, struct page *page,
mm/rmap.c +46 -14
83 lines not on this page · raw message
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 36/38 mm: Convert do_set_pte() to set_pte_range()
In reply to Matthew Wilcox (Oracle)From: Yin Fengwei <[email protected]>
set_pte_range() allows to setup page table entries for a specific range. It takes advantage of batched rmap update for large folio. It now takes care of calling update_mmu_cache_range().
--- Documentation/filesystems/locking.rst | 2 +- include/linux/mm.h | 3 ++- mm/filemap.c | 3 +-- mm/memory.c | 37 +++++++++++++++++---------- 4 files changed, 28 insertions(+), 17 deletions(-)
Documentation/filesystems/locking.rst +1 -1
@@ -670,7 +670,7 @@ locked. The VM will unlock the page. Filesystem should find and map pages associated with offsets from "start_pgoff" till "end_pgoff". ->map_pages() is called with the RCU lock held and must not block. If it's not possible to reach a page without blocking, -filesystem should skip it. Filesystem should use do_set_pte() to setup +filesystem should skip it. Filesystem should use set_pte_range() to setup page table entry. Pointer to entry associated with the page is passed in "pte" field in vm_fault structure. Pointers to entries for other offsets should be calculated relative to "pte".
include/linux/mm.h +2 -1
@@ -1346,7 +1346,8 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) }vm_fault_t do_set_pmd (struct vm_fault *vmf,struct page *page); -void do_set_pte (struct vm_fault *vmf,struct page *page,unsigned long addr); +void set_pte_range (struct vm_fault *vmf,struct folio *folio, +struct page *page,unsigned int nr,unsigned long addr);vm_fault_t finish_fault (struct vm_fault *vmf);vm_fault_t finish_mkwrite_fault (struct vm_fault *vmf);
mm/filemap.c +1 -2
@@ -3506,8 +3506,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, ret =VM_FAULT_NOPAGE ; ref_count++; -do_set_pte (vmf, page, addr); -update_mmu_cache (vma, addr, vmf->pte); +set_pte_range (vmf, folio, page,1 , addr); }while (vmf->pte++, page++, addr +=PAGE_SIZE , ++count < nr_pages);/* Restore the vmf->pte */
mm/memory.c +24 -13
@@ -4465,15 +4465,24 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) }#endif -void do_set_pte (struct vm_fault *vmf,struct page *page,unsigned long addr) +/** +* set_pte_range - Set a range of PTEs to point to pages in a folio. +* @vmf: Fault decription. +* @folio: The folio that contains @page. +* @page: The first page to create a PTE for. +* @nr: The number of PTEs to create. +* @addr: The first address to create a PTE for. +*/ +void set_pte_range (struct vm_fault *vmf,struct folio *folio, +struct page *page,unsigned int nr,unsigned long addr) {struct vm_area_struct *vma = vmf->vma;bool uffd_wp =vmf_orig_pte_uffd_wp (vmf);bool write = vmf->flags &FAULT_FLAG_WRITE ; -bool prefault = vmf->address != addr; +bool prefault =in_range (vmf->address, addr, nr *PAGE_SIZE );pte_t entry; -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr); entry =mk_pte (page, vma->vm_page_prot);if (prefault &&arch_wants_old_prefaulted_pte ())
@@ -4487,14 +4496,18 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) entry =pte_mkuffd_wp (entry);/* copy-on-write page */ if (write && !(vma->vm_flags &VM_SHARED )) { -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, addr); -lru_cache_add_inactive_or_unevictable (page, vma); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr); +VM_BUG_ON_FOLIO (nr !=1 , folio); +folio_add_new_anon_rmap (folio, vma, addr); +folio_add_lru_vma (folio, vma); }else { -inc_mm_counter (vma->vm_mm,mm_counter_file (page)); -page_add_file_rmap (page, vma, false); +add_mm_counter (vma->vm_mm,mm_counter_file (page), nr); +folio_add_file_rmap_range (folio, page, nr, vma, false); } -set_pte_at (vma->vm_mm, addr, vmf->pte, entry); +set_ptes (vma->vm_mm, addr, vmf->pte, entry, nr); + +/* no need to invalidate: a not-present page won't be cached */ +update_mmu_cache_range (vmf, vma, addr, vmf->pte, nr); }static bool vmf_pte_changed (struct vm_fault *vmf)
@@ -4562,11 +4575,9 @@ vm_fault_t finish_fault(struct vm_fault *vmf)/* Re-check under ptl */ if (likely (!vmf_pte_changed (vmf))) { -do_set_pte (vmf, page, vmf->address); - -/* no need to invalidate: a not-present page won't be cached */ -update_mmu_cache (vma, vmf->address, vmf->pte); +struct folio *folio =page_folio (page); +set_pte_range (vmf, folio, page,1 , vmf->address); ret =0 ; }else {update_mmu_tlb (vma, vmf->address, vmf->pte);
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 37/38 filemap: Batch PTE mappings
In reply to Matthew Wilcox (Oracle)From: Yin Fengwei <[email protected]>
Call set_pte_range() once per contiguous range of the folio instead of once per page. This batches the updates to mm counters and the rmap.
With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]) got 15% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs.
Perf data collected before/after the change:
18.73%--page_add_file_rmap
|
--11.60%--__mod_lruvec_page_state
|
|--7.40%--__mod_memcg_lruvec_state
| |
| --5.58%--cgroup_rstat_updated
|
--2.53%--__mod_lruvec_state
|
--1.48%--__mod_node_page_state9.93%--page_add_file_rmap_range
|
--2.67%--__mod_lruvec_page_state
|
|--1.95%--__mod_memcg_lruvec_state
| |
| --1.57%--cgroup_rstat_updated
|
--0.61%--__mod_lruvec_state
|
--0.54%--__mod_node_page_stateThe running time of __mode_lruvec_page_state() is reduced about 9%.
[1]: https://github.com/antonblanchard/will-it-scale/pull/37
--- mm/filemap.c | 43 +++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 14 deletions(-)
mm/filemap.c +29 -14
@@ -3485,11 +3485,12 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,struct file *file = vma->vm_file;struct page *page =folio_page (folio, start);unsigned int mmap_miss =READ_ONCE (file->f_ra.mmap_miss); -unsigned int ref_count =0 , count =0 ; +unsigned int count =0 ; +pte_t *old_ptep = vmf->pte; do { - if (PageHWPoison(page)) - continue; + if (PageHWPoison(page + count)) + goto skip; if (mmap_miss > 0) mmap_miss--;
@@ -3499,20 +3500,34 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,* handled in the specific fault path, and it'll prohibit the * fault-around logic. */ -if (!pte_none (*vmf->pte)) -continue ; - -if (vmf->address == addr) - ret =VM_FAULT_NOPAGE ; +if (!pte_none (vmf->pte[count])) + goto skip; - ref_count++; -set_pte_range (vmf, folio, page,1 , addr); - }while (vmf->pte++, page++, addr +=PAGE_SIZE , ++count < nr_pages); + count++; +continue ; +skip: +if (count) { +set_pte_range (vmf, folio, page, count, addr); +folio_ref_add (folio, count); +if (in_range (vmf->address, addr, count)) + ret =VM_FAULT_NOPAGE ; + } -/* Restore the vmf->pte */ - vmf->pte -= nr_pages; + count++; + page += count; + vmf->pte += count; + addr += count *PAGE_SIZE ; + count =0 ; + }while (--nr_pages >0 ); + +if (count) { +set_pte_range (vmf, folio, page, count, addr); +folio_ref_add (folio, count); +if (in_range (vmf->address, addr, count)) + ret =VM_FAULT_NOPAGE ; + } -folio_ref_add (folio, ref_count); + vmf->pte = old_ptep;WRITE_ONCE (file->f_ra.mmap_miss, mmap_miss);return ret;
Signature · 1 line
2.40.1
Matthew Wilcox (Oracle)
maintainerPATCH v6 38/38 mm: Call update_mmu_cache_range() in more page fault handling paths
In reply to Matthew Wilcox (Oracle)Pass the vm_fault to the architecture to help it make smarter decisions about which PTEs to insert into the TLB.
--- mm/memory.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
mm/memory.c +8 -7
@@ -2997,7 +2997,7 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, entry =pte_mkyoung (vmf->orig_pte);if (ptep_set_access_flags (vma, addr, vmf->pte, entry,0 )) -update_mmu_cache (vma, addr, vmf->pte); +update_mmu_cache_range (vmf, vma, addr, vmf->pte,1 ); }/*
@@ -3174,7 +3174,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) entry =pte_mkyoung (vmf->orig_pte); entry =maybe_mkwrite (pte_mkdirty (entry), vma);if (ptep_set_access_flags (vma, vmf->address, vmf->pte, entry,1 )) -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 );pte_unmap_unlock (vmf->pte, vmf->ptl);count_vm_event (PGREUSE ); }
@@ -3298,7 +3298,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)*/ BUG_ON (unshare &&pte_write (entry));set_pte_at_notify (mm, vmf->address, vmf->pte, entry); -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 );if (old_folio) {/* * Only after switching the pte to the new page may
@@ -4181,7 +4181,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
@@ -4305,7 +4305,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry);/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
@@ -4994,7 +4994,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)if (writable) pte =pte_mkwrite (pte);ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte); -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 );pte_unmap_unlock (vmf->pte, vmf->ptl); goto out; }
@@ -5165,7 +5165,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) entry =pte_mkyoung (entry);if (ptep_set_access_flags (vmf->vma, vmf->address, vmf->pte, entry, vmf->flags &FAULT_FLAG_WRITE )) { -update_mmu_cache (vmf->vma, vmf->address, vmf->pte); +update_mmu_cache_range (vmf, vmf->vma, vmf->address, + vmf->pte,1 ); }else {/* Skip spurious TLB flush for retried page fault */ if (vmf->flags &FAULT_FLAG_TRIED )
Signature · 1 line
2.40.1
Andrew Morton
maintainerQuoted 4 lines · On Wed, 2 Aug 2023 16:13:28 +0100 "Matthew Wilcox (Oracle)"…
For some reason, I didn't get an email when v5 was dropped from linux-next (I got an email from Andrew saying he was going to, but when I didn't get the automated emails, I assumed he'd changed his mind). So here's v6, far later than I would have resent it.
Sorry - I try to reduce the amount of email I spray out by reducing it to a single email.