incoming
Subsystems (91)
Andrew Morton
maintainer146 patches, based on df0cc57e057f18e44dac8e6c18aba47ab53202f9 ("Linux 5.16")
Subsystems affected by this patch series:
kthread
ia64
scripts
ntfs
squashfs
ocfs2
vfs
mm/slab-generic
mm/slab
mm/kmemleak
mm/dax
mm/kasan
mm/debug
mm/pagecache
mm/gup
mm/shmem
mm/frontswap
mm/memremap
mm/memcg
mm/selftests
mm/pagemap
mm/dma
mm/vmalloc
mm/memory-failure
mm/hugetlb
mm/userfaultfd
mm/vmscan
mm/mempolicy
mm/oom-kill
mm/hugetlbfs
mm/migration
mm/thp
mm/ksm
mm/page-poison
mm/percpu
mm/rmap
mm/zswap
mm/zram
mm/cleanups
mm/hmm
mm/damon
Subsystem: kthread
Cai Huoqing <[email protected]>:
kthread: add the helper function kthread_run_on_cpu() RDMA/siw: make use of the helper function kthread_run_on_cpu()
ring-buffer: make use of the helper function kthread_run_on_cpu()
rcutorture: make use of the helper function kthread_run_on_cpu() trace/osnoise: make use of the helper function kthread_run_on_cpu() trace/hwlat: make use of the helper function kthread_run_on_cpu()
Subsystem: ia64
Yang Guang <[email protected]>:
ia64: module: use swap() to make code cleaner arch/ia64/kernel/setup.c: use swap() to make code cleaner
Jason Wang <[email protected]>:
ia64: fix typo in a comment
Greg Kroah-Hartman <[email protected]>:
ia64: topology: use default_groups in kobj_type
Subsystem: scripts
Drew Fustini <[email protected]>:
scripts/spelling.txt: add "oveflow"
Subsystem: ntfs
Yang Li <[email protected]>:
fs/ntfs/attrib.c: fix one kernel-doc comment
Subsystem: squashfs
Zheng Liang <[email protected]>:
squashfs: provide backing_dev_info in order to disable read-ahead
Subsystem: ocfs2
Zhang Mingyu <[email protected]>:
ocfs2: use BUG_ON instead of if condition followed by BUG.
Joseph Qi <[email protected]>:
ocfs2: clearly handle ocfs2_grab_pages_for_write() return value
Greg Kroah-Hartman <[email protected]>:
ocfs2: use default_groups in kobj_type
Colin Ian King <[email protected]>:
ocfs2: remove redundant assignment to pointer root_bh
Greg Kroah-Hartman <[email protected]>:
ocfs2: cluster: use default_groups in kobj_type
Colin Ian King <[email protected]>:
ocfs2: remove redundant assignment to variable free_space
Subsystem: vfs
Amit Daniel Kachhap <[email protected]>:
fs/ioctl: remove unnecessary __user annotation
Subsystem: mm/slab-generic
Marco Elver <[email protected]>:
mm/slab_common: use WARN() if cache still has objects on destroy
Subsystem: mm/slab
Muchun Song <[email protected]>:
mm: slab: make slab iterator functions static
Subsystem: mm/kmemleak
Kuan-Ying Lee <[email protected]>:
kmemleak: fix kmemleak false positive report with HW tag-based kasan enable
Calvin Zhang <[email protected]>:
mm: kmemleak: alloc gray object for reserved region with direct map
Kefeng Wang <[email protected]>:
mm: defer kmemleak object creation of module_alloc()
Subsystem: mm/dax
Joao Martins <[email protected]>:
Patch series "mm, device-dax: Introduce compound pages in devmap", v7:
mm/page_alloc: split prep_compound_page into head and tail subparts mm/page_alloc: refactor memmap_init_zone_device() page init mm/memremap: add ZONE_DEVICE support for compound pages
device-dax: use ALIGN() for determining pgoff
device-dax: use struct_size()
device-dax: ensure dev_dax->pgmap is valid for dynamic devices
device-dax: factor out page mapping initialization
device-dax: set mapping prior to vmf_insert_pfn{,_pmd,pud}()
device-dax: remove pfn from __dev_dax_{pte,pmd,pud}_fault()
device-dax: compound devmap support
Subsystem: mm/kasan
Marco Elver <[email protected]>:
kasan: test: add globals left-out-of-bounds test
kasan: add ability to detect double-kmem_cache_destroy()
kasan: test: add test case for double-kmem_cache_destroy()
Andrey Konovalov <[email protected]>:
kasan: fix quarantine conflicting with init_on_free
Subsystem: mm/debug
"Matthew Wilcox (Oracle)" <[email protected]>:
mm,fs: split dump_mapping() out from dump_page()
Anshuman Khandual <[email protected]>:
mm/debug_vm_pgtable: update comments regarding migration swap entries
Subsystem: mm/pagecache
chiminghao <[email protected]>:
mm/truncate.c: remove unneeded variable
Subsystem: mm/gup
Christophe Leroy <[email protected]>:
gup: avoid multiple user access locking/unlocking in fault_in_{read/write}able
Li Xinhai <[email protected]>:
mm/gup.c: stricter check on THP migration entry during follow_pmd_mask
Subsystem: mm/shmem
Yang Shi <[email protected]>:
mm: shmem: don't truncate page if memory failure happens
Gang Li <[email protected]>:
shmem: fix a race between shmem_unused_huge_shrink and shmem_evict_inode
Subsystem: mm/frontswap
Christophe JAILLET <[email protected]>:
mm/frontswap.c: use non-atomic '__set_bit()' when possible
Subsystem: mm/memremap
Subsystem: mm/memcg
Muchun Song <[email protected]>:
mm: memcontrol: make cgroup_memory_nokmem static
Donghai Qiao <[email protected]>:
mm/page_counter: remove an incorrect call to propagate_protected_usage()
Dan Schatzberg <[email protected]>:
mm/memcg: add oom_group_kill memory event
Shakeel Butt <[email protected]>:
memcg: better bounds on the memcg stats updates
Wang Weiyang <[email protected]>:
mm/memcg: use struct_size() helper in kzalloc()
Shakeel Butt <[email protected]>:
memcg: add per-memcg vmalloc stat
Subsystem: mm/selftests
chiminghao <[email protected]>:
tools/testing/selftests/vm/userfaultfd.c: use swap() to make code cleaner
Subsystem: mm/pagemap
Qi Zheng <[email protected]>:
mm: remove redundant check about FAULT_FLAG_ALLOW_RETRY bit
Colin Cross <[email protected]>:
Patch series "mm: rearrange madvise code to allow for reuse", v11:
mm: rearrange madvise code to allow for reuse
mm: add a field to store names for private anonymous memory
Suren Baghdasaryan <[email protected]>:
mm: add anonymous vma name refcounting
Arnd Bergmann <[email protected]>:
mm: move anon_vma declarations to linux/mm_inline.h
mm: move tlb_flush_pending inline helpers to mm_inline.h
Suren Baghdasaryan <[email protected]>:
mm: protect free_pgtables with mmap_lock write lock in exit_mmap
mm: document locking restrictions for vm_operations_struct::close mm/oom_kill: allow process_mrelease to run under mmap_lock protection
Shuah Khan <[email protected]>:
docs/vm: add vmalloced-kernel-stacks document
Pasha Tatashin <[email protected]>:
Patch series "page table check", v3:
mm: change page type prior to adding page table entry
mm: ptep_clear() page table helper
mm: page table check
x86: mm: add x86_64 support for page table check
"Matthew Wilcox (Oracle)" <[email protected]>:
mm: remove last argument of reuse_swap_page()
mm: remove the total_mapcount argument from page_trans_huge_map_swapcount()
mm: remove the total_mapcount argument from page_trans_huge_mapcount()
Subsystem: mm/dma
Christian König <[email protected]>:
mm/dmapool.c: revert "make dma pool to use kmalloc_node"
Subsystem: mm/vmalloc
Michal Hocko <[email protected]>:
Patch series "extend vmalloc support for constrained allocations", v2:
mm/vmalloc: alloc GFP_NO{FS,IO} for vmalloc
mm/vmalloc: add support for __GFP_NOFAIL
mm/vmalloc: be more explicit about supported gfp flags.
mm: allow !GFP_KERNEL allocations for kvmalloc
mm: make slab and vmalloc allocators __GFP_NOLOCKDEP aware
"NeilBrown" <[email protected]>:
mm: introduce memalloc_retry_wait()
Suren Baghdasaryan <[email protected]>:
mm/pagealloc: sysctl: change watermark_scale_factor max limit to 30%
Changcheng Deng <[email protected]>:
mm: fix boolreturn.cocci warning
Xiongwei Song <[email protected]>:
mm: page_alloc: fix building error on -Werror=array-compare
Michal Hocko <[email protected]>:
mm: drop node from alloc_pages_vma
Miles Chen <[email protected]>:
include/linux/gfp.h: further document GFP_DMA32
Anshuman Khandual <[email protected]>:
mm/page_alloc.c: modify the comment section for alloc_contig_pages()
Baoquan He <[email protected]>:
Patch series "Handle warning of allocation failure on DMA zone w/o managed pages", v4:
mm_zone: add function to check if managed dma zone exists
dma/pool: create dma atomic pool only if dma zone has managed pages mm/page_alloc.c: do not warn allocation failure on zone DMA if no managed pages
Subsystem: mm/memory-failure
Subsystem: mm/hugetlb
Mina Almasry <[email protected]>:
hugetlb: add hugetlb.*.numa_stat file
Yosry Ahmed <[email protected]>:
mm, hugepages: make memory size variable in hugepage-mremap selftest
Yang Yang <[email protected]>:
mm/vmstat: add events for THP max_ptes_* exceeds
Waiman Long <[email protected]>:
selftests/vm: make charge_reserved_hugetlb.sh work with existing cgroup setting
Subsystem: mm/userfaultfd
Peter Xu <[email protected]>:
selftests/uffd: allow EINTR/EAGAIN
Mike Kravetz <[email protected]>:
userfaultfd/selftests: clean up hugetlb allocation code
Subsystem: mm/vmscan
Gang Li <[email protected]>:
vmscan: make drop_slab_node static
Chen Wandun <[email protected]>:
mm/page_isolation: unset migratetype directly for non Buddy page
Subsystem: mm/mempolicy
"Aneesh Kumar K.V" <[email protected]>:
Patch series "mm: add new syscall set_mempolicy_home_node", v6:
mm/mempolicy: use policy_node helper with MPOL_PREFERRED_MANY mm/mempolicy: add set_mempolicy_home_node syscall mm/mempolicy: wire up syscall set_mempolicy_home_node
Randy Dunlap <[email protected]>:
mm/mempolicy: fix all kernel-doc warnings
Subsystem: mm/oom-kill
Jann Horn <[email protected]>:
mm, oom: OOM sysrq should always kill a process
Subsystem: mm/hugetlbfs
Sean Christopherson <[email protected]>:
hugetlbfs: fix off-by-one error in hugetlb_vmdelete_list()
Subsystem: mm/migration
Baolin Wang <[email protected]>:
Patch series "Improve the migration stats":
mm: migrate: fix the return value of migrate_pages()
mm: migrate: correct the hugetlb migration stats
mm: compaction: fix the migration stats in trace_mm_compaction_migratepages()
mm: migrate: support multiple target nodes demotion
mm: migrate: add more comments for selecting target node randomly
Huang Ying <[email protected]>:
mm/migrate: move node demotion code to near its user
Colin Ian King <[email protected]>:
mm/migrate: remove redundant variables used in a for-loop
Subsystem: mm/thp
Anshuman Khandual <[email protected]>:
mm/thp: drop unused trace events hugepage_[invalidate|splitting]
Subsystem: mm/ksm
Nanyong Sun <[email protected]>:
mm: ksm: fix use-after-free kasan report in ksm_might_need_to_copy
Subsystem: mm/page-poison
Naoya Horiguchi <[email protected]>:
Patch series "mm/hwpoison: fix unpoison_memory()", v4:
mm/hwpoison: mf_mutex for soft offline and unpoison mm/hwpoison: remove MF_MSG_BUDDY_2ND and MF_MSG_POISONED_HUGE mm/hwpoison: fix unpoison_memory()
Subsystem: mm/percpu
Qi Zheng <[email protected]>:
mm: memcg/percpu: account extra objcg space to memory cgroups
Subsystem: mm/rmap
Huang Ying <[email protected]>:
mm/rmap: fix potential batched TLB flush race
Subsystem: mm/zswap
Zhaoyu Liu <[email protected]>:
zpool: remove the list of pools_head
Subsystem: mm/zram
Luis Chamberlain <[email protected]>:
zram: use ATTRIBUTE_GROUPS
Subsystem: mm/cleanups
Quanfa Fu <[email protected]>:
mm: fix some comment errors
Ting Liu <[email protected]>:
mm: make some vars and functions static or __init
Subsystem: mm/hmm
Alistair Popple <[email protected]>:
mm/hmm.c: allow VM_MIXEDMAP to work with hmm_range_fault
Subsystem: mm/damon
Xin Hao <[email protected]>:
Patch series "mm/damon: Do some small changes", v4:
mm/damon: unified access_check function naming rules mm/damon: add 'age' of region tracepoint support mm/damon/core: use abs() instead of diff_of() mm/damon: remove some unneeded function definitions in damon.h
Yihao Han <[email protected]>:
mm/damon/vaddr: remove swap_ranges() and replace it with swap()
Xin Hao <[email protected]>:
mm/damon/schemes: add the validity judgment of thresholds mm/damon: move damon_rand() definition into damon.h mm/damon: modify damon_rand() macro to static inline function
SeongJae Park <[email protected]>:
Patch series "mm/damon: Misc cleanups":
mm/damon: convert macro functions to static inline functions Docs/admin-guide/mm/damon/usage: update for scheme quotas and watermarks Docs/admin-guide/mm/damon/usage: remove redundant information Docs/admin-guide/mm/damon/usage: mention tracepoint at the beginning Docs/admin-guide/mm/damon/usage: update for kdamond_pid and (mk|rm)_contexts mm/damon: remove a mistakenly added comment for a future feature
Patch series "mm/damon/schemes: Extend stats for better online analysis and tuning":
mm/damon/schemes: account scheme actions that successfully applied mm/damon/schemes: account how many times quota limit has exceeded mm/damon/reclaim: provide reclamation statistics Docs/admin-guide/mm/damon/reclaim: document statistics parameters mm/damon/dbgfs: support all DAMOS stats Docs/admin-guide/mm/damon/usage: update for schemes statistics
Baolin Wang <[email protected]>:
mm/damon: add access checking for hugetlb pages
Guoqing Jiang <[email protected]>:
mm/damon: move the implementation of damon_insert_region to damon.h
SeongJae Park <[email protected]>:
Patch series "mm/damon: Hide unnecessary information disclosures":
mm/damon/dbgfs: remove an unnecessary variable
mm/damon/vaddr: use pr_debug() for damon_va_three_regions() failure logging mm/damon/vaddr: hide kernel pointer from damon_va_three_regions() failure log mm/damon: hide kernel pointer from tracepoint event
Documentation/admin-guide/cgroup-v1/hugetlb.rst | 4 Documentation/admin-guide/cgroup-v2.rst | 11 Documentation/admin-guide/mm/damon/reclaim.rst | 25 Documentation/admin-guide/mm/damon/usage.rst | 235 +++++-- Documentation/admin-guide/mm/numa_memory_policy.rst | 16 Documentation/admin-guide/sysctl/vm.rst | 2 Documentation/filesystems/proc.rst | 6 Documentation/vm/arch_pgtable_helpers.rst | 20 Documentation/vm/index.rst | 2 Documentation/vm/page_migration.rst | 12 Documentation/vm/page_table_check.rst | 56 + Documentation/vm/vmalloced-kernel-stacks.rst | 153 ++++ MAINTAINERS | 9 arch/Kconfig | 3 arch/alpha/kernel/syscalls/syscall.tbl | 1 arch/alpha/mm/fault.c | 16 arch/arc/mm/fault.c | 3 arch/arm/mm/fault.c | 2 arch/arm/tools/syscall.tbl | 1 arch/arm64/include/asm/unistd.h | 2 arch/arm64/include/asm/unistd32.h | 2 arch/arm64/kernel/module.c | 4 arch/arm64/mm/fault.c | 6 arch/hexagon/mm/vm_fault.c | 8 arch/ia64/kernel/module.c | 6 arch/ia64/kernel/setup.c | 5 arch/ia64/kernel/syscalls/syscall.tbl | 1 arch/ia64/kernel/topology.c | 3 arch/ia64/kernel/uncached.c | 2 arch/ia64/mm/fault.c | 16 arch/m68k/kernel/syscalls/syscall.tbl | 1 arch/m68k/mm/fault.c | 18 arch/microblaze/kernel/syscalls/syscall.tbl | 1 arch/microblaze/mm/fault.c | 18 arch/mips/kernel/syscalls/syscall_n32.tbl | 1 arch/mips/kernel/syscalls/syscall_n64.tbl | 1 arch/mips/kernel/syscalls/syscall_o32.tbl | 1 arch/mips/mm/fault.c | 19 arch/nds32/mm/fault.c | 16 arch/nios2/mm/fault.c | 18 arch/openrisc/mm/fault.c | 18 arch/parisc/kernel/syscalls/syscall.tbl | 1 arch/parisc/mm/fault.c | 18 arch/powerpc/kernel/syscalls/syscall.tbl | 1 arch/powerpc/mm/fault.c | 6 arch/riscv/mm/fault.c | 2 arch/s390/kernel/module.c | 5 arch/s390/kernel/syscalls/syscall.tbl | 1 arch/s390/mm/fault.c | 28 arch/sh/kernel/syscalls/syscall.tbl | 1 arch/sh/mm/fault.c | 18 arch/sparc/kernel/syscalls/syscall.tbl | 1 arch/sparc/mm/fault_32.c | 16 arch/sparc/mm/fault_64.c | 16 arch/um/kernel/trap.c | 8 arch/x86/Kconfig | 1 arch/x86/entry/syscalls/syscall_32.tbl | 1 arch/x86/entry/syscalls/syscall_64.tbl | 1 arch/x86/include/asm/pgtable.h | 31 - arch/x86/kernel/module.c | 7 arch/x86/mm/fault.c | 3 arch/xtensa/kernel/syscalls/syscall.tbl | 1 arch/xtensa/mm/fault.c | 17 drivers/block/zram/zram_drv.c | 11 drivers/dax/bus.c | 32 + drivers/dax/bus.h | 1 drivers/dax/device.c | 140 ++-- drivers/infiniband/sw/siw/siw_main.c | 7 drivers/of/fdt.c | 6 fs/ext4/extents.c | 8 fs/ext4/inline.c | 5 fs/ext4/page-io.c | 9 fs/f2fs/data.c | 4 fs/f2fs/gc.c | 5 fs/f2fs/inode.c | 4 fs/f2fs/node.c | 4 fs/f2fs/recovery.c | 6 fs/f2fs/segment.c | 9 fs/f2fs/super.c | 5 fs/hugetlbfs/inode.c | 7 fs/inode.c | 49 + fs/ioctl.c | 2 fs/ntfs/attrib.c | 2 fs/ocfs2/alloc.c | 2 fs/ocfs2/aops.c | 26 fs/ocfs2/cluster/masklog.c | 11 fs/ocfs2/dir.c | 2 fs/ocfs2/filecheck.c | 3 fs/ocfs2/journal.c | 6 fs/proc/task_mmu.c | 13 fs/squashfs/super.c | 33 + fs/userfaultfd.c | 8 fs/xfs/kmem.c | 3 fs/xfs/xfs_buf.c | 2 include/linux/ceph/libceph.h | 1 include/linux/damon.h | 93 +-- include/linux/fs.h | 1 include/linux/gfp.h | 12 include/linux/hugetlb.h | 4 include/linux/hugetlb_cgroup.h | 7 include/linux/kasan.h | 4 include/linux/kthread.h | 25 include/linux/memcontrol.h | 22 include/linux/mempolicy.h | 1 include/linux/memremap.h | 11 include/linux/mm.h | 76 -- include/linux/mm_inline.h | 136 ++++ include/linux/mm_types.h | 252 +++----- include/linux/mmzone.h | 9 include/linux/page-flags.h | 6 include/linux/page_idle.h | 1 include/linux/page_table_check.h | 147 ++++ include/linux/pgtable.h | 8 include/linux/sched/mm.h | 26 include/linux/swap.h | 8 include/linux/syscalls.h | 3 include/linux/vm_event_item.h | 3 include/linux/vmalloc.h | 7 include/ras/ras_event.h | 2 include/trace/events/compaction.h | 24 include/trace/events/damon.h | 15 include/trace/events/thp.h | 35 - include/uapi/asm-generic/unistd.h | 5 include/uapi/linux/prctl.h | 3 kernel/dma/pool.c | 4 kernel/fork.c | 3 kernel/kthread.c | 1 kernel/rcu/rcutorture.c | 7 kernel/sys.c | 63 ++ kernel/sys_ni.c | 1 kernel/sysctl.c | 3 kernel/trace/ring_buffer.c | 7 kernel/trace/trace_hwlat.c | 6 kernel/trace/trace_osnoise.c | 3 lib/test_hmm.c | 24 lib/test_kasan.c | 30 mm/Kconfig | 14 mm/Kconfig.debug | 24 mm/Makefile | 1 mm/compaction.c | 7 mm/damon/core.c | 45 - mm/damon/dbgfs.c | 20 mm/damon/paddr.c | 24 mm/damon/prmtv-common.h | 4 mm/damon/reclaim.c | 46 + mm/damon/vaddr.c | 186 ++++-- mm/debug.c | 52 - mm/debug_vm_pgtable.c | 6 mm/dmapool.c | 2 mm/frontswap.c | 4 mm/gup.c | 31 - mm/hmm.c | 5 mm/huge_memory.c | 32 - mm/hugetlb.c | 6 mm/hugetlb_cgroup.c | 133 +++- mm/internal.h | 7 mm/kasan/quarantine.c | 11 mm/kasan/shadow.c | 9 mm/khugepaged.c | 23 mm/kmemleak.c | 21 mm/ksm.c | 5 mm/madvise.c | 510 ++++++++++------ mm/mapping_dirty_helpers.c | 1 mm/memcontrol.c | 44 - mm/memory-failure.c | 189 +++--- mm/memory.c | 12 mm/mempolicy.c | 95 ++- mm/memremap.c | 18 mm/migrate.c | 527 ++++++++++------- mm/mlock.c | 2 mm/mmap.c | 55 + mm/mmu_gather.c | 1 mm/mprotect.c | 2 mm/oom_kill.c | 30 mm/page_alloc.c | 198 ++++-- mm/page_counter.c | 1 mm/page_ext.c | 8 mm/page_isolation.c | 2 mm/page_owner.c | 4 mm/page_table_check.c | 270 ++++++++ mm/percpu-internal.h | 18 mm/percpu.c | 10 mm/pgtable-generic.c | 1 mm/rmap.c | 43 + mm/shmem.c | 91 ++ mm/slab.h | 5 mm/slab_common.c | 34 - mm/swap.c | 2 mm/swapfile.c | 46 - mm/truncate.c | 5 mm/userfaultfd.c | 5 mm/util.c | 15 mm/vmalloc.c | 75 +- mm/vmscan.c | 2 mm/vmstat.c | 3 mm/zpool.c | 12 net/ceph/buffer.c | 4 net/ceph/ceph_common.c | 27 net/ceph/crypto.c | 2 net/ceph/messenger.c | 2 net/ceph/messenger_v2.c | 2 net/ceph/osdmap.c | 12 net/sunrpc/svc_xprt.c | 3 scripts/spelling.txt | 1 tools/testing/selftests/vm/charge_reserved_hugetlb.sh | 34 - tools/testing/selftests/vm/hmm-tests.c | 42 + tools/testing/selftests/vm/hugepage-mremap.c | 46 - tools/testing/selftests/vm/hugetlb_reparenting_test.sh | 21 tools/testing/selftests/vm/run_vmtests.sh | 2 tools/testing/selftests/vm/userfaultfd.c | 33 - tools/testing/selftests/vm/write_hugetlb_memory.sh | 2 211 files changed, 3980 insertions(+), 1759 deletions(-)
Andrew Morton
maintainerPATCH 1/146 kthread: add the helper function kthread_run_on_cpu()
From: Cai Huoqing <[email protected]>
Subject: kthread: add the helper function kthread_run_on_cpu()
the helper function kthread_run_on_cpu() includes kthread_create_on_cpu/wake_up_process(). In some cases, use kthread_run_on_cpu() directly instead of kthread_create_on_node/kthread_bind/wake_up_process() or kthread_create_on_cpu/wake_up_process() or kthreadd_create/kthread_bind/wake_up_process() to simplify the code.
---
include/linux/kthread.h | 25 +++++++++++++++++++++++++ kernel/kthread.c | 1 + 2 files changed, 26 insertions(+)
include/linux/kthread.h renamed from include/linux/kthread.h~kthread-add-the-helper-function-kthread_run_on_cpu +25 -0
@@ -56,6 +56,31 @@ bool kthread_is_per_cpu(struct task_stru __k; \ }) +/** +* kthread_run_on_cpu - create and wake a cpu bound thread. +* @threadfn: the function to run until signal_pending(current). +* @data: data ptr for @threadfn. +* @cpu: The cpu on which the thread should be bound, +* @namefmt: printf-style name for the thread. Format is restricted +* to "name.*%u". Code fills in cpu number. +* +* Description: Convenient wrapper for kthread_create_on_cpu() +* followed by wake_up_process(). Returns the kthread or +* ERR_PTR(-ENOMEM). +*/ +static inline struct task_struct * +kthread_run_on_cpu (int (*threadfn)(void *data),void *data, +unsigned int cpu,const char *namefmt) +{ +struct task_struct *p; + + p =kthread_create_on_cpu (threadfn, data, cpu, namefmt); +if (!IS_ERR (p)) +wake_up_process (p); + +return p; +} +void free_kthread_struct (struct task_struct *k);void kthread_bind (struct task_struct *k,unsigned int cpu);void kthread_bind_mask (struct task_struct *k,const struct cpumask *mask);
kernel/kthread.c renamed from kernel/kthread.c~kthread-add-the-helper-function-kthread_run_on_cpu +1 -0
@@ -523,6 +523,7 @@ struct task_struct *kthread_create_on_cpto_kthread (p)->cpu = cpu;return p; } +EXPORT_SYMBOL (kthread_create_on_cpu);void kthread_set_per_cpu (struct task_struct *k,int cpu) {
_
Andrew Morton
maintainerPATCH 2/146 RDMA/siw: make use of the helper function kthread_run_on_cpu()
In reply to Andrew MortonFrom: Cai Huoqing <[email protected]>
Subject: RDMA/siw: make use of the helper function kthread_run_on_cpu()
Replace kthread_create/kthread_bind/wake_up_process() with kthread_run_on_cpu() to simplify the code.
---
drivers/infiniband/sw/siw/siw_main.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/infiniband/sw/siw/siw_main.c renamed from drivers/infiniband/sw/siw/siw_main.c~rdma-siw-make-use-of-the-helper-function-kthread_run_on_cpu +3 -4
@@ -98,15 +98,14 @@ static int siw_create_tx_threads(void)continue ; siw_tx_thread[cpu] = -kthread_create (siw_run_sq, (unsigned long *)(long )cpu, -"siw_tx/%d" , cpu); +kthread_run_on_cpu (siw_run_sq, + (unsigned long *)(long )cpu, + cpu,"siw_tx/%u" );if (IS_ERR (siw_tx_thread[cpu])) { siw_tx_thread[cpu] =NULL ;continue ; } -kthread_bind (siw_tx_thread[cpu], cpu); -wake_up_process (siw_tx_thread[cpu]); assigned++; }return assigned;
_
RE: [patch 002/146] RDMA/siw: make use of the helper function kthread_run_on_cpu()
Quoted 68 lines
-----Original Message-----
From: Andrew Morton <[email protected]>
Sent: Friday, 14 January 2022 23:03
To: [email protected]; Bernard Metzler <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; mm- [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXTERNAL] [patch 002/146] RDMA/siw: make use of the helper functionkthread_run_on_cpu()From: Cai Huoqing <[email protected]>
Subject: RDMA/siw: make use of the helper functionkthread_run_on_cpu()Replace kthread_create/kthread_bind/wake_up_process() with
kthread_run_on_cpu()to simplify the code.Link: INVALID URI REMOVED 3A__lkml.kernel.org_r_20211022025711.3673-2D3-2Dcaihuoqing- 40baidu.com&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T- r8ZO1PP1alNwU_QJcRRLfmYTAgd3QCvqSc&m=T1BlvlT0L- L5ZQY32ETtxzEwyFIAWcF0pGOthKHP_68&s=n4Cgq9aYj-341vdV-_8BZDAkCcFoRR- 7G1vpwblTuBE&e=
Signed-off-by Cai Huoqing <[email protected]>Cc Bernard Metzler <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Davidlohr Bueso <[email protected]>Cc Doug Ledford <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Joel Fernandes (Google) <[email protected]>Cc Josh Triplett <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc "Paul E . McKenney" <[email protected]>Cc Steven Rostedt <[email protected]>Signed-off-by Andrew Morton <[email protected]>---
drivers/infiniband/sw/siw/siw_main.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)--- a/drivers/infiniband/sw/siw/siw_main.c~rdma-siw-make-use-of-the- helper-function-kthread_run_on_cpu
+++a/drivers/infiniband/sw/siw/siw_main.cdrivers/infiniband/sw/siw/siw_main.c renamed from drivers/infiniband/sw/siw/siw_main.c~rdma-siw-make-use-of-the-helper-function-kthread_run_on_cpu +3
-4@@ -98,15 +98,14 @@ static int siw_create_tx_threads(void)continue ; siw_tx_thread[cpu] = -kthread_create (siw_run_sq, (unsigned long *)(long )cpu, -"siw_tx/%d" , cpu); +kthread_run_on_cpu (siw_run_sq, + (unsigned long *)(long )cpu, + cpu,"siw_tx/%u" );if (IS_ERR (siw_tx_thread[cpu])) { siw_tx_thread[cpu] =NULL ;continue ; } -kthread_bind (siw_tx_thread[cpu], cpu); -wake_up_process (siw_tx_thread[cpu]); assigned++; }return assigned;
Quoted 1 line
_
Thank you.
Andrew Morton
maintainerPATCH 3/146 ring-buffer: make use of the helper function kthread_run_on_cpu()
In reply to Andrew MortonFrom: Cai Huoqing <[email protected]>
Subject: ring-buffer: make use of the helper function kthread_run_on_cpu()
Replace kthread_create/kthread_bind/wake_up_process() with kthread_run_on_cpu() to simplify the code.
---
kernel/trace/ring_buffer.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
kernel/trace/ring_buffer.c renamed from kernel/trace/ring_buffer.c~ring-buffer-make-use-of-the-helper-function-kthread_run_on_cpu +2 -5
@@ -5898,16 +5898,13 @@ static __init int test_ringbuffer(void) rb_data[cpu].buffer = buffer; rb_data[cpu].cpu = cpu; rb_data[cpu].cnt = cpu; - rb_threads[cpu] =kthread_create (rb_test, &rb_data[cpu], -"rbtester/%d" , cpu); + rb_threads[cpu] =kthread_run_on_cpu (rb_test, &rb_data[cpu], + cpu,"rbtester/%u" );if (WARN_ON (IS_ERR (rb_threads[cpu]))) {pr_cont ("FAILED\n" ); ret =PTR_ERR (rb_threads[cpu]); goto out_free; } - -kthread_bind (rb_threads[cpu], cpu); -wake_up_process (rb_threads[cpu]); }/* Now create the rb hammer! */
_
Andrew Morton
maintainerPATCH 4/146 rcutorture: make use of the helper function kthread_run_on_cpu()
In reply to Andrew MortonFrom: Cai Huoqing <[email protected]>
Subject: rcutorture: make use of the helper function kthread_run_on_cpu()
Replace kthread_create_on_node/kthread_bind/wake_up_process() with kthread_run_on_cpu() to simplify the code.
---
kernel/rcu/rcutorture.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
kernel/rcu/rcutorture.c renamed from kernel/rcu/rcutorture.c~rcutorture-make-use-of-the-helper-function-kthread_run_on_cpu +2 -5
@@ -1992,9 +1992,8 @@ static int rcutorture_booster_init(unsigmutex_lock (&boost_mutex);rcu_torture_disable_rt_throttle ();VERBOSE_TOROUT_STRING ("Creating rcu_torture_boost task" ); - boost_tasks[cpu] =kthread_create_on_node (rcu_torture_boost,NULL , -cpu_to_node (cpu), -"rcu_torture_boost" ); + boost_tasks[cpu] =kthread_run_on_cpu (rcu_torture_boost,NULL , + cpu,"rcu_torture_boost_%u" );if (IS_ERR (boost_tasks[cpu])) { retval =PTR_ERR (boost_tasks[cpu]);VERBOSE_TOROUT_STRING ("rcu_torture_boost task create failed" );
@@ -2003,8 +2002,6 @@ static int rcutorture_booster_init(unsigmutex_unlock (&boost_mutex);return retval; } -kthread_bind (boost_tasks[cpu], cpu); -wake_up_process (boost_tasks[cpu]);mutex_unlock (&boost_mutex);return 0 ; }
_
Andrew Morton
maintainerPATCH 5/146 trace/osnoise: make use of the helper function kthread_run_on_cpu()
In reply to Andrew MortonFrom: Cai Huoqing <[email protected]>
Subject: trace/osnoise: make use of the helper function kthread_run_on_cpu()
Replace kthread_create_on_cpu/wake_up_process() with kthread_run_on_cpu() to simplify the code.
---
kernel/trace/trace_osnoise.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/trace/trace_osnoise.c renamed from kernel/trace/trace_osnoise.c~trace-osnoise-make-use-of-the-helper-function-kthread_run_on_cpu +1 -2
@@ -1701,7 +1701,7 @@ static int start_kthread(unsigned int cpsnprintf (comm,24 ,"osnoise/%d" , cpu); } - kthread =kthread_create_on_cpu (main,NULL , cpu, comm); + kthread =kthread_run_on_cpu (main,NULL , cpu, comm);if (IS_ERR (kthread)) {pr_err (BANNER "could not start sampling thread\n" );
@@ -1710,7 +1710,6 @@ static int start_kthread(unsigned int cp }per_cpu (per_cpu_osnoise_var, cpu).kthread = kthread; -wake_up_process (kthread);return 0 ; }
_
Andrew Morton
maintainerPATCH 6/146 trace/hwlat: make use of the helper function kthread_run_on_cpu()
In reply to Andrew MortonFrom: Cai Huoqing <[email protected]>
Subject: trace/hwlat: make use of the helper function kthread_run_on_cpu()
Replace kthread_create_on_cpu/wake_up_process() with kthread_run_on_cpu() to simplify the code.
---
kernel/trace/trace_hwlat.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
kernel/trace/trace_hwlat.c renamed from kernel/trace/trace_hwlat.c~trace-hwlat-make-use-of-the-helper-function-kthread_run_on_cpu +1 -5
@@ -491,18 +491,14 @@ static void stop_per_cpu_kthreads(void)static int start_cpu_kthread (unsigned int cpu) {struct task_struct *kthread; -char comm[24 ]; -snprintf (comm,24 ,"hwlatd/%d" , cpu); - - kthread =kthread_create_on_cpu (kthread_fn,NULL , cpu, comm); + kthread =kthread_run_on_cpu (kthread_fn,NULL , cpu,"hwlatd/%u" );if (IS_ERR (kthread)) {pr_err (BANNER "could not start sampling thread\n" );return -ENOMEM ; }per_cpu (hwlat_per_cpu_data, cpu).kthread = kthread; -wake_up_process (kthread);return 0 ; }
_
Andrew Morton
maintainerPATCH 7/146 ia64: module: use swap() to make code cleaner
In reply to Andrew MortonFrom: Yang Guang <[email protected]>
Subject: ia64: module: use swap() to make code cleaner
Use the macro 'swap()' defined in 'include/linux/minmax.h' to avoid opencoding it.
---
arch/ia64/kernel/module.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
arch/ia64/kernel/module.c renamed from arch/ia64/kernel/module.c~ia64-module-use-swap-to-make-code-cleaner +2 -4
@@ -848,7 +848,7 @@ register_unwind_table (struct module *mo {struct unw_table_entry *start = (void *) mod->arch.unwind->sh_addr;struct unw_table_entry *end = start + mod->arch.unwind->sh_size /sizeof (*start); -struct unw_table_entry tmp, *e1, *e2, *core, *init; +struct unw_table_entry *e1, *e2, *core, *init;unsigned long num_init =0 , num_core =0 ;/* First, count how many init and core unwind-table entries there are. */
@@ -865,9 +865,7 @@ register_unwind_table (struct module *mofor (e1 = start; e1 < end; ++e1) {for (e2 = e1 +1 ; e2 < end; ++e2) {if (e2->start_offset < e1->start_offset) { - tmp = *e1; - *e1 = *e2; - *e2 = tmp; +swap (*e1, *e2); } } }
_
Andrew Morton
maintainerPATCH 8/146 arch/ia64/kernel/setup.c: use swap() to make code cleaner
In reply to Andrew MortonFrom: Yang Guang <[email protected]>
Subject: arch/ia64/kernel/setup.c: use swap() to make code cleaner
Use the macro 'swap()' defined in 'include/linux/minmax.h' to avoid opencoding it.
---
arch/ia64/kernel/setup.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
arch/ia64/kernel/setup.c renamed from arch/ia64/kernel/setup.c~ia64-use-swap-to-make-code-cleaner +1 -4
@@ -208,10 +208,7 @@ sort_regions (struct rsvd_region *rsvd_rwhile (max--) {for (j =0 ; j < max; ++j) {if (rsvd_region[j].start > rsvd_region[j+1 ].start) { -struct rsvd_region tmp; - tmp = rsvd_region[j]; - rsvd_region[j] = rsvd_region[j +1 ]; - rsvd_region[j +1 ] = tmp; +swap (rsvd_region[j], rsvd_region[j +1 ]); } } }
_
Andrew Morton
maintainerPATCH 9/146 ia64: fix typo in a comment
In reply to Andrew MortonFrom: Jason Wang <[email protected]>
Subject: ia64: fix typo in a comment
The double `the' in a comment is repeated, thus it should be removed.
---
arch/ia64/kernel/uncached.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/ia64/kernel/uncached.c renamed from arch/ia64/kernel/uncached.c~ia64-fix-typo-in-a-comment +1 -1
@@ -171,7 +171,7 @@ failed:* @n_pages: number of contiguous pages to allocate * * Allocate the specified number of contiguous uncached pages on the -* the requested node. If not enough contiguous uncached pages are available +* requested node. If not enough contiguous uncached pages are available * on the requested node, roundrobin starting with the next higher node. */ unsigned long uncached_alloc_page (int starting_nid,int n_pages)
_
Andrew Morton
maintainerPATCH 10/146 ia64: topology: use default_groups in kobj_type
In reply to Andrew MortonFrom: Greg Kroah-Hartman <[email protected]>
Subject: ia64: topology: use default_groups in kobj_type
There are currently 2 ways to create a set of sysfs files for a kobj_type, through the default_attrs field, and the default_groups field. Move the ia64 topology sysfs code to use default_groups field which has been the preferred way since aa30f47cf666 ("kobject: Add support for default attribute groups to kobj_type") so that we can soon get rid of the obsolete default_attrs field.
---
arch/ia64/kernel/topology.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/ia64/kernel/topology.c renamed from arch/ia64/kernel/topology.c~ia64-topology-use-default_groups-in-kobj_type +2 -1
@@ -264,6 +264,7 @@ static struct attribute * cache_default_ &shared_cpu_map.attr, NULL }; +ATTRIBUTE_GROUPS (cache_default);#define to_object (k) container_of(k, struct cache_info, kobj)#define to_attr (a) container_of(a, struct cache_attr, attr)
@@ -284,7 +285,7 @@ static const struct sysfs_ops cache_sysfstatic struct kobj_type cache_ktype = { .sysfs_ops = &cache_sysfs_ops, - .default_attrs = cache_default_attrs, + .default_groups = cache_default_groups, };static struct kobj_type cache_ktype_percpu_entry = {
_
Andrew Morton
maintainerPATCH 11/146 scripts/spelling.txt: add "oveflow"
In reply to Andrew MortonFrom: Drew Fustini <[email protected]>
Subject: scripts/spelling.txt: add "oveflow"
Add typo "oveflow" for "overflow". This typo was found and fixed in tools/testing/selftests/bpf/prog_tests/btf_dump.c
---
scripts/spelling.txt | 1 + 1 file changed, 1 insertion(+)
scripts/spelling.txt renamed from scripts/spelling.txt~scripts-spellingtxt-add-oveflow +1 -0
@@ -1046,6 +1046,7 @@ oustanding||outstanding overaall||overall overhread||overhead overlaping||overlapping +oveflow||overflow overflw||overflow overlfow||overflow overide||override
_
Andrew Morton
maintainerPATCH 12/146 fs/ntfs/attrib.c: fix one kernel-doc comment
In reply to Andrew MortonFrom: Yang Li <[email protected]>
Subject: fs/ntfs/attrib.c: fix one kernel-doc comment
/**
* attrib.c - NTFS attribute operations. Part of the Linux-NTFS
The comments for the file should not be in kernel-doc format, which causes it to be incorrectly identified for function ntfs_map_runlist_nolock(), causing some warnings found by running scripts/kernel-doc.
fs/ntfs/attrib.c:25: warning: Incorrect use of kernel-doc format: * ntfs_map_runlist_nolock - map (a part of) a runlist of an ntfs inode fs/ntfs/attrib.c:71: warning: Function parameter or member 'ni' not described in 'ntfs_map_runlist_nolock' fs/ntfs/attrib.c:71: warning: Function parameter or member 'vcn' not described in 'ntfs_map_runlist_nolock' fs/ntfs/attrib.c:71: warning: Function parameter or member 'ctx' not described in 'ntfs_map_runlist_nolock' fs/ntfs/attrib.c:71: warning: expecting prototype for attrib.c - NTFS attribute operations. Part of the Linux(). Prototype was for ntfs_map_runlist_nolock() instead
---
fs/ntfs/attrib.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ntfs/attrib.c renamed from fs/ntfs/attrib.c~ntfs-fix-one-kernel-doc-comment +1 -1
@@ -1,5 +1,5 @@// SPDX-License-Identifier: GPL-2.0-or-later -/** +/* * attrib.c - NTFS attribute operations. Part of the Linux-NTFS project. * * Copyright (c) 2001-2012 Anton Altaparmakov and Tuxera Inc.
_
Andrew Morton
maintainerPATCH 13/146 squashfs: provide backing_dev_info in order to disable read-ahead
In reply to Andrew MortonFrom: Zheng Liang <[email protected]>
Subject: squashfs: provide backing_dev_info in order to disable read-ahead
Commit c1f6925e1091 ("mm: put readahead pages in cache earlier") causes the read performance of squashfs to deteriorate.Through testing, we find that the performance will be back by closing the readahead of squashfs. So we want to learn the way of ubifs, provides backing_dev_info and disable read-ahead.
--------------------------------------------------------------------
We tested the following data by fio.
squashfs image blocksize=128K
test command:
fio --name basic --bs=? --filename="/mnt/test_file" --rw=? --iodepth=1 --ioengine=psync --runtime=200 --time_based
turn on squashfs readahead in 5.10 kernel
bs(k) read/randread MB/s 4 randread 271 128 randread 231 1024 randread 246 4 read 310 128 read 245 1024 read 247
turn off squashfs readahead in 5.10 kernel
bs(k) read/randread MB/s 4 randread 293 128 randread 330 1024 randread 363 4 read 338 128 read 360 1024 read 365
turn on squashfs readahead and revert the
commit c1f6925e1091("mm: put readahead
pages in cache earlier") in 5.10 kernel
bs(k) read/randread MB/s 4 randread 289 128 randread 306 1024 randread 335 4 read 337 128 read 336 1024 read 338
---
fs/squashfs/super.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
fs/squashfs/super.c renamed from fs/squashfs/super.c~squashfs-provides-backing_dev_info-in-order-to-disable-read-ahead +33 -0
@@ -29,6 +29,7 @@#include <linux/module.h> #include <linux/magic.h> #include <linux/xattr.h> +#include <linux/backing-dev.h> #include "squashfs_fs.h" #include "squashfs_fs_sb.h"
@@ -112,6 +113,24 @@ static const struct squashfs_decompressoreturn decompressor; } +static int squashfs_bdi_init (struct super_block *sb) +{ +int err; +unsigned int major =MAJOR (sb->s_dev); +unsigned int minor =MINOR (sb->s_dev); + +bdi_put (sb->s_bdi); + sb->s_bdi = &noop_backing_dev_info; + + err =super_setup_bdi_name (sb,"squashfs_%u_%u" , major, minor); +if (err) +return err; + + sb->s_bdi->ra_pages =0 ; + sb->s_bdi->io_pages =0 ; + +return 0 ; +}static int squashfs_fill_super (struct super_block *sb,struct fs_context *fc) {
@@ -127,6 +146,20 @@ static int squashfs_fill_super(struct suTRACE ("Entered squashfs_fill_superblock\n" ); +/* +* squashfs provides 'backing_dev_info' in order to disable read-ahead. For +* squashfs, I/O is not deferred, it is done immediately in readpage, +* which means the user would always have to wait their own I/O. So the effect +* of readahead is very weak for squashfs. squashfs_bdi_init will set +* sb->s_bdi->ra_pages and sb->s_bdi->io_pages to 0 and close readahead for +* squashfs. +*/ + err =squashfs_bdi_init (sb); +if (err) { +errorf (fc,"squashfs init bdi failed" ); +return err; + } + sb->s_fs_info =kzalloc (sizeof (*msblk),GFP_KERNEL );if (sb->s_fs_info ==NULL ) {ERROR ("Failed to allocate squashfs_sb_info\n" );
_
Andrew Morton
maintainerPATCH 14/146 ocfs2: use BUG_ON instead of if condition followed by BUG.
In reply to Andrew MortonFrom: Zhang Mingyu <[email protected]>
Subject: ocfs2: use BUG_ON instead of if condition followed by BUG.
This issue was detected with the help of Coccinelle.
---
fs/ocfs2/journal.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
fs/ocfs2/journal.c renamed from fs/ocfs2/journal.c~ocfs2-use-bug_on-instead-of-if-condition-followed-by-bug +2 -4
@@ -1669,8 +1669,7 @@ static int ocfs2_replay_journal(struct o status =jbd2_journal_load (journal);if (status <0 ) {mlog_errno (status); -if (!igrab (inode)) -BUG (); +BUG_ON (!igrab (inode));jbd2_journal_destroy (journal); goto done; }
@@ -1699,8 +1698,7 @@ static int ocfs2_replay_journal(struct oif (status <0 )mlog_errno (status); -if (!igrab (inode)) -BUG (); +BUG_ON (!igrab (inode));jbd2_journal_destroy (journal);
_
Andrew Morton
maintainerPATCH 15/146 ocfs2: clearly handle ocfs2_grab_pages_for_write() return value
In reply to Andrew MortonFrom: Joseph Qi <[email protected]>
Subject: ocfs2: clearly handle ocfs2_grab_pages_for_write() return value
ocfs2_grab_pages_for_write() may return -EAGAIN if write context type is mmap and it could not lock the target page. In this case, we exit with no error and no target page. And then trigger the caller page_mkwrite() to retry.
Since there are other caller types, e.g. buffer and direct io, make the return value handling more clear.
---
fs/ocfs2/aops.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
fs/ocfs2/aops.c renamed from fs/ocfs2/aops.c~ocfs2-clearly-handle-ocfs2_grab_pages_for_write-return-value +13 -13
@@ -1799,20 +1799,20 @@ try_again:*/ ret =ocfs2_grab_pages_for_write (mapping, wc, wc->w_cpos, pos, len, cluster_of_pages, mmap_page); -if (ret && ret != -EAGAIN ) { -mlog_errno (ret); - goto out_quota; - } +if (ret) { +/* +* ocfs2_grab_pages_for_write() returns -EAGAIN if it could not lock +* the target page. In this case, we exit with no error and no target +* page. This will trigger the caller, page_mkwrite(), to re-try +* the operation. +*/ +if (type ==OCFS2_WRITE_MMAP && ret == -EAGAIN ) { +BUG_ON (wc->w_target_page); + ret =0 ; + goto out_quota; + } -/* -* ocfs2_grab_pages_for_write() returns -EAGAIN if it could not lock -* the target page. In this case, we exit with no error and no target -* page. This will trigger the caller, page_mkwrite(), to re-try -* the operation. -*/ -if (ret == -EAGAIN ) { -BUG_ON (wc->w_target_page); - ret =0 ; +mlog_errno (ret); goto out_quota; }
_
Andrew Morton
maintainerPATCH 16/146 ocfs2: use default_groups in kobj_type
In reply to Andrew MortonFrom: Greg Kroah-Hartman <[email protected]>
Subject: ocfs2: use default_groups in kobj_type
There are currently 2 ways to create a set of sysfs files for a kobj_type, through the default_attrs field, and the default_groups field. Move the ocfs2 code to use default_groups field which has been the preferred way since aa30f47cf666 ("kobject: Add support for default attribute groups to kobj_type") so that we can soon get rid of the obsolete default_attrs field.
---
fs/ocfs2/filecheck.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/ocfs2/filecheck.c renamed from fs/ocfs2/filecheck.c~ocfs2-use-default_groups-in-kobj_type +2 -1
@@ -94,6 +94,7 @@ static struct attribute *ocfs2_filecheck &ocfs2_filecheck_attr_set.attr, NULL }; +ATTRIBUTE_GROUPS (ocfs2_filecheck);static void ocfs2_filecheck_release (struct kobject *kobj) {
@@ -138,7 +139,7 @@ static const struct sysfs_ops ocfs2_file };static struct kobj_type ocfs2_ktype_filecheck = { - .default_attrs = ocfs2_filecheck_attrs, + .default_groups = ocfs2_filecheck_groups, .sysfs_ops = &ocfs2_filecheck_ops, .release = ocfs2_filecheck_release, };
_
Andrew Morton
maintainerPATCH 17/146 ocfs2: remove redundant assignment to pointer root_bh
In reply to Andrew MortonFrom: Colin Ian King <[email protected]>
Subject: ocfs2: remove redundant assignment to pointer root_bh
Pointer root_bh is being initialized with a value that is not read, it is being re-assigned later on closer to its use. The early initialization is redundant and can be removed.
---
fs/ocfs2/alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ocfs2/alloc.c renamed from fs/ocfs2/alloc.c~ocfs2-remove-redundant-assignment-to-pointer-root_bh +1 -1
@@ -2040,7 +2040,7 @@ static void ocfs2_complete_edge_insert(hint i, idx;struct ocfs2_extent_list *el, *left_el, *right_el;struct ocfs2_extent_rec *left_rec, *right_rec; -struct buffer_head *root_bh = left_path->p_node[subtree_index].bh; +struct buffer_head *root_bh;/* * Update the counts and position values within all the
_
Andrew Morton
maintainerPATCH 18/146 ocfs2: cluster: use default_groups in kobj_type
In reply to Andrew MortonFrom: Greg Kroah-Hartman <[email protected]>
Subject: ocfs2: cluster: use default_groups in kobj_type
There are currently 2 ways to create a set of sysfs files for a kobj_type, through the default_attrs field, and the default_groups field. Move the ocfs2 cluster sysfs code to use default_groups field which has been the preferred way since aa30f47cf666 ("kobject: Add support for default attribute groups to kobj_type") so that we can soon get rid of the obsolete default_attrs field.
---
fs/ocfs2/cluster/masklog.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
fs/ocfs2/cluster/masklog.c renamed from fs/ocfs2/cluster/masklog.c~ocfs2-cluster-use-default_groups-in-kobj_type +6 -5
@@ -120,7 +120,8 @@ static struct mlog_attribute mlog_attrs[ define_mask(KTHREAD), }; -static struct attribute *mlog_attr_ptrs[MLOG_MAX_BITS ] = {NULL , }; +static struct attribute *mlog_default_attrs[MLOG_MAX_BITS ] = {NULL , }; +ATTRIBUTE_GROUPS (mlog_default);static ssize_t mlog_show (struct kobject *obj,struct attribute *attr,char *buf)
@@ -144,8 +145,8 @@ static const struct sysfs_ops mlog_attr_ };static struct kobj_type mlog_ktype = { - .default_attrs = mlog_attr_ptrs, - .sysfs_ops = &mlog_attr_ops, + .default_groups = mlog_default_groups, + .sysfs_ops = &mlog_attr_ops, };static struct kset mlog_kset = {
@@ -157,10 +158,10 @@ int mlog_sys_init(struct kset *o2cb_ksetint i =0 ;while (mlog_attrs[i].attr.mode) { - mlog_attr_ptrs[i] = &mlog_attrs[i].attr; + mlog_default_attrs[i] = &mlog_attrs[i].attr; i++; } - mlog_attr_ptrs[i] =NULL ; + mlog_default_attrs[i] =NULL ;kobject_set_name (&mlog_kset.kobj,"logmask" ); mlog_kset.kobj.kset = o2cb_kset;
_
Andrew Morton
maintainerPATCH 19/146 ocfs2: remove redundant assignment to variable free_space
In reply to Andrew MortonFrom: Colin Ian King <[email protected]>
Subject: ocfs2: remove redundant assignment to variable free_space
Variable free_space is being initialized with a value that is not read, it is being re-assigned later in the two paths of an if statement. The early initialization is redundant and can be removed.
---
fs/ocfs2/dir.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ocfs2/dir.c renamed from fs/ocfs2/dir.c~ocfs2-remove-redundant-assignment-to-variable-free_space +1 -1
@@ -3343,7 +3343,7 @@ static int ocfs2_find_dir_space_id(strucstruct ocfs2_dir_entry *de, *last_de =NULL ;char *de_buf, *limit;unsigned long offset =0 ; -unsigned int rec_len, new_rec_len, free_space = dir->i_sb->s_blocksize; +unsigned int rec_len, new_rec_len, free_space;/* * This calculates how many free bytes we'd have in block zero, should
_
Andrew Morton
maintainerPATCH 20/146 fs/ioctl: remove unnecessary __user annotation
In reply to Andrew MortonFrom: Amit Daniel Kachhap <[email protected]>
Subject: fs/ioctl: remove unnecessary __user annotation
__user annotations are used by the checker (e.g sparse) to mark user pointers. However here __user is applied to a struct directly, without a pointer being directly involved.
Although the presence of __user does not cause sparse to emit a warning, __user should be removed for consistency with other uses of offsetof().
Note: No functional changes intended.
---
fs/ioctl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ioctl.c renamed from fs/ioctl.c~fs-ioctl-remove-unnecessary-__user-annotation +1 -1
@@ -430,7 +430,7 @@ static int ioctl_file_dedupe_range(struc goto out; } - size = offsetof(struct file_dedupe_range __user, info[count]); + size = offsetof(struct file_dedupe_range, info[count]);if (size >PAGE_SIZE ) { ret = -ENOMEM ; goto out;
_
Andrew Morton
maintainerPATCH 21/146 mm/slab_common: use WARN() if cache still has objects on destroy
In reply to Andrew MortonFrom: Marco Elver <[email protected]>
Subject: mm/slab_common: use WARN() if cache still has objects on destroy
Calling kmem_cache_destroy() while the cache still has objects allocated is a kernel bug, and will usually result in the entire cache being leaked. While the message in kmem_cache_destroy() resembles a warning, it is currently not implemented using a real WARN().
This is problematic for infrastructure testing the kernel, all of which rely on the specific format of WARN()s to pick up on bugs.
Some 13 years ago this used to be a simple WARN_ON() in slub, but d629d8195793 ("slub: improve kmem_cache_destroy() error message") changed it into an open-coded warning to avoid confusion with a bug in slub itself.
Instead, turn the open-coded warning into a real WARN() with the message preserved, so that test systems can actually identify these issues, and we get all the other benefits of using a normal WARN(). The warning message is extended with "when called from <caller-ip>" to make it even clearer where the fault lies.
For most configurations this is only a cosmetic change, however, note that WARN() here will now also respect panic_on_warn.
---
mm/slab_common.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
mm/slab_common.c renamed from mm/slab_common.c~mm-slab_common-use-warn-if-cache-still-has-objects-on-destroy +3 -8
@@ -489,8 +489,6 @@ void slab_kmem_cache_release(struct kmemvoid kmem_cache_destroy (struct kmem_cache *s) { -int err; -if (unlikely (!s))return ;
@@ -501,12 +499,9 @@ void kmem_cache_destroy(struct kmem_cachif (s->refcount) goto out_unlock; - err =shutdown_cache (s); -if (err) { -pr_err ("%s %s: Slab cache still has objects\n" , - __func__, s->name); -dump_stack (); - } +WARN (shutdown_cache (s), +"%s %s: Slab cache still has objects when called from %pS" , + __func__, s->name, (void *)_RET_IP_); out_unlock:mutex_unlock (&slab_mutex);cpus_read_unlock ();
_
Andrew Morton
maintainerPATCH 22/146 mm: slab: make slab iterator functions static
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: slab: make slab iterator functions static
There is no external users of slab_start/next/stop(), so make them static. And the memory.kmem.slabinfo is deprecated, which outputs nothing now, so move memcg_slab_show() into mm/memcontrol.c and rename it to mem_cgroup_slab_show to be consistent with other function names.
---
mm/memcontrol.c | 13 ++++++++++++- mm/slab.h | 5 ----- mm/slab_common.c | 17 +++-------------- 3 files changed, 15 insertions(+), 20 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-slab-make-slab-iterator-functions-static +12 -1
@@ -4845,6 +4845,17 @@ out_kfree:return ret; } +#if defined(CONFIG_MEMCG_KMEM ) && (defined(CONFIG_SLAB ) || defined(CONFIG_SLUB_DEBUG )) +static int mem_cgroup_slab_show (struct seq_file *m,void *p) +{ +/* +* Deprecated. +* Please, take a look at tools/cgroup/slabinfo.py . +*/ +return 0 ; +} +#endif +static struct cftype mem_cgroup_legacy_files[] = { { .name ="usage_in_bytes" ,
@@ -4945,7 +4956,7 @@ static struct cftype mem_cgroup_legacy_f (defined(CONFIG_SLAB) || defined(CONFIG_SLUB_DEBUG)) { .name = "kmem.slabinfo", - .seq_show = memcg_slab_show, + .seq_show = mem_cgroup_slab_show, },#endif {
mm/slab_common.c renamed from mm/slab_common.c~mm-slab-make-slab-iterator-functions-static +3 -14
@@ -1039,18 +1039,18 @@ static void print_slabinfo_header(structseq_putc (m,'\n' ); } -void *slab_start (struct seq_file *m,loff_t *pos) +static void *slab_start (struct seq_file *m,loff_t *pos) {mutex_lock (&slab_mutex);return seq_list_start (&slab_caches, *pos); } -void *slab_next (struct seq_file *m,void *p,loff_t *pos) +static void *slab_next (struct seq_file *m,void *p,loff_t *pos) {return seq_list_next (p, &slab_caches, pos); } -void slab_stop (struct seq_file *m,void *p) +static void slab_stop (struct seq_file *m,void *p) {mutex_unlock (&slab_mutex); }
@@ -1118,17 +1118,6 @@ void dump_unreclaimable_slab(void)mutex_unlock (&slab_mutex); } -#if defined(CONFIG_MEMCG_KMEM ) -int memcg_slab_show (struct seq_file *m,void *p) -{ -/* -* Deprecated. -* Please, take a look at tools/cgroup/slabinfo.py . -*/ -return 0 ; -} -#endif -/* * slabinfo_op - iterator that generates /proc/slabinfo *
mm/slab.h renamed from mm/slab.h~mm-slab-make-slab-iterator-functions-static +0 -5
@@ -575,11 +575,6 @@ static inline struct kmem_cache_node *ge#endif -void *slab_start (struct seq_file *m,loff_t *pos); -void *slab_next (struct seq_file *m,void *p,loff_t *pos); -void slab_stop (struct seq_file *m,void *p); -int memcg_slab_show (struct seq_file *m,void *p); -#if defined(CONFIG_SLAB ) || defined(CONFIG_SLUB_DEBUG )void dump_unreclaimable_slab (void );#else
_
Andrew Morton
maintainerPATCH 23/146 kmemleak: fix kmemleak false positive report with HW tag-based kasan enable
In reply to Andrew MortonFrom: Kuan-Ying Lee <[email protected]>
Subject: kmemleak: fix kmemleak false positive report with HW tag-based kasan enable
With HW tag-based kasan enable, We will get the warning when we free object whose address starts with 0xFF.
It is because kmemleak rbtree stores tagged object and this freeing object's tag does not match with rbtree object.
In the example below, kmemleak rbtree stores the tagged object in the kmalloc(), and kfree() gets the pointer with 0xFF tag.
Call sequence:
ptr = kmalloc(size, GFP_KERNEL);
page = virt_to_page(ptr);
offset = offset_in_page(ptr);kfree(page_address(page) + offset);
ptr = kmalloc(size, GFP_KERNEL);
Call sequence like that may cause the warning as following:
1) Freeing unknown object:
In kfree(), we will get free unknown object warning in kmemleak_free(). Because object(0xFx) in kmemleak rbtree and pointer(0xFF) in kfree() have different tag.
2) Overlap existing:
When we allocate that object with the same hw-tag again, we will find the overlap in the kmemleak rbtree and kmemleak thread will be killed.
[ 116.685312] kmemleak: Freeing unknown object at 0xffff000003f88000 [ 116.686422] CPU: 5 PID: 177 Comm: cat Not tainted 5.16.0-rc1-dirty #21 [ 116.687067] Hardware name: linux,dummy-virt (DT) [ 116.687496] Call trace: [ 116.687792] dump_backtrace+0x0/0x1ac [ 116.688255] show_stack+0x1c/0x30 [ 116.688663] dump_stack_lvl+0x68/0x84 [ 116.689096] dump_stack+0x1c/0x38 [ 116.689499] kmemleak_free+0x6c/0x70 [ 116.689919] slab_free_freelist_hook+0x104/0x200 [ 116.690420] kmem_cache_free+0xa8/0x3d4 [ 116.690845] test_version_show+0x270/0x3a0 [ 116.691344] module_attr_show+0x28/0x40 [ 116.691789] sysfs_kf_seq_show+0xb0/0x130 [ 116.692245] kernfs_seq_show+0x30/0x40 [ 116.692678] seq_read_iter+0x1bc/0x4b0 [ 116.692678] seq_read_iter+0x1bc/0x4b0 [ 116.693114] kernfs_fop_read_iter+0x144/0x1c0 [ 116.693586] generic_file_splice_read+0xd0/0x184 [ 116.694078] do_splice_to+0x90/0xe0 [ 116.694498] splice_direct_to_actor+0xb8/0x250 [ 116.694975] do_splice_direct+0x88/0xd4 [ 116.695409] do_sendfile+0x2b0/0x344 [ 116.695829] __arm64_sys_sendfile64+0x164/0x16c [ 116.696306] invoke_syscall+0x48/0x114 [ 116.696735] el0_svc_common.constprop.0+0x44/0xec [ 116.697263] do_el0_svc+0x74/0x90 [ 116.697665] el0_svc+0x20/0x80 [ 116.698261] el0t_64_sync_handler+0x1a8/0x1b0 [ 116.698695] el0t_64_sync+0x1ac/0x1b0 ... [ 117.520301] kmemleak: Cannot insert 0xf2ff000003f88000 into the object search tree (overlaps existing) [ 117.521118] CPU: 5 PID: 178 Comm: cat Not tainted 5.16.0-rc1-dirty #21 [ 117.521827] Hardware name: linux,dummy-virt (DT) [ 117.522287] Call trace: [ 117.522586] dump_backtrace+0x0/0x1ac [ 117.523053] show_stack+0x1c/0x30 [ 117.523578] dump_stack_lvl+0x68/0x84 [ 117.524039] dump_stack+0x1c/0x38 [ 117.524472] create_object.isra.0+0x2d8/0x2fc [ 117.524975] kmemleak_alloc+0x34/0x40 [ 117.525416] kmem_cache_alloc+0x23c/0x2f0 [ 117.525914] test_version_show+0x1fc/0x3a0 [ 117.526379] module_attr_show+0x28/0x40 [ 117.526827] sysfs_kf_seq_show+0xb0/0x130 [ 117.527363] kernfs_seq_show+0x30/0x40 [ 117.527848] seq_read_iter+0x1bc/0x4b0 [ 117.528320] kernfs_fop_read_iter+0x144/0x1c0 [ 117.528809] generic_file_splice_read+0xd0/0x184 [ 117.529316] do_splice_to+0x90/0xe0 [ 117.529734] splice_direct_to_actor+0xb8/0x250 [ 117.530227] do_splice_direct+0x88/0xd4 [ 117.530686] do_sendfile+0x2b0/0x344 [ 117.531154] __arm64_sys_sendfile64+0x164/0x16c [ 117.531673] invoke_syscall+0x48/0x114 [ 117.532111] el0_svc_common.constprop.0+0x44/0xec [ 117.532621] do_el0_svc+0x74/0x90 [ 117.533048] el0_svc+0x20/0x80 [ 117.533461] el0t_64_sync_handler+0x1a8/0x1b0 [ 117.533950] el0t_64_sync+0x1ac/0x1b0 [ 117.534625] kmemleak: Kernel memory leak detector disabled [ 117.535201] kmemleak: Object 0xf2ff000003f88000 (size 128): [ 117.535761] kmemleak: comm "cat", pid 177, jiffies 4294921177 [ 117.536339] kmemleak: min_count = 1 [ 117.536718] kmemleak: count = 0 [ 117.537068] kmemleak: flags = 0x1 [ 117.537429] kmemleak: checksum = 0 [ 117.537806] kmemleak: backtrace: [ 117.538211] kmem_cache_alloc+0x23c/0x2f0 [ 117.538924] test_version_show+0x1fc/0x3a0 [ 117.539393] module_attr_show+0x28/0x40 [ 117.539844] sysfs_kf_seq_show+0xb0/0x130 [ 117.540304] kernfs_seq_show+0x30/0x40 [ 117.540750] seq_read_iter+0x1bc/0x4b0 [ 117.541206] kernfs_fop_read_iter+0x144/0x1c0 [ 117.541687] generic_file_splice_read+0xd0/0x184 [ 117.542182] do_splice_to+0x90/0xe0 [ 117.542611] splice_direct_to_actor+0xb8/0x250 [ 117.543097] do_splice_direct+0x88/0xd4 [ 117.543544] do_sendfile+0x2b0/0x344 [ 117.543983] __arm64_sys_sendfile64+0x164/0x16c [ 117.544471] invoke_syscall+0x48/0x114 [ 117.544917] el0_svc_common.constprop.0+0x44/0xec [ 117.545416] do_el0_svc+0x74/0x90 [ 117.554100] kmemleak: Automatic memory scanning thread ended
---
mm/kmemleak.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)
mm/kmemleak.c renamed from mm/kmemleak.c~kmemleak-fix-kmemleak-false-positive-report-with-hw-tag-based-kasan-enable +14 -7
@@ -381,15 +381,20 @@ static void dump_object_info(struct kmemstatic struct kmemleak_object *lookup_object (unsigned long ptr,int alias) {struct rb_node *rb = object_tree_root.rb_node; +unsigned long untagged_ptr = (unsigned long )kasan_reset_tag ((void *)ptr);while (rb) { -struct kmemleak_object *object = - rb_entry(rb, struct kmemleak_object, rb_node); -if (ptr < object->pointer) +struct kmemleak_object *object; +unsigned long untagged_objp; + + object = rb_entry(rb, struct kmemleak_object, rb_node); + untagged_objp = (unsigned long )kasan_reset_tag ((void *)object->pointer); + +if (untagged_ptr < untagged_objp) rb = object->rb_node.rb_left; -else if (object->pointer + object->size <= ptr) +else if (untagged_objp + object->size <= untagged_ptr) rb = object->rb_node.rb_right; -else if (object->pointer == ptr || alias) +else if (untagged_objp == untagged_ptr || alias)return object;else { kmemleak_warn("Found object by alias at 0x%08lx\n",
@@ -576,6 +581,7 @@ static struct kmemleak_object *create_obstruct kmemleak_object *object, *parent;struct rb_node **link, *rb_parent;unsigned long untagged_ptr; +unsigned long untagged_objp; object =mem_pool_alloc (gfp);if (!object) {
@@ -629,9 +635,10 @@ static struct kmemleak_object *create_obwhile (*link) { rb_parent = *link; parent = rb_entry(rb_parent, struct kmemleak_object, rb_node); -if (ptr + size <= parent->pointer) + untagged_objp = (unsigned long )kasan_reset_tag ((void *)parent->pointer); +if (untagged_ptr + size <= untagged_objp) link = &parent->rb_node.rb_left; -else if (parent->pointer + parent->size <= ptr) +else if (untagged_objp + parent->size <= untagged_ptr) link = &parent->rb_node.rb_right;else { kmemleak_stop("Cannot insert 0x%lx into the object search tree (overlaps existing)\n",
_
Andrew Morton
maintainerPATCH 24/146 mm: kmemleak: alloc gray object for reserved region with direct map
In reply to Andrew MortonFrom: Calvin Zhang <[email protected]>
Subject: mm: kmemleak: alloc gray object for reserved region with direct map
Reserved regions with direct mapping may contain references to other regions. CMA region with fixed location is reserved without creating kmemleak_object for it.
So add them as gray kmemleak objects.
---
drivers/of/fdt.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/of/fdt.c renamed from drivers/of/fdt.c~mm-kmemleak-alloc-gray-object-for-reserved-region-with-direct-map +5 -1
@@ -26,6 +26,7 @@#include <linux/serial_core.h> #include <linux/sysfs.h> #include <linux/random.h> +#include <linux/kmemleak.h> #include <asm/setup.h> /* for COMMAND_LINE_SIZE */ #include <asm/page.h>
@@ -522,9 +523,12 @@ static int __init __reserved_mem_reserve size =dt_mem_next_cell (dt_root_size_cells, &prop);if (size && -early_init_dt_reserve_memory_arch (base, size, nomap) ==0 ) +early_init_dt_reserve_memory_arch (base, size, nomap) ==0 ) {pr_debug ("Reserved memory: reserved region for node '%s': base %pa, size %lu MiB\n" , uname, &base, (unsigned long )(size /SZ_1M )); +if (!nomap) +kmemleak_alloc_phys (base, size,0 ,0 ); + }else pr_info ("Reserved memory: failed to reserve memory for node '%s': base %pa, size %lu MiB\n" , uname, &base, (unsigned long )(size /SZ_1M ));
_
Andrew Morton
maintainerPATCH 25/146 mm: defer kmemleak object creation of module_alloc()
In reply to Andrew MortonFrom: Kefeng Wang <[email protected]>
Subject: mm: defer kmemleak object creation of module_alloc()
Yongqiang reports a kmemleak panic when module insmod/rmmod with KASAN enabled(without KASAN_VMALLOC) on x86[1].
When the module area allocates memory, it's kmemleak_object is created successfully, but the KASAN shadow memory of module allocation is not ready, so when kmemleak scan the module's pointer, it will panic due to no shadow memory with KASAN check.
module_alloc
__vmalloc_node_range
kmemleak_vmalloc
kmemleak_scan
update_checksum
kasan_module_alloc
kmemleak_ignore
Note, there is no problem if KASAN_VMALLOC enabled, the modules area entire shadow memory is preallocated. Thus, the bug only exits on ARCH which supports dynamic allocation of module area per module load, for now, only x86/arm64/s390 are involved.
Add a VM_DEFER_KMEMLEAK flags, defer vmalloc'ed object register of kmemleak in module_alloc() to fix this issue.
[1] https://lore.kernel.org/all/[email protected]/
---
arch/arm64/kernel/module.c | 4 ++-- arch/s390/kernel/module.c | 5 +++-- arch/x86/kernel/module.c | 7 ++++--- include/linux/kasan.h | 4 ++-- include/linux/vmalloc.h | 7 +++++++ mm/kasan/shadow.c | 9 +++++++-- mm/vmalloc.c | 3 ++- 7 files changed, 27 insertions(+), 12 deletions(-)
arch/arm64/kernel/module.c renamed from arch/arm64/kernel/module.c~mm-defer-kmemleak-object-creation-of-module_alloc +2 -2
@@ -36,7 +36,7 @@ void *module_alloc(unsigned long size) module_alloc_end =MODULES_END ; p =__vmalloc_node_range (size,MODULE_ALIGN , module_alloc_base, - module_alloc_end, gfp_mask,PAGE_KERNEL ,0 , + module_alloc_end, gfp_mask,PAGE_KERNEL ,VM_DEFER_KMEMLEAK ,NUMA_NO_NODE ,__builtin_return_address (0 )); if (!p && IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) &&
@@ -58,7 +58,7 @@ void *module_alloc(unsigned long size)PAGE_KERNEL ,0 ,NUMA_NO_NODE , __builtin_return_address(0)); -if (p && (kasan_module_alloc (p, size) <0 )) { +if (p && (kasan_module_alloc (p, size, gfp_mask) <0 )) {vfree (p);return NULL ; }
arch/s390/kernel/module.c renamed from arch/s390/kernel/module.c~mm-defer-kmemleak-object-creation-of-module_alloc +3 -2
@@ -37,14 +37,15 @@void *module_alloc (unsigned long size) { +gfp_t gfp_mask =GFP_KERNEL ;void *p;if (PAGE_ALIGN (size) >MODULES_LEN )return NULL ; p =__vmalloc_node_range (size,MODULE_ALIGN ,MODULES_VADDR ,MODULES_END , -GFP_KERNEL ,PAGE_KERNEL_EXEC ,0 ,NUMA_NO_NODE , + gfp_mask,PAGE_KERNEL_EXEC ,VM_DEFER_KMEMLEAK ,NUMA_NO_NODE ,__builtin_return_address (0 )); -if (p && (kasan_module_alloc (p, size) <0 )) { +if (p && (kasan_module_alloc (p, size, gfp_mask) <0 )) {vfree (p);return NULL ; }
arch/x86/kernel/module.c renamed from arch/x86/kernel/module.c~mm-defer-kmemleak-object-creation-of-module_alloc +4 -3
@@ -67,6 +67,7 @@ static unsigned long int get_module_loadvoid *module_alloc (unsigned long size) { +gfp_t gfp_mask =GFP_KERNEL ;void *p;if (PAGE_ALIGN (size) >MODULES_LEN )
@@ -74,10 +75,10 @@ void *module_alloc(unsigned long size) p =__vmalloc_node_range (size,MODULE_ALIGN ,MODULES_VADDR +get_module_load_offset (), -MODULES_END ,GFP_KERNEL , -PAGE_KERNEL ,0 ,NUMA_NO_NODE , +MODULES_END , gfp_mask, +PAGE_KERNEL ,VM_DEFER_KMEMLEAK ,NUMA_NO_NODE ,__builtin_return_address (0 )); -if (p && (kasan_module_alloc (p, size) <0 )) { +if (p && (kasan_module_alloc (p, size, gfp_mask) <0 )) {vfree (p);return NULL ; }
include/linux/kasan.h renamed from include/linux/kasan.h~mm-defer-kmemleak-object-creation-of-module_alloc +2 -2
@@ -474,12 +474,12 @@ static inline void kasan_populate_early_* allocations with real shadow memory. With KASAN vmalloc, the special * case is unnecessary, as the work is handled in the generic case. */ -int kasan_module_alloc (void *addr,size_t size); +int kasan_module_alloc (void *addr,size_t size,gfp_t gfp_mask);void kasan_free_shadow (const struct vm_struct *vm);#else /* (CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) && !CONFIG_KASAN_VMALLOC */ -static inline int kasan_module_alloc (void *addr,size_t size) {return 0 ; } +static inline int kasan_module_alloc (void *addr,size_t size,gfp_t gfp_mask) {return 0 ; }static inline void kasan_free_shadow (const struct vm_struct *vm) {}#endif /* (CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS) && !CONFIG_KASAN_VMALLOC */
include/linux/vmalloc.h renamed from include/linux/vmalloc.h~mm-defer-kmemleak-object-creation-of-module_alloc +7 -0
@@ -28,6 +28,13 @@ struct notifier_block; /* in notifier.h#define VM_MAP_PUT_PAGES 0x00000200/* put pages and free array in vfree */ #define VM_NO_HUGE_VMAP 0x00000400/* force PAGE_SIZE pte mapping */ +#if (defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS )) && \ + !defined(CONFIG_KASAN_VMALLOC ) +#define VM_DEFER_KMEMLEAK 0x00000800/* defer kmemleak object creation */ +#else +#define VM_DEFER_KMEMLEAK 0 +#endif +/* * VM_KASAN is used slightly differently depending on CONFIG_KASAN_VMALLOC. *
mm/kasan/shadow.c renamed from mm/kasan/shadow.c~mm-defer-kmemleak-object-creation-of-module_alloc +7 -2
@@ -498,7 +498,7 @@ void kasan_release_vmalloc(unsigned long#else /* CONFIG_KASAN_VMALLOC */ -int kasan_module_alloc (void *addr,size_t size) +int kasan_module_alloc (void *addr,size_t size,gfp_t gfp_mask) {void *ret;size_t scaled_size;
@@ -520,9 +520,14 @@ int kasan_module_alloc(void *addr, size_ __builtin_return_address(0));if (ret) { +struct vm_struct *vm =find_vm_area (addr);__memset (ret,KASAN_SHADOW_INIT , shadow_size); -find_vm_area (addr)->flags |=VM_KASAN ; + vm->flags |=VM_KASAN ;kmemleak_ignore (ret); + +if (vm->flags &VM_DEFER_KMEMLEAK ) +kmemleak_vmalloc (vm, size, gfp_mask); +return 0 ; }
mm/vmalloc.c renamed from mm/vmalloc.c~mm-defer-kmemleak-object-creation-of-module_alloc +2 -1
@@ -3074,7 +3074,8 @@ again:clear_vm_uninitialized_flag (area); size =PAGE_ALIGN (size); -kmemleak_vmalloc (area, size, gfp_mask); +if (!(vm_flags &VM_DEFER_KMEMLEAK )) +kmemleak_vmalloc (area, size, gfp_mask);return addr;
_
Andrew Morton
maintainerPATCH 26/146 mm/page_alloc: split prep_compound_page into head and tail subparts
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: mm/page_alloc: split prep_compound_page into head and tail subparts
Patch series "mm, device-dax: Introduce compound pages in devmap", v7.
This series converts device-dax to use compound pages, and moves away from the 'struct page per basepage on PMD/PUD' that is done today. Doing so,
1) unlocks a few noticeable improvements on unpin_user_pages() and makes device-dax+altmap case 4x times faster in pinning (numbers below and in last patch) 2) as mentioned in various other threads it's one important step towards cleaning up ZONE_DEVICE refcounting.
I've split the compound pages on devmap part from the rest based on recent discussions on devmap pending and future work planned[5][6]. There is consensus that device-dax should be using compound pages to represent its PMD/PUDs just like HugeTLB and THP, and that leads to less specialization of the dax parts. I will pursue the rest of the work in parallel once this part is merged, particular the GUP-{slow,fast} improvements [7] and the tail struct page deduplication memory savings part[8].
To summarize what the series does:
Patch 1: Prepare hwpoisoning to work with dax compound pages.
Patches 2-3: Split the current utility function of prep_compound_page() into head and tail and use those two helpers where appropriate to take advantage of caches being warm after __init_single_page(). This is used when initializing zone device when we bring up device-dax namespaces.
Patches 4-10: Add devmap support for compound pages in device-dax. memmap_init_zone_device() initialize its metadata as compound pages, and it introduces a new devmap property known as vmemmap_shift which outlines how the vmemmap is structured (defaults to base pages as done today). The property describe the page order of the metadata essentially. While at it do a few cleanups in device-dax in patches 5-9. Finally enable device-dax usage of devmap @vmemmap_shift to a value based on its own @align property. @vmemmap_shift returns 0 by default (which is today's case of base pages in devmap, like fsdax or the others) and the usage of compound devmap is optional. Starting with device-dax (*not* fsdax) we enable it by default. There are a few pinning improvements particular on the unpinning case and altmap, as well as unpin_user_page_range_dirty_lock() being just as effective as THP/hugetlb[0] pages.
$ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S -a -n 512 -w
(pin_user_pages_fast 2M pages) put:~71 ms -> put:~22 ms
[altmap]
(pin_user_pages_fast 2M pages) get:~524ms put:~525 ms -> get: ~127ms put:~71ms $ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S -a -n 512 -w
(pin_user_pages_fast 2M pages) put:~513 ms -> put:~188 ms
[altmap with -m 127004]
(pin_user_pages_fast 2M pages) get:~4.1 secs put:~4.12 secs -> get:~1sec put:~563msTested on x86 with 1Tb+ of pmem (alongside registering it with RDMA with and without altmap), alongside gup_test selftests with dynamic dax regions and static dax regions. Coupled with ndctl unit tests for dynamic dax devices that exercise all of this. Note, for dynamic dax regions I had to revert commit 8aa83e6395 ("x86/setup: Call early_reserve_memory() earlier"), it is a known issue that this commit broke efi_fake_mem=.
This patch (of 11):
Split the utility function prep_compound_page() into head and tail counterparts, and use them accordingly.
This is in preparation for sharing the storage for compound page metadata.
---
mm/page_alloc.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-split-prep_compound_page-into-head-and-tail-subparts +20 -10
@@ -726,23 +726,33 @@ void free_compound_page(struct page *pagfree_the_page (page,compound_order (page)); } +static void prep_compound_head (struct page *page,unsigned int order) +{ +set_compound_page_dtor (page,COMPOUND_PAGE_DTOR ); +set_compound_order (page, order); +atomic_set (compound_mapcount_ptr (page),-1 ); +if (hpage_pincount_available (page)) +atomic_set (compound_pincount_ptr (page),0 ); +} + +static void prep_compound_tail (struct page *head,int tail_idx) +{ +struct page *p = head + tail_idx; + + p->mapping =TAIL_MAPPING ; +set_compound_head (p, head); +} +void prep_compound_page (struct page *page,unsigned int order) {int i;int nr_pages =1 << order;__SetPageHead (page); -for (i =1 ; i < nr_pages; i++) { -struct page *p = page + i; - p->mapping =TAIL_MAPPING ; -set_compound_head (p, page); - } +for (i =1 ; i < nr_pages; i++) +prep_compound_tail (page, i); -set_compound_page_dtor (page,COMPOUND_PAGE_DTOR ); -set_compound_order (page, order); -atomic_set (compound_mapcount_ptr (page),-1 ); -if (hpage_pincount_available (page)) -atomic_set (compound_pincount_ptr (page),0 ); +prep_compound_head (page, order); }#ifdef CONFIG_DEBUG_PAGEALLOC
_
Andrew Morton
maintainerPATCH 27/146 mm/page_alloc: refactor memmap_init_zone_device() page init
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: mm/page_alloc: refactor memmap_init_zone_device() page init
Move struct page init to an helper function __init_zone_device_page().
This is in preparation for sharing the storage for compound page metadata.
---
mm/page_alloc.c | 74 +++++++++++++++++++++++++--------------------- 1 file changed, 41 insertions(+), 33 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-refactor-memmap_init_zone_device-page-init +41 -33
@@ -6572,6 +6572,46 @@ void __meminit memmap_init_range(unsigne }#ifdef CONFIG_ZONE_DEVICE +static void __ref __init_zone_device_page (struct page *page,unsigned long pfn, +unsigned long zone_idx,int nid, +struct dev_pagemap *pgmap) +{ + +__init_single_page (page, pfn, zone_idx, nid); + +/* +* Mark page reserved as it will need to wait for onlining +* phase for it to be fully associated with a zone. +* +* We can use the non-atomic __set_bit operation for setting +* the flag as we are still initializing the pages. +*/ +__SetPageReserved (page); + +/* +* ZONE_DEVICE pages union ->lru with a ->pgmap back pointer +* and zone_device_data. It is a bug if a ZONE_DEVICE page is +* ever freed or placed on a driver-private list. +*/ + page->pgmap = pgmap; + page->zone_device_data =NULL ; + +/* +* Mark the block movable so that blocks are reserved for +* movable at startup. This will force kernel allocations +* to reserve their blocks rather than leaking throughout +* the address space during boot when many long-lived +* kernel allocations are made. +* +* Please note that MEMINIT_HOTPLUG path doesn't clear memmap +* because this is done early in section_activate() +*/ +if (IS_ALIGNED (pfn, pageblock_nr_pages)) { +set_pageblock_migratetype (page,MIGRATE_MOVABLE ); +cond_resched (); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages,
@@ -6600,39 +6640,7 @@ void __ref memmap_init_zone_device(strucfor (pfn = start_pfn; pfn < end_pfn; pfn++) {struct page *page =pfn_to_page (pfn); -__init_single_page (page, pfn, zone_idx, nid); - -/* -* Mark page reserved as it will need to wait for onlining -* phase for it to be fully associated with a zone. -* -* We can use the non-atomic __set_bit operation for setting -* the flag as we are still initializing the pages. -*/ -__SetPageReserved (page); - -/* -* ZONE_DEVICE pages union ->lru with a ->pgmap back pointer -* and zone_device_data. It is a bug if a ZONE_DEVICE page is -* ever freed or placed on a driver-private list. -*/ - page->pgmap = pgmap; - page->zone_device_data =NULL ; - -/* -* Mark the block movable so that blocks are reserved for -* movable at startup. This will force kernel allocations -* to reserve their blocks rather than leaking throughout -* the address space during boot when many long-lived -* kernel allocations are made. -* -* Please note that MEMINIT_HOTPLUG path doesn't clear memmap -* because this is done early in section_activate() -*/ -if (IS_ALIGNED (pfn, pageblock_nr_pages)) { -set_pageblock_migratetype (page,MIGRATE_MOVABLE ); -cond_resched (); - } +__init_zone_device_page (page, pfn, zone_idx, nid, pgmap); } pr_info("%s initialised %lu pages in %ums\n", __func__,
_
Andrew Morton
maintainerPATCH 28/146 mm/memremap: add ZONE_DEVICE support for compound pages
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: mm/memremap: add ZONE_DEVICE support for compound pages
Add a new @vmemmap_shift property for struct dev_pagemap which specifies that a devmap is composed of a set of compound pages of order @vmemmap_shift, instead of base pages. When a compound page devmap is requested, all but the first page are initialised as tail pages instead of order-0 pages.
For certain ZONE_DEVICE users like device-dax which have a fixed page size, this creates an opportunity to optimize GUP and GUP-fast walkers, treating it the same way as THP or hugetlb pages.
Additionally, commit 7118fc2906e2 ("hugetlb: address ref count racing in prep_compound_gigantic_page") removed set_page_count() because the setting of page ref count to zero was redundant. devmap pages don't come from page allocator though and only head page refcount is used for compound pages, hence initialize tail page count to zero.
---
include/linux/memremap.h | 11 ++++++++++ mm/memremap.c | 18 +++++++++++------ mm/page_alloc.c | 38 ++++++++++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 7 deletions(-)
include/linux/memremap.h renamed from include/linux/memremap.h~mm-memremap-add-zone_device-support-for-compound-pages +11 -0
@@ -99,6 +99,11 @@ struct dev_pagemap_ops {* @done: completion for @internal_ref * @type: memory type: see MEMORY_* in memory_hotplug.h * @flags: PGMAP_* flags to specify defailed behavior +* @vmemmap_shift: structural definition of how the vmemmap page metadata +* is populated, specifically the metadata page order. +* A zero value (default) uses base pages as the vmemmap metadata +* representation. A bigger value will set up compound struct pages +* of the requested order value. * @ops: method table * @owner: an opaque pointer identifying the entity that manages this * instance. Used by various helpers to make sure that no
@@ -114,6 +119,7 @@ struct dev_pagemap {struct completion done;enum memory_type type;unsigned int flags; +unsigned long vmemmap_shift;const struct dev_pagemap_ops *ops;void *owner;int nr_range;
@@ -130,6 +136,11 @@ static inline struct vmem_altmap *pgmap_return NULL ; } +static inline unsigned long pgmap_vmemmap_nr (struct dev_pagemap *pgmap) +{ +return 1 << pgmap->vmemmap_shift; +} +#ifdef CONFIG_ZONE_DEVICE void *memremap_pages (struct dev_pagemap *pgmap,int nid);void memunmap_pages (struct dev_pagemap *pgmap);
mm/memremap.c renamed from mm/memremap.c~mm-memremap-add-zone_device-support-for-compound-pages +12 -6
@@ -102,15 +102,22 @@ static unsigned long pfn_end(struct dev_return (range->start +range_len (range)) >>PAGE_SHIFT ; } -static unsigned long pfn_next (unsigned long pfn) +static unsigned long pfn_next (struct dev_pagemap *pgmap,unsigned long pfn) { -if (pfn %1024 ==0 ) +if (pfn % (1024 << pgmap->vmemmap_shift))cond_resched (); -return pfn +1 ; +return pfn +pgmap_vmemmap_nr (pgmap); +} + +static unsigned long pfn_len (struct dev_pagemap *pgmap,unsigned long range_id) +{ +return (pfn_end (pgmap, range_id) - +pfn_first (pgmap, range_id)) >> pgmap->vmemmap_shift; }#define for_each_device_pfn (pfn, map, i) \ - for (pfn = pfn_first(map, i); pfn < pfn_end(map, i); pfn = pfn_next(pfn)) + for (pfn = pfn_first(map, i); pfn < pfn_end(map, i); \ + pfn = pfn_next(map, pfn))static void dev_pagemap_kill (struct dev_pagemap *pgmap) {
@@ -295,8 +302,7 @@ static int pagemap_range(struct dev_pagememmap_init_zone_device (&NODE_DATA (nid)->node_zones[ZONE_DEVICE ],PHYS_PFN (range->start),PHYS_PFN (range_len (range)), pgmap); -percpu_ref_get_many (pgmap->ref,pfn_end (pgmap, range_id) - -pfn_first (pgmap, range_id)); +percpu_ref_get_many (pgmap->ref,pfn_len (pgmap, range_id));return 0 ; err_add_memory:
mm/page_alloc.c renamed from mm/page_alloc.c~mm-memremap-add-zone_device-support-for-compound-pages +37 -1
@@ -6612,6 +6612,35 @@ static void __ref __init_zone_device_pag } } +static void __ref memmap_init_compound (struct page *head, +unsigned long head_pfn, +unsigned long zone_idx,int nid, +struct dev_pagemap *pgmap, +unsigned long nr_pages) +{ +unsigned long pfn, end_pfn = head_pfn + nr_pages; +unsigned int order = pgmap->vmemmap_shift; + +__SetPageHead (head); +for (pfn = head_pfn +1 ; pfn < end_pfn; pfn++) { +struct page *page =pfn_to_page (pfn); + +__init_zone_device_page (page, pfn, zone_idx, nid, pgmap); +prep_compound_tail (head, pfn - head_pfn); +set_page_count (page,0 ); + +/* +* The first tail page stores compound_mapcount_ptr() and +* compound_order() and the second tail page stores +* compound_pincount_ptr(). Call prep_compound_head() after +* the first and second tail pages have been initialized to +* not have the data overwritten. +*/ +if (pfn == head_pfn +2 ) +prep_compound_head (head, order); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages,
@@ -6620,6 +6649,7 @@ void __ref memmap_init_zone_device(strucunsigned long pfn, end_pfn = start_pfn + nr_pages;struct pglist_data *pgdat = zone->zone_pgdat;struct vmem_altmap *altmap =pgmap_altmap (pgmap); +unsigned int pfns_per_compound =pgmap_vmemmap_nr (pgmap);unsigned long zone_idx =zone_idx (zone);unsigned long start = jiffies;int nid = pgdat->node_id;
@@ -6637,10 +6667,16 @@ void __ref memmap_init_zone_device(struc nr_pages = end_pfn - start_pfn; } -for (pfn = start_pfn; pfn < end_pfn; pfn++) { +for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {struct page *page =pfn_to_page (pfn);__init_zone_device_page (page, pfn, zone_idx, nid, pgmap); + +if (pfns_per_compound ==1 ) +continue ; + +memmap_init_compound (page, pfn, zone_idx, nid, pgmap, + pfns_per_compound); } pr_info("%s initialised %lu pages in %ums\n", __func__,
_
Andrew Morton
maintainerPATCH 29/146 device-dax: use ALIGN() for determining pgoff
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: device-dax: use ALIGN() for determining pgoff
Rather than calculating @pgoff manually, switch to ALIGN() instead.
---
drivers/dax/device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/dax/device.c renamed from drivers/dax/device.c~device-dax-use-align-for-determining-pgoff +2 -2
@@ -234,8 +234,8 @@ static vm_fault_t dev_dax_huge_fault(str* mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff =linear_page_index (vmf->vma, vmf->address - & ~(fault_size -1 )); + pgoff =linear_page_index (vmf->vma, +ALIGN (vmf->address, fault_size));for (i =0 ; i < fault_size /PAGE_SIZE ; i++) {struct page *page;
_
Andrew Morton
maintainerPATCH 30/146 device-dax: use struct_size()
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: device-dax: use struct_size()
Use the struct_size() helper for the size of a struct with variable array member at the end, rather than manually calculating it.
---
drivers/dax/device.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/dax/device.c renamed from drivers/dax/device.c~device-dax-use-struct_size +3 -2
@@ -404,8 +404,9 @@ int dev_dax_probe(struct dev_dax *dev_dareturn -EINVAL ;if (!pgmap) { - pgmap =devm_kzalloc (dev,sizeof (*pgmap) +sizeof (struct range ) - * (dev_dax->nr_range -1 ),GFP_KERNEL ); + pgmap =devm_kzalloc (dev, +struct_size (pgmap, ranges, dev_dax->nr_range -1 ), +GFP_KERNEL );if (!pgmap)return -ENOMEM ; pgmap->nr_range = dev_dax->nr_range;
_
Andrew Morton
maintainerPATCH 31/146 device-dax: ensure dev_dax->pgmap is valid for dynamic devices
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: device-dax: ensure dev_dax->pgmap is valid for dynamic devices
Right now, only static dax regions have a valid @pgmap pointer in its struct dev_dax. Dynamic dax case however, do not.
In preparation for device-dax compound devmap support, make sure that dev_dax pgmap field is set after it has been allocated and initialized.
dynamic dax device have the @pgmap is allocated at probe() and it's managed by devm (contrast to static dax region which a pgmap is provided and dax core kfrees it). So in addition to ensure a valid @pgmap, clear the pgmap when the dynamic dax device is released to avoid the same pgmap ranges to be re-requested across multiple region device reconfigs.
Add a static_dev_dax() and use that helper in dev_dax_probe() to ensure the initialization differences between dynamic and static regions are more explicit. While at it, consolidate the ranges initialization when we allocate the @pgmap for the dynamic dax region case. Also take the opportunity to document the differences between static and dynamic da regions.
---
drivers/dax/bus.c | 32 ++++++++++++++++++++++++++++++++ drivers/dax/bus.h | 1 + drivers/dax/device.c | 29 +++++++++++++++++++++-------- 3 files changed, 54 insertions(+), 8 deletions(-)
drivers/dax/bus.c renamed from drivers/dax/bus.c~device-dax-ensure-dev_dax-pgmap-is-valid-for-dynamic-devices +32 -0
@@ -129,11 +129,35 @@ ATTRIBUTE_GROUPS(dax_drv);static int dax_bus_match (struct device *dev,struct device_driver *drv); +/* +* Static dax regions are regions created by an external subsystem +* nvdimm where a single range is assigned. Its boundaries are by the external +* subsystem and are usually limited to one physical memory range. For example, +* for PMEM it is usually defined by NVDIMM Namespace boundaries (i.e. a +* single contiguous range) +* +* On dynamic dax regions, the assigned region can be partitioned by dax core +* into multiple subdivisions. A subdivision is represented into one +* /dev/daxN.M device composed by one or more potentially discontiguous ranges. +* +* When allocating a dax region, drivers must set whether it's static +* (IORESOURCE_DAX_STATIC). On static dax devices, the @pgmap is pre-assigned +* to dax core when calling devm_create_dev_dax(), whereas in dynamic dax +* devices it is NULL but afterwards allocated by dax core on device ->probe(). +* Care is needed to make sure that dynamic dax devices are torn down with a +* cleared @pgmap field (see kill_dev_dax()). +*/ static bool is_static (struct dax_region *dax_region) {return (dax_region->res.flags &IORESOURCE_DAX_STATIC ) !=0 ; } +bool static_dev_dax (struct dev_dax *dev_dax) +{ +return is_static (dev_dax->region); +} +EXPORT_SYMBOL_GPL (static_dev_dax); +static u64 dev_dax_size (struct dev_dax *dev_dax) {u64 size =0 ;
@@ -363,6 +387,14 @@ void kill_dev_dax(struct dev_dax *dev_dakill_dax (dax_dev);unmap_mapping_range (inode->i_mapping,0 ,0 ,1 ); + +/* +* Dynamic dax region have the pgmap allocated via dev_kzalloc() +* and thus freed by devm. Clear the pgmap to not have stale pgmap +* ranges on probe() from previous reconfigurations of region devices. +*/ +if (!static_dev_dax (dev_dax)) + dev_dax->pgmap =NULL ; }EXPORT_SYMBOL_GPL (kill_dev_dax);
drivers/dax/bus.h renamed from drivers/dax/bus.h~device-dax-ensure-dev_dax-pgmap-is-valid-for-dynamic-devices +1 -0
@@ -48,6 +48,7 @@ int __dax_driver_register(struct dax_dev __dax_driver_register(driver, THIS_MODULE, KBUILD_MODNAME)void dax_driver_unregister (struct dax_device_driver *dax_drv);void kill_dev_dax (struct dev_dax *dev_dax); +bool static_dev_dax (struct dev_dax *dev_dax);#if IS_ENABLED (CONFIG_DEV_DAX_PMEM_COMPAT )int dev_dax_probe (struct dev_dax *dev_dax);
drivers/dax/device.c renamed from drivers/dax/device.c~device-dax-ensure-dev_dax-pgmap-is-valid-for-dynamic-devices +21 -8
@@ -398,18 +398,34 @@ int dev_dax_probe(struct dev_dax *dev_davoid *addr;int rc, i; - pgmap = dev_dax->pgmap; -if (dev_WARN_ONCE (dev, pgmap && dev_dax->nr_range >1 , -"static pgmap / multi-range device conflict\n" )) -return -EINVAL ; +if (static_dev_dax (dev_dax)) { +if (dev_dax->nr_range >1 ) { +dev_warn (dev, +"static pgmap / multi-range device conflict\n" ); +return -EINVAL ; + } + + pgmap = dev_dax->pgmap; + }else { +if (dev_dax->pgmap) { +dev_warn (dev, +"dynamic-dax with pre-populated page map\n" ); +return -EINVAL ; + } -if (!pgmap) { pgmap =devm_kzalloc (dev,struct_size (pgmap, ranges, dev_dax->nr_range -1 ),GFP_KERNEL );if (!pgmap)return -ENOMEM ; + pgmap->nr_range = dev_dax->nr_range; + dev_dax->pgmap = pgmap; + +for (i =0 ; i < dev_dax->nr_range; i++) { +struct range *range = &dev_dax->ranges[i].range; + pgmap->ranges[i] = *range; + } }for (i =0 ; i < dev_dax->nr_range; i++) {
@@ -421,9 +437,6 @@ int dev_dax_probe(struct dev_dax *dev_da i, range->start, range->end);return -EBUSY ; } -/* don't update the range for static pgmap */ -if (!dev_dax->pgmap) - pgmap->ranges[i] = *range; } pgmap->type =MEMORY_DEVICE_GENERIC ;
_
Andrew Morton
maintainerPATCH 32/146 device-dax: factor out page mapping initialization
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: device-dax: factor out page mapping initialization
Move initialization of page->mapping into a separate helper.
This is in preparation to move the mapping set to be prior to inserting the page table entry and also for tidying up compound page handling into one helper.
---
drivers/dax/device.c | 45 ++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 22 deletions(-)
drivers/dax/device.c renamed from drivers/dax/device.c~device-dax-factor-out-page-mapping-initialization +23 -22
@@ -73,6 +73,27 @@ __weak phys_addr_t dax_pgoff_to_phys(strreturn -1 ; } +static void dax_set_mapping (struct vm_fault *vmf,pfn_t pfn, +unsigned long fault_size) +{ +unsigned long i, nr_pages = fault_size /PAGE_SIZE ; +struct file *filp = vmf->vma->vm_file; +pgoff_t pgoff; + + pgoff =linear_page_index (vmf->vma, +ALIGN (vmf->address, fault_size)); + +for (i =0 ; i < nr_pages; i++) { +struct page *page =pfn_to_page (pfn_t_to_pfn (pfn) + i); + +if (page->mapping) +continue ; + + page->mapping = filp->f_mapping; + page->index = pgoff + i; + } +} +static vm_fault_t __dev_dax_pte_fault (struct dev_dax *dev_dax,struct vm_fault *vmf,pfn_t *pfn) {
@@ -224,28 +245,8 @@ static vm_fault_t dev_dax_huge_fault(str rc =VM_FAULT_SIGBUS ; } -if (rc ==VM_FAULT_NOPAGE ) { -unsigned long i; -pgoff_t pgoff; - -/* -* In the device-dax case the only possibility for a -* VM_FAULT_NOPAGE result is when device-dax capacity is -* mapped. No need to consider the zero page, or racing -* conflicting mappings. -*/ - pgoff =linear_page_index (vmf->vma, -ALIGN (vmf->address, fault_size)); -for (i =0 ; i < fault_size /PAGE_SIZE ; i++) { -struct page *page; - - page =pfn_to_page (pfn_t_to_pfn (pfn) + i); -if (page->mapping) -continue ; - page->mapping = filp->f_mapping; - page->index = pgoff + i; - } - } +if (rc ==VM_FAULT_NOPAGE ) +dax_set_mapping (vmf, pfn, fault_size);dax_read_unlock (id);return rc;
_
Andrew Morton
maintainerPATCH 33/146 device-dax: set mapping prior to vmf_insert_pfn{,_pmd,pud}()
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: device-dax: set mapping prior to vmf_insert_pfn{,_pmd,pud}()
Normally, the @page mapping is set prior to inserting the page into a page table entry. Make device-dax adhere to the same ordering, rather than setting mapping after the PTE is inserted.
The address_space never changes and it is always associated with the same inode and underlying pages. So, the page mapping is set once but cleared when the struct pages are removed/freed (i.e. after {devm_}memunmap_pages()).
---
drivers/dax/device.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/dax/device.c renamed from drivers/dax/device.c~device-dax-set-mapping-prior-to-vmf_insert_pfn_pmdpud +6 -6
@@ -121,6 +121,8 @@ static vm_fault_t __dev_dax_pte_fault(st *pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); +dax_set_mapping (vmf, *pfn, fault_size); +return vmf_insert_mixed (vmf->vma, vmf->address, *pfn); }
@@ -161,6 +163,8 @@ static vm_fault_t __dev_dax_pmd_fault(st *pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); +dax_set_mapping (vmf, *pfn, fault_size); +return vmf_insert_pfn_pmd (vmf, *pfn, vmf->flags &FAULT_FLAG_WRITE ); }
@@ -203,6 +207,8 @@ static vm_fault_t __dev_dax_pud_fault(st *pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); +dax_set_mapping (vmf, *pfn, fault_size); + return vmf_insert_pfn_pud(vmf, *pfn, vmf->flags & FAULT_FLAG_WRITE); }#else
@@ -217,7 +223,6 @@ static vm_fault_t dev_dax_huge_fault(str enum page_entry_size pe_size) {struct file *filp = vmf->vma->vm_file; -unsigned long fault_size;vm_fault_t rc =VM_FAULT_SIGBUS ;int id;pfn_t pfn;
@@ -230,23 +235,18 @@ static vm_fault_t dev_dax_huge_fault(str id =dax_read_lock ();switch (pe_size) {case PE_SIZE_PTE : - fault_size =PAGE_SIZE ; rc =__dev_dax_pte_fault (dev_dax, vmf, &pfn);break ;case PE_SIZE_PMD : - fault_size =PMD_SIZE ; rc =__dev_dax_pmd_fault (dev_dax, vmf, &pfn);break ;case PE_SIZE_PUD : - fault_size =PUD_SIZE ; rc =__dev_dax_pud_fault (dev_dax, vmf, &pfn);break ;default : rc =VM_FAULT_SIGBUS ; } -if (rc ==VM_FAULT_NOPAGE ) -dax_set_mapping (vmf, pfn, fault_size);dax_read_unlock (id);return rc;
_
Andrew Morton
maintainerPATCH 34/146 device-dax: remove pfn from __dev_dax_{pte,pmd,pud}_fault()
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: device-dax: remove pfn from __dev_dax_{pte,pmd,pud}_fault()
After moving the page mapping to be set prior to pte insertion, the pfn in dev_dax_huge_fault() no longer is necessary. Remove it, as well as the @pfn argument passed to the internal fault handler helpers.
---
drivers/dax/device.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-)
drivers/dax/device.c renamed from drivers/dax/device.c~device-dax-remove-pfn-from-__dev_dax_ptepmdpud_fault +19 -17
@@ -95,10 +95,11 @@ static void dax_set_mapping(struct vm_fa }static vm_fault_t __dev_dax_pte_fault (struct dev_dax *dev_dax, -struct vm_fault *vmf,pfn_t *pfn) +struct vm_fault *vmf) {struct device *dev = &dev_dax->dev;phys_addr_t phys; +pfn_t pfn;unsigned int fault_size =PAGE_SIZE ;if (check_vma (dev_dax, vmf->vma, __func__))
@@ -119,20 +120,21 @@ static vm_fault_t __dev_dax_pte_fault(streturn VM_FAULT_SIGBUS ; } - *pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); + pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); -dax_set_mapping (vmf, *pfn, fault_size); +dax_set_mapping (vmf, pfn, fault_size); -return vmf_insert_mixed (vmf->vma, vmf->address, *pfn); +return vmf_insert_mixed (vmf->vma, vmf->address, pfn); }static vm_fault_t __dev_dax_pmd_fault (struct dev_dax *dev_dax, -struct vm_fault *vmf,pfn_t *pfn) +struct vm_fault *vmf) {unsigned long pmd_addr = vmf->address &PMD_MASK ;struct device *dev = &dev_dax->dev;phys_addr_t phys;pgoff_t pgoff; +pfn_t pfn;unsigned int fault_size =PMD_SIZE ;if (check_vma (dev_dax, vmf->vma, __func__))
@@ -161,21 +163,22 @@ static vm_fault_t __dev_dax_pmd_fault(streturn VM_FAULT_SIGBUS ; } - *pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); + pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); -dax_set_mapping (vmf, *pfn, fault_size); +dax_set_mapping (vmf, pfn, fault_size); -return vmf_insert_pfn_pmd (vmf, *pfn, vmf->flags &FAULT_FLAG_WRITE ); +return vmf_insert_pfn_pmd (vmf, pfn, vmf->flags &FAULT_FLAG_WRITE ); }#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD static vm_fault_t __dev_dax_pud_fault (struct dev_dax *dev_dax, -struct vm_fault *vmf,pfn_t *pfn) +struct vm_fault *vmf) {unsigned long pud_addr = vmf->address &PUD_MASK ;struct device *dev = &dev_dax->dev;phys_addr_t phys;pgoff_t pgoff; +pfn_t pfn;unsigned int fault_size =PUD_SIZE ;
@@ -205,15 +208,15 @@ static vm_fault_t __dev_dax_pud_fault(st return VM_FAULT_SIGBUS; } - *pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); + pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); -dax_set_mapping (vmf, *pfn, fault_size); +dax_set_mapping (vmf, pfn, fault_size); -return vmf_insert_pfn_pud (vmf, *pfn, vmf->flags &FAULT_FLAG_WRITE ); +return vmf_insert_pfn_pud (vmf, pfn, vmf->flags &FAULT_FLAG_WRITE ); }#else static vm_fault_t __dev_dax_pud_fault (struct dev_dax *dev_dax, -struct vm_fault *vmf,pfn_t *pfn) +struct vm_fault *vmf) {return VM_FAULT_FALLBACK ; }
@@ -225,7 +228,6 @@ static vm_fault_t dev_dax_huge_fault(strstruct file *filp = vmf->vma->vm_file;vm_fault_t rc =VM_FAULT_SIGBUS ;int id; -pfn_t pfn;struct dev_dax *dev_dax = filp->private_data; dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) size = %d\n", current->comm,
@@ -235,13 +237,13 @@ static vm_fault_t dev_dax_huge_fault(str id =dax_read_lock ();switch (pe_size) {case PE_SIZE_PTE : - rc =__dev_dax_pte_fault (dev_dax, vmf, &pfn); + rc =__dev_dax_pte_fault (dev_dax, vmf);break ;case PE_SIZE_PMD : - rc =__dev_dax_pmd_fault (dev_dax, vmf, &pfn); + rc =__dev_dax_pmd_fault (dev_dax, vmf);break ;case PE_SIZE_PUD : - rc =__dev_dax_pud_fault (dev_dax, vmf, &pfn); + rc =__dev_dax_pud_fault (dev_dax, vmf);break ;default : rc =VM_FAULT_SIGBUS ;
_
Andrew Morton
maintainerPATCH 35/146 device-dax: compound devmap support
In reply to Andrew MortonFrom: Joao Martins <[email protected]>
Subject: device-dax: compound devmap support
Use the newly added compound devmap facility which maps the assigned dax ranges as compound pages at a page size of @align.
dax devices are created with a fixed @align (huge page size) which is enforced through as well at mmap() of the device. Faults, consequently happen too at the specified @align specified at the creation, and those don't change throughout dax device lifetime. MCEs unmap a whole dax huge page, as well as splits occurring at the configured page size.
Performance measured by gup_test improves considerably for unpin_user_pages() and altmap with NVDIMMs:
$ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S -a -n 512 -w (pin_user_pages_fast 2M pages) put:~71 ms -> put:~22 ms [altmap] (pin_user_pages_fast 2M pages) get:~524ms put:~525 ms -> get: ~127ms put:~71ms
$ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S -a -n 512 -w (pin_user_pages_fast 2M pages) put:~513 ms -> put:~188 ms [altmap with -m 127004] (pin_user_pages_fast 2M pages) get:~4.1 secs put:~4.12 secs -> get:~1sec put:~563ms
.. as well as unpin_user_page_range_dirty_lock() being just as effective as THP/hugetlb[0] pages.
[0] https://lore.kernel.org/linux-mm/[email protected]/
---
drivers/dax/device.c | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/dax/device.c renamed from drivers/dax/device.c~device-dax-compound-devmap-support +9 -0
@@ -78,14 +78,20 @@ static void dax_set_mapping(struct vm_fa {unsigned long i, nr_pages = fault_size /PAGE_SIZE ;struct file *filp = vmf->vma->vm_file; +struct dev_dax *dev_dax = filp->private_data;pgoff_t pgoff; +/* mapping is only set on the head */ +if (dev_dax->pgmap->vmemmap_shift) + nr_pages =1 ; + pgoff =linear_page_index (vmf->vma,ALIGN (vmf->address, fault_size));for (i =0 ; i < nr_pages; i++) {struct page *page =pfn_to_page (pfn_t_to_pfn (pfn) + i); + page =compound_head (page);if (page->mapping)continue ;
@@ -443,6 +449,9 @@ int dev_dax_probe(struct dev_dax *dev_da } pgmap->type =MEMORY_DEVICE_GENERIC ; +if (dev_dax->align >PAGE_SIZE ) + pgmap->vmemmap_shift = +order_base_2 (dev_dax->align >>PAGE_SHIFT ); addr =devm_memremap_pages (dev, pgmap);if (IS_ERR (addr))return PTR_ERR (addr);
_
Andrew Morton
maintainerPATCH 36/146 kasan: test: add globals left-out-of-bounds test
In reply to Andrew MortonFrom: Marco Elver <[email protected]>
Subject: kasan: test: add globals left-out-of-bounds test
Add a test checking that KASAN generic can also detect out-of-bounds accesses to the left of globals.
Unfortunately it seems that GCC doesn't catch this (tested GCC 10, 11). The main difference between GCC's globals redzoning and Clang's is that GCC relies on using increased alignment to producing padding, where Clang's redzoning implementation actually adds real data after the global and doesn't rely on alignment to produce padding. I believe this is the main reason why GCC can't reliably catch globals out-of-bounds in this case.
Given this is now a known issue, to avoid failing the whole test suite, skip this test case with GCC.
---
lib/test_kasan.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
lib/test_kasan.c renamed from lib/test_kasan.c~kasan-test-add-globals-left-out-of-bounds-test +17 -2
@@ -700,7 +700,7 @@ static void kmem_cache_bulk(struct kunitstatic char global_array[10 ]; -static void kasan_global_oob (struct kunit *test) +static void kasan_global_oob_right (struct kunit *test) {/* * Deliberate out-of-bounds access. To prevent CONFIG_UBSAN_LOCAL_BOUNDS
@@ -723,6 +723,20 @@ static void kasan_global_oob(struct kuniKUNIT_EXPECT_KASAN_FAIL (test, *(volatile char *)p); } +static void kasan_global_oob_left (struct kunit *test) +{ +char *volatile array = global_array; +char *p = array -3 ; + +/* +* GCC is known to fail this test, skip it. +* See https://bugzilla.kernel.org/show_bug.cgi?id=215051. +*/ +KASAN_TEST_NEEDS_CONFIG_ON (test,CONFIG_CC_IS_CLANG ); +KASAN_TEST_NEEDS_CONFIG_ON (test,CONFIG_KASAN_GENERIC ); +KUNIT_EXPECT_KASAN_FAIL (test, *(volatile char *)p); +} +/* Check that ksize() makes the whole object accessible. */ static void ksize_unpoisons_memory (struct kunit *test) {
@@ -1162,7 +1176,8 @@ static struct kunit_case kasan_kunit_tesKUNIT_CASE (kmem_cache_oob),KUNIT_CASE (kmem_cache_accounted),KUNIT_CASE (kmem_cache_bulk), -KUNIT_CASE (kasan_global_oob), +KUNIT_CASE (kasan_global_oob_right), +KUNIT_CASE (kasan_global_oob_left),KUNIT_CASE (kasan_stack_oob),KUNIT_CASE (kasan_alloca_oob_left), KUNIT_CASE(kasan_alloca_oob_right),
_
Andrew Morton
maintainerPATCH 37/146 kasan: add ability to detect double-kmem_cache_destroy()
In reply to Andrew MortonFrom: Marco Elver <[email protected]>
Subject: kasan: add ability to detect double-kmem_cache_destroy()
Because mm/slab_common.c is not instrumented with software KASAN modes, it is not possible to detect use-after-free of the kmem_cache passed into kmem_cache_destroy(). In particular, because of the s->refcount-- and subsequent early return if non-zero, KASAN would never be able to see the double-free via kmem_cache_free(kmem_cache, s). To be able to detect a double-kmem_cache_destroy(), check accessibility of the kmem_cache, and in case of failure return early.
While KASAN_HW_TAGS is able to detect such bugs, by checking accessibility and returning early we fail more gracefully and also avoid corrupting reused objects (where tags mismatch).
A recent case of a double-kmem_cache_destroy() was detected by KFENCE:
https://lkml.kernel.org/r/[email protected], which was not detectable by software KASAN modes.
---
mm/slab_common.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/slab_common.c renamed from mm/slab_common.c~kasan-add-ability-to-detect-double-kmem_cache_destroy +1 -1
@@ -489,7 +489,7 @@ void slab_kmem_cache_release(struct kmemvoid kmem_cache_destroy (struct kmem_cache *s) { -if (unlikely (!s)) +if (unlikely (!s) || !kasan_check_byte (s))return ;cpus_read_lock ();
_
Andrew Morton
maintainerPATCH 38/146 kasan: test: add test case for double-kmem_cache_destroy()
In reply to Andrew MortonFrom: Marco Elver <[email protected]>
Subject: kasan: test: add test case for double-kmem_cache_destroy()
Add a test case for double-kmem_cache_destroy() detection.
---
lib/test_kasan.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
lib/test_kasan.c renamed from lib/test_kasan.c~kasan-test-add-test-case-for-double-kmem_cache_destroy +11 -0
@@ -866,6 +866,16 @@ static void kmem_cache_invalid_free(strukmem_cache_destroy (cache); } +static void kmem_cache_double_destroy (struct kunit *test) +{ +struct kmem_cache *cache; + + cache =kmem_cache_create ("test_cache" ,200 ,0 ,0 ,NULL ); +KUNIT_ASSERT_NOT_ERR_OR_NULL (test, cache); +kmem_cache_destroy (cache); +KUNIT_EXPECT_KASAN_FAIL (test,kmem_cache_destroy (cache)); +} +static void kasan_memchr (struct kunit *test) {char *ptr;
@@ -1185,6 +1195,7 @@ static struct kunit_case kasan_kunit_tesKUNIT_CASE (ksize_uaf),KUNIT_CASE (kmem_cache_double_free),KUNIT_CASE (kmem_cache_invalid_free), +KUNIT_CASE (kmem_cache_double_destroy),KUNIT_CASE (kasan_memchr),KUNIT_CASE (kasan_memcmp), KUNIT_CASE(kasan_strings),
_
Andrew Morton
maintainerPATCH 39/146 kasan: fix quarantine conflicting with init_on_free
In reply to Andrew MortonFrom: Andrey Konovalov <[email protected]>
Subject: kasan: fix quarantine conflicting with init_on_free
KASAN's quarantine might save its metadata inside freed objects. As this happens after the memory is zeroed by the slab allocator when init_on_free is enabled, the memory coming out of quarantine is not properly zeroed.
This causes lib/test_meminit.c tests to fail with Generic KASAN.
Zero the metadata when the object is removed from quarantine.
---
mm/kasan/quarantine.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
mm/kasan/quarantine.c renamed from mm/kasan/quarantine.c~kasan-fix-quarantine-conflicting-with-init_on_free +11 -0
@@ -132,12 +132,23 @@ static void *qlink_to_object(struct qlisstatic void qlink_free (struct qlist_node *qlink,struct kmem_cache *cache) {void *object =qlink_to_object (qlink, cache); +struct kasan_free_meta *meta =kasan_get_free_meta (cache, object);unsigned long flags;if (IS_ENABLED (CONFIG_SLAB ))local_irq_save (flags);/* +* If init_on_free is enabled and KASAN's free metadata is stored in +* the object, zero the metadata. Otherwise, the object's memory will +* not be properly zeroed, as KASAN saves the metadata after the slab +* allocator zeroes the object. +*/ +if (slab_want_init_on_free (cache) && + cache->kasan_info.free_meta_offset ==0 ) +memzero_explicit (meta,sizeof (*meta)); + +/* * As the object now gets freed from the quarantine, assume that its * free track is no longer valid. */
_
Andrew Morton
maintainerPATCH 40/146 mm,fs: split dump_mapping() out from dump_page()
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm,fs: split dump_mapping() out from dump_page()
dump_mapping() is a big chunk of dump_page(), and it'd be handy to be able to call it when we don't have a struct page. Split it out and move it to fs/inode.c. Take the opportunity to simplify some of the debug messages a little.
---
fs/inode.c | 49 ++++++++++++++++++++++++++++++++++++++++ include/linux/fs.h | 1 mm/debug.c | 52 +------------------------------------------ 3 files changed, 52 insertions(+), 50 deletions(-)
fs/inode.c renamed from fs/inode.c~mmfs-split-dump_mapping-out-from-dump_page +49 -0
@@ -526,6 +526,55 @@ void __remove_inode_hash(struct inode *i }EXPORT_SYMBOL (__remove_inode_hash); +void dump_mapping (const struct address_space *mapping) +{ +struct inode *host; +const struct address_space_operations *a_ops; +struct hlist_node *dentry_first; +struct dentry *dentry_ptr; +struct dentry dentry; +unsigned long ino; + +/* +* If mapping is an invalid pointer, we don't want to crash +* accessing it, so probe everything depending on it carefully. +*/ +if (get_kernel_nofault (host, &mapping->host) || +get_kernel_nofault (a_ops, &mapping->a_ops)) { +pr_warn ("invalid mapping:%px\n" , mapping); +return ; + } + +if (!host) { +pr_warn ("aops:%ps\n" , a_ops); +return ; + } + +if (get_kernel_nofault (dentry_first, &host->i_dentry.first) || +get_kernel_nofault (ino, &host->i_ino)) { +pr_warn ("aops:%ps invalid inode:%px\n" , a_ops, host); +return ; + } + +if (!dentry_first) { +pr_warn ("aops:%ps ino:%lx\n" , a_ops, ino); +return ; + } + + dentry_ptr = container_of(dentry_first, struct dentry, d_u.d_alias); +if (get_kernel_nofault (dentry, dentry_ptr)) { +pr_warn ("aops:%ps ino:%lx invalid dentry:%px\n" , + a_ops, ino, dentry_ptr); +return ; + } + +/* +* if dentry is corrupted, the %pd handler may still crash, +* but it's unlikely that we reach here with a corrupt mapping +*/ +pr_warn ("aops:%ps ino:%lx dentry name:\"%pd\"\n" , a_ops, ino, &dentry); +} +void clear_inode (struct inode *inode) {/*
include/linux/fs.h renamed from include/linux/fs.h~mmfs-split-dump_mapping-out-from-dump_page +1 -0
@@ -3152,6 +3152,7 @@ extern void unlock_new_inode(struct inodextern void discard_new_inode (struct inode *);extern unsigned int get_next_ino (void );extern void evict_inodes (struct super_block *sb); +void dump_mapping (const struct address_space *);/* * Userspace may rely on the the inode number being non-zero. For example, glibc
mm/debug.c renamed from mm/debug.c~mmfs-split-dump_mapping-out-from-dump_page +2 -50
58 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 41/146 mm/debug_vm_pgtable: update comments regarding migration swap entries
In reply to Andrew MortonFrom: Anshuman Khandual <[email protected]>
Subject: mm/debug_vm_pgtable: update comments regarding migration swap entries
The commit 4dd845b5a3e5 ("mm/swapops: rework swap entry manipulation code") had changed migtation entry related helpers. Just update debug_vm_pgatble() synced documentation to reflect those changes.
---
Documentation/vm/arch_pgtable_helpers.rst | 14 +++++++------- mm/debug_vm_pgtable.c | 4 ++-- 2 files changed, 9 insertions(+), 9 deletions(-)
Documentation/vm/arch_pgtable_helpers.rst renamed from Documentation/vm/arch_pgtable_helpers.rst~mm-debug_vm_pgtable-update-comments-regarding-migration-swap-entries +7 -7
@@ -247,12 +247,12 @@ SWAP Page Table Helpers | __swp_to_pmd_entry | Creates a mapped PMD from a swapped entry (arch) | +---------------------------+--------------------------------------------------+ | is_migration_entry | Tests a migration (read or write) swapped entry | -+---------------------------+--------------------------------------------------+ -| is_write_migration_entry | Tests a write migration swapped entry | -+---------------------------+--------------------------------------------------+ -| make_migration_entry_read | Converts into read migration swapped entry | -+---------------------------+--------------------------------------------------+ -| make_migration_entry | Creates a migration swapped entry (read or write)| -+---------------------------+--------------------------------------------------+ ++-------------------------------+----------------------------------------------+ +| is_writable_migration_entry | Tests a write migration swapped entry | ++-------------------------------+----------------------------------------------+ +| make_readable_migration_entry | Creates a read migration swapped entry | ++-------------------------------+----------------------------------------------+ +| make_writable_migration_entry | Creates a write migration swapped entry | ++-------------------------------+----------------------------------------------+ [1] https://lore.kernel.org/linux-mm/[email protected]/
mm/debug_vm_pgtable.c renamed from mm/debug_vm_pgtable.c~mm-debug_vm_pgtable-update-comments-regarding-migration-swap-entries +2 -2
@@ -888,8 +888,8 @@ static void __init swap_migration_tests(pr_debug ("Validating swap migration\n" );/* -* make_migration_entry() expects given page to be -* locked, otherwise it stumbles upon a BUG_ON(). +* make_[readable|writable]_migration_entry() expects given page to +* be locked, otherwise it stumbles upon a BUG_ON(). */ __SetPageLocked (page); swp =make_writable_migration_entry (page_to_pfn (page));
_
Andrew Morton
maintainerPATCH 42/146 mm/truncate.c: remove unneeded variable
In reply to Andrew MortonFrom: chiminghao <[email protected]>
Subject: mm/truncate.c: remove unneeded variable
Return value directly instead of taking this in another redundant variable.
---
mm/truncate.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
mm/truncate.c renamed from mm/truncate.c~mm-remove-unneeded-variable +1 -4
@@ -205,7 +205,6 @@ static void truncate_cleanup_page(structstatic int invalidate_complete_page (struct address_space *mapping,struct page *page) { -int ret;if (page->mapping != mapping)return 0 ;
@@ -213,9 +212,7 @@ invalidate_complete_page(struct address_if (page_has_private (page) && !try_to_release_page (page,0 ))return 0 ; - ret =remove_mapping (mapping, page); - -return ret; +return remove_mapping (mapping, page); }int truncate_inode_page (struct address_space *mapping,struct page *page)
_
Andrew Morton
maintainerPATCH 43/146 gup: avoid multiple user access locking/unlocking in fault_in_{read/write}able
In reply to Andrew MortonFrom: Christophe Leroy <[email protected]>
Subject: gup: avoid multiple user access locking/unlocking in fault_in_{read/write}able
fault_in_readable() and fault_in_writeable() perform __get_user() and __put_user() in a loop, implying multiple user access locking/unlocking.
To avoid that, use user access blocks.
---
mm/gup.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
mm/gup.c renamed from mm/gup.c~gup-avoid-multiple-user-access-locking-unlocking-in-fault_in_read-writeable +10 -8
@@ -1672,21 +1672,22 @@ size_t fault_in_writeable(char __user *uif (unlikely (size ==0 ))return 0 ; +if (!user_write_access_begin (uaddr, size)) +return size;if (!PAGE_ALIGNED (uaddr)) { -if (unlikely (__put_user (0 , uaddr) !=0 )) -return size; +unsafe_put_user (0 , uaddr, out); uaddr = (char __user *)PAGE_ALIGN ((unsigned long )uaddr); } end = (char __user *)PAGE_ALIGN ((unsigned long )start + size);if (unlikely (end < start)) end =NULL ;while (uaddr != end) { -if (unlikely (__put_user (0 , uaddr) !=0 )) - goto out; +unsafe_put_user (0 , uaddr, out); uaddr +=PAGE_SIZE ; } out: +user_write_access_end ();if (size > uaddr - start)return size - (uaddr - start);return 0 ;
@@ -1771,21 +1772,22 @@ size_t fault_in_readable(const char __usif (unlikely (size ==0 ))return 0 ; +if (!user_read_access_begin (uaddr, size)) +return size;if (!PAGE_ALIGNED (uaddr)) { -if (unlikely (__get_user (c, uaddr) !=0 )) -return size; +unsafe_get_user (c, uaddr, out); uaddr = (const char __user *)PAGE_ALIGN ((unsigned long )uaddr); } end = (const char __user *)PAGE_ALIGN ((unsigned long )start + size);if (unlikely (end < start)) end =NULL ;while (uaddr != end) { -if (unlikely (__get_user (c, uaddr) !=0 )) - goto out; +unsafe_get_user (c, uaddr, out); uaddr +=PAGE_SIZE ; } out: +user_read_access_end (); (void )c;if (size > uaddr - start)return size - (uaddr - start);
_
Andrew Morton
maintainerPATCH 44/146 mm/gup.c: stricter check on THP migration entry during follow_pmd_mask
In reply to Andrew MortonFrom: Li Xinhai <[email protected]>
Subject: mm/gup.c: stricter check on THP migration entry during follow_pmd_mask
When BUG_ON check for THP migration entry, the existing code only check thp_migration_supported case, but not for !thp_migration_supported case. If !thp_migration_supported() and !pmd_present(), the original code may dead loop in theory. To make the BUG_ON check consistent, we need catch both cases.
Move the BUG_ON check one step earlier, because if the bug happen we should know it instead of depend on FOLL_MIGRATION been used by caller.
Because pmdval instead of *pmd is read by the is_pmd_migration_entry() check, the existing code don't help to avoid useless locking within pmd_migration_entry_wait(), so remove that check.
---
mm/gup.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
mm/gup.c renamed from mm/gup.c~mm-gupc-stricter-check-on-thp-migration-entry-during-follow_pmd_mask +9 -4
@@ -642,12 +642,17 @@ static struct page *follow_pmd_mask(stru } retry:if (!pmd_present (pmdval)) { +/* +* Should never reach here, if thp migration is not supported; +* Otherwise, it must be a thp migration entry. +*/ +VM_BUG_ON (!thp_migration_supported () || + !is_pmd_migration_entry (pmdval)); +if (likely (!(flags &FOLL_MIGRATION )))return no_page_table (vma, flags); -VM_BUG_ON (thp_migration_supported () && - !is_pmd_migration_entry (pmdval)); -if (is_pmd_migration_entry (pmdval)) -pmd_migration_entry_wait (mm, pmd); + +pmd_migration_entry_wait (mm, pmd); pmdval =READ_ONCE (*pmd);/* * MADV_DONTNEED may convert the pmd to null because
_
Andrew Morton
maintainerPATCH 45/146 mm: shmem: don't truncate page if memory failure happens
In reply to Andrew MortonFrom: Yang Shi <[email protected]>
Subject: mm: shmem: don't truncate page if memory failure happens
The current behavior of memory failure is to truncate the page cache regardless of dirty or clean. If the page is dirty the later access will get the obsolete data from disk without any notification to the users. This may cause silent data loss. It is even worse for shmem since shmem is in-memory filesystem, truncating page cache means discarding data blocks. The later read would return all zero.
The right approach is to keep the corrupted page in page cache, any later access would return error for syscalls or SIGBUS for page fault, until the file is truncated, hole punched or removed. The regular storage backed filesystems would be more complicated so this patch is focused on shmem. This also unblock the support for soft offlining shmem THP.
[Fix invalid pointer dereference in shmem_read_mapping_page_gfp() with a
slight different implementation from what Ajay Garg <[email protected]> and Muchun Song <[email protected]> proposed and reworked the error handling of shmem_write_begin() suggested by Linus]
---
mm/memory-failure.c | 14 +++++++++-- mm/shmem.c | 51 +++++++++++++++++++++++++++++++++++++----- mm/userfaultfd.c | 5 ++++ 3 files changed, 61 insertions(+), 9 deletions(-)
mm/memory-failure.c renamed from mm/memory-failure.c~mm-shmem-dont-truncate-page-if-memory-failure-happens +11 -3
@@ -58,6 +58,7 @@#include <linux/ratelimit.h> #include <linux/page-isolation.h> #include <linux/pagewalk.h> +#include <linux/shmem_fs.h> #include "internal.h" #include "ras/ras_event.h"
@@ -867,6 +868,7 @@ static int me_pagecache_clean(struct pag {int ret;struct address_space *mapping; +bool extra_pins;delete_from_lru_cache (p);
@@ -896,17 +898,23 @@ static int me_pagecache_clean(struct pag }/* +* The shmem page is kept in page cache instead of truncating +* so is expected to have an extra refcount after error-handling. +*/ + extra_pins =shmem_mapping (mapping); + +/* * Truncation is a bit tricky. Enable it per file system for now. * * Open: to take i_rwsem or not for this? Right now we don't. */ ret =truncate_error_page (p,page_to_pfn (p), mapping); +if (has_extra_refcount (ps, p, extra_pins)) + ret =MF_FAILED ; + out:unlock_page (p); -if (has_extra_refcount (ps, p, false)) - ret =MF_FAILED ; -return ret; }
mm/shmem.c renamed from mm/shmem.c~mm-shmem-dont-truncate-page-if-memory-failure-happens +45 -6
99 lines not on this page · raw message
mm/userfaultfd.c renamed from mm/userfaultfd.c~mm-shmem-dont-truncate-page-if-memory-failure-happens +5 -0
@@ -232,6 +232,11 @@ static int mcontinue_atomic_pte(struct m goto out; } +if (PageHWPoison (page)) { + ret = -EIO ; + goto out_release; + } + ret =mfill_atomic_install_pte (dst_mm, dst_pmd, dst_vma, dst_addr, page, false, wp_copy);if (ret)
_
Andrew Morton
maintainerPATCH 46/146 shmem: fix a race between shmem_unused_huge_shrink and shmem_evict_inode
In reply to Andrew MortonFrom: Gang Li <[email protected]>
Subject: shmem: fix a race between shmem_unused_huge_shrink and shmem_evict_inode
Fix a data race in commit 779750d20b93 ("shmem: split huge pages beyond i_size under memory pressure").
Here are call traces causing race:
Call Trace 1:
shmem_unused_huge_shrink+0x3ae/0x410
? __list_lru_walk_one.isra.5+0x33/0x160
super_cache_scan+0x17c/0x190
shrink_slab.part.55+0x1ef/0x3f0
shrink_node+0x10e/0x330
kswapd+0x380/0x740
kthread+0xfc/0x130
? mem_cgroup_shrink_node+0x170/0x170
? kthread_create_on_node+0x70/0x70
ret_from_fork+0x1f/0x30Call Trace 2:
shmem_evict_inode+0xd8/0x190
evict+0xbe/0x1c0
do_unlinkat+0x137/0x330
do_syscall_64+0x76/0x120
entry_SYSCALL_64_after_hwframe+0x3d/0xa2A simple explanation:
Image there are 3 items in the local list (@list). In the first traversal, A is not deleted from @list.
1) A->B->C
^
|
pos (leave)In the second traversal, B is deleted from @list. Concurrently, A is deleted from @list through shmem_evict_inode() since last reference counter of inode is dropped by other thread. Then the @list is corrupted.
2) A->B->C
^ ^
| |
evict pos (drop)We should make sure the inode is either on the global list or deleted from any local list before iput().
Fixed by moving inodes back to global list before we put them.
---
mm/shmem.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-)
mm/shmem.c renamed from mm/shmem.c~shmem-fix-a-race-between-shmem_unused_huge_shrink-and-shmem_evict_inode +21 -16
@@ -554,7 +554,7 @@ static unsigned long shmem_unused_huge_sstruct shmem_inode_info *info;struct page *page;unsigned long batch = sc ? sc->nr_to_scan :128 ; -int removed =0 , split =0 ; +int split =0 ;if (list_empty (&sbinfo->shrinklist))return SHRINK_STOP ;
@@ -569,7 +569,6 @@ static unsigned long shmem_unused_huge_s/* inode is about to be evicted */ if (!inode) {list_del_init (&info->shrinklist); - removed++; goto next; }
@@ -577,12 +576,12 @@ static unsigned long shmem_unused_huge_sif (round_up (inode->i_size,PAGE_SIZE ) ==round_up (inode->i_size,HPAGE_PMD_SIZE )) {list_move (&info->shrinklist, &to_remove); - removed++; goto next; }list_move (&info->shrinklist, &list); next: + sbinfo->shrinklist_len--;if (!--batch)break ; }
@@ -602,7 +601,7 @@ next: inode = &info->vfs_inode;if (nr_to_split && split >= nr_to_split) - goto leave; + goto move_back; page =find_get_page (inode->i_mapping, (inode->i_size &HPAGE_PMD_MASK ) >>PAGE_SHIFT );
@@ -616,38 +615,44 @@ next: }/* -* Leave the inode on the list if we failed to lock -* the page at this time. +* Move the inode on the list back to shrinklist if we failed +* to lock the page at this time. * * Waiting for the lock may lead to deadlock in the * reclaim path. */ if (!trylock_page (page)) {put_page (page); - goto leave; + goto move_back; } ret =split_huge_page (page);unlock_page (page);put_page (page); -/* If split failed leave the inode on the list */ +/* If split failed move the inode on the list back to shrinklist */ if (ret) - goto leave; + goto move_back; split++; drop:list_del_init (&info->shrinklist); - removed++; -leave: + goto put; +move_back: +/* +* Make sure the inode is either on the global list or deleted +* from any local list before iput() since it could be deleted +* in another thread once we put the inode (then the local list +* is corrupted). +*/ +spin_lock (&sbinfo->shrinklist_lock); +list_move (&info->shrinklist, &sbinfo->shrinklist); + sbinfo->shrinklist_len++; +spin_unlock (&sbinfo->shrinklist_lock); +put:iput (inode); } -spin_lock (&sbinfo->shrinklist_lock); -list_splice_tail (&list, &sbinfo->shrinklist); - sbinfo->shrinklist_len -= removed; -spin_unlock (&sbinfo->shrinklist_lock); -return split; }
_
Andrew Morton
maintainerPATCH 47/146 mm/frontswap.c: use non-atomic '__set_bit()' when possible
In reply to Andrew MortonFrom: Christophe JAILLET <[email protected]>
Subject: mm/frontswap.c: use non-atomic '__set_bit()' when possible
the 'a' and 'b' bitmaps are local to this function, so no concurrent access can occur. So the non-atomic '__set_bit()' can be used to save a few cycles.
---
mm/frontswap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/frontswap.c renamed from mm/frontswap.c~mm-mempool-use-non-atomic-__set_bit-when-possible +2 -2
@@ -127,7 +127,7 @@ void frontswap_register_ops(struct frontspin_lock (&swap_lock);plist_for_each_entry (si, &swap_active_head, list) {if (!WARN_ON (!si->frontswap_map)) -set_bit (si->type, a); +__set_bit (si->type, a); }spin_unlock (&swap_lock);
@@ -149,7 +149,7 @@ void frontswap_register_ops(struct frontspin_lock (&swap_lock);plist_for_each_entry (si, &swap_active_head, list) {if (si->frontswap_map) -set_bit (si->type, b); +__set_bit (si->type, b); }spin_unlock (&swap_lock);
_
Andrew Morton
maintainerPATCH 48/146 mm: memcontrol: make cgroup_memory_nokmem static
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: make cgroup_memory_nokmem static
commit 494c1dfe855e ("mm: memcg/slab: create a new set of kmalloc-cg-<n> caches") makes cgroup_memory_nokmem global, however, it is unnecessary because there is already a function mem_cgroup_kmem_disabled() which exports it.
Just make it static and replace it with mem_cgroup_kmem_disabled() in mm/slab_common.c.
---
mm/internal.h | 5 ----- mm/memcontrol.c | 2 +- mm/slab_common.c | 2 +- 3 files changed, 2 insertions(+), 7 deletions(-)
mm/internal.h renamed from mm/internal.h~mm-memcontrol-make-cgroup_memory_nokmem-static +0 -5
@@ -158,11 +158,6 @@ extern void reclaim_throttle(pg_data_t *extern pmd_t *mm_find_pmd (struct mm_struct *mm,unsigned long address);/* -* in mm/memcontrol.c: -*/ -extern bool cgroup_memory_nokmem; - -/* * in mm/page_alloc.c */
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-make-cgroup_memory_nokmem-static +1 -1
@@ -84,7 +84,7 @@ EXPORT_PER_CPU_SYMBOL_GPL(int_active_memstatic bool cgroup_memory_nosocket__ro_after_init ;/* Kernel memory accounting disabled? */ -bool cgroup_memory_nokmem__ro_after_init ; +static bool cgroup_memory_nokmem__ro_after_init ;/* Whether the swap controller is active */ #ifdef CONFIG_MEMCG_SWAP
mm/slab_common.c renamed from mm/slab_common.c~mm-memcontrol-make-cgroup_memory_nokmem-static +1 -1
@@ -844,7 +844,7 @@ new_kmalloc_cache(int idx, enum kmalloc_if (type ==KMALLOC_RECLAIM ) { flags |=SLAB_RECLAIM_ACCOUNT ; }else if (IS_ENABLED (CONFIG_MEMCG_KMEM ) && (type ==KMALLOC_CGROUP )) { -if (cgroup_memory_nokmem) { +if (mem_cgroup_kmem_disabled ()) { kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL ][idx];return ; }
_
Andrew Morton
maintainerPATCH 49/146 mm/page_counter: remove an incorrect call to propagate_protected_usage()
In reply to Andrew MortonFrom: Donghai Qiao <[email protected]>
Subject: mm/page_counter: remove an incorrect call to propagate_protected_usage()
propagate_protected_usage() is called to propagate the usage change in the page_counter structure. But there is a call to this function from page_counter_try_charge() when there is actually no usage change. Hence this call should be removed.
---
mm/page_counter.c | 1 - 1 file changed, 1 deletion(-)
mm/page_counter.c renamed from mm/page_counter.c~mm-page_counter-remove-an-incorrect-call-to-propagate_protected_usage +0 -1
@@ -120,7 +120,6 @@ bool page_counter_try_charge(struct page new =atomic_long_add_return (nr_pages, &c->usage);if (new > c->max) {atomic_long_sub (nr_pages, &c->usage); -propagate_protected_usage (c, new);/* * This is racy, but we can live with some * inaccuracy in the failcnt which is only used
_
Andrew Morton
maintainerPATCH 50/146 mm/memcg: add oom_group_kill memory event
In reply to Andrew MortonFrom: Dan Schatzberg <[email protected]>
Subject: mm/memcg: add oom_group_kill memory event
Our container agent wants to know when a container exits if it was OOM killed or not to report to the user. We use memory.oom.group = 1 to ensure that OOM kills within the container's cgroup kill everything. Existing memory.events are insufficient for knowing if this triggered:
1) Our current approach reads memory.events oom_kill and reports the container was killed if the value is non-zero. This is erroneous in some cases where containers create their children cgroups with memory.oom.group=1 as such OOM kills will get counted against the container cgroup's oom_kill counter despite not actually OOM killing the entire container.
2) Reading memory.events.local will fail to identify OOM kills in leaf cgroups (that don't set memory.oom.group) within the container cgroup.
This patch adds a new oom_group_kill event when memory.oom.group triggers to allow userspace to cleanly identify when an entire cgroup is oom killed.
---
Documentation/admin-guide/cgroup-v2.rst | 3 +++ include/linux/memcontrol.h | 1 + mm/memcontrol.c | 2 ++ mm/oom_kill.c | 1 + 4 files changed, 7 insertions(+)
Documentation/admin-guide/cgroup-v2.rst renamed from Documentation/admin-guide/cgroup-v2.rst~mm-add-group_oom_kill-memory-event +3 -0
@@ -1268,6 +1268,9 @@ PAGE_SIZE multiple when read back. The number of processes belonging to this cgroup killed by any kind of OOM killer. + oom_group_kill + The number of times a group OOM has occurred. + memory.events.local Similar to memory.events but the fields in the file are local to the cgroup i.e. not hierarchical. The file modified event
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-add-group_oom_kill-memory-event +1 -0
@@ -42,6 +42,7 @@ enum memcg_memory_event {MEMCG_MAX ,MEMCG_OOM ,MEMCG_OOM_KILL , +MEMCG_OOM_GROUP_KILL ,MEMCG_SWAP_HIGH ,MEMCG_SWAP_MAX , MEMCG_SWAP_FAIL,
mm/memcontrol.c renamed from mm/memcontrol.c~mm-add-group_oom_kill-memory-event +2 -0
@@ -6318,6 +6318,8 @@ static void __memory_events_show(structseq_printf (m,"oom %lu\n" ,atomic_long_read (&events[MEMCG_OOM ]));seq_printf (m,"oom_kill %lu\n" ,atomic_long_read (&events[MEMCG_OOM_KILL ])); +seq_printf (m,"oom_group_kill %lu\n" , +atomic_long_read (&events[MEMCG_OOM_GROUP_KILL ])); }static int memory_events_show (struct seq_file *m,void *v)
mm/oom_kill.c renamed from mm/oom_kill.c~mm-add-group_oom_kill-memory-event +1 -0
@@ -994,6 +994,7 @@ static void oom_kill_process(struct oom_* If necessary, kill all tasks in the selected memory cgroup. */ if (oom_group) { +memcg_memory_event (oom_group,MEMCG_OOM_GROUP_KILL );mem_cgroup_print_oom_group (oom_group);mem_cgroup_scan_tasks (oom_group, oom_kill_memcg_member, (void *)message);
_
Andrew Morton
maintainerPATCH 51/146 memcg: better bounds on the memcg stats updates
In reply to Andrew MortonFrom: Shakeel Butt <[email protected]>
Subject: memcg: better bounds on the memcg stats updates
The commit 11192d9c124d ("memcg: flush stats only if updated") added tracking of memcg stats updates which is used by the readers to flush only if the updates are over a certain threshold. However each individual update can correspond to a large value change for a given stat. For example adding or removing a hugepage to an LRU changes the stat by thp_nr_pages (512 on x86_64). Treating the update related to THP as one can keep the stat off, in theory, by (thp_nr_pages * nr_cpus * CHARGE_BATCH) before flush.
To handle such scenarios, this patch adds consideration of the stat update value as well instead of just the update event. In addition let the asyn flusher unconditionally flush the stats to put time limit on the stats skew and hopefully a lot less readers would need to flush.
---
mm/memcontrol.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~memcg-better-bounds-on-the-memcg-stats-updates +13 -7
@@ -629,11 +629,17 @@ static DEFINE_SPINLOCK(stats_flush_lock) static DEFINE_PER_CPU(unsigned int, stats_updates);static atomic_t stats_flush_threshold =ATOMIC_INIT (0 ); -static inline void memcg_rstat_updated (struct mem_cgroup *memcg) +static inline void memcg_rstat_updated (struct mem_cgroup *memcg,int val) { +unsigned int x; +cgroup_rstat_updated (memcg->css.cgroup,smp_processor_id ()); -if (!(__this_cpu_inc_return (stats_updates) %MEMCG_CHARGE_BATCH )) -atomic_inc (&stats_flush_threshold); + + x =__this_cpu_add_return (stats_updates,abs (val)); +if (x >MEMCG_CHARGE_BATCH ) { +atomic_add (x /MEMCG_CHARGE_BATCH , &stats_flush_threshold); +__this_cpu_write (stats_updates,0 ); + } }static void __mem_cgroup_flush_stats (void )
@@ -656,7 +662,7 @@ void mem_cgroup_flush_stats(void)static void flush_memcg_stats_dwork (struct work_struct *w) { -mem_cgroup_flush_stats (); +__mem_cgroup_flush_stats ();queue_delayed_work (system_unbound_wq, &stats_flush_dwork,2UL *HZ ); }
@@ -672,7 +678,7 @@ void __mod_memcg_state(struct mem_cgroupreturn ;__this_cpu_add (memcg->vmstats_percpu->state[idx], val); -memcg_rstat_updated (memcg); +memcg_rstat_updated (memcg, val); }/* idx can be of type enum memcg_stat_item or node_stat_item. */
@@ -705,7 +711,7 @@ void __mod_memcg_lruvec_state(struct lru/* Update lruvec */ __this_cpu_add (pn->lruvec_stats_percpu->state[idx], val); -memcg_rstat_updated (memcg); +memcg_rstat_updated (memcg, val); }/**
@@ -789,7 +795,7 @@ void __count_memcg_events(struct mem_cgrreturn ;__this_cpu_add (memcg->vmstats_percpu->events[idx], count); -memcg_rstat_updated (memcg); +memcg_rstat_updated (memcg, count); }static unsigned long memcg_events (struct mem_cgroup *memcg,int event)
_
Andrew Morton
maintainerPATCH 52/146 mm/memcg: use struct_size() helper in kzalloc()
In reply to Andrew MortonFrom: Wang Weiyang <[email protected]>
Subject: mm/memcg: use struct_size() helper in kzalloc()
Make use of the struct_size() helper instead of an open-coded version, in order to avoid any potential type mistakes or integer overflows that, in the worst scenario, could lead to heap overflows.
---
mm/memcontrol.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-use-struct_size-helper-in-kzalloc +1 -5
@@ -5122,15 +5122,11 @@ static void mem_cgroup_free(struct mem_cstatic struct mem_cgroup *mem_cgroup_alloc (void ) {struct mem_cgroup *memcg; -unsigned int size;int node;int __maybe_unused i;long error = -ENOMEM ; - size =sizeof (struct mem_cgroup ); - size += nr_node_ids *sizeof (struct mem_cgroup_per_node *); - - memcg =kzalloc (size,GFP_KERNEL ); + memcg =kzalloc (struct_size (memcg, nodeinfo, nr_node_ids),GFP_KERNEL );if (!memcg)return ERR_PTR (error);
_
Andrew Morton
maintainerPATCH 53/146 memcg: add per-memcg vmalloc stat
In reply to Andrew MortonFrom: Shakeel Butt <[email protected]>
Subject: memcg: add per-memcg vmalloc stat
The kvmalloc* allocation functions can fallback to vmalloc allocations and more often on long running machines. In addition the kernel does have __GFP_ACCOUNT kvmalloc* calls. So, often on long running machines, the memory.stat does not tell the complete picture which type of memory is charged to the memcg. So add a per-memcg vmalloc stat.
---
Documentation/admin-guide/cgroup-v2.rst | 3 +++ include/linux/memcontrol.h | 21 +++++++++++++++++++++ mm/memcontrol.c | 1 + mm/vmalloc.c | 13 +++++++++++-- 4 files changed, 36 insertions(+), 2 deletions(-)
Documentation/admin-guide/cgroup-v2.rst renamed from Documentation/admin-guide/cgroup-v2.rst~memcg-add-per-memcg-vmalloc-stat +3 -0
@@ -1314,6 +1314,9 @@ PAGE_SIZE multiple when read back. sock (npn) Amount of memory used in network transmission buffers + vmalloc (npn) + Amount of memory used for vmap backed memory. + shmem Amount of cached filesystem data that is swap-backed, such as tmpfs, shm segments, shared anonymous mmap()s
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~memcg-add-per-memcg-vmalloc-stat +21 -0
@@ -33,6 +33,7 @@ enum memcg_stat_item {MEMCG_SWAP =NR_VM_NODE_STAT_ITEMS ,MEMCG_SOCK ,MEMCG_PERCPU_B , +MEMCG_VMALLOC , MEMCG_NR_STAT, };
@@ -992,6 +993,21 @@ static inline void mod_memcg_state(struclocal_irq_restore (flags); } +static inline void mod_memcg_page_state (struct page *page, +int idx,int val) +{ +struct mem_cgroup *memcg; + +if (mem_cgroup_disabled ()) +return ; + +rcu_read_lock (); + memcg =page_memcg (page); +if (memcg) +mod_memcg_state (memcg, idx, val); +rcu_read_unlock (); +} +static inline unsigned long memcg_page_state (struct mem_cgroup *memcg,int idx) {return READ_ONCE (memcg->vmstats.state[idx]);
@@ -1447,6 +1463,11 @@ static inline void mod_memcg_state(struc { } +static inline void mod_memcg_page_state (struct page *page, +int idx,int val) +{ +} +static inline unsigned long memcg_page_state (struct mem_cgroup *memcg,int idx) {return 0 ;
mm/memcontrol.c renamed from mm/memcontrol.c~memcg-add-per-memcg-vmalloc-stat +1 -0
@@ -1375,6 +1375,7 @@ static const struct memory_stat memory_s { "pagetables", NR_PAGETABLE }, { "percpu", MEMCG_PERCPU_B }, { "sock", MEMCG_SOCK }, + { "vmalloc", MEMCG_VMALLOC }, { "shmem", NR_SHMEM }, { "file_mapped", NR_FILE_MAPPED }, { "file_dirty", NR_FILE_DIRTY },
mm/vmalloc.c renamed from mm/vmalloc.c~memcg-add-per-memcg-vmalloc-stat +11 -2
@@ -31,6 +31,7 @@#include <linux/kmemleak.h> #include <linux/atomic.h> #include <linux/compiler.h> +#include <linux/memcontrol.h> #include <linux/llist.h> #include <linux/bitops.h> #include <linux/rbtree_augmented.h>
@@ -2623,12 +2624,13 @@ static void __vunmap(const void *addr, iif (deallocate_pages) {unsigned int page_order =vm_area_page_order (area); -int i; +int i, step =1U << page_order; -for (i =0 ; i < area->nr_pages; i +=1U << page_order) { +for (i =0 ; i < area->nr_pages; i += step) {struct page *page = area->pages[i];BUG_ON (!page); +mod_memcg_page_state (page,MEMCG_VMALLOC , -step);__free_pages (page, page_order);cond_resched (); }
@@ -2955,6 +2957,13 @@ static void *__vmalloc_area_node(struct page_order, nr_small_pages, area->pages);atomic_long_add (area->nr_pages, &nr_vmalloc_pages); +if (gfp_mask & __GFP_ACCOUNT) { +int i, step =1U << page_order; + +for (i =0 ; i < area->nr_pages; i += step) +mod_memcg_page_state (area->pages[i],MEMCG_VMALLOC , + step); + }/* * If not enough pages were obtained to accomplish an
_
Andrew Morton
maintainerPATCH 54/146 tools/testing/selftests/vm/userfaultfd.c: use swap() to make code cleaner
In reply to Andrew MortonFrom: chiminghao <[email protected]>
Subject: tools/testing/selftests/vm/userfaultfd.c: use swap() to make code cleaner
Fix the following coccicheck REVIEW:
./tools/testing/selftests/vm/userfaultfd.c:1531:21-22:use swap() to make code cleaner
---
tools/testing/selftests/vm/userfaultfd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
tools/testing/selftests/vm/userfaultfd.c renamed from tools/testing/selftests/vm/userfaultfd.c~selftests-vm-use-swap-to-make-code-cleaner +3 -8
@@ -1417,7 +1417,6 @@ static void userfaultfd_pagemap_test(unsstatic int userfaultfd_stress (void ) {void *area; -char *tmp_area;unsigned long nr;struct uffdio_register uffdio_register;struct uffd_stats uffd_stats[nr_cpus];
@@ -1528,13 +1527,9 @@ static int userfaultfd_stress(void) count_verify[nr], nr);/* prepare next bounce */ - tmp_area = area_src; - area_src = area_dst; - area_dst = tmp_area; - - tmp_area = area_src_alias; - area_src_alias = area_dst_alias; - area_dst_alias = tmp_area; +swap (area_src, area_dst); + +swap (area_src_alias, area_dst_alias);uffd_stats_report (uffd_stats, nr_cpus); }
_
Andrew Morton
maintainerPATCH 55/146 mm: remove redundant check about FAULT_FLAG_ALLOW_RETRY bit
In reply to Andrew MortonFrom: Qi Zheng <[email protected]>
Subject: mm: remove redundant check about FAULT_FLAG_ALLOW_RETRY bit
Since commit 4064b9827063 ("mm: allow VM_FAULT_RETRY for multiple times") allowed VM_FAULT_RETRY for multiple times, the FAULT_FLAG_ALLOW_RETRY bit of fault_flag will not be changed in the page fault path, so the following check is no longer needed:
flags & FAULT_FLAG_ALLOW_RETRY
So just remove it.
---
arch/alpha/mm/fault.c | 16 +++++++--------- arch/arc/mm/fault.c | 3 +-- arch/arm/mm/fault.c | 2 +- arch/arm64/mm/fault.c | 6 ++---- arch/hexagon/mm/vm_fault.c | 8 +++----- arch/ia64/mm/fault.c | 16 +++++++--------- arch/m68k/mm/fault.c | 18 ++++++++---------- arch/microblaze/mm/fault.c | 18 ++++++++---------- arch/mips/mm/fault.c | 19 +++++++++---------- arch/nds32/mm/fault.c | 16 +++++++--------- arch/nios2/mm/fault.c | 18 ++++++++---------- arch/openrisc/mm/fault.c | 18 ++++++++---------- arch/parisc/mm/fault.c | 18 ++++++++---------- arch/powerpc/mm/fault.c | 6 ++---- arch/riscv/mm/fault.c | 2 +- arch/s390/mm/fault.c | 28 ++++++++++++++-------------- arch/sh/mm/fault.c | 18 ++++++++---------- arch/sparc/mm/fault_32.c | 16 +++++++--------- arch/sparc/mm/fault_64.c | 16 +++++++--------- arch/um/kernel/trap.c | 8 +++----- arch/x86/mm/fault.c | 3 +-- arch/xtensa/mm/fault.c | 17 ++++++++--------- 22 files changed, 128 insertions(+), 162 deletions(-)
arch/alpha/mm/fault.c renamed from arch/alpha/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +7 -9
@@ -165,17 +165,15 @@ retry:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/arc/mm/fault.c renamed from arch/arc/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +1 -2
@@ -149,8 +149,7 @@ retry:/* * Fault retry nuances, mmap_lock already relinquished by core mm */ -if (unlikely ((fault &VM_FAULT_RETRY ) && - (flags &FAULT_FLAG_ALLOW_RETRY ))) { +if (unlikely (fault &VM_FAULT_RETRY )) { flags |=FAULT_FLAG_TRIED ; goto retry; }
arch/arm64/mm/fault.c renamed from arch/arm64/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +2 -4
@@ -606,10 +606,8 @@ retry: }if (fault &VM_FAULT_RETRY ) { -if (mm_flags &FAULT_FLAG_ALLOW_RETRY ) { - mm_flags |=FAULT_FLAG_TRIED ; - goto retry; - } + mm_flags |=FAULT_FLAG_TRIED ; + goto retry; }mmap_read_unlock (mm);
arch/arm/mm/fault.c renamed from arch/arm/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +1 -1
@@ -312,7 +312,7 @@ retry:return 0 ; } -if (!(fault &VM_FAULT_ERROR ) && flags &FAULT_FLAG_ALLOW_RETRY ) { +if (!(fault &VM_FAULT_ERROR )) {if (fault &VM_FAULT_RETRY ) { flags |=FAULT_FLAG_TRIED ; goto retry;
arch/hexagon/mm/vm_fault.c renamed from arch/hexagon/mm/vm_fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +3 -5
@@ -98,11 +98,9 @@ good_area:/* The most common case -- we are done. */ if (likely (!(fault &VM_FAULT_ERROR ))) { -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; - goto retry; - } +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; + goto retry; }mmap_read_unlock (mm);
arch/ia64/mm/fault.c renamed from arch/ia64/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +7 -9
@@ -156,17 +156,15 @@ retry:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/m68k/mm/fault.c renamed from arch/m68k/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +8 -10
@@ -153,18 +153,16 @@ good_area:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* -* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* +* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/microblaze/mm/fault.c renamed from arch/microblaze/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +8 -10
@@ -232,18 +232,16 @@ good_area:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* -* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* +* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/mips/mm/fault.c renamed from arch/mips/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +9 -10
@@ -171,18 +171,17 @@ good_area: goto do_sigbus;BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; -/* -* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; - goto retry; - } +/* +* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ + + goto retry; }mmap_read_unlock (mm);
arch/nds32/mm/fault.c renamed from arch/nds32/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +7 -9
@@ -230,16 +230,14 @@ good_area: goto bad_area; } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ - goto retry; - } +/* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ + goto retry; }mmap_read_unlock (mm);
arch/nios2/mm/fault.c renamed from arch/nios2/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +8 -10
@@ -149,18 +149,16 @@ good_area:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* -* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* +* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/openrisc/mm/fault.c renamed from arch/openrisc/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +8 -10
@@ -177,18 +177,16 @@ good_area:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -/*RGD modeled on Cris */ -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +/*RGD modeled on Cris */ +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/parisc/mm/fault.c renamed from arch/parisc/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +8 -10
@@ -324,16 +324,14 @@ good_area: goto bad_area;BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { -/* -* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ - flags |=FAULT_FLAG_TRIED ; - goto retry; - } +if (fault &VM_FAULT_RETRY ) { +/* +* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ + flags |=FAULT_FLAG_TRIED ; + goto retry; }mmap_read_unlock (mm);return ;
arch/powerpc/mm/fault.c renamed from arch/powerpc/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +2 -4
@@ -516,10 +516,8 @@ retry:* case. */ if (unlikely (fault &VM_FAULT_RETRY )) { -if (flags &FAULT_FLAG_ALLOW_RETRY ) { - flags |=FAULT_FLAG_TRIED ; - goto retry; - } + flags |=FAULT_FLAG_TRIED ; + goto retry; }mmap_read_unlock (current->mm);
arch/riscv/mm/fault.c renamed from arch/riscv/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +1 -1
@@ -330,7 +330,7 @@ good_area:if (fault_signal_pending (fault, regs))return ; -if (unlikely ((fault &VM_FAULT_RETRY ) && (flags &FAULT_FLAG_ALLOW_RETRY ))) { +if (unlikely (fault &VM_FAULT_RETRY )) { flags |=FAULT_FLAG_TRIED ;/*
arch/s390/mm/fault.c renamed from arch/s390/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +14 -14
@@ -452,21 +452,21 @@ retry:if (unlikely (fault &VM_FAULT_ERROR )) goto out_up; -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { -if (IS_ENABLED (CONFIG_PGSTE ) && gmap && - (flags &FAULT_FLAG_RETRY_NOWAIT )) { -/* FAULT_FLAG_RETRY_NOWAIT has been set, -* mmap_lock has not been released */ - current->thread.gmap_pfault =1 ; - fault =VM_FAULT_PFAULT ; - goto out_up; - } - flags &= ~FAULT_FLAG_RETRY_NOWAIT ; - flags |=FAULT_FLAG_TRIED ; -mmap_read_lock (mm); - goto retry; +if (fault &VM_FAULT_RETRY ) { +if (IS_ENABLED (CONFIG_PGSTE ) && gmap && + (flags &FAULT_FLAG_RETRY_NOWAIT )) { +/* +* FAULT_FLAG_RETRY_NOWAIT has been set, mmap_lock has +* not been released +*/ + current->thread.gmap_pfault =1 ; + fault =VM_FAULT_PFAULT ; + goto out_up; } + flags &= ~FAULT_FLAG_RETRY_NOWAIT ; + flags |=FAULT_FLAG_TRIED ; +mmap_read_lock (mm); + goto retry; }if (IS_ENABLED (CONFIG_PGSTE ) && gmap) { address = __gmap_link(gmap, current->thread.gmap_addr,
arch/sh/mm/fault.c renamed from arch/sh/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +8 -10
@@ -485,17 +485,15 @@ good_area:if (mm_fault_error (regs, error_code, address, fault))return ; -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* -* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ - goto retry; - } +/* +* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ + goto retry; }mmap_read_unlock (mm);
arch/sparc/mm/fault_32.c renamed from arch/sparc/mm/fault_32.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +7 -9
@@ -200,17 +200,15 @@ good_area:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/sparc/mm/fault_64.c renamed from arch/sparc/mm/fault_64.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +7 -9
@@ -437,17 +437,15 @@ good_area:BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; -/* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +/* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ - goto retry; - } + goto retry; }mmap_read_unlock (mm);
arch/um/kernel/trap.c renamed from arch/um/kernel/trap.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +3 -5
@@ -87,12 +87,10 @@ good_area: }BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; - goto retry; - } + goto retry; } pmd =pmd_off (mm, address);
arch/x86/mm/fault.c renamed from arch/x86/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +1 -2
@@ -1413,8 +1413,7 @@ good_area:* and if there is a fatal signal pending there is no guarantee * that we made any progress. Handle this case first. */ -if (unlikely ((fault &VM_FAULT_RETRY ) && - (flags &FAULT_FLAG_ALLOW_RETRY ))) { +if (unlikely (fault &VM_FAULT_RETRY )) { flags |=FAULT_FLAG_TRIED ; goto retry; }
arch/xtensa/mm/fault.c renamed from arch/xtensa/mm/fault.c~mm-remove-redundant-check-about-fault_flag_allow_retry-bit +8 -9
@@ -127,17 +127,16 @@ good_area: goto do_sigbus;BUG (); } -if (flags &FAULT_FLAG_ALLOW_RETRY ) { -if (fault &VM_FAULT_RETRY ) { - flags |=FAULT_FLAG_TRIED ; -/* No need to mmap_read_unlock(mm) as we would -* have already released it in __lock_page_or_retry -* in mm/filemap.c. -*/ +if (fault &VM_FAULT_RETRY ) { + flags |=FAULT_FLAG_TRIED ; - goto retry; - } +/* No need to mmap_read_unlock(mm) as we would +* have already released it in __lock_page_or_retry +* in mm/filemap.c. +*/ + + goto retry; }mmap_read_unlock (mm);
_
Andrew Morton
maintainerPATCH 56/146 mm: rearrange madvise code to allow for reuse
In reply to Andrew MortonFrom: Colin Cross <[email protected]>
Subject: mm: rearrange madvise code to allow for reuse
Patch series "mm: rearrange madvise code to allow for reuse", v11.
Speed up fork() by up to 40% by refcounting the anon vma name field.
I checked the image sizes with allnoconfig builds:
unpatched Linus' ToT
text data bss dec hex filename
1324759 32 73928 1398719 1557bf vmlinux
After the first patch is applied (madvise refactoring)
text data bss dec hex filename
1322346 32 73928 1396306 154e52 vmlinux
Quoted 1 line
Quoted 1 line
Quoted 1 line
2413 bytes decrease vs ToT <<<
After all patches applied with CONFIG_ANON_VMA_NAME=n
text data bss dec hex filename
1322337 32 73928 1396297 154e49 vmlinux
Quoted 1 line
Quoted 1 line
Quoted 1 line
2422 bytes decrease vs ToT <<<
After all patches applied with CONFIG_ANON_VMA_NAME=y
text data bss dec hex filename
1325228 32 73928 1399188 155994 vmlinux
Quoted 1 line
Quoted 1 line
Quoted 1 line
469 bytes increase vs ToT <<<
This patch (of 3):
Refactor the madvise syscall to allow for parts of it to be reused by a prctl syscall that affects vmas.
Move the code that walks vmas in a virtual address range into a function that takes a function pointer as a parameter. The only caller for now is sys_madvise, which uses it to call madvise_vma_behavior on each vma, but the next patch will add an additional caller.
Move handling all vma behaviors inside madvise_behavior, and rename it to madvise_vma_behavior.
Move the code that updates the flags on a vma, including splitting or merging the vma as necessary, into a new function called madvise_update_vma. The next patch will add support for updating a new anon_name field as well.
---
mm/madvise.c | 338 +++++++++++++++++++++++++------------------------ 1 file changed, 178 insertions(+), 160 deletions(-)
mm/madvise.c renamed from mm/madvise.c~mm-rearrange-madvise-code-to-allow-for-reuse +178 -160
@@ -63,76 +63,20 @@ static int madvise_need_mmap_write(int b }/* -* We can potentially split a vm area into separate -* areas, each area with its own behavior. +* Update the vm_flags on region of a vma, splitting it or merging it as +* necessary. Must be called with mmap_sem held for writing; */ -static long madvise_behavior (struct vm_area_struct *vma, -struct vm_area_struct **prev, -unsigned long start,unsigned long end,int behavior) +static int madvise_update_vma (struct vm_area_struct *vma, +struct vm_area_struct **prev,unsigned long start, +unsigned long end,unsigned long new_flags) {struct mm_struct *mm = vma->vm_mm; -int error =0 ; +int error;pgoff_t pgoff; -unsigned long new_flags = vma->vm_flags; - -switch (behavior) { -case MADV_NORMAL : - new_flags = new_flags & ~VM_RAND_READ & ~VM_SEQ_READ ; -break ; -case MADV_SEQUENTIAL : - new_flags = (new_flags & ~VM_RAND_READ ) |VM_SEQ_READ ; -break ; -case MADV_RANDOM : - new_flags = (new_flags & ~VM_SEQ_READ ) |VM_RAND_READ ; -break ; -case MADV_DONTFORK : - new_flags |=VM_DONTCOPY ; -break ; -case MADV_DOFORK : -if (vma->vm_flags &VM_IO ) { - error = -EINVAL ; - goto out; - } - new_flags &= ~VM_DONTCOPY ; -break ; -case MADV_WIPEONFORK : -/* MADV_WIPEONFORK is only supported on anonymous memory. */ -if (vma->vm_file || vma->vm_flags &VM_SHARED ) { - error = -EINVAL ; - goto out; - } - new_flags |=VM_WIPEONFORK ; -break ; -case MADV_KEEPONFORK : - new_flags &= ~VM_WIPEONFORK ; -break ; -case MADV_DONTDUMP : - new_flags |=VM_DONTDUMP ; -break ; -case MADV_DODUMP : -if (!is_vm_hugetlb_page (vma) && new_flags &VM_SPECIAL ) { - error = -EINVAL ; - goto out; - } - new_flags &= ~VM_DONTDUMP ; -break ; -case MADV_MERGEABLE : -case MADV_UNMERGEABLE : - error =ksm_madvise (vma, start, end, behavior, &new_flags); -if (error) - goto out_convert_errno; -break ; -case MADV_HUGEPAGE : -case MADV_NOHUGEPAGE : - error =hugepage_madvise (vma, &new_flags, behavior); -if (error) - goto out_convert_errno; -break ; - }if (new_flags == vma->vm_flags) { *prev = vma; - goto out; +return 0 ; } pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT );
@@ -147,23 +91,19 @@ static long madvise_behavior(struct vm_a *prev = vma;if (start != vma->vm_start) { -if (unlikely (mm->map_count >= sysctl_max_map_count)) { - error = -ENOMEM ; - goto out; - } +if (unlikely (mm->map_count >= sysctl_max_map_count)) +return -ENOMEM ; error =__split_vma (mm, vma, start,1 );if (error) - goto out_convert_errno; +return error; }if (end != vma->vm_end) { -if (unlikely (mm->map_count >= sysctl_max_map_count)) { - error = -ENOMEM ; - goto out; - } +if (unlikely (mm->map_count >= sysctl_max_map_count)) +return -ENOMEM ; error =__split_vma (mm, vma, end,0 );if (error) - goto out_convert_errno; +return error; } success:
@@ -172,15 +112,7 @@ success:*/ vma->vm_flags = new_flags; -out_convert_errno: -/* -* madvise() returns EAGAIN if kernel resources, such as -* slab, are temporarily unavailable. -*/ -if (error == -ENOMEM ) - error = -EAGAIN ; -out: -return error; +return 0 ; }#ifdef CONFIG_SWAP
@@ -930,6 +862,94 @@ static long madvise_remove(struct vm_arereturn error; } +/* +* Apply an madvise behavior to a region of a vma. madvise_update_vma +* will handle splitting a vm area into separate areas, each area with its own +* behavior. +*/ +static int madvise_vma_behavior (struct vm_area_struct *vma, +struct vm_area_struct **prev, +unsigned long start,unsigned long end, +unsigned long behavior) +{ +int error; +unsigned long new_flags = vma->vm_flags; + +switch (behavior) { +case MADV_REMOVE : +return madvise_remove (vma, prev, start, end); +case MADV_WILLNEED : +return madvise_willneed (vma, prev, start, end); +case MADV_COLD : +return madvise_cold (vma, prev, start, end); +case MADV_PAGEOUT : +return madvise_pageout (vma, prev, start, end); +case MADV_FREE : +case MADV_DONTNEED : +return madvise_dontneed_free (vma, prev, start, end, behavior); +case MADV_POPULATE_READ : +case MADV_POPULATE_WRITE : +return madvise_populate (vma, prev, start, end, behavior); +case MADV_NORMAL : + new_flags = new_flags & ~VM_RAND_READ & ~VM_SEQ_READ ; +break ; +case MADV_SEQUENTIAL : + new_flags = (new_flags & ~VM_RAND_READ ) |VM_SEQ_READ ; +break ; +case MADV_RANDOM : + new_flags = (new_flags & ~VM_SEQ_READ ) |VM_RAND_READ ; +break ; +case MADV_DONTFORK : + new_flags |=VM_DONTCOPY ; +break ; +case MADV_DOFORK : +if (vma->vm_flags &VM_IO ) +return -EINVAL ; + new_flags &= ~VM_DONTCOPY ; +break ; +case MADV_WIPEONFORK : +/* MADV_WIPEONFORK is only supported on anonymous memory. */ +if (vma->vm_file || vma->vm_flags &VM_SHARED ) +return -EINVAL ; + new_flags |=VM_WIPEONFORK ; +break ; +case MADV_KEEPONFORK : + new_flags &= ~VM_WIPEONFORK ; +break ; +case MADV_DONTDUMP : + new_flags |=VM_DONTDUMP ; +break ; +case MADV_DODUMP : +if (!is_vm_hugetlb_page (vma) && new_flags &VM_SPECIAL ) +return -EINVAL ; + new_flags &= ~VM_DONTDUMP ; +break ; +case MADV_MERGEABLE : +case MADV_UNMERGEABLE : + error =ksm_madvise (vma, start, end, behavior, &new_flags); +if (error) + goto out; +break ; +case MADV_HUGEPAGE : +case MADV_NOHUGEPAGE : + error =hugepage_madvise (vma, &new_flags, behavior); +if (error) + goto out; +break ; + } + + error =madvise_update_vma (vma, prev, start, end, new_flags); + +out: +/* +* madvise() returns EAGAIN if kernel resources, such as +* slab, are temporarily unavailable. +*/ +if (error == -ENOMEM ) + error = -EAGAIN ; +return error; +} +#ifdef CONFIG_MEMORY_FAILURE /* * Error injection support for memory error handling.
@@ -978,30 +998,6 @@ static int madvise_inject_error(int beha }#endif -static long -madvise_vma (struct vm_area_struct *vma,struct vm_area_struct **prev, -unsigned long start,unsigned long end,int behavior) -{ -switch (behavior) { -case MADV_REMOVE : -return madvise_remove (vma, prev, start, end); -case MADV_WILLNEED : -return madvise_willneed (vma, prev, start, end); -case MADV_COLD : -return madvise_cold (vma, prev, start, end); -case MADV_PAGEOUT : -return madvise_pageout (vma, prev, start, end); -case MADV_FREE : -case MADV_DONTNEED : -return madvise_dontneed_free (vma, prev, start, end, behavior); -case MADV_POPULATE_READ : -case MADV_POPULATE_WRITE : -return madvise_populate (vma, prev, start, end, behavior); -default : -return madvise_behavior (vma, prev, start, end, behavior); - } -} -static bool madvise_behavior_valid (int behavior) {
@@ -1056,6 +1052,73 @@ process_madvise_behavior_valid(int behav }/* +* Walk the vmas in range [start,end), and call the visit function on each one. +* The visit function will get start and end parameters that cover the overlap +* between the current vma and the original range. Any unmapped regions in the +* original range will result in this function returning -ENOMEM while still +* calling the visit function on all of the existing vmas in the range. +* Must be called with the mmap_lock held for reading or writing. +*/ +static +int madvise_walk_vmas (struct mm_struct *mm,unsigned long start, +unsigned long end,unsigned long arg, +int (*visit)(struct vm_area_struct *vma, +struct vm_area_struct **prev,unsigned long start, +unsigned long end,unsigned long arg)) +{ +struct vm_area_struct *vma; +struct vm_area_struct *prev; +unsigned long tmp; +int unmapped_error =0 ; + +/* +* If the interval [start,end) covers some unmapped address +* ranges, just ignore them, but return -ENOMEM at the end. +* - different from the way of handling in mlock etc. +*/ + vma =find_vma_prev (mm, start, &prev); +if (vma && start > vma->vm_start) + prev = vma; + +for (;;) { +int error; + +/* Still start < end. */ +if (!vma) +return -ENOMEM ; + +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + unmapped_error = -ENOMEM ; + start = vma->vm_start; +if (start >= end) +break ; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =visit (vma, &prev, start, tmp, arg); +if (error) +return error; + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; +if (prev) + vma = prev->vm_next; +else /* madvise_remove dropped mmap_lock */ + vma =find_vma (mm, start); + } + +return unmapped_error; +} + +/* * The madvise(2) system call. * * Applications can use madvise() to advise the kernel how it should
@@ -1127,10 +1190,8 @@ process_madvise_behavior_valid(int behav*/ int do_madvise (struct mm_struct *mm,unsigned long start,size_t len_in,int behavior) { -unsigned long end, tmp; -struct vm_area_struct *vma, *prev; -int unmapped_error =0 ; -int error = -EINVAL ; +unsigned long end; +int error;int write;size_t len;struct blk_plug plug;
@@ -1138,23 +1199,22 @@ int do_madvise(struct mm_struct *mm, uns start =untagged_addr (start);if (!madvise_behavior_valid (behavior)) -return error; +return -EINVAL ;if (!PAGE_ALIGNED (start)) -return error; +return -EINVAL ; len =PAGE_ALIGN (len_in);/* Check to see whether len was rounded up from small -ve to zero */ if (len_in && !len) -return error; +return -EINVAL ; end = start + len;if (end < start) -return error; +return -EINVAL ; - error =0 ;if (end == start) -return error; +return 0 ;#ifdef CONFIG_MEMORY_FAILURE if (behavior ==MADV_HWPOISON || behavior ==MADV_SOFT_OFFLINE )
@@ -1169,51 +1229,9 @@ int do_madvise(struct mm_struct *mm, unsmmap_read_lock (mm); } -/* -* If the interval [start,end) covers some unmapped address -* ranges, just ignore them, but return -ENOMEM at the end. -* - different from the way of handling in mlock etc. -*/ - vma =find_vma_prev (mm, start, &prev); -if (vma && start > vma->vm_start) - prev = vma; -blk_start_plug (&plug); -for (;;) { -/* Still start < end. */ - error = -ENOMEM ; -if (!vma) - goto out; - -/* Here start < (end|vma->vm_end). */ -if (start < vma->vm_start) { - unmapped_error = -ENOMEM ; - start = vma->vm_start; -if (start >= end) - goto out; - } - -/* Here vma->vm_start <= start < (end|vma->vm_end) */ - tmp = vma->vm_end; -if (end < tmp) - tmp = end; - -/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ - error =madvise_vma (vma, &prev, start, tmp, behavior); -if (error) - goto out; - start = tmp; -if (prev && start < prev->vm_end) - start = prev->vm_end; - error = unmapped_error; -if (start >= end) - goto out; -if (prev) - vma = prev->vm_next; -else /* madvise_remove dropped mmap_lock */ - vma =find_vma (mm, start); - } -out: + error =madvise_walk_vmas (mm, start, end, behavior, + madvise_vma_behavior);blk_finish_plug (&plug);if (write)mmap_write_unlock (mm);
_
Re: [patch 056/146] mm: rearrange madvise code to allow for reuse
Quoted 2 lines · Andrew Morton
Speed up
fork()by up to 40% by refcounting the anon vma name field.
What?
No.
This doesn't speed up anything at all.
The refcounting of the anon-vma name field avoids a 40% regression that comes from adding the field in the first place, but this commit message makes it sound like this series is speeding up fork() by 40%.
I don't mind the series, but I absolutely mind these kinds of horribly misleading commit messages.
This is literally the first commit in the series - and the series in no way improves performance by 40% in the end, it just first makes it worse, and then fixes the regression.
I can speed up any function by a thousand percent - if I'm just allowed to make it horribly slow first, and only count the final speedup win when I remove the overhead of the garbage I added.
Linus
Suren Baghdasaryan
reviewerRe: [patch 056/146] mm: rearrange madvise code to allow for reuse
Quoted 19 lines · Linus Torvalds
Quoted 2 lines · Andrew Morton
Speed up
fork()by up to 40% by refcounting the anon vma name field.What?
No.
This doesn't speed up anything at all.
The refcounting of the anon-vma name field avoids a 40% regression that comes from adding the field in the first place, but this commit message makes it sound like this series is speeding up
fork()by 40%.I don't mind the series, but I absolutely mind these kinds of horribly misleading commit messages.
Quoted 7 lines
This is literally the first commit in the series - and the series in no way improves performance by 40% in the end, it just first makes it worse, and then fixes the regression.
I can speed up any function by a thousand percent - if I'm just allowed to make it horribly slow first, and only count the final speedup win when I remove the overhead of the garbage I added.
Thanks for fixing it! I see in your tree it is changed to:
"Avoid performance regression of the new anon vma name field refcounting it." Please let me know if anything else is left to be done here.
Quoted 2 lines
Linus
Andrew Morton
maintainerPATCH 57/146 mm: add a field to store names for private anonymous memory
In reply to Andrew MortonFrom: Colin Cross <[email protected]>
Subject: mm: add a field to store names for private anonymous memory
In many userspace applications, and especially in VM based applications like Android uses heavily, there are multiple different allocators in use. At a minimum there is libc malloc and the stack, and in many cases there are libc malloc, the stack, direct syscalls to mmap anonymous memory, and multiple VM heaps (one for small objects, one for big objects, etc.). Each of these layers usually has its own tools to inspect its usage; malloc by compiling a debug version, the VM through heap inspection tools, and for direct syscalls there is usually no way to track them.
On Android we heavily use a set of tools that use an extended version of the logic covered in Documentation/vm/pagemap.txt to walk all pages mapped in userspace and slice their usage by process, shared (COW) vs. unique mappings, backing, etc. This can account for real physical memory usage even in cases like fork without exec (which Android uses heavily to share as many private COW pages as possible between processes), Kernel SamePage Merging, and clean zero pages. It produces a measurement of the pages that only exist in that process (USS, for unique), and a measurement of the physical memory usage of that process with the cost of shared pages being evenly split between processes that share them (PSS).
If all anonymous memory is indistinguishable then figuring out the real physical memory usage (PSS) of each heap requires either a pagemap walking tool that can understand the heap debugging of every layer, or for every layer's heap debugging tools to implement the pagemap walking logic, in which case it is hard to get a consistent view of memory across the whole system.
Tracking the information in userspace leads to all sorts of problems. It either needs to be stored inside the process, which means every process has to have an API to export its current heap information upon request, or it has to be stored externally in a filesystem that somebody needs to clean up on crashes. It needs to be readable while the process is still running, so it has to have some sort of synchronization with every layer of userspace. Efficiently tracking the ranges requires reimplementing something like the kernel vma trees, and linking to it from every layer of userspace. It requires more memory, more syscalls, more runtime cost, and more complexity to separately track regions that the kernel is already tracking.
This patch adds a field to /proc/pid/maps and /proc/pid/smaps to show a userspace-provided name for anonymous vmas. The names of named anonymous vmas are shown in /proc/pid/maps and /proc/pid/smaps as [anon:<name>].
Userspace can set the name for a region of memory by calling prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, start, len, (unsigned long)name); Setting the name to NULL clears it. The name length limit is 80 bytes including NUL-terminator and is checked to contain only printable ascii characters (including space), except '[',']','\','$' and '`'. Ascii strings are being used to have a descriptive identifiers for vmas, which can be understood by the users reading /proc/pid/maps or /proc/pid/smaps. Names can be standardized for a given system and they can include some variable parts such as the name of the allocator or a library, tid of the thread using it, etc.
The name is stored in a pointer in the shared union in vm_area_struct that points to a null terminated string. Anonymous vmas with the same name (equivalent strings) and are otherwise mergeable will be merged. The name pointers are not shared between vmas even if they contain the same name. The name pointer is stored in a union with fields that are only used on file-backed mappings, so it does not increase memory usage.
CONFIG_ANON_VMA_NAME kernel configuration is introduced to enable this feature. It keeps the feature disabled by default to prevent any additional memory overhead and to avoid confusing procfs parsers on systems which are not ready to support named anonymous vmas.
The patch is based on the original patch developed by Colin Cross, more specifically on its latest version [1] posted upstream by Sumit Semwal. It used a userspace pointer to store vma names. In that design, name pointers could be shared between vmas. However during the last upstreaming attempt, Kees Cook raised concerns [2] about this approach and suggested to copy the name into kernel memory space, perform validity checks [3] and store as a string referenced from vm_area_struct. One big concern is about fork() performance which would need to strdup anonymous vma names. Dave Hansen suggested experimenting with worst-case scenario of forking a process with 64k vmas having longest possible names [4]. I ran this experiment on an ARM64 Android device and recorded a worst-case regression of almost 40% when forking such a process. This regression is addressed in the followup patch which replaces the pointer to a name with a refcounted structure that allows sharing the name pointer between vmas of the same name. Instead of duplicating the string during fork() or when splitting a vma it increments the refcount.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/202009031031.D32EF57ED@keescook/
[3] https://lore.kernel.org/linux-mm/202009031022.3834F692@keescook/
[4] https://lore.kernel.org/linux-mm/[email protected]/
Changes for prctl(2) manual page (in the options section):
PR_SET_VMA
Sets an attribute specified in arg2 for virtual memory areas starting from the address specified in arg3 and spanning the
size specified in arg4. arg5 specifies the value of the attribute
to be set. Note that assigning an attribute to a virtual memory area might prevent it from being merged with adjacent virtual memory areas due to the difference in that attribute's value.
Currently, arg2 must be one of:
PR_SET_VMA_ANON_NAME
Set a name for anonymous virtual memory areas. arg5 should be a pointer to a null-terminated string containing the name. The name length including null byte cannot exceed 80 bytes. If arg5 is NULL, the name of the appropriate anonymous virtual memory areas will be reset. The name can contain only printable ascii characters (including space), except '[',']','\','$' and '`'.
This feature is available only if the kernel is built with the CONFIG_ANON_VMA_NAME option enabled.
[surenb: rebased over v5.15-rc6, replaced userpointer with a kernel copy,
added input sanitization and CONFIG_ANON_VMA_NAME config. The bulk of the work here was done by Colin Cross, therefore, with his permission, keeping him as the author]
---
Documentation/filesystems/proc.rst | 6 - fs/proc/task_mmu.c | 12 ++ fs/userfaultfd.c | 7 - include/linux/mm.h | 13 ++ include/linux/mm_types.h | 64 ++++++++++++- include/uapi/linux/prctl.h | 3 kernel/fork.c | 2 kernel/sys.c | 63 +++++++++++++ mm/Kconfig | 14 ++ mm/madvise.c | 129 ++++++++++++++++++++++++++- mm/mempolicy.c | 3 mm/mlock.c | 2 mm/mmap.c | 38 ++++--- mm/mprotect.c | 2 14 files changed, 324 insertions(+), 34 deletions(-)
Documentation/filesystems/proc.rst renamed from Documentation/filesystems/proc.rst~mm-add-a-field-to-store-names-for-private-anonymous-memory +4 -2
@@ -426,12 +426,14 @@ with the memory region, as the case woul The "pathname" shows the name associated file for this mapping. If the mapping is not associated with a file: - ======= ==================================== + ============= ==================================== [heap] the heap of the program [stack] the stack of the main process [vdso] the "virtual dynamic shared object", the kernel system call handler - ======= ==================================== + [anon:<name>] an anonymous mapping that has been + named by userspace + ============= ==================================== or if empty, the mapping is anonymous.
fs/proc/task_mmu.c renamed from fs/proc/task_mmu.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +11 -1
@@ -308,6 +308,8 @@ show_map_vma(struct seq_file *m, struct name =arch_vma_name (vma);if (!name) { +const char *anon_name; +if (!mm) { name ="[vdso]" ; goto done;
@@ -319,8 +321,16 @@ show_map_vma(struct seq_file *m, struct goto done; } -if (is_stack (vma)) +if (is_stack (vma)) { name ="[stack]" ; + goto done; + } + + anon_name =vma_anon_name (vma); +if (anon_name) { +seq_pad (m,' ' ); +seq_printf (m,"[anon:%s]" , anon_name); + } } done:
fs/userfaultfd.c renamed from fs/userfaultfd.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +4 -3
@@ -877,7 +877,7 @@ static int userfaultfd_release(struct in new_flags, vma->anon_vma, vma->vm_file, vma->vm_pgoff,vma_policy (vma), - NULL_VM_UFFD_CTX); + NULL_VM_UFFD_CTX, vma_anon_name(vma));if (prev) vma = prev;else
@@ -1436,7 +1436,8 @@ static int userfaultfd_register(struct u prev =vma_merge (mm, prev, start, vma_end, new_flags, vma->anon_vma, vma->vm_file, vma->vm_pgoff,vma_policy (vma), - ((struct vm_userfaultfd_ctx ){ ctx })); + ((struct vm_userfaultfd_ctx ){ ctx }), +vma_anon_name (vma));if (prev) { vma = prev; goto next;
@@ -1613,7 +1614,7 @@ static int userfaultfd_unregister(struct prev =vma_merge (mm, prev, start, vma_end, new_flags, vma->anon_vma, vma->vm_file, vma->vm_pgoff,vma_policy (vma), -NULL_VM_UFFD_CTX ); +NULL_VM_UFFD_CTX ,vma_anon_name (vma));if (prev) { vma = prev; goto next;
include/linux/mm.h renamed from include/linux/mm.h~mm-add-a-field-to-store-names-for-private-anonymous-memory +12 -1
@@ -2658,7 +2658,7 @@ static inline int vma_adjust(struct vm_aextern struct vm_area_struct *vma_merge (struct mm_struct *,struct vm_area_struct *prev,unsigned long addr,unsigned long end,unsigned long vm_flags,struct anon_vma *,struct file *,pgoff_t , -struct mempolicy *,struct vm_userfaultfd_ctx ); +struct mempolicy *,struct vm_userfaultfd_ctx ,const char *);extern struct anon_vma *find_mergeable_anon_vma (struct vm_area_struct *);extern int __split_vma (struct mm_struct *,struct vm_area_struct *,unsigned long addr,int new_below);
@@ -3391,5 +3391,16 @@ static inline int seal_check_future_writreturn 0 ; } +#ifdef CONFIG_ANON_VMA_NAME +int madvise_set_anon_name (struct mm_struct *mm,unsigned long start, +unsigned long len_in,const char *name); +#else +static inline int +madvise_set_anon_name (struct mm_struct *mm,unsigned long start, +unsigned long len_in,const char *name) { +return 0 ; +} +#endif +#endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */
include/linux/mm_types.h renamed from include/linux/mm_types.h~mm-add-a-field-to-store-names-for-private-anonymous-memory +60 -4
75 lines not on this page · raw message
include/uapi/linux/prctl.h renamed from include/uapi/linux/prctl.h~mm-add-a-field-to-store-names-for-private-anonymous-memory +3 -0
@@ -272,4 +272,7 @@ struct prctl_mm_map {# define PR_SCHED_CORE_SCOPE_THREAD_GROUP 1# define PR_SCHED_CORE_SCOPE_PROCESS_GROUP 2 +#define PR_SET_VMA 0x53564d41 +# define PR_SET_VMA_ANON_NAME 0 +#endif /* _LINUX_PRCTL_H */
kernel/fork.c renamed from kernel/fork.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +2 -0
@@ -365,12 +365,14 @@ struct vm_area_struct *vm_area_dup(struc *new =data_race (*orig);INIT_LIST_HEAD (&new->anon_vma_chain); new->vm_next = new->vm_prev =NULL ; +dup_vma_anon_name (orig, new); }return new; }void vm_area_free (struct vm_area_struct *vma) { +free_vma_anon_name (vma);kmem_cache_free (vm_area_cachep, vma); }
kernel/sys.c renamed from kernel/sys.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +63 -0
75 lines not on this page · raw message
mm/Kconfig renamed from mm/Kconfig~mm-add-a-field-to-store-names-for-private-anonymous-memory +14 -0
@@ -900,6 +900,20 @@ config IO_MAPPING config SECRETMEM def_bool ARCH_HAS_SET_DIRECT_MAP && !EMBEDDED +config ANON_VMA_NAME + bool "Anonymous VMA name support" + depends on PROC_FS && ADVISE_SYSCALLS && MMU + + help + Allow naming anonymous virtual memory areas. + + This feature allows assigning names to virtual memory areas. Assigned + names can be later retrieved from /proc/pid/maps and /proc/pid/smaps + and help identifying individual anonymous memory areas. + Assigning a name to anonymous virtual memory area might prevent that + area from being merged with adjacent virtual memory areas due to the + difference in their name. + source "mm/damon/Kconfig" endmenu
mm/madvise.c renamed from mm/madvise.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +125 -4
176 lines not on this page · raw message
mm/mempolicy.c renamed from mm/mempolicy.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +2 -1
@@ -810,7 +810,8 @@ static int mbind_range(struct mm_struct ((vmstart - vma->vm_start) >>PAGE_SHIFT ); prev =vma_merge (mm, prev, vmstart, vmend, vma->vm_flags, vma->anon_vma, vma->vm_file, pgoff, - new_pol, vma->vm_userfaultfd_ctx); + new_pol, vma->vm_userfaultfd_ctx, +vma_anon_name (vma));if (prev) { vma = prev; next = vma->vm_next;
mm/mlock.c renamed from mm/mlock.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +1 -1
@@ -512,7 +512,7 @@ static int mlock_fixup(struct vm_area_st pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, newflags, vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), - vma->vm_userfaultfd_ctx); + vma->vm_userfaultfd_ctx,vma_anon_name (vma));if (*prev) { vma = *prev; goto success;
mm/mmap.c renamed from mm/mmap.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +22 -16
@@ -1029,7 +1029,8 @@ again:*/ static inline int is_mergeable_vma (struct vm_area_struct *vma,struct file *file,unsigned long vm_flags, -struct vm_userfaultfd_ctx vm_userfaultfd_ctx) +struct vm_userfaultfd_ctx vm_userfaultfd_ctx, +const char *anon_name) {/* * VM_SOFTDIRTY should not prevent from VMA merging, if we
@@ -1047,6 +1048,8 @@ static inline int is_mergeable_vma(strucreturn 0 ;if (!is_mergeable_vm_userfaultfd_ctx (vma, vm_userfaultfd_ctx))return 0 ; +if (!is_same_vma_anon_name (vma, anon_name)) +return 0 ;return 1 ; }
@@ -1079,9 +1082,10 @@ static int can_vma_merge_before(struct vm_area_struct *vma, unsigned long vm_flags, struct anon_vma *anon_vma, struct file *file, pgoff_t vm_pgoff, - struct vm_userfaultfd_ctx vm_userfaultfd_ctx) + struct vm_userfaultfd_ctx vm_userfaultfd_ctx, + const char *anon_name) { - if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx) && + if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx, anon_name) && is_mergeable_anon_vma(anon_vma, vma->anon_vma, vma)) { if (vma->vm_pgoff == vm_pgoff) return 1;
@@ -1100,9 +1104,10 @@ static int can_vma_merge_after(struct vm_area_struct *vma, unsigned long vm_flags, struct anon_vma *anon_vma, struct file *file, pgoff_t vm_pgoff, - struct vm_userfaultfd_ctx vm_userfaultfd_ctx) + struct vm_userfaultfd_ctx vm_userfaultfd_ctx, + const char *anon_name) { - if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx) && + if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx, anon_name) && is_mergeable_anon_vma(anon_vma, vma->anon_vma, vma)) { pgoff_t vm_pglen; vm_pglen =vma_pages (vma);
@@ -1113,9 +1118,9 @@ can_vma_merge_after(struct vm_area_struc }/* -* Given a mapping request (addr,end,vm_flags,file,pgoff), figure out -* whether that can be merged with its predecessor or its successor. -* Or both (it neatly fills a hole). +* Given a mapping request (addr,end,vm_flags,file,pgoff,anon_name), +* figure out whether that can be merged with its predecessor or its +* successor. Or both (it neatly fills a hole). * * In most cases - when called for mmap, brk or mremap - [addr,end) is * certain not to be mapped by the time vma_merge is called; but when
@@ -1160,7 +1165,8 @@ struct vm_area_struct *vma_merge(struct unsigned long end, unsigned long vm_flags, struct anon_vma *anon_vma, struct file *file, pgoff_t pgoff, struct mempolicy *policy, - struct vm_userfaultfd_ctx vm_userfaultfd_ctx) + struct vm_userfaultfd_ctx vm_userfaultfd_ctx, + const char *anon_name) { pgoff_t pglen = (end - addr) >> PAGE_SHIFT;struct vm_area_struct *area, *next;
@@ -1190,7 +1196,7 @@ struct vm_area_struct *vma_merge(structmpol_equal (vma_policy (prev), policy) &&can_vma_merge_after (prev, vm_flags, anon_vma, file, pgoff, - vm_userfaultfd_ctx)) { + vm_userfaultfd_ctx, anon_name)) {/* * OK, it can. Can we now merge in the successor as well? */
@@ -1199,7 +1205,7 @@ struct vm_area_struct *vma_merge(structcan_vma_merge_before (next, vm_flags, anon_vma, file, pgoff+pglen, - vm_userfaultfd_ctx) && + vm_userfaultfd_ctx, anon_name) &&is_mergeable_anon_vma (prev->anon_vma, next->anon_vma, NULL)) {/* cases 1, 6 */
@@ -1222,7 +1228,7 @@ struct vm_area_struct *vma_merge(structmpol_equal (policy,vma_policy (next)) &&can_vma_merge_before (next, vm_flags, anon_vma, file, pgoff+pglen, - vm_userfaultfd_ctx)) { + vm_userfaultfd_ctx, anon_name)) { if (prev && addr < prev->vm_end) /* case 4 */ err = __vma_adjust(prev, prev->vm_start, addr, prev->vm_pgoff,NULL , next);
@@ -1754,7 +1760,7 @@ unsigned long mmap_region(struct file *f* Can we just expand an old mapping? */ vma =vma_merge (mm, prev, addr, addr + len, vm_flags, -NULL , file, pgoff,NULL ,NULL_VM_UFFD_CTX ); +NULL , file, pgoff,NULL ,NULL_VM_UFFD_CTX ,NULL );if (vma) goto out;
@@ -1803,7 +1809,7 @@ unsigned long mmap_region(struct file *f*/ if (unlikely (vm_flags != vma->vm_flags && prev)) { merge =vma_merge (mm, prev, vma->vm_start, vma->vm_end, vma->vm_flags, -NULL , vma->vm_file, vma->vm_pgoff,NULL ,NULL_VM_UFFD_CTX ); +NULL , vma->vm_file, vma->vm_pgoff,NULL ,NULL_VM_UFFD_CTX ,NULL );if (merge) {/* ->mmap() can change vma->vm_file and fput the original file. So * fput the vma->vm_file here or we would add an extra fput for file
@@ -3056,7 +3062,7 @@ static int do_brk_flags(unsigned long ad/* Can we just expand an old private anonymous mapping? */ vma =vma_merge (mm, prev, addr, addr + len, flags, -NULL ,NULL , pgoff,NULL ,NULL_VM_UFFD_CTX ); +NULL ,NULL , pgoff,NULL ,NULL_VM_UFFD_CTX ,NULL );if (vma) goto out;
@@ -3249,7 +3255,7 @@ struct vm_area_struct *copy_vma(struct vreturn NULL ;/* should never get here */ new_vma =vma_merge (mm, prev, addr, addr + len, vma->vm_flags, vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), - vma->vm_userfaultfd_ctx); + vma->vm_userfaultfd_ctx,vma_anon_name (vma));if (new_vma) {/* * Source vma may have been merged into new_vma
mm/mprotect.c renamed from mm/mprotect.c~mm-add-a-field-to-store-names-for-private-anonymous-memory +1 -1
@@ -464,7 +464,7 @@ mprotect_fixup(struct vm_area_struct *vm pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *pprev =vma_merge (mm, *pprev, start, end, newflags, vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), - vma->vm_userfaultfd_ctx); + vma->vm_userfaultfd_ctx,vma_anon_name (vma));if (*pprev) { vma = *pprev;VM_WARN_ON ((vma->vm_flags ^ newflags) & ~VM_SOFTDIRTY );
_
Andrew Morton
maintainerPATCH 58/146 mm: add anonymous vma name refcounting
In reply to Andrew MortonFrom: Suren Baghdasaryan <[email protected]>
Subject: mm: add anonymous vma name refcounting
While forking a process with high number (64K) of named anonymous vmas the overhead caused by strdup() is noticeable. Experiments with ARM64 Android device show up to 40% performance regression when forking a process with 64k unpopulated anonymous vmas using the max name lengths vs the same process with the same number of anonymous vmas having no name.
Introduce anon_vma_name refcounted structure to avoid the overhead of copying vma names during fork() and when splitting named anonymous vmas. When a vma is duplicated, instead of copying the name we increment the refcount of this structure. Multiple vmas can point to the same anon_vma_name as long as they increment the refcount. The name member of anon_vma_name structure is assigned at structure allocation time and is never changed. If vma name changes then the refcount of the original structure is dropped, a new anon_vma_name structure is allocated to hold the new name and the vma pointer is updated to point to the new structure.
With this approach the fork() performance regressions is reduced 3-4x times and with usecases using more reasonable number of VMAs (a few thousand) the regressions is not measurable.
---
include/linux/mm_types.h | 9 +++++++ mm/madvise.c | 42 +++++++++++++++++++++++++++++++------ 2 files changed, 44 insertions(+), 7 deletions(-)
include/linux/mm_types.h renamed from include/linux/mm_types.h~mm-add-anonymous-vma-name-refcounting +8 -1
@@ -5,6 +5,7 @@#include <linux/mm_types_task.h> #include <linux/auxvec.h> +#include <linux/kref.h> #include <linux/list.h> #include <linux/spinlock.h> #include <linux/rbtree.h>
@@ -386,6 +387,12 @@ struct vm_userfaultfd_ctx {struct vm_userfaultfd_ctx {};#endif /* CONFIG_USERFAULTFD */ +struct anon_vma_name { +struct kref kref; +/* The name needs to be at the end because it is dynamically sized. */ +char name[]; +}; +/* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory
@@ -437,7 +444,7 @@ struct vm_area_struct {unsigned long rb_subtree_last; } shared;/* Serialized by mmap_sem. */ -char *anon_name; +struct anon_vma_name *anon_name; };/*
mm/madvise.c renamed from mm/madvise.c~mm-add-anonymous-vma-name-refcounting +36 -6
83 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 59/146 mm: move anon_vma declarations to linux/mm_inline.h
In reply to Andrew MortonFrom: Arnd Bergmann <[email protected]>
Subject: mm: move anon_vma declarations to linux/mm_inline.h
The patch to add anonymous vma names causes a build failure in some configurations:
include/linux/mm_types.h: In function 'is_same_vma_anon_name':include/linux/mm_types.h:924:37: error: implicit declaration of function 'strcmp' [-Werror=implicit-function-declaration]
924 | return name && vma_name && !strcmp(name, vma_name);
| ^~~~~~include/linux/mm_types.h:22:1: note: 'strcmp' is defined in header '<string.h>'; did you forget to '#include <string.h>'?
This should not really be part of linux/mm_types.h in the first place, as that header is meant to only contain structure defintions and need a minimum set of indirect includes itself. While the header clearly includes more than it should at this point, let's not make it worse by including string.h as well, which would pull in the expensive (compile-speed wise) fortify-string logic.
Move the new functions into a separate header that only needs to be included in a couple of locations.
---
fs/proc/task_mmu.c | 1 fs/userfaultfd.c | 1 include/linux/mm_inline.h | 50 ++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 48 ---------------------------------- kernel/fork.c | 1 mm/madvise.c | 1 mm/mmap.c | 1 7 files changed, 55 insertions(+), 48 deletions(-)
fs/proc/task_mmu.c renamed from fs/proc/task_mmu.c~mm-move-anon_vma-declarations-to-linux-mm_inlineh +1 -0
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/pagewalk.h> #include <linux/vmacache.h> +#include <linux/mm_inline.h> #include <linux/hugetlb.h> #include <linux/huge_mm.h> #include <linux/mount.h>
fs/userfaultfd.c renamed from fs/userfaultfd.c~mm-move-anon_vma-declarations-to-linux-mm_inlineh +1 -0
@@ -15,6 +15,7 @@#include <linux/sched/signal.h> #include <linux/sched/mm.h> #include <linux/mm.h> +#include <linux/mm_inline.h> #include <linux/mmu_notifier.h> #include <linux/poll.h> #include <linux/slab.h>
include/linux/mm_inline.h renamed from include/linux/mm_inline.h~mm-move-anon_vma-declarations-to-linux-mm_inlineh +50 -0
60 lines not on this page · raw message
include/linux/mm_types.h renamed from include/linux/mm_types.h~mm-move-anon_vma-declarations-to-linux-mm_inlineh +0 -48
52 lines not on this page · raw message
kernel/fork.c renamed from kernel/fork.c~mm-move-anon_vma-declarations-to-linux-mm_inlineh +1 -0
@@ -42,6 +42,7 @@#include <linux/mmu_notifier.h> #include <linux/fs.h> #include <linux/mm.h> +#include <linux/mm_inline.h> #include <linux/vmacache.h> #include <linux/nsproxy.h> #include <linux/capability.h>
mm/madvise.c renamed from mm/madvise.c~mm-move-anon_vma-declarations-to-linux-mm_inlineh +1 -0
@@ -18,6 +18,7 @@#include <linux/fadvise.h> #include <linux/sched.h> #include <linux/sched/mm.h> +#include <linux/mm_inline.h> #include <linux/string.h> #include <linux/uio.h> #include <linux/ksm.h>
mm/mmap.c renamed from mm/mmap.c~mm-move-anon_vma-declarations-to-linux-mm_inlineh +1 -0
@@ -13,6 +13,7 @@#include <linux/slab.h> #include <linux/backing-dev.h> #include <linux/mm.h> +#include <linux/mm_inline.h> #include <linux/vmacache.h> #include <linux/shm.h> #include <linux/mman.h>
_
Andrew Morton
maintainerPATCH 60/146 mm: move tlb_flush_pending inline helpers to mm_inline.h
In reply to Andrew MortonFrom: Arnd Bergmann <[email protected]>
Subject: mm: move tlb_flush_pending inline helpers to mm_inline.h
linux/mm_types.h should only define structure definitions, to make it cheap to include elsewhere. The atomic_t helper function definitions are particularly large, so it's better to move the helpers using those into the existing linux/mm_inline.h and only include that where needed.
As a follow-up, we may want to go through all the indirect includes in mm_types.h and reduce them as much as possible.
---
arch/x86/include/asm/pgtable.h | 2 include/linux/mm.h | 45 ---------- include/linux/mm_inline.h | 86 ++++++++++++++++++++ include/linux/mm_types.h | 129 ++++++++++--------------------- mm/ksm.c | 1 mm/mapping_dirty_helpers.c | 1 mm/memory.c | 1 mm/mmu_gather.c | 1 mm/pgtable-generic.c | 1 9 files changed, 137 insertions(+), 130 deletions(-)
arch/x86/include/asm/pgtable.h renamed from arch/x86/include/asm/pgtable.h~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +1 -1
@@ -752,7 +752,7 @@ static inline bool pte_accessible(structreturn true;if ((pte_flags (a) & _PAGE_PROTNONE) && -mm_tlb_flush_pending (mm)) +atomic_read (&mm->tlb_flush_pending))return true;return false;
include/linux/mm.h renamed from include/linux/mm.h~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +0 -45
51 lines not on this page · raw message
include/linux/mm_inline.h renamed from include/linux/mm_inline.h~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +86 -0
96 lines not on this page · raw message
include/linux/mm_types.h renamed from include/linux/mm_types.h~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +45 -84
139 lines not on this page · raw message
mm/ksm.c renamed from mm/ksm.c~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +1 -0
@@ -15,6 +15,7 @@#include <linux/errno.h> #include <linux/mm.h> +#include <linux/mm_inline.h> #include <linux/fs.h> #include <linux/mman.h> #include <linux/sched.h>
mm/mapping_dirty_helpers.c renamed from mm/mapping_dirty_helpers.c~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +1 -0
@@ -3,6 +3,7 @@#include <linux/hugetlb.h> #include <linux/bitops.h> #include <linux/mmu_notifier.h> +#include <linux/mm_inline.h> #include <asm/cacheflush.h> #include <asm/tlbflush.h>
mm/memory.c renamed from mm/memory.c~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +1 -0
@@ -41,6 +41,7 @@#include <linux/kernel_stat.h> #include <linux/mm.h> +#include <linux/mm_inline.h> #include <linux/sched/mm.h> #include <linux/sched/coredump.h> #include <linux/sched/numa_balancing.h>
mm/mmu_gather.c renamed from mm/mmu_gather.c~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +1 -0
@@ -3,6 +3,7 @@#include <linux/kernel.h> #include <linux/mmdebug.h> #include <linux/mm_types.h> +#include <linux/mm_inline.h> #include <linux/pagemap.h> #include <linux/rcupdate.h> #include <linux/smp.h>
mm/pgtable-generic.c renamed from mm/pgtable-generic.c~mm-move-tlb_flush_pending-inline-helpers-to-mm_inlineh +1 -0
@@ -10,6 +10,7 @@#include <linux/pagemap.h> #include <linux/hugetlb.h> #include <linux/pgtable.h> +#include <linux/mm_inline.h> #include <asm/tlb.h> /*
_
Andrew Morton
maintainerPATCH 61/146 mm: protect free_pgtables with mmap_lock write lock in exit_mmap
In reply to Andrew MortonFrom: Suren Baghdasaryan <[email protected]>
Subject: mm: protect free_pgtables with mmap_lock write lock in exit_mmap
oom-reaper and process_mrelease system call should protect against races with exit_mmap which can destroy page tables while they walk the VMA tree. oom-reaper protects from that race by setting MMF_OOM_VICTIM and by relying on exit_mmap to set MMF_OOM_SKIP before taking and releasing mmap_write_lock. process_mrelease has to elevate mm->mm_users to prevent such race.
Both oom-reaper and process_mrelease hold mmap_read_lock when walking the VMA tree. The locking rules and mechanisms could be simpler if exit_mmap takes mmap_write_lock while executing destructive operations such as free_pgtables.
Change exit_mmap to hold the mmap_write_lock when calling unlock_range, free_pgtables and remove_vma. Note also that because oom-reaper checks VM_LOCKED flag, unlock_range() should not be allowed to race with it.
Before this patch, remove_vma used to be called with no locks held, however with fput being executed asynchronously and vm_ops->close not being allowed to hold mmap_lock (it is called from __split_vma with mmap_sem held for write), changing that should be fine.
In most cases this lock should be uncontended. Previously, Kirill reported ~4% regression caused by a similar change [1]. We reran the same test and although the individual results are quite noisy, the percentiles show lower regression with 1.6% being the worst case [2]. The change allows oom-reaper and process_mrelease to execute safely under mmap_read_lock without worries that exit_mmap might destroy page tables from under them.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/CAJuCfpGC9-c9P40x7oy=jy5SphMcd0o0G_6U1-+JAziGKG6dGA@mail.gmail.com/
---
mm/mmap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
mm/mmap.c renamed from mm/mmap.c~mm-protect-free_pgtables-with-mmap_lock-write-lock-in-exit_mmap +8 -8
@@ -3149,25 +3149,27 @@ void exit_mmap(struct mm_struct *mm)* to mmu_notifier_release(mm) ensures mmu notifier callbacks in * __oom_reap_task_mm() will not block. * -* This needs to be done before calling munlock_vma_pages_all(), +* This needs to be done before calling unlock_range(), * which clears VM_LOCKED, otherwise the oom reaper cannot * reliably test it. */ (void )__oom_reap_task_mm (mm);set_bit (MMF_OOM_SKIP , &mm->flags); -mmap_write_lock (mm); -mmap_write_unlock (mm); } +mmap_write_lock (mm);if (mm->locked_vm)unlock_range (mm->mmap,ULONG_MAX );arch_exit_mmap (mm); vma = mm->mmap; -if (!vma)/* Can happen if dup_mmap() received an OOM */ +if (!vma) { +/* Can happen if dup_mmap() received an OOM */ +mmap_write_unlock (mm);return ; + }lru_add_drain ();flush_cache_mm (mm);
@@ -3178,16 +3180,14 @@ void exit_mmap(struct mm_struct *mm)free_pgtables (&tlb, vma,FIRST_USER_ADDRESS ,USER_PGTABLES_CEILING );tlb_finish_mmu (&tlb); -/* -* Walk the list again, actually closing and freeing it, -* with preemption enabled, without holding any MM locks. -*/ +/* Walk the list again, actually closing and freeing it. */ while (vma) {if (vma->vm_flags &VM_ACCOUNT ) nr_accounted +=vma_pages (vma); vma =remove_vma (vma);cond_resched (); } +mmap_write_unlock (mm);vm_unacct_memory (nr_accounted); }
_
Andrew Morton
maintainerPATCH 62/146 mm: document locking restrictions for vm_operations_struct::close
In reply to Andrew MortonFrom: Suren Baghdasaryan <[email protected]>
Subject: mm: document locking restrictions for vm_operations_struct::close
Add comments for vm_operations_struct::close documenting locking requirements for this callback and its callers.
---
include/linux/mm.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/mm.h renamed from include/linux/mm.h~mm-document-locking-restrictions-for-vm_operations_struct-close +4 -0
@@ -532,6 +532,10 @@ enum page_entry_size {*/ struct vm_operations_struct {void (*open)(struct vm_area_struct * area); +/** +* @close: Called when the VMA is being removed from the MM. +* Context: User context. May sleep. Caller holds mmap_lock. +*/ void (*close)(struct vm_area_struct * area);/* Called any time before splitting to check if it's allowed */ int (*may_split)(struct vm_area_struct *area,unsigned long addr);
_
Andrew Morton
maintainerPATCH 63/146 mm/oom_kill: allow process_mrelease to run under mmap_lock protection
In reply to Andrew MortonFrom: Suren Baghdasaryan <[email protected]>
Subject: mm/oom_kill: allow process_mrelease to run under mmap_lock protection
With exit_mmap holding mmap_write_lock during free_pgtables call, process_mrelease does not need to elevate mm->mm_users in order to prevent exit_mmap from destrying pagetables while __oom_reap_task_mm is walking the VMA tree. The change prevents process_mrelease from calling the last mmput, which can lead to waiting for IO completion in exit_aio.
---
mm/oom_kill.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-)
mm/oom_kill.c renamed from mm/oom_kill.c~mm-oom_kill-allow-process_mrelease-to-run-under-mmap_lock-protection +15 -12
@@ -1170,15 +1170,15 @@ SYSCALL_DEFINE2(process_mrelease, int, p goto put_task; } -if (mmget_not_zero (p->mm)) { - mm = p->mm; -if (task_will_free_mem (p)) - reap = true; -else { -/* Error only if the work has not been done already */ -if (!test_bit (MMF_OOM_SKIP , &mm->flags)) - ret = -EINVAL ; - } + mm = p->mm; +mmgrab (mm); + +if (task_will_free_mem (p)) + reap = true; +else { +/* Error only if the work has not been done already */ +if (!test_bit (MMF_OOM_SKIP , &mm->flags)) + ret = -EINVAL ; }task_unlock (p);
@@ -1189,13 +1189,16 @@ SYSCALL_DEFINE2(process_mrelease, int, p ret = -EINTR ; goto drop_mm; } -if (!__oom_reap_task_mm (mm)) +/* +* Check MMF_OOM_SKIP again under mmap_read_lock protection to ensure +* possible change in exit_mmap is seen +*/ +if (!test_bit (MMF_OOM_SKIP , &mm->flags) && !__oom_reap_task_mm (mm)) ret = -EAGAIN ;mmap_read_unlock (mm); drop_mm: -if (mm) -mmput (mm); +mmdrop (mm); put_task:put_task_struct (task);return ret;
_
Andrew Morton
maintainerPATCH 64/146 docs/vm: add vmalloced-kernel-stacks document
In reply to Andrew MortonFrom: Shuah Khan <[email protected]>
Subject: docs/vm: add vmalloced-kernel-stacks document
Add a new document to explain Virtually Mapped Kernel Stack Support. This is a compilation of information from the code and original patch series that introduced the Virtually Mapped Kernel Stacks feature.
This document summarizes the feature and provides details on allocation, free, and stack overflow handling. Provides reference to available tests.
---
Documentation/vm/index.rst | 1 Documentation/vm/vmalloced-kernel-stacks.rst | 153 +++++++++++++++++ 2 files changed, 154 insertions(+)
Documentation/vm/index.rst renamed from Documentation/vm/index.rst~docs-vm-add-vmalloced-kernel-stacks-document +1 -0
@@ -36,5 +36,6 @@ algorithms. If you are looking for advi split_page_table_lock transhuge unevictable-lru + vmalloced-kernel-stacks z3fold zsmalloc
Documentation/vm/vmalloced-kernel-stacks.rst new file +153 -0
153 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 65/146 mm: change page type prior to adding page table entry
In reply to Andrew MortonFrom: Pasha Tatashin <[email protected]>
Subject: mm: change page type prior to adding page table entry
Patch series "page table check", v3.
Ensure that some memory corruptions are prevented by checking at the time of insertion of entries into user page tables that there is no illegal sharing.
We have recently found a problem [1] that existed in kernel since 4.14. The problem was caused by broken page ref count and led to memory leaking from one process into another. The problem was accidentally detected by studying a dump of one process and noticing that one page contains memory that should not belong to this process.
There are some other page->_refcount related problems that were recently
fixed: [2], [3] which potentially could also lead to illegal sharing.
In addition to hardening refcount [4] itself, this work is an attempt to prevent this class of memory corruption issues.
It uses a simple state machine that is independent from regular MM logic to check for illegal sharing at time pages are inserted and removed from page tables.
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/[email protected]
[4] https://lore.kernel.org/all/[email protected]
This patch (of 4):
There are a few places where we first update the entry in the user page table, and later change the struct page to indicate that this is anonymous or file page.
In most places, however, we first configure the page metadata and then insert entries into the page table. Page table check, will use the information from struct page to verify the type of entry is inserted.
Change the order in all places to first update struct page, and later to update page table.
This means that we first do calls that may change the type of page (anon or file):
page_move_anon_rmap
page_add_anon_rmap
do_page_add_anon_rmap
page_add_new_anon_rmap
page_add_file_rmap
hugepage_add_anon_rmap
hugepage_add_new_anon_rmap
And after that do calls that add entries to the page table:
set_huge_pte_at
set_pte_at
---
mm/hugetlb.c | 6 +++--- mm/memory.c | 9 +++++---- mm/migrate.c | 5 ++--- mm/swapfile.c | 4 ++-- 4 files changed, 12 insertions(+), 12 deletions(-)
mm/hugetlb.c renamed from mm/hugetlb.c~mm-change-page-type-prior-to-adding-page-table-entry +3 -3
@@ -4684,8 +4684,8 @@ hugetlb_install_page(struct vm_area_stru struct page *new_page) {__SetPageUptodate (new_page); -set_huge_pte_at (vma->vm_mm, addr, ptep,make_huge_pte (vma, new_page,1 ));hugepage_add_new_anon_rmap (new_page, vma, addr); +set_huge_pte_at (vma->vm_mm, addr, ptep,make_huge_pte (vma, new_page,1 ));hugetlb_count_add (pages_per_huge_page (hstate_vma (vma)), vma->vm_mm);ClearHPageRestoreReserve (new_page);SetHPageMigratable (new_page);
@@ -5259,10 +5259,10 @@ retry_avoidcopy:/* Break COW */ huge_ptep_clear_flush (vma, haddr, ptep);mmu_notifier_invalidate_range (mm, range.start, range.end); -set_huge_pte_at (mm, haddr, ptep, -make_huge_pte (vma, new_page,1 ));page_remove_rmap (old_page, true);hugepage_add_new_anon_rmap (new_page, vma, haddr); +set_huge_pte_at (mm, haddr, ptep, +make_huge_pte (vma, new_page,1 ));SetHPageMigratable (new_page);/* Make the old page be freed below */ new_page = old_page;
mm/memory.c renamed from mm/memory.c~mm-change-page-type-prior-to-adding-page-table-entry +5 -4
@@ -720,8 +720,6 @@ static void restore_exclusive_pte(structelse if (is_writable_device_exclusive_entry (entry)) pte =maybe_mkwrite (pte_mkdirty (pte), vma); -set_pte_at (vma->vm_mm, address, ptep, pte); -/* * No need to take a page reference as one was already * created when the swap entry was made.
@@ -735,6 +733,8 @@ static void restore_exclusive_pte(struct*/ WARN_ON_ONCE (!PageAnon (page)); +set_pte_at (vma->vm_mm, address, ptep, pte); +if (vma->vm_flags &VM_LOCKED )mlock_vma_page (page);
@@ -3640,8 +3640,6 @@ vm_fault_t do_swap_page(struct vm_fault pte =pte_mkuffd_wp (pte); pte =pte_wrprotect (pte); } -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); vmf->orig_pte = pte;/* ksm created a completely new copy */
@@ -3652,6 +3650,9 @@ vm_fault_t do_swap_page(struct vm_faultdo_page_add_anon_rmap (page, vma, vmf->address, exclusive); } +set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); +arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +swap_free (entry);if (mem_cgroup_swap_full (page) || (vma->vm_flags &VM_LOCKED ) ||PageMlocked (page))
mm/migrate.c renamed from mm/migrate.c~mm-change-page-type-prior-to-adding-page-table-entry +2 -3
@@ -236,20 +236,19 @@ static bool remove_migration_pte(struct pte =pte_mkhuge (pte); pte =arch_make_huge_pte (pte, shift, vma->vm_flags); -set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte);if (PageAnon (new))hugepage_add_anon_rmap (new, vma, pvmw.address);else page_dup_rmap (new, true); +set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte); } else#endif { -set_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte); -if (PageAnon (new))page_add_anon_rmap (new, vma, pvmw.address, false);else page_add_file_rmap (new, false); +set_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte); }if (vma->vm_flags &VM_LOCKED && !PageTransCompound (new))mlock_vma_page (new);
mm/swapfile.c renamed from mm/swapfile.c~mm-change-page-type-prior-to-adding-page-table-entry +2 -2
@@ -1917,14 +1917,14 @@ static int unuse_pte(struct vm_area_strudec_mm_counter (vma->vm_mm,MM_SWAPENTS );inc_mm_counter (vma->vm_mm,MM_ANONPAGES );get_page (page); -set_pte_at (vma->vm_mm, addr, pte, -pte_mkold (mk_pte (page, vma->vm_page_prot)));if (page == swapcache) {page_add_anon_rmap (page, vma, addr, false); }else {/* ksm created a completely new copy */ page_add_new_anon_rmap (page, vma, addr, false);lru_cache_add_inactive_or_unevictable (page, vma); } +set_pte_at (vma->vm_mm, addr, pte, +pte_mkold (mk_pte (page, vma->vm_page_prot)));swap_free (entry); out:pte_unmap_unlock (pte, ptl);
_
Andrew Morton
maintainerPATCH 66/146 mm: ptep_clear() page table helper
In reply to Andrew MortonFrom: Pasha Tatashin <[email protected]>
Subject: mm: ptep_clear() page table helper
We have ptep_get_and_clear() and ptep_get_and_clear_full() helpers to clear PTE from user page tables, but there is no variant for simple clear of a present PTE from user page tables without using a low level pte_clear() which can be either native or para-virtualised.
Add a new ptep_clear() that can be used in common code to clear PTEs from page table. We will need this call later in order to add a hook for page table check.
---
Documentation/vm/arch_pgtable_helpers.rst | 6 ++++-- include/linux/pgtable.h | 8 ++++++++ mm/debug_vm_pgtable.c | 2 +- mm/khugepaged.c | 12 ++---------- 4 files changed, 15 insertions(+), 13 deletions(-)
Documentation/vm/arch_pgtable_helpers.rst renamed from Documentation/vm/arch_pgtable_helpers.rst~mm-ptep_clear-page-table-helper +4 -2
@@ -66,9 +66,11 @@ PTE Page Table Helpers +---------------------------+--------------------------------------------------+ | pte_mknotpresent | Invalidates a mapped PTE | +---------------------------+--------------------------------------------------+ -| ptep_get_and_clear | Clears a PTE | +| ptep_clear | Clears a PTE | +---------------------------+--------------------------------------------------+ -| ptep_get_and_clear_full | Clears a PTE | +| ptep_get_and_clear | Clears and returns PTE | ++---------------------------+--------------------------------------------------+ +| ptep_get_and_clear_full | Clears and returns PTE (batched PTE unmap) | +---------------------------+--------------------------------------------------+ | ptep_test_and_clear_young | Clears young from a PTE | +---------------------------+--------------------------------------------------+
include/linux/pgtable.h renamed from include/linux/pgtable.h~mm-ptep_clear-page-table-helper +8 -0
@@ -258,6 +258,14 @@ static inline int pmdp_clear_flush_young#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif +#ifndef __HAVE_ARCH_PTEP_CLEAR +static inline void ptep_clear (struct mm_struct *mm,unsigned long addr, +pte_t *ptep) +{ +pte_clear (mm, addr, ptep); +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
mm/debug_vm_pgtable.c renamed from mm/debug_vm_pgtable.c~mm-ptep_clear-page-table-helper +1 -1
@@ -652,7 +652,7 @@ static void __init pte_clear_tests(strucset_pte_at (args->mm, args->vaddr, args->ptep, pte);flush_dcache_page (page);barrier (); -pte_clear (args->mm, args->vaddr, args->ptep); +ptep_clear (args->mm, args->vaddr, args->ptep); pte =ptep_get (args->ptep);WARN_ON (!pte_none (pte)); }
mm/khugepaged.c renamed from mm/khugepaged.c~mm-ptep_clear-page-table-helper +2 -10
@@ -756,11 +756,7 @@ static void __collapse_huge_page_copy(pt* ptl mostly unnecessary. */ spin_lock (ptl); -/* -* paravirt calls inside pte_clear here are -* superfluous. -*/ -pte_clear (vma->vm_mm, address, _pte); +ptep_clear (vma->vm_mm, address, _pte);spin_unlock (ptl); } }else {
@@ -774,11 +770,7 @@ static void __collapse_huge_page_copy(pt* inside page_remove_rmap(). */ spin_lock (ptl); -/* -* paravirt calls inside pte_clear here are -* superfluous. -*/ -pte_clear (vma->vm_mm, address, _pte); +ptep_clear (vma->vm_mm, address, _pte);page_remove_rmap (src_page, false);spin_unlock (ptl);free_page_and_swap_cache (src_page);
_
Andrew Morton
maintainerPATCH 67/146 mm: page table check
In reply to Andrew MortonFrom: Pasha Tatashin <[email protected]>
Subject: mm: page table check
Check user page table entries at the time they are added and removed.
Allows to synchronously catch memory corruption issues related to double mapping.
When a pte for an anonymous page is added into page table, we verify that this pte does not already point to a file backed page, and vice versa if this is a file backed page that is being added we verify that this page does not have an anonymous mapping
We also enforce that read-only sharing for anonymous pages is allowed (i.e. cow after fork). All other sharing must be for file pages.
Page table check allows to protect and debug cases where "struct page" metadata became corrupted for some reason. For example, when refcnt or mapcount become invalid.
---
Documentation/vm/index.rst | 1 Documentation/vm/page_table_check.rst | 56 ++++ MAINTAINERS | 9 arch/Kconfig | 3 include/linux/page_table_check.h | 147 +++++++++++++ mm/Kconfig.debug | 24 ++ mm/Makefile | 1 mm/page_alloc.c | 4 mm/page_ext.c | 4 mm/page_table_check.c | 270 ++++++++++++++++++++++++ 10 files changed, 519 insertions(+)
arch/Kconfig renamed from arch/Kconfig~mm-page-table-check +3 -0
@@ -1297,6 +1297,9 @@ config HAVE_ARCH_PFN_VALID config ARCH_SUPPORTS_DEBUG_PAGEALLOC bool +config ARCH_SUPPORTS_PAGE_TABLE_CHECK + bool + config ARCH_SPLIT_ARG64 bool help
Documentation/vm/index.rst renamed from Documentation/vm/index.rst~mm-page-table-check +1 -0
@@ -31,6 +31,7 @@ algorithms. If you are looking for advi page_migration page_frags page_owner + page_table_check remap_file_pages slub split_page_table_lock
Documentation/vm/page_table_check.rst new file +56 -0
56 lines not on this page · raw message
include/linux/page_table_check.h new file +147 -0
147 lines not on this page · raw message
MAINTAINERS renamed from MAINTAINERS~mm-page-table-check +9 -0
@@ -14387,6 +14387,15 @@ F: include/net/page_pool.h F: include/trace/events/page_pool.h F: net/core/page_pool.c +PAGE TABLE CHECK +M: Pasha Tatashin <[email protected]> +M: Andrew Morton <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/vm/page_table_check.rst +F: include/linux/page_table_check.h +F: mm/page_table_check.c + PANASONIC LAPTOP ACPI EXTRAS DRIVER M: Kenneth Chan <[email protected]> L: [email protected]
mm/Kconfig.debug renamed from mm/Kconfig.debug~mm-page-table-check +24 -0
@@ -62,6 +62,30 @@ config PAGE_OWNER If unsure, say N. +config PAGE_TABLE_CHECK + bool "Check for invalid mappings in user page tables" + depends on ARCH_SUPPORTS_PAGE_TABLE_CHECK + select PAGE_EXTENSION + help + Check that anonymous page is not being mapped twice with read write + permissions. Check that anonymous and file pages are not being + erroneously shared. Since the checking is performed at the time + entries are added and removed to user page tables, leaking, corruption + and double mapping problems are detected synchronously. + + If unsure say "n". + +config PAGE_TABLE_CHECK_ENFORCED + bool "Enforce the page table checking by default" + depends on PAGE_TABLE_CHECK + help + Always enable page table checking. By default the page table checking + is disabled, and can be optionally enabled via page_table_check=on + kernel parameter. This config enforces that page table check is always + enabled. + + If unsure say "n". + config PAGE_POISONING bool "Poison pages after freeing" help
mm/Makefile renamed from mm/Makefile~mm-page-table-check +1 -0
@@ -112,6 +112,7 @@ obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += e obj-$(CONFIG_CMA) += cma.o obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o +obj-$(CONFIG_PAGE_TABLE_CHECK) += page_table_check.o obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o obj-$(CONFIG_SECRETMEM) += secretmem.o obj-$(CONFIG_CMA_SYSFS) += cma_sysfs.o
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page-table-check +4 -0
@@ -63,6 +63,7 @@#include <linux/sched/rt.h> #include <linux/sched/mm.h> #include <linux/page_owner.h> +#include <linux/page_table_check.h> #include <linux/kthread.h> #include <linux/memcontrol.h> #include <linux/ftrace.h>
@@ -1307,6 +1308,7 @@ static __always_inline bool free_pages_pif (memcg_kmem_enabled () &&PageMemcgKmem (page))__memcg_kmem_uncharge_page (page, order);reset_page_owner (page, order); +page_table_check_free (page, order);return false; }
@@ -1346,6 +1348,7 @@ static __always_inline bool free_pages_ppage_cpupid_reset_last (page); page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP ;reset_page_owner (page, order); +page_table_check_free (page, order);if (!PageHighMem (page)) { debug_check_no_locks_freed(page_address(page),
@@ -2420,6 +2423,7 @@ inline void post_alloc_hook(struct page }set_page_owner (page, order, gfp_flags); +page_table_check_alloc (page, order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
mm/page_ext.c renamed from mm/page_ext.c~mm-page-table-check +4 -0
@@ -8,6 +8,7 @@#include <linux/kmemleak.h> #include <linux/page_owner.h> #include <linux/page_idle.h> +#include <linux/page_table_check.h> /* * struct page extension
@@ -75,6 +76,9 @@ static struct page_ext_operations *page_#if defined(CONFIG_PAGE_IDLE_FLAG ) && !defined(CONFIG_64BIT ) &page_idle_ops,#endif +#ifdef CONFIG_PAGE_TABLE_CHECK + &page_table_check_ops, +#endif };unsigned long page_ext_size =sizeof (struct page_ext );
mm/page_table_check.c new file +270 -0
270 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 68/146 x86: mm: add x86_64 support for page table check
In reply to Andrew MortonFrom: Pasha Tatashin <[email protected]>
Subject: x86: mm: add x86_64 support for page table check
Add page table check hooks into routines that modify user page tables.
---
arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 2 deletions(-)
arch/x86/include/asm/pgtable.h renamed from arch/x86/include/asm/pgtable.h~x86-mm-add-x86_64-support-for-page-table-check +27 -2
@@ -26,6 +26,7 @@#include <asm/pkru.h> #include <asm/fpu/api.h> #include <asm-generic/pgtable_uffd.h> +#include <linux/page_table_check.h> extern pgd_t early_top_pgt[PTRS_PER_PGD ];bool __init __early_make_pgtable (unsigned long address,pmdval_t pmd);
@@ -1006,18 +1007,21 @@ static inline pud_t native_local_pudp_gestatic inline void set_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte) { +page_table_check_pte_set (mm, addr, ptep, pte);set_pte (ptep, pte); }static inline void set_pmd_at (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,pmd_t pmd) { +page_table_check_pmd_set (mm, addr, pmdp, pmd);set_pmd (pmdp, pmd); }static inline void set_pud_at (struct mm_struct *mm,unsigned long addr,pud_t *pudp,pud_t pud) { +page_table_check_pud_set (mm, addr, pudp, pud);native_set_pud (pudp, pud); }
@@ -1048,6 +1052,7 @@ static inline pte_t ptep_get_and_clear(s pte_t *ptep) {pte_t pte =native_ptep_get_and_clear (ptep); +page_table_check_pte_clear (mm, addr, pte);return pte; }
@@ -1063,12 +1068,23 @@ static inline pte_t ptep_get_and_clear_f* care about updates and native needs no locking */ pte =native_local_ptep_get_and_clear (ptep); +page_table_check_pte_clear (mm, addr, pte); }else { pte =ptep_get_and_clear (mm, addr, ptep); }return pte; } +#define __HAVE_ARCH_PTEP_CLEAR +static inline void ptep_clear (struct mm_struct *mm,unsigned long addr, +pte_t *ptep) +{ +if (IS_ENABLED (CONFIG_PAGE_TABLE_CHECK )) +ptep_get_and_clear (mm, addr, ptep); +else +pte_clear (mm, addr, ptep); +} +#define __HAVE_ARCH_PTEP_SET_WRPROTECTstatic inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
@@ -1109,14 +1125,22 @@ static inline int pmd_write(pmd_t pmd)static inline pmd_t pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp) { -return native_pmdp_get_and_clear (pmdp); +pmd_t pmd =native_pmdp_get_and_clear (pmdp); + +page_table_check_pmd_clear (mm, addr, pmd); + +return pmd; }#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEARstatic inline pud_t pudp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pud_t *pudp) { -return native_pudp_get_and_clear (pudp); +pud_t pud =native_pudp_get_and_clear (pudp); + +page_table_check_pud_clear (mm, addr, pud); + +return pud; }#define __HAVE_ARCH_PMDP_SET_WRPROTECT
@@ -1137,6 +1161,7 @@ static inline int pud_write(pud_t pud)static inline pmd_t pmdp_establish (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp,pmd_t pmd) { +page_table_check_pmd_set (vma->vm_mm, address, pmdp, pmd);if (IS_ENABLED (CONFIG_SMP )) {return xchg (pmdp, pmd); }else {
arch/x86/Kconfig renamed from arch/x86/Kconfig~x86-mm-add-x86_64-support-for-page-table-check +1 -0
@@ -104,6 +104,7 @@ config X86 select ARCH_SUPPORTS_ACPI select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_DEBUG_PAGEALLOC + select ARCH_SUPPORTS_PAGE_TABLE_CHECK if X86_64 select ARCH_SUPPORTS_NUMA_BALANCING if X86_64 select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096 select ARCH_SUPPORTS_LTO_CLANG
_
Andrew Morton
maintainerPATCH 69/146 mm: remove last argument of reuse_swap_page()
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm: remove last argument of reuse_swap_page()
None of the callers care about the total_map_swapcount() any more.
---
include/linux/swap.h | 6 +++--- mm/huge_memory.c | 2 +- mm/khugepaged.c | 2 +- mm/memory.c | 2 +- mm/swapfile.c | 8 +------- 5 files changed, 7 insertions(+), 13 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~mm-remove-last-argument-of-reuse_swap_page +3 -3
@@ -514,7 +514,7 @@ extern int __swp_swapcount(swp_entry_t eextern int swp_swapcount (swp_entry_t entry);extern struct swap_info_struct *page_swap_info (struct page *);extern struct swap_info_struct *swp_swap_info (swp_entry_t entry); -extern bool reuse_swap_page (struct page *,int *); +extern bool reuse_swap_page (struct page *);extern int try_to_free_swap (struct page *);struct backing_dev_info ;extern int init_swap_address_space (unsigned int type,unsigned long nr_pages);
@@ -680,8 +680,8 @@ static inline int swp_swapcount(swp_entrreturn 0 ; } -#define reuse_swap_page (page, total_map_swapcount) \ - (page_trans_huge_mapcount(page, total_map_swapcount) == 1) +#define reuse_swap_page (page) \ + (page_trans_huge_mapcount(page, NULL) == 1)static inline int try_to_free_swap (struct page *page) {
mm/huge_memory.c renamed from mm/huge_memory.c~mm-remove-last-argument-of-reuse_swap_page +1 -1
@@ -1322,7 +1322,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm* We can only reuse the page if nobody else maps the huge page or it's * part. */ -if (reuse_swap_page (page,NULL )) { +if (reuse_swap_page (page)) {pmd_t entry; entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma);
mm/khugepaged.c renamed from mm/khugepaged.c~mm-remove-last-argument-of-reuse_swap_page +1 -1
@@ -681,7 +681,7 @@ static int __collapse_huge_page_isolate( goto out; }if (!pte_write (pteval) &&PageSwapCache (page) && - !reuse_swap_page (page,NULL )) { + !reuse_swap_page (page)) {/* * Page is in the swap cache and cannot be re-used. * It cannot be collapsed into a THP.
mm/memory.c renamed from mm/memory.c~mm-remove-last-argument-of-reuse_swap_page +1 -1
@@ -3627,7 +3627,7 @@ vm_fault_t do_swap_page(struct vm_faultinc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );dec_mm_counter_fast (vma->vm_mm,MM_SWAPENTS ); pte =mk_pte (page, vma->vm_page_prot); -if ((vmf->flags &FAULT_FLAG_WRITE ) &&reuse_swap_page (page,NULL )) { +if ((vmf->flags &FAULT_FLAG_WRITE ) &&reuse_swap_page (page)) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; ret |=VM_FAULT_WRITE ;
mm/swapfile.c renamed from mm/swapfile.c~mm-remove-last-argument-of-reuse_swap_page +1 -7
@@ -1668,12 +1668,8 @@ static int page_trans_huge_map_swapcount* to it. And as a side-effect, free up its swap: because the old content * on disk will never be read, and seeking back there to write new content * later would only waste time away from clustering. -* -* NOTE: total_map_swapcount should not be relied upon by the caller if -* reuse_swap_page() returns false, but it may be always overwritten -* (see the other implementation for CONFIG_SWAP=n). */ -bool reuse_swap_page (struct page *page,int *total_map_swapcount) +bool reuse_swap_page (struct page *page) {int count, total_mapcount, total_swapcount;
@@ -1682,8 +1678,6 @@ bool reuse_swap_page(struct page *page,return false; count =page_trans_huge_map_swapcount (page, &total_mapcount, &total_swapcount); -if (total_map_swapcount) - *total_map_swapcount = total_mapcount + total_swapcount; if (count == 1 && PageSwapCache(page) && (likely(!PageTransCompound(page)) ||/* The remaining swap count will be freed soon */
_
Andrew Morton
maintainerPATCH 70/146 mm: remove the total_mapcount argument from page_trans_huge_map_swapcount()
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm: remove the total_mapcount argument from page_trans_huge_map_swapcount()
Now that we don't report it to the caller of reuse_swap_page(), we don't need to request it from page_trans_huge_map_swapcount().
---
mm/swapfile.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-)
mm/swapfile.c renamed from mm/swapfile.c~mm-remove-the-total_mapcount-argument-from-page_trans_huge_map_swapcount +12 -20
@@ -1601,31 +1601,30 @@ static bool page_swapped(struct page *pareturn false; } -static int page_trans_huge_map_swapcount (struct page *page,int *total_mapcount, +static int page_trans_huge_map_swapcount (struct page *page,int *total_swapcount) { -int i, map_swapcount, _total_mapcount, _total_swapcount; +int i, map_swapcount, _total_swapcount;unsigned long offset =0 ;struct swap_info_struct *si;struct swap_cluster_info *ci =NULL ;unsigned char *map =NULL ; -int mapcount, swapcount =0 ; +int swapcount =0 ;/* hugetlbfs shouldn't call it */ VM_BUG_ON_PAGE (PageHuge (page), page);if (!IS_ENABLED (CONFIG_THP_SWAP ) ||likely (!PageTransCompound (page))) { - mapcount =page_trans_huge_mapcount (page, total_mapcount);if (PageSwapCache (page)) swapcount =page_swapcount (page);if (total_swapcount) *total_swapcount = swapcount; -return mapcount + swapcount; +return swapcount +page_trans_huge_mapcount (page,NULL ); } page =compound_head (page); - _total_mapcount = _total_swapcount = map_swapcount =0 ; + _total_swapcount = map_swapcount =0 ;if (PageSwapCache (page)) {swp_entry_t entry;
@@ -1639,8 +1638,7 @@ static int page_trans_huge_map_swapcountif (map) ci =lock_cluster (si, offset);for (i =0 ; i <HPAGE_PMD_NR ; i++) { - mapcount =atomic_read (&page[i]._mapcount) +1 ; - _total_mapcount += mapcount; +int mapcount =atomic_read (&page[i]._mapcount) +1 ;if (map) { swapcount =swap_count (map[offset + i]); _total_swapcount += swapcount;
@@ -1648,19 +1646,14 @@ static int page_trans_huge_map_swapcount map_swapcount =max (map_swapcount, mapcount + swapcount); }unlock_cluster (ci); -if (PageDoubleMap (page)) { + +if (PageDoubleMap (page)) map_swapcount -=1 ; - _total_mapcount -=HPAGE_PMD_NR ; - } - mapcount =compound_mapcount (page); - map_swapcount += mapcount; - _total_mapcount += mapcount; -if (total_mapcount) - *total_mapcount = _total_mapcount; +if (total_swapcount) *total_swapcount = _total_swapcount; -return map_swapcount; +return map_swapcount +compound_mapcount (page); }/*
@@ -1671,13 +1664,12 @@ static int page_trans_huge_map_swapcount*/ bool reuse_swap_page (struct page *page) { -int count, total_mapcount, total_swapcount; +int count, total_swapcount;VM_BUG_ON_PAGE (!PageLocked (page), page);if (unlikely (PageKsm (page)))return false; - count =page_trans_huge_map_swapcount (page, &total_mapcount, - &total_swapcount); + count =page_trans_huge_map_swapcount (page, &total_swapcount); if (count == 1 && PageSwapCache(page) && (likely(!PageTransCompound(page)) ||/* The remaining swap count will be freed soon */
_
Andrew Morton
maintainerPATCH 71/146 mm: remove the total_mapcount argument from page_trans_huge_mapcount()
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm: remove the total_mapcount argument from page_trans_huge_mapcount()
All callers pass NULL, so we can stop calculating the value we would store in it.
---
include/linux/mm.h | 10 +++------- include/linux/swap.h | 2 +- mm/huge_memory.c | 30 ++++++++++-------------------- mm/swapfile.c | 2 +- 4 files changed, 15 insertions(+), 29 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-remove-the-total_mapcount-argument-from-page_trans_huge_mapcount +3 -7
@@ -799,19 +799,15 @@ static inline int page_mapcount(struct p#ifdef CONFIG_TRANSPARENT_HUGEPAGE int total_mapcount (struct page *page); -int page_trans_huge_mapcount (struct page *page,int *total_mapcount); +int page_trans_huge_mapcount (struct page *page);#else static inline int total_mapcount (struct page *page) {return page_mapcount (page); } -static inline int page_trans_huge_mapcount (struct page *page, -int *total_mapcount) +static inline int page_trans_huge_mapcount (struct page *page) { -int mapcount =page_mapcount (page); -if (total_mapcount) - *total_mapcount = mapcount; -return mapcount; +return page_mapcount (page); }#endif
include/linux/swap.h renamed from include/linux/swap.h~mm-remove-the-total_mapcount-argument-from-page_trans_huge_mapcount +1 -1
@@ -681,7 +681,7 @@ static inline int swp_swapcount(swp_entr }#define reuse_swap_page (page) \ - (page_trans_huge_mapcount(page, NULL) == 1) + (page_trans_huge_mapcount(page) == 1)static inline int try_to_free_swap (struct page *page) {
mm/huge_memory.c renamed from mm/huge_memory.c~mm-remove-the-total_mapcount-argument-from-page_trans_huge_mapcount +10 -20
@@ -2542,38 +2542,28 @@ int total_mapcount(struct page *page)* need full accuracy to avoid breaking page pinning, because * page_trans_huge_mapcount() is slower than page_mapcount(). */ -int page_trans_huge_mapcount (struct page *page,int *total_mapcount) +int page_trans_huge_mapcount (struct page *page) { -int i, ret, _total_mapcount, mapcount; +int i, ret;/* hugetlbfs shouldn't call it */ VM_BUG_ON_PAGE (PageHuge (page), page); -if (likely (!PageTransCompound (page))) { - mapcount =atomic_read (&page->_mapcount) +1 ; -if (total_mapcount) - *total_mapcount = mapcount; -return mapcount; - } +if (likely (!PageTransCompound (page))) +return atomic_read (&page->_mapcount) +1 ; page =compound_head (page); - _total_mapcount = ret =0 ; + ret =0 ;for (i =0 ; i <thp_nr_pages (page); i++) { - mapcount =atomic_read (&page[i]._mapcount) +1 ; +int mapcount =atomic_read (&page[i]._mapcount) +1 ; ret =max (ret, mapcount); - _total_mapcount += mapcount; } -if (PageDoubleMap (page)) { + +if (PageDoubleMap (page)) ret -=1 ; - _total_mapcount -=thp_nr_pages (page); - } - mapcount =compound_mapcount (page); - ret += mapcount; - _total_mapcount += mapcount; -if (total_mapcount) - *total_mapcount = _total_mapcount; -return ret; + +return ret +compound_mapcount (page); }/* Racy check whether the huge page can be split */
mm/swapfile.c renamed from mm/swapfile.c~mm-remove-the-total_mapcount-argument-from-page_trans_huge_mapcount +1 -1
@@ -1619,7 +1619,7 @@ static int page_trans_huge_map_swapcount swapcount =page_swapcount (page);if (total_swapcount) *total_swapcount = swapcount; -return swapcount +page_trans_huge_mapcount (page,NULL ); +return swapcount +page_trans_huge_mapcount (page); } page =compound_head (page);
_
Andrew Morton
maintainerPATCH 72/146 mm/dmapool.c: revert "make dma pool to use kmalloc_node"
In reply to Andrew MortonFrom: Christian König <[email protected]>
Subject: mm/dmapool.c: revert "make dma pool to use kmalloc_node"
This reverts commit 2618c60b8b5836 ("dma: make dma pool to use kmalloc_node").
While working myself into the dmapool code I've found this little odd kmalloc_node().
What basically happens here is that we allocate the housekeeping structure on the numa node where the device is attached to. Since the device is never doing DMA to or from that memory this doesn't seem to make sense at all.
So while this doesn't seem to cause much harm it's probably cleaner to revert the change for consistency.
---
mm/dmapool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/dmapool.c renamed from mm/dmapool.c~dma-revert-make-dma-pool-to-use-kmalloc_node +1 -1
@@ -152,7 +152,7 @@ struct dma_pool *dma_pool_create(const celse if ((boundary < size) || (boundary & (boundary -1 )))return NULL ; - retval =kmalloc_node (sizeof (*retval),GFP_KERNEL ,dev_to_node (dev)); + retval =kmalloc (sizeof (*retval),GFP_KERNEL );if (!retval)return retval;
_
Andrew Morton
maintainerPATCH 73/146 mm/vmalloc: alloc GFP_NO{FS,IO} for vmalloc
In reply to Andrew MortonFrom: Michal Hocko <[email protected]>
Subject: mm/vmalloc: alloc GFP_NO{FS,IO} for vmalloc
Patch series "extend vmalloc support for constrained allocations", v2.
Based on a recent discussion with Dave and Neil [1] I have tried to implement NOFS, NOIO, NOFAIL support for the vmalloc to make life of kvmalloc users easier.
A requirement for NOFAIL support for kvmalloc was new to me but this seems to be really needed by the xfs code.
NOFS/NOIO was a known and a long term problem which was hoped to be handled by the scope API. Those scope should have been used at the reclaim recursion boundaries both to document them and also to remove the necessity of NOFS/NOIO constrains for all allocations within that scope. Instead workarounds were developed to wrap a single allocation instead (like ceph_kvmalloc).
First patch implements NOFS/NOIO support for vmalloc. The second one adds NOFAIL support and the third one bundles all together into kvmalloc and drops ceph_kvmalloc which can use kvmalloc directly now.
[1] http://lkml.kernel.org/r/[email protected]
This patch (of 4):
vmalloc historically hasn't supported GFP_NO{FS,IO} requests because page table allocations do not support externally provided gfp mask and performed GFP_KERNEL like allocations.
Since few years we have scope (memalloc_no{fs,io}_{save,restore}) APIs to enforce NOFS and NOIO constrains implicitly to all allocators within the scope. There was a hope that those scopes would be defined on a higher level when the reclaim recursion boundary starts/stops (e.g. when a lock required during the memory reclaim is required etc.). It seems that not all NOFS/NOIO users have adopted this approach and instead they have taken a workaround approach to wrap a single [k]vmalloc allocation by a scope API.
These workarounds do not serve the purpose of a better reclaim recursion documentation and reduction of explicit GFP_NO{FS,IO} usege so let's just provide them with the semantic they are asking for without a need for workarounds.
Add support for GFP_NOFS and GFP_NOIO to vmalloc directly. All internal allocations already comply with the given gfp_mask. The only current exception is vmap_pages_range which maps kernel page tables. Infer the proper scope API based on the given gfp mask.
---
mm/vmalloc.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
mm/vmalloc.c renamed from mm/vmalloc.c~mm-vmalloc-alloc-gfp_nofsio-for-vmalloc +21 -2
@@ -39,6 +39,7 @@#include <linux/pgtable.h> #include <linux/uaccess.h> #include <linux/hugetlb.h> +#include <linux/sched/mm.h> #include <asm/tlbflush.h> #include <asm/shmparam.h>
@@ -2928,6 +2929,8 @@ static void *__vmalloc_area_node(structunsigned long array_size;unsigned int nr_small_pages = size >>PAGE_SHIFT ;unsigned int page_order; +unsigned int flags; +int ret; array_size = (unsigned long )nr_small_pages *sizeof (struct page *); gfp_mask |= __GFP_NOWARN;
@@ -2976,8 +2979,24 @@ static void *__vmalloc_area_node(struct goto fail; } -if (vmap_pages_range (addr, addr + size, prot, area->pages, - page_shift) <0 ) { +/* +* page tables allocations ignore external gfp mask, enforce it +* by the scope API +*/ +if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) + flags =memalloc_nofs_save (); +else if ((gfp_mask & (__GFP_FS | __GFP_IO)) ==0 ) + flags =memalloc_noio_save (); + + ret =vmap_pages_range (addr, addr + size, prot, area->pages, + page_shift); + +if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) +memalloc_nofs_restore (flags); +else if ((gfp_mask & (__GFP_FS | __GFP_IO)) ==0 ) +memalloc_noio_restore (flags); + +if (ret <0 ) {warn_alloc (orig_gfp_mask,NULL ,"vmalloc error: size %lu, failed to map pages" , area->nr_pages *PAGE_SIZE );
_
Andrew Morton
maintainerPATCH 74/146 mm/vmalloc: add support for __GFP_NOFAIL
In reply to Andrew MortonFrom: Michal Hocko <[email protected]>
Subject: mm/vmalloc: add support for __GFP_NOFAIL
Dave Chinner has mentioned that some of the xfs code would benefit from kvmalloc support for __GFP_NOFAIL because they have allocations that cannot fail and they do not fit into a single page.
The large part of the vmalloc implementation already complies with the given gfp flags so there is no work for those to be done. The area and page table allocations are an exception to that. Implement a retry loop for those.
Add a short sleep before retrying. 1 jiffy is a completely random timeout. Ideally the retry would wait for an explicit event - e.g. a change to the vmalloc space change if the failure was caused by the space fragmentation or depletion. But there are multiple different reasons to retry and this could become much more complex. Keep the retry simple for now and just sleep to prevent from hogging CPUs.
---
mm/vmalloc.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
mm/vmalloc.c renamed from mm/vmalloc.c~mm-vmalloc-add-support-for-__gfp_nofail +17 -5
@@ -2847,6 +2847,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,* more permissive. */ if (!order) { +gfp_t bulk_gfp = gfp & ~__GFP_NOFAIL; +while (nr_allocated < nr_pages) {unsigned int nr, nr_pages_request;
@@ -2864,12 +2866,12 @@ vm_area_alloc_pages(gfp_t gfp, int nid,* but mempolcy want to alloc memory by interleaving. */ if (IS_ENABLED (CONFIG_NUMA ) && nid ==NUMA_NO_NODE ) - nr =alloc_pages_bulk_array_mempolicy (gfp, + nr =alloc_pages_bulk_array_mempolicy (bulk_gfp, nr_pages_request, pages + nr_allocated);else - nr =alloc_pages_bulk_array_node (gfp, nid, + nr =alloc_pages_bulk_array_node (bulk_gfp, nid, nr_pages_request, pages + nr_allocated);
@@ -2924,6 +2926,7 @@ static void *__vmalloc_area_node(struct {const gfp_t nested_gfp = (gfp_mask &GFP_RECLAIM_MASK ) | __GFP_ZERO;const gfp_t orig_gfp_mask = gfp_mask; +bool nofail = gfp_mask & __GFP_NOFAIL;unsigned long addr = (unsigned long )area->addr;unsigned long size =get_vm_area_size (area);unsigned long array_size;
@@ -2988,8 +2991,12 @@ static void *__vmalloc_area_node(structelse if ((gfp_mask & (__GFP_FS | __GFP_IO)) ==0 ) flags =memalloc_noio_save (); - ret =vmap_pages_range (addr, addr + size, prot, area->pages, +do { + ret =vmap_pages_range (addr, addr + size, prot, area->pages, page_shift); +if (nofail && (ret <0 )) +schedule_timeout_uninterruptible (1 ); + }while (nofail && (ret <0 ));if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)memalloc_nofs_restore (flags);
@@ -3084,9 +3091,14 @@ again:VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller);if (!area) { +bool nofail = gfp_mask & __GFP_NOFAIL;warn_alloc (gfp_mask,NULL , -"vmalloc error: size %lu, vm_struct allocation failed" , - real_size); +"vmalloc error: size %lu, vm_struct allocation failed%s" , + real_size, (nofail) ?". Retrying." :"" ); +if (nofail) { +schedule_timeout_uninterruptible (1 ); + goto again; + } goto fail; }
_
Andrew Morton
maintainerPATCH 75/146 mm/vmalloc: be more explicit about supported gfp flags.
In reply to Andrew MortonFrom: Michal Hocko <[email protected]>
Subject: mm/vmalloc: be more explicit about supported gfp flags.
b7d90e7a5ea8 ("mm/vmalloc: be more explicit about supported gfp flags") has been merged prematurely without the rest of the series and without addressed review feedback from Neil. Fix that up now. Only wording is changed slightly.
---
mm/vmalloc.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
mm/vmalloc.c renamed from mm/vmalloc.c~mm-vmalloc-be-more-explicit-about-supported-gfp-flags +8 -6
@@ -3031,12 +3031,14 @@ fail:* * Allocate enough pages to cover @size from the page level * allocator with @gfp_mask flags. Please note that the full set of gfp -* flags are not supported. GFP_KERNEL would be a preferred allocation mode -* but GFP_NOFS and GFP_NOIO are supported as well. Zone modifiers are not -* supported. From the reclaim modifiers__GFP_DIRECT_RECLAIM is required (aka -* GFP_NOWAIT is not supported) and only __GFP_NOFAIL is supported (aka -* __GFP_NORETRY and __GFP_RETRY_MAYFAIL are not supported). -* __GFP_NOWARN can be used to suppress error messages about failures. +* flags are not supported. GFP_KERNEL, GFP_NOFS and GFP_NOIO are all +* supported. +* Zone modifiers are not supported. From the reclaim modifiers +* __GFP_DIRECT_RECLAIM is required (aka GFP_NOWAIT is not supported) +* and only __GFP_NOFAIL is supported (i.e. __GFP_NORETRY and +* __GFP_RETRY_MAYFAIL are not supported). +* +* __GFP_NOWARN can be used to suppress failures messages. * * Map them into contiguous kernel virtual space, using a pagetable * protection of @prot.
_
Andrew Morton
maintainerPATCH 76/146 mm: allow !GFP_KERNEL allocations for kvmalloc
In reply to Andrew MortonFrom: Michal Hocko <[email protected]>
Subject: mm: allow !GFP_KERNEL allocations for kvmalloc
Support for GFP_NO{FS,IO} and __GFP_NOFAIL has been implemented by previous patches so we can allow the support for kvmalloc. This will allow some external users to simplify or completely remove their helpers.
GFP_NOWAIT semantic hasn't been supported so far but it hasn't been explicitly documented so let's add a note about that.
ceph_kvmalloc is the first helper to be dropped and changed to kvmalloc.
---
include/linux/ceph/libceph.h | 1 - mm/util.c | 15 ++++----------- net/ceph/buffer.c | 4 ++-- net/ceph/ceph_common.c | 27 --------------------------- net/ceph/crypto.c | 2 +- net/ceph/messenger.c | 2 +- net/ceph/messenger_v2.c | 2 +- net/ceph/osdmap.c | 12 ++++++------ 8 files changed, 15 insertions(+), 50 deletions(-)
include/linux/ceph/libceph.h renamed from include/linux/ceph/libceph.h~mm-allow-gfp_kernel-allocations-for-kvmalloc +0 -1
@@ -295,7 +295,6 @@ extern bool libceph_compatible(void *datextern const char *ceph_msg_type_name (int type);extern int ceph_check_fsid (struct ceph_client *client,struct ceph_fsid *fsid); -extern void *ceph_kvmalloc (size_t size,gfp_t flags);struct fs_parameter ;struct fc_log ;
mm/util.c renamed from mm/util.c~mm-allow-gfp_kernel-allocations-for-kvmalloc +4 -11
@@ -549,13 +549,10 @@ EXPORT_SYMBOL(vm_mmap);* Uses kmalloc to get the memory but if the allocation fails then falls back * to the vmalloc allocator. Use kvfree for freeing the memory. * -* Reclaim modifiers - __GFP_NORETRY and __GFP_NOFAIL are not supported. +* GFP_NOWAIT and GFP_ATOMIC are not supported, neither is the __GFP_NORETRY modifier. * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is * preferable to the vmalloc fallback, due to visible performance drawbacks. * -* Please note that any use of gfp flags outside of GFP_KERNEL is careful to not -* fall back to vmalloc. -* * Return: pointer to the allocated memory of %NULL in case of failure */ void *kvmalloc_node (size_t size,gfp_t flags,int node)
@@ -564,13 +561,6 @@ void *kvmalloc_node(size_t size, gfp_t fvoid *ret;/* -* vmalloc uses GFP_KERNEL for some internal allocations (e.g page tables) -* so the given set of flags has to be compatible. -*/ -if ((flags &GFP_KERNEL ) !=GFP_KERNEL ) -return kmalloc_node (size, flags, node); - -/* * We want to attempt a large physically contiguous block first because * it is less likely to fragment multiple larger blocks and therefore * contribute to a long term fragmentation less than vmalloc fallback.
@@ -582,6 +572,9 @@ void *kvmalloc_node(size_t size, gfp_t fif (!(kmalloc_flags & __GFP_RETRY_MAYFAIL)) kmalloc_flags |= __GFP_NORETRY; + +/* nofail semantic is implemented by the vmalloc fallback */ + kmalloc_flags &= ~__GFP_NOFAIL; } ret =kmalloc_node (size, kmalloc_flags, node);
net/ceph/buffer.c renamed from net/ceph/buffer.c~mm-allow-gfp_kernel-allocations-for-kvmalloc +2 -2
@@ -7,7 +7,7 @@#include <linux/ceph/buffer.h> #include <linux/ceph/decode.h> -#include <linux/ceph/libceph.h> /* for ceph_kvmalloc */ +#include <linux/ceph/libceph.h> /* for kvmalloc */ struct ceph_buffer *ceph_buffer_new (size_t len,gfp_t gfp) {
@@ -17,7 +17,7 @@ struct ceph_buffer *ceph_buffer_new(sizeif (!b)return NULL ; - b->vec.iov_base =ceph_kvmalloc (len, gfp); + b->vec.iov_base =kvmalloc (len, gfp);if (!b->vec.iov_base) {kfree (b);return NULL ;
net/ceph/ceph_common.c renamed from net/ceph/ceph_common.c~mm-allow-gfp_kernel-allocations-for-kvmalloc +0 -27
@@ -190,33 +190,6 @@ int ceph_compare_options(struct ceph_opt }EXPORT_SYMBOL (ceph_compare_options); -/* -* kvmalloc() doesn't fall back to the vmalloc allocator unless flags are -* compatible with (a superset of) GFP_KERNEL. This is because while the -* actual pages are allocated with the specified flags, the page table pages -* are always allocated with GFP_KERNEL. -* -* ceph_kvmalloc() may be called with GFP_KERNEL, GFP_NOFS or GFP_NOIO. -*/ -void *ceph_kvmalloc (size_t size,gfp_t flags) -{ -void *p; - -if ((flags & (__GFP_IO | __GFP_FS)) == (__GFP_IO | __GFP_FS)) { - p =kvmalloc (size, flags); - }else if ((flags & (__GFP_IO | __GFP_FS)) == __GFP_IO) { -unsigned int nofs_flag =memalloc_nofs_save (); - p =kvmalloc (size,GFP_KERNEL ); -memalloc_nofs_restore (nofs_flag); - }else { -unsigned int noio_flag =memalloc_noio_save (); - p =kvmalloc (size,GFP_KERNEL ); -memalloc_noio_restore (noio_flag); - } - -return p; -} -static int parse_fsid (const char *str,struct ceph_fsid *fsid) {int i =0 ;
net/ceph/crypto.c renamed from net/ceph/crypto.c~mm-allow-gfp_kernel-allocations-for-kvmalloc +1 -1
@@ -147,7 +147,7 @@ void ceph_crypto_key_destroy(struct cephstatic const u8 *aes_iv = (u8 *)CEPH_AES_IV ;/* -* Should be used for buffers allocated with ceph_kvmalloc(). +* Should be used for buffers allocated with kvmalloc(). * Currently these are encrypt out-buffer (ceph_buffer) and decrypt * in-buffer (msg front). *
net/ceph/messenger.c renamed from net/ceph/messenger.c~mm-allow-gfp_kernel-allocations-for-kvmalloc +1 -1
@@ -1920,7 +1920,7 @@ struct ceph_msg *ceph_msg_new2(int type,/* front */ if (front_len) { - m->front.iov_base =ceph_kvmalloc (front_len, flags); + m->front.iov_base =kvmalloc (front_len, flags);if (m->front.iov_base ==NULL ) {dout ("ceph_msg_new can't allocate %d bytes\n" , front_len);
net/ceph/messenger_v2.c renamed from net/ceph/messenger_v2.c~mm-allow-gfp_kernel-allocations-for-kvmalloc +1 -1
@@ -308,7 +308,7 @@ static void *alloc_conn_buf(struct ceph_if (WARN_ON (con->v2.conn_buf_cnt >=ARRAY_SIZE (con->v2.conn_bufs)))return NULL ; - buf =ceph_kvmalloc (len,GFP_NOIO ); + buf =kvmalloc (len,GFP_NOIO );if (!buf)return NULL ;
net/ceph/osdmap.c renamed from net/ceph/osdmap.c~mm-allow-gfp_kernel-allocations-for-kvmalloc +6 -6
@@ -980,7 +980,7 @@ static struct crush_work *alloc_workspac work_size =crush_work_size (c,CEPH_PG_MAX_SIZE );dout ("%s work_size %zu bytes\n" , __func__, work_size); - work =ceph_kvmalloc (work_size,GFP_NOIO ); + work =kvmalloc (work_size,GFP_NOIO );if (!work)return NULL ;
@@ -1190,9 +1190,9 @@ static int osdmap_set_max_osd(struct cepif (max == map->max_osd)return 0 ; - state =ceph_kvmalloc (array_size (max,sizeof (*state)),GFP_NOFS ); - weight =ceph_kvmalloc (array_size (max,sizeof (*weight)),GFP_NOFS ); - addr =ceph_kvmalloc (array_size (max,sizeof (*addr)),GFP_NOFS ); + state =kvmalloc (array_size (max,sizeof (*state)),GFP_NOFS ); + weight =kvmalloc (array_size (max,sizeof (*weight)),GFP_NOFS ); + addr =kvmalloc (array_size (max,sizeof (*addr)),GFP_NOFS );if (!state || !weight || !addr) {kvfree (state);kvfree (weight);
@@ -1222,7 +1222,7 @@ static int osdmap_set_max_osd(struct cepif (map->osd_primary_affinity) {u32 *affinity; - affinity =ceph_kvmalloc (array_size (max,sizeof (*affinity)), + affinity =kvmalloc (array_size (max,sizeof (*affinity)),GFP_NOFS );if (!affinity)return -ENOMEM ;
@@ -1503,7 +1503,7 @@ static int set_primary_affinity(struct cif (!map->osd_primary_affinity) {int i; - map->osd_primary_affinity =ceph_kvmalloc ( + map->osd_primary_affinity =kvmalloc (array_size (map->max_osd,sizeof (*map->osd_primary_affinity)),GFP_NOFS );if (!map->osd_primary_affinity)
_
Andrew Morton
maintainerPATCH 77/146 mm: make slab and vmalloc allocators __GFP_NOLOCKDEP aware
In reply to Andrew MortonFrom: Michal Hocko <[email protected]>
Subject: mm: make slab and vmalloc allocators __GFP_NOLOCKDEP aware
sl?b and vmalloc allocators reduce the given gfp mask for their internal needs. For that they use GFP_RECLAIM_MASK to preserve the reclaim behavior and constrains.
__GFP_NOLOCKDEP is not a part of that mask because it doesn't really control the reclaim behavior strictly speaking. On the other hand it tells the underlying page allocator to disable reclaim recursion detection so arguably it should be part of the mask.
Having __GFP_NOLOCKDEP in the mask will not alter the behavior in any form so this change is safe pretty much by definition. It also adds a support for this flag to SL?B and vmalloc allocators which will in turn allow its use to kvmalloc as well. A lack of the support has been noticed recently in http://lkml.kernel.org/r/[email protected]
---
mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h renamed from mm/internal.h~mm-make-slab-and-vmalloc-allocators-__gfp_nolockdep-aware +1 -1
@@ -21,7 +21,7 @@#define GFP_RECLAIM_MASK (__GFP_RECLAIM|__GFP_HIGH|__GFP_IO|__GFP_FS|\ __GFP_NOWARN|__GFP_RETRY_MAYFAIL|__GFP_NOFAIL|\ __GFP_NORETRY|__GFP_MEMALLOC|__GFP_NOMEMALLOC|\ - __GFP_ATOMIC) + __GFP_ATOMIC|__GFP_NOLOCKDEP)/* The GFP flags allowed during early boot */ #define GFP_BOOT_MASK (__GFP_BITS_MASK & ~(__GFP_RECLAIM|__GFP_IO|__GFP_FS))
_
Andrew Morton
maintainerPATCH 78/146 mm: introduce memalloc_retry_wait()
In reply to Andrew MortonFrom: "NeilBrown" <[email protected]>
Subject: mm: introduce memalloc_retry_wait()
Various places in the kernel - largely in filesystems - respond to a memory allocation failure by looping around and re-trying. Some of these cannot conveniently use __GFP_NOFAIL, for reasons such as:
- a GFP_ATOMIC allocation, which __GFP_NOFAIL doesn't work on
- a need to check for the process being signalled between failures
- the possibility that other recovery actions could be performed
- the allocation is quite deep in support code, and passing down an extra flag to say if __GFP_NOFAIL is wanted would be clumsy.
Many of these currently use congestion_wait() which (in almost all cases) simply waits the given timeout - congestion isn't tracked for most devices.
It isn't clear what the best delay is for loops, but it is clear that the various filesystems shouldn't be responsible for choosing a timeout.
This patch introduces memalloc_retry_wait() with takes on that responsibility. Code that wants to retry a memory allocation can call this function passing the GFP flags that were used. It will wait however is appropriate.
For now, it only considers __GFP_NORETRY and whatever gfpflags_allow_blocking() tests. If blocking is allowed without __GFP_NORETRY, then alloc_page either made some reclaim progress, or waited for a while, before failing. So there is no need for much further waiting. memalloc_retry_wait() will wait until the current jiffie ends. If this condition is not met, then alloc_page() won't have waited much if at all. In that case memalloc_retry_wait() waits about 200ms. This is the delay that most current loops uses.
linux/sched/mm.h needs to be included in some files now, but linux/backing-dev.h does not.
---
fs/ext4/extents.c | 8 +++----- fs/ext4/inline.c | 5 ++--- fs/ext4/page-io.c | 9 +++++---- fs/f2fs/data.c | 4 ++-- fs/f2fs/gc.c | 5 ++--- fs/f2fs/inode.c | 4 ++-- fs/f2fs/node.c | 4 ++-- fs/f2fs/recovery.c | 6 +++--- fs/f2fs/segment.c | 9 +++------ fs/f2fs/super.c | 5 ++--- fs/xfs/kmem.c | 3 +-- fs/xfs/xfs_buf.c | 2 +- include/linux/sched/mm.h | 26 ++++++++++++++++++++++++++ net/sunrpc/svc_xprt.c | 3 ++- 14 files changed, 56 insertions(+), 37 deletions(-)
fs/ext4/extents.c renamed from fs/ext4/extents.c~mm-introduce-memalloc_retry_wait +3 -5
@@ -27,8 +27,8 @@#include <linux/slab.h> #include <linux/uaccess.h> #include <linux/fiemap.h> -#include <linux/backing-dev.h> #include <linux/iomap.h> +#include <linux/sched/mm.h> #include "ext4_jbd2.h" #include "ext4_extents.h" #include "xattr.h"
@@ -4407,8 +4407,7 @@ retry: err =ext4_es_remove_extent (inode, last_block,EXT_MAX_BLOCKS - last_block);if (err == -ENOMEM ) { -cond_resched (); -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); +memalloc_retry_wait (GFP_ATOMIC ); goto retry; }if (err)
@@ -4416,8 +4415,7 @@ retry: retry_remove_space: err =ext4_ext_remove_space (inode, last_block,EXT_MAX_BLOCKS -1 );if (err == -ENOMEM ) { -cond_resched (); -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); +memalloc_retry_wait (GFP_ATOMIC ); goto retry_remove_space; }return err;
fs/ext4/inline.c renamed from fs/ext4/inline.c~mm-introduce-memalloc_retry_wait +2 -3
@@ -7,7 +7,7 @@#include <linux/iomap.h> #include <linux/fiemap.h> #include <linux/iversion.h> -#include <linux/backing-dev.h> +#include <linux/sched/mm.h> #include "ext4_jbd2.h" #include "ext4.h"
@@ -1929,8 +1929,7 @@ int ext4_inline_data_truncate(struct ino retry: err =ext4_es_remove_extent (inode,0 ,EXT_MAX_BLOCKS );if (err == -ENOMEM ) { -cond_resched (); -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); +memalloc_retry_wait (GFP_ATOMIC ); goto retry; }if (err)
fs/ext4/page-io.c renamed from fs/ext4/page-io.c~mm-introduce-memalloc_retry_wait +5 -4
@@ -24,7 +24,7 @@#include <linux/kernel.h> #include <linux/slab.h> #include <linux/mm.h> -#include <linux/backing-dev.h> +#include <linux/sched/mm.h> #include "ext4_jbd2.h" #include "xattr.h"
@@ -523,12 +523,13 @@ int ext4_bio_write_page(struct ext4_io_s ret =PTR_ERR (bounce_page);if (ret == -ENOMEM && (io->io_bio || wbc->sync_mode ==WB_SYNC_ALL )) { - gfp_flags =GFP_NOFS ; +gfp_t new_gfp_flags =GFP_NOFS ;if (io->io_bio)ext4_io_submit (io);else - gfp_flags |= __GFP_NOFAIL; -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); + new_gfp_flags |= __GFP_NOFAIL; +memalloc_retry_wait (gfp_flags); + gfp_flags = new_gfp_flags; goto retry_encrypt; }
fs/f2fs/data.c renamed from fs/f2fs/data.c~mm-introduce-memalloc_retry_wait +2 -2
@@ -8,9 +8,9 @@#include <linux/fs.h> #include <linux/f2fs_fs.h> #include <linux/buffer_head.h> +#include <linux/sched/mm.h> #include <linux/mpage.h> #include <linux/writeback.h> -#include <linux/backing-dev.h> #include <linux/pagevec.h> #include <linux/blkdev.h> #include <linux/bio.h>
@@ -2542,7 +2542,7 @@ retry_encrypt:/* flush pending IOs and wait for a while in the ENOMEM case */ if (PTR_ERR (fio->encrypted_page) == -ENOMEM ) {f2fs_flush_merged_writes (fio->sbi); -congestion_wait (BLK_RW_ASYNC ,DEFAULT_IO_TIMEOUT ); +memalloc_retry_wait (GFP_NOFS ); gfp_flags |= __GFP_NOFAIL; goto retry_encrypt; }
fs/f2fs/gc.c renamed from fs/f2fs/gc.c~mm-introduce-memalloc_retry_wait +2 -3
@@ -7,7 +7,6 @@*/ #include <linux/fs.h> #include <linux/module.h> -#include <linux/backing-dev.h> #include <linux/init.h> #include <linux/f2fs_fs.h> #include <linux/kthread.h>
@@ -15,6 +14,7 @@#include <linux/freezer.h> #include <linux/sched/signal.h> #include <linux/random.h> +#include <linux/sched/mm.h> #include "f2fs.h" #include "node.h"
@@ -1375,8 +1375,7 @@ retry:if (err) {clear_page_private_gcing (page);if (err == -ENOMEM ) { -congestion_wait (BLK_RW_ASYNC , -DEFAULT_IO_TIMEOUT ); +memalloc_retry_wait (GFP_NOFS ); goto retry; }if (is_dirty)
fs/f2fs/inode.c renamed from fs/f2fs/inode.c~mm-introduce-memalloc_retry_wait +2 -2
@@ -8,8 +8,8 @@#include <linux/fs.h> #include <linux/f2fs_fs.h> #include <linux/buffer_head.h> -#include <linux/backing-dev.h> #include <linux/writeback.h> +#include <linux/sched/mm.h> #include "f2fs.h" #include "node.h"
@@ -562,7 +562,7 @@ retry: inode =f2fs_iget (sb, ino);if (IS_ERR (inode)) {if (PTR_ERR (inode) == -ENOMEM ) { -congestion_wait (BLK_RW_ASYNC ,DEFAULT_IO_TIMEOUT ); +memalloc_retry_wait (GFP_NOFS ); goto retry; } }
fs/f2fs/node.c renamed from fs/f2fs/node.c~mm-introduce-memalloc_retry_wait +2 -2
@@ -8,7 +8,7 @@#include <linux/fs.h> #include <linux/f2fs_fs.h> #include <linux/mpage.h> -#include <linux/backing-dev.h> +#include <linux/sched/mm.h> #include <linux/blkdev.h> #include <linux/pagevec.h> #include <linux/swap.h>
@@ -2750,7 +2750,7 @@ int f2fs_recover_inode_page(struct f2fs_ retry: ipage =f2fs_grab_cache_page (NODE_MAPPING (sbi), ino, false);if (!ipage) { -congestion_wait (BLK_RW_ASYNC ,DEFAULT_IO_TIMEOUT ); +memalloc_retry_wait (GFP_NOFS ); goto retry; }
fs/f2fs/recovery.c renamed from fs/f2fs/recovery.c~mm-introduce-memalloc_retry_wait +3 -3
@@ -8,6 +8,7 @@#include <asm/unaligned.h> #include <linux/fs.h> #include <linux/f2fs_fs.h> +#include <linux/sched/mm.h> #include "f2fs.h" #include "node.h" #include "segment.h"
@@ -587,7 +588,7 @@ retry_dn: err =f2fs_get_dnode_of_data (&dn, start,ALLOC_NODE );if (err) {if (err == -ENOMEM ) { -congestion_wait (BLK_RW_ASYNC ,DEFAULT_IO_TIMEOUT ); +memalloc_retry_wait (GFP_NOFS ); goto retry_dn; } goto out;
@@ -670,8 +671,7 @@ retry_prev: err =check_index_in_prev_nodes (sbi, dest, &dn);if (err) {if (err == -ENOMEM ) { -congestion_wait (BLK_RW_ASYNC , -DEFAULT_IO_TIMEOUT ); +memalloc_retry_wait (GFP_NOFS ); goto retry_prev; } goto err;
fs/f2fs/segment.c renamed from fs/f2fs/segment.c~mm-introduce-memalloc_retry_wait +3 -6
@@ -9,6 +9,7 @@#include <linux/f2fs_fs.h> #include <linux/bio.h> #include <linux/blkdev.h> +#include <linux/sched/mm.h> #include <linux/prefetch.h> #include <linux/kthread.h> #include <linux/swap.h>
@@ -245,9 +246,7 @@ retry: LOOKUP_NODE);if (err) {if (err == -ENOMEM ) { -congestion_wait (BLK_RW_ASYNC , -DEFAULT_IO_TIMEOUT ); -cond_resched (); +memalloc_retry_wait (GFP_NOFS ); goto retry; } err = -EAGAIN ;
@@ -424,9 +423,7 @@ retry: err =f2fs_do_write_data_page (&fio);if (err) {if (err == -ENOMEM ) { -congestion_wait (BLK_RW_ASYNC , -DEFAULT_IO_TIMEOUT ); -cond_resched (); +memalloc_retry_wait (GFP_NOFS ); goto retry; }unlock_page (page);
fs/f2fs/super.c renamed from fs/f2fs/super.c~mm-introduce-memalloc_retry_wait +2 -3
@@ -8,9 +8,9 @@#include <linux/module.h> #include <linux/init.h> #include <linux/fs.h> +#include <linux/sched/mm.h> #include <linux/statfs.h> #include <linux/buffer_head.h> -#include <linux/backing-dev.h> #include <linux/kthread.h> #include <linux/parser.h> #include <linux/mount.h>
@@ -2415,8 +2415,7 @@ repeat: page =read_cache_page_gfp (mapping, blkidx,GFP_NOFS );if (IS_ERR (page)) {if (PTR_ERR (page) == -ENOMEM ) { -congestion_wait (BLK_RW_ASYNC , -DEFAULT_IO_TIMEOUT ); +memalloc_retry_wait (GFP_NOFS ); goto repeat; }set_sbi_flag (F2FS_SB (sb),SBI_QUOTA_NEED_REPAIR );
fs/xfs/kmem.c renamed from fs/xfs/kmem.c~mm-introduce-memalloc_retry_wait +1 -2
@@ -4,7 +4,6 @@* All Rights Reserved. */ #include "xfs.h" -#include <linux/backing-dev.h> #include "xfs_message.h" #include "xfs_trace.h"
@@ -26,6 +25,6 @@ kmem_alloc(size_t size, xfs_km_flags_t f"%s(%u) possible memory allocation deadlock size %u in %s (mode:0x%x)" , current->comm, current->pid, (unsigned int)size, __func__, lflags); -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); +memalloc_retry_wait (lflags); }while (1 ); }
fs/xfs/xfs_buf.c renamed from fs/xfs/xfs_buf.c~mm-introduce-memalloc_retry_wait +1 -1
@@ -394,7 +394,7 @@ xfs_buf_alloc_pages( }XFS_STATS_INC (bp->b_mount, xb_page_retries); -congestion_wait (BLK_RW_ASYNC ,HZ /50 ); +memalloc_retry_wait (gfp_mask); }return 0 ; }
include/linux/sched/mm.h renamed from include/linux/sched/mm.h~mm-introduce-memalloc_retry_wait +26 -0
@@ -214,6 +214,32 @@ static inline void fs_reclaim_acquire(gfstatic inline void fs_reclaim_release (gfp_t gfp_mask) { }#endif +/* Any memory-allocation retry loop should use +* memalloc_retry_wait(), and pass the flags for the most +* constrained allocation attempt that might have failed. +* This provides useful documentation of where loops are, +* and a central place to fine tune the waiting as the MM +* implementation changes. +*/ +static inline void memalloc_retry_wait (gfp_t gfp_flags) +{ +/* We use io_schedule_timeout because waiting for memory +* typically included waiting for dirty pages to be +* written out, which requires IO. +*/ +__set_current_state (TASK_UNINTERRUPTIBLE ); + gfp_flags =current_gfp_context (gfp_flags); +if (gfpflags_allow_blocking (gfp_flags) && + !(gfp_flags & __GFP_NORETRY)) +/* Probably waited already, no need for much more */ +io_schedule_timeout (1 ); +else +/* Probably didn't wait, and has now released a lock, +* so now is a good time to wait +*/ +io_schedule_timeout (HZ /50 ); +} +/** * might_alloc - Mark possible allocation sites * @gfp_mask: gfp_t flags that would be used to allocate
net/sunrpc/svc_xprt.c renamed from net/sunrpc/svc_xprt.c~mm-introduce-memalloc_retry_wait +2 -1
@@ -6,6 +6,7 @@*/ #include <linux/sched.h> +#include <linux/sched/mm.h> #include <linux/errno.h> #include <linux/freezer.h> #include <linux/kthread.h>
@@ -688,7 +689,7 @@ static int svc_alloc_arg(struct svc_rqstreturn -EINTR ; }trace_svc_alloc_arg_err (pages); -schedule_timeout (msecs_to_jiffies (500 )); +memalloc_retry_wait (GFP_KERNEL ); } rqstp->rq_page_end = &rqstp->rq_pages[pages]; rqstp->rq_pages[pages] =NULL ;/* this might be seen in nfsd_splice_actor() */
_
Andrew Morton
maintainerPATCH 79/146 mm/pagealloc: sysctl: change watermark_scale_factor max limit to 30%
In reply to Andrew MortonFrom: Suren Baghdasaryan <[email protected]>
Subject: mm/pagealloc: sysctl: change watermark_scale_factor max limit to 30%
For embedded systems with low total memory, having to run applications with relatively large memory requirements, 10% max limitation for watermark_scale_factor poses an issue of triggering direct reclaim every time such application is started. This results in slow application startup times and bad end-user experience.
By increasing watermark_scale_factor max limit we allow vendors more flexibility to choose the right level of kswapd aggressiveness for their device and workload requirements.
---
Documentation/admin-guide/sysctl/vm.rst | 2 +- kernel/sysctl.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
Documentation/admin-guide/sysctl/vm.rst renamed from Documentation/admin-guide/sysctl/vm.rst~sysctl-change-watermark_scale_factor-max-limit-to-30% +1 -1
@@ -948,7 +948,7 @@ how much memory needs to be free before The unit is in fractions of 10,000. The default value of 10 means the distances between watermarks are 0.1% of the available memory in the -node/system. The maximum value is 1000, or 10% of memory. +node/system. The maximum value is 3000, or 30% of memory. A high rate of threads entering direct reclaim (allocstall) or kswapd going to sleep prematurely (kswapd_low_wmark_hit_quickly) can indicate
kernel/sysctl.c renamed from kernel/sysctl.c~sysctl-change-watermark_scale_factor-max-limit-to-30% +2 -1
@@ -122,6 +122,7 @@ static unsigned long long_max = LONG_MAXstatic int one_hundred =100 ;static int two_hundred =200 ;static int one_thousand =1000 ; +static int three_thousand =3000 ;#ifdef CONFIG_PRINTK static int ten_thousand =10000 ;#endif
@@ -2959,7 +2960,7 @@ static struct ctl_table vm_table[] = { .mode = 0644, .proc_handler = watermark_scale_factor_sysctl_handler, .extra1 = SYSCTL_ONE, - .extra2 = &one_thousand, + .extra2 = &three_thousand, }, { .procname = "percpu_pagelist_high_fraction",
_
Andrew Morton
maintainerPATCH 80/146 mm: fix boolreturn.cocci warning
In reply to Andrew MortonFrom: Changcheng Deng <[email protected]>
Subject: mm: fix boolreturn.cocci warning
Return statements in functions returning bool should use true/false instead of 1/0.
---
include/linux/page-flags.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/page-flags.h renamed from include/linux/page-flags.h~mm-fix-boolreturncocci-warning +1 -1
@@ -383,7 +383,7 @@ static __always_inline int TestClearPage TESTCLEARFLAG(uname, lname, policy)#define TESTPAGEFLAG_FALSE (uname, lname) \ -static inline bool folio_test_##lname(const struct folio *folio) { return 0; } \ +static inline bool folio_test_##lname(const struct folio *folio) { return false; } \ static inline int Page##uname(const struct page *page) { return 0; }#define SETPAGEFLAG_NOOP (uname, lname) \
_
Andrew Morton
maintainerPATCH 81/146 mm: page_alloc: fix building error on -Werror=array-compare
In reply to Andrew MortonFrom: Xiongwei Song <[email protected]>
Subject: mm: page_alloc: fix building error on -Werror=array-compare
Arthur Marsh reported we would hit the error below when building kernel with gcc-12:
CC mm/page_alloc.omm/page_alloc.c: In function `mem_init_print_info':mm/page_alloc.c:8173:27: error: comparison between two arrays [-Werror=array-compare]
8173 | if (start <= pos && pos < end && size > adj) \
|In C++20, the comparision between arrays should be warned.
---
mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-fix-building-error-on-werror=array-compare +1 -1
@@ -8228,7 +8228,7 @@ void __init mem_init_print_info(void)*/ #define adj_init_size (start, end, size, pos, adj) \ do { \ - if (start <= pos && pos < end && size > adj) \ + if (&start[0] <= &pos[0] && &pos[0] < &end[0] && size > adj) \ size -= adj; \ } while (0)
_
Andrew Morton
maintainerPATCH 82/146 mm: drop node from alloc_pages_vma
In reply to Andrew MortonFrom: Michal Hocko <[email protected]>
Subject: mm: drop node from alloc_pages_vma
alloc_pages_vma is meant to allocate a page with a vma specific memory policy. The initial node parameter is always a local node so it is pointless to waste a function argument for this. Drop the parameter.
---
include/linux/gfp.h | 8 ++++---- mm/mempolicy.c | 3 ++- mm/shmem.c | 3 +-- 3 files changed, 7 insertions(+), 7 deletions(-)
include/linux/gfp.h renamed from include/linux/gfp.h~mm-drop-node-from-alloc_pages_vma +4 -4
@@ -598,9 +598,9 @@ struct page *alloc_pages(gfp_t gfp, unsistruct folio *folio_alloc (gfp_t gfp,unsigned order);extern struct page *alloc_pages_vma (gfp_t gfp_mask,int order,struct vm_area_struct *vma,unsigned long addr, -int node,bool hugepage); +bool hugepage);#define alloc_hugepage_vma (gfp_mask, vma, addr, order) \ - alloc_pages_vma(gfp_mask, order, vma, addr, numa_node_id(), true) + alloc_pages_vma(gfp_mask, order, vma, addr, true)#else static inline struct page *alloc_pages (gfp_t gfp_mask,unsigned int order) {
@@ -610,14 +610,14 @@ static inline struct folio *folio_alloc( {return __folio_alloc_node (gfp, order,numa_node_id ()); } -#define alloc_pages_vma (gfp_mask, order, vma, addr, node, false)\ +#define alloc_pages_vma (gfp_mask, order, vma, addr, false)\ alloc_pages(gfp_mask, order)#define alloc_hugepage_vma (gfp_mask, vma, addr, order) \ alloc_pages(gfp_mask, order)#endif #define alloc_page (gfp_mask) alloc_pages(gfp_mask, 0)#define alloc_page_vma (gfp_mask, vma, addr) \ - alloc_pages_vma(gfp_mask, 0, vma, addr, numa_node_id(), false) + alloc_pages_vma(gfp_mask, 0, vma, addr, false)extern unsigned long __get_free_pages (gfp_t gfp_mask,unsigned int order);extern unsigned long get_zeroed_page (gfp_t gfp_mask);
mm/mempolicy.c renamed from mm/mempolicy.c~mm-drop-node-from-alloc_pages_vma +2 -1
@@ -2084,9 +2084,10 @@ static struct page *alloc_pages_preferre* Return: The page on success or NULL if allocation fails. */ struct page *alloc_pages_vma (gfp_t gfp,int order,struct vm_area_struct *vma, -unsigned long addr,int node,bool hugepage) +unsigned long addr,bool hugepage) {struct mempolicy *pol; +int node =numa_node_id ();struct page *page;int preferred_nid;nodemask_t *nmask;
mm/shmem.c renamed from mm/shmem.c~mm-drop-node-from-alloc_pages_vma +1 -2
@@ -1564,8 +1564,7 @@ static struct page *shmem_alloc_hugepagereturn NULL ;shmem_pseudo_vma_init (&pvma, info, hindex); - page =alloc_pages_vma (gfp,HPAGE_PMD_ORDER , &pvma,0 ,numa_node_id (), - true); + page =alloc_pages_vma (gfp,HPAGE_PMD_ORDER , &pvma,0 , true);shmem_pseudo_vma_destroy (&pvma);if (page)prep_transhuge_page (page);
_
Andrew Morton
maintainerPATCH 83/146 include/linux/gfp.h: further document GFP_DMA32
In reply to Andrew MortonFrom: Miles Chen <[email protected]>
Subject: include/linux/gfp.h: further document GFP_DMA32
kmalloc(..., GFP_DMA32) does not return DMA32 memory because the DMA32 kmalloc cache array is not implemented. (Reason: there is no such user in kernel).
Put a short comment about this so people can understand this by reading the comment.
[1] https://lists.linuxfoundation.org/pipermail/iommu/2018-December/031696.html
---
include/linux/gfp.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
include/linux/gfp.h renamed from include/linux/gfp.h~gfp-further-document-gfp_dma32 +3 -1
@@ -302,7 +302,9 @@ struct vm_area_struct;* lowest zone as a type of emergency reserve. * * %GFP_DMA32 is similar to %GFP_DMA except that the caller requires a 32-bit -* address. +* address. Note that kmalloc(..., GFP_DMA32) does not return DMA32 memory +* because the DMA32 kmalloc cache array is not implemented. +* (Reason: there is no such user in kernel). * * %GFP_HIGHUSER is for userspace allocations that may be mapped to userspace, * do not need to be directly accessible by the kernel but that cannot
_
Andrew Morton
maintainerPATCH 84/146 mm/page_alloc.c: modify the comment section for alloc_contig_pages()
In reply to Andrew MortonFrom: Anshuman Khandual <[email protected]>
Subject: mm/page_alloc.c: modify the comment section for alloc_contig_pages()
Clarify that the alloc_contig_pages() allocated range will always be aligned to the requested nr_pages.
---
mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-modify-the-comment-section-for-alloc_contig_pages +2 -2
@@ -9272,8 +9272,8 @@ static bool zone_spans_last_pfn(const st* for allocation requests which can not be fulfilled with the buddy allocator. * * The allocated memory is always aligned to a page boundary. If nr_pages is a -* power of two then the alignment is guaranteed to be to the given nr_pages -* (e.g. 1GB request would be aligned to 1GB). +* power of two, then allocated range is also guaranteed to be aligned to same +* nr_pages (e.g. 1GB request would be aligned to 1GB). * * Allocated pages can be freed with free_contig_range() or by manually calling * __free_page() on each allocated page.
_
Andrew Morton
maintainerPATCH 85/146 mm_zone: add function to check if managed dma zone exists
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: mm_zone: add function to check if managed dma zone exists
Patch series "Handle warning of allocation failure on DMA zone w/o managed pages", v4.
**Problem observed:
On x86_64, when crash is triggered and entering into kdump kernel, page allocation failure can always be seen.
---------------------------------
DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
swapper/0: page allocation failure: order:5, mode:0xcc1(GFP_KERNEL|GFP_DMA), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 PID: 1 Comm: swapper/0 Call Trace: dump_stack+0x7f/0xa1 warn_alloc.cold+0x72/0xd6 ...... __alloc_pages+0x24d/0x2c0 ...... dma_atomic_pool_init+0xdb/0x176 do_one_initcall+0x67/0x320 ? rcu_read_lock_sched_held+0x3f/0x80 kernel_init_freeable+0x290/0x2dc ? rest_init+0x24f/0x24f kernel_init+0xa/0x111 ret_from_fork+0x22/0x30 Mem-Info: ------------------------------------
***Root cause:
In the current kernel, it assumes that DMA zone must have managed pages and try to request pages if CONFIG_ZONE_DMA is enabled. While this is not always true. E.g in kdump kernel of x86_64, only low 1M is presented and locked down at very early stage of boot, so that this low 1M won't be added into buddy allocator to become managed pages of DMA zone. This exception will always cause page allocation failure if page is requested from DMA zone.
***Investigation:
This failure happens since below commit merged into linus's tree.
1a6a9044b967 x86/setup: Remove CONFIG_X86_RESERVE_LOW and reservelow= options 23721c8e92f7 x86/crash: Remove crash_reserve_low_1M()
f1d4d47c5851 x86/setup: Always reserve the first 1M of RAM
7c321eb2b843 x86/kdump: Remove the backup region handling
6f599d84231f x86/kdump: Always reserve the low 1M when the crashkernel option is specified
Before them, on x86_64, the low 640K area will be reused by kdump kernel. So in kdump kernel, the content of low 640K area is copied into a backup region for dumping before jumping into kdump. Then except of those firmware reserved region in [0, 640K], the left area will be added into buddy allocator to become available managed pages of DMA zone.
However, after above commits applied, in kdump kernel of x86_64, the low 1M is reserved by memblock, but not released to buddy allocator. So any later page allocation requested from DMA zone will fail.
At the beginning, if crashkernel is reserved, the low 1M need be locked down because AMD SME encrypts memory making the old backup region mechanims impossible when switching into kdump kernel.
Later, it was also observed that there are BIOSes corrupting memory under 1M. To solve this, in commit f1d4d47c5851, the entire region of low 1M is always reserved after the real mode trampoline is allocated.
Besides, recently, Intel engineer mentioned their TDX (Trusted domain extensions) which is under development in kernel also needs to lock down the low 1M. So we can't simply revert above commits to fix the page allocation failure from DMA zone as someone suggested.
***Solution:
Currently, only DMA atomic pool and dma-kmalloc will initialize and request page allocation with GFP_DMA during bootup.
So only initializ DMA atomic pool when DMA zone has available managed pages, otherwise just skip the initialization.
For dma-kmalloc(), for the time being, let's mute the warning of allocation failure if requesting pages from DMA zone while no manged pages. Meanwhile, change code to use dma_alloc_xx/dma_map_xx API to replace kmalloc(GFP_DMA), or do not use GFP_DMA when calling kmalloc() if not necessary. Christoph is posting patches to fix those under drivers/scsi/. Finally, we can remove the need of dma-kmalloc() as people suggested.
This patch (of 3):
In some places of the current kernel, it assumes that dma zone must have managed pages if CONFIG_ZONE_DMA is enabled. While this is not always true. E.g in kdump kernel of x86_64, only low 1M is presented and locked down at very early stage of boot, so that there's no managed pages at all in DMA zone. This exception will always cause page allocation failure if page is requested from DMA zone.
Here add function has_managed_dma() and the relevant helper functions to check if there's DMA zone with managed pages. It will be used in later patches.
---
include/linux/mmzone.h | 9 +++++++++ mm/page_alloc.c | 15 +++++++++++++++ 2 files changed, 24 insertions(+)
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm_zone-add-function-to-check-if-managed-dma-zone-exists +9 -0
@@ -1047,6 +1047,15 @@ static inline int is_highmem_idx(enum zo#endif } +#ifdef CONFIG_ZONE_DMA +bool has_managed_dma (void ); +#else +static inline bool has_managed_dma (void ) +{ +return false; +} +#endif +/** * is_highmem - helper function to quickly check if a struct zone is a * highmem zone or not. This is an attempt to keep references
mm/page_alloc.c renamed from mm/page_alloc.c~mm_zone-add-function-to-check-if-managed-dma-zone-exists +15 -0
@@ -9518,3 +9518,18 @@ bool take_page_off_buddy(struct page *pa return ret; }#endif + +#ifdef CONFIG_ZONE_DMA +bool has_managed_dma (void ) +{ +struct pglist_data *pgdat; + +for_each_online_pgdat (pgdat) { +struct zone *zone = &pgdat->node_zones[ZONE_DMA ]; + +if (managed_zone (zone)) +return true; + } +return false; +} +#endif /* CONFIG_ZONE_DMA */
_
Andrew Morton
maintainerPATCH 86/146 dma/pool: create dma atomic pool only if dma zone has managed pages
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: dma/pool: create dma atomic pool only if dma zone has managed pages
Currently three dma atomic pools are initialized as long as the relevant kernel codes are built in. While in kdump kernel of x86_64, this is not right when trying to create atomic_pool_dma, because there's no managed pages in DMA zone. In the case, DMA zone only has low 1M memory presented and locked down by memblock allocator. So no pages are added into buddy of DMA zone. Please check commit f1d4d47c5851 ("x86/setup: Always reserve the first 1M of RAM").
Then in kdump kernel of x86_64, it always prints below failure message:
DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
swapper/0: page allocation failure: order:5, mode:0xcc1(GFP_KERNEL|GFP_DMA), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.13.0-0.rc5.20210611git929d931f2b40.42.fc35.x86_64 #1 Hardware name: Dell Inc. PowerEdge R910/0P658H, BIOS 2.12.0 06/04/2018 Call Trace: dump_stack+0x7f/0xa1 warn_alloc.cold+0x72/0xd6 ? _raw_spin_unlock_irq+0x24/0x40 ? __alloc_pages_direct_compact+0x90/0x1b0 __alloc_pages_slowpath.constprop.0+0xf29/0xf50 ? __cond_resched+0x16/0x50 ? prepare_alloc_pages.constprop.0+0x19d/0x1b0 __alloc_pages+0x24d/0x2c0 ? __dma_atomic_pool_init+0x93/0x93 alloc_page_interleave+0x13/0xb0 atomic_pool_expand+0x118/0x210 ? __dma_atomic_pool_init+0x93/0x93 __dma_atomic_pool_init+0x45/0x93 dma_atomic_pool_init+0xdb/0x176 do_one_initcall+0x67/0x320 ? rcu_read_lock_sched_held+0x3f/0x80 kernel_init_freeable+0x290/0x2dc ? rest_init+0x24f/0x24f kernel_init+0xa/0x111 ret_from_fork+0x22/0x30 Mem-Info: ...... DMA: failed to allocate 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocation DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
Here, let's check if DMA zone has managed pages, then create atomic_pool_dma if yes. Otherwise just skip it.
---
kernel/dma/pool.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/dma/pool.c renamed from kernel/dma/pool.c~dma-pool-create-dma-atomic-pool-only-if-dma-zone-has-managed-pages +2 -2
@@ -203,7 +203,7 @@ static int __init dma_atomic_pool_init(v GFP_KERNEL);if (!atomic_pool_kernel) ret = -ENOMEM ; -if (IS_ENABLED (CONFIG_ZONE_DMA )) { +if (has_managed_dma ()) { atomic_pool_dma =__dma_atomic_pool_init (atomic_pool_size,GFP_KERNEL |GFP_DMA );if (!atomic_pool_dma)
@@ -226,7 +226,7 @@ static inline struct gen_pool *dma_guessif (prev ==NULL ) {if (IS_ENABLED (CONFIG_ZONE_DMA32 ) && (gfp &GFP_DMA32 ))return atomic_pool_dma32; -if (IS_ENABLED (CONFIG_ZONE_DMA ) && (gfp &GFP_DMA )) +if (atomic_pool_dma && (gfp &GFP_DMA ))return atomic_pool_dma;return atomic_pool_kernel; }
_
Andrew Morton
maintainerPATCH 87/146 mm/page_alloc.c: do not warn allocation failure on zone DMA if no managed pages
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: mm/page_alloc.c: do not warn allocation failure on zone DMA if no managed pages
In kdump kernel of x86_64, page allocation failure is observed:
kworker/u2:2: page allocation failure: order:0, mode:0xcc1(GFP_KERNEL|GFP_DMA), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 PID: 55 Comm: kworker/u2:2 Not tainted 5.16.0-rc4+ #5 Hardware name: AMD Dinar/Dinar, BIOS RDN1505B 06/05/2013 Workqueue: events_unbound async_run_entry_fn Call Trace: <TASK> dump_stack_lvl+0x48/0x5e warn_alloc.cold+0x72/0xd6 __alloc_pages_slowpath.constprop.0+0xc69/0xcd0 __alloc_pages+0x1df/0x210 new_slab+0x389/0x4d0 ___slab_alloc+0x58f/0x770 __slab_alloc.constprop.0+0x4a/0x80 kmem_cache_alloc_trace+0x24b/0x2c0 sr_probe+0x1db/0x620 ...... device_add+0x405/0x920 ...... __scsi_add_device+0xe5/0x100 ata_scsi_scan_host+0x97/0x1d0 async_run_entry_fn+0x30/0x130 process_one_work+0x1e8/0x3c0 worker_thread+0x50/0x3b0 ? rescuer_thread+0x350/0x350 kthread+0x16b/0x190 ? set_kthread_struct+0x40/0x40 ret_from_fork+0x22/0x30 </TASK> Mem-Info: ......
The above failure happened when calling kmalloc() to allocate buffer with GFP_DMA. It requests to allocate slab page from DMA zone while no managed pages at all in there.
sr_probe()
--> get_capabilities()
--> buffer = kmalloc(512, GFP_KERNEL | GFP_DMA);
Because in the current kernel, dma-kmalloc will be created as long as CONFIG_ZONE_DMA is enabled. However, kdump kernel of x86_64 doesn't have managed pages on DMA zone since commit 6f599d84231f ("x86/kdump: Always reserve the low 1M when the crashkernel option is specified"). The failure can be always reproduced.
For now, let's mute the warning of allocation failure if requesting pages from DMA zone while no managed pages.
---
mm/page_alloc.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-do-not-warn-allocation-failure-on-zone-dma-if-no-managed-pages +3 -1
@@ -4218,7 +4218,9 @@ void warn_alloc(gfp_t gfp_mask, nodemaskva_list args; static DEFINE_RATELIMIT_STATE(nopage_rs, 10*HZ, 1); -if ((gfp_mask & __GFP_NOWARN) || !__ratelimit (&nopage_rs)) +if ((gfp_mask & __GFP_NOWARN) || + !__ratelimit (&nopage_rs) || + ((gfp_mask & __GFP_DMA) && !has_managed_dma ()))return ;va_start (args, fmt);
_
Andrew Morton
maintainerPATCH 88/146 hugetlb: add hugetlb.*.numa_stat file
In reply to Andrew MortonFrom: Mina Almasry <[email protected]>
Subject: hugetlb: add hugetlb.*.numa_stat file
For hugetlb backed jobs/VMs it's critical to understand the numa information for the memory backing these jobs to deliver optimal performance.
Currently this technically can be queried from /proc/self/numa_maps, but there are significant issues with that. Namely:
1. Memory can be mapped or unmapped.
2. numa_maps are per process and need to be aggregated across all processes in the cgroup. For shared memory this is more involved as the userspace needs to make sure it doesn't double count shared mappings.
3. I believe querying numa_maps needs to hold the mmap_lock which adds to the contention on this lock.
For these reasons I propose simply adding hugetlb.*.numa_stat file,
which shows the numa information of the cgroup similarly to memory.numa_stat.
On cgroup-v2:
cat /sys/fs/cgroup/unified/test/hugetlb.2MB.numa_stat total=2097152 N0=2097152 N1=0
On cgroup-v1:
cat /sys/fs/cgroup/hugetlb/test/hugetlb.2MB.numa_stat total=2097152 N0=2097152 N1=0 hierarichal_total=2097152 N0=2097152 N1=0
This patch was tested manually by allocating hugetlb memory and querying the hugetlb.*.numa_stat file of the cgroup and its parents.
---
Documentation/admin-guide/cgroup-v1/hugetlb.rst | 4 Documentation/admin-guide/cgroup-v2.rst | 5 include/linux/hugetlb.h | 4 include/linux/hugetlb_cgroup.h | 7 mm/hugetlb_cgroup.c | 133 ++++++++++++-- 5 files changed, 141 insertions(+), 12 deletions(-)
Documentation/admin-guide/cgroup-v1/hugetlb.rst renamed from Documentation/admin-guide/cgroup-v1/hugetlb.rst~hugetlb-add-hugetlbnuma_stat-file +4 -0
@@ -29,12 +29,14 @@ Brief summary of control files:: hugetlb.<hugepagesize>.max_usage_in_bytes # show max "hugepagesize" hugetlb usage recorded hugetlb.<hugepagesize>.usage_in_bytes # show current usage for "hugepagesize" hugetlb hugetlb.<hugepagesize>.failcnt # show the number of allocation failure due to HugeTLB usage limit + hugetlb.<hugepagesize>.numa_stat # show the numa information of the hugetlb memory charged to this cgroup For a system supporting three hugepage sizes (64k, 32M and 1G), the control files include:: hugetlb.1GB.limit_in_bytes hugetlb.1GB.max_usage_in_bytes + hugetlb.1GB.numa_stat hugetlb.1GB.usage_in_bytes hugetlb.1GB.failcnt hugetlb.1GB.rsvd.limit_in_bytes
@@ -43,6 +45,7 @@ files include:: hugetlb.1GB.rsvd.failcnt hugetlb.64KB.limit_in_bytes hugetlb.64KB.max_usage_in_bytes + hugetlb.64KB.numa_stat hugetlb.64KB.usage_in_bytes hugetlb.64KB.failcnt hugetlb.64KB.rsvd.limit_in_bytes
@@ -51,6 +54,7 @@ files include:: hugetlb.64KB.rsvd.failcnt hugetlb.32MB.limit_in_bytes hugetlb.32MB.max_usage_in_bytes + hugetlb.32MB.numa_stat hugetlb.32MB.usage_in_bytes hugetlb.32MB.failcnt hugetlb.32MB.rsvd.limit_in_bytes
Documentation/admin-guide/cgroup-v2.rst renamed from Documentation/admin-guide/cgroup-v2.rst~hugetlb-add-hugetlbnuma_stat-file +5 -0
@@ -2266,6 +2266,11 @@ HugeTLB Interface Files are local to the cgroup i.e. not hierarchical. The file modified event generated on this file reflects only the local events. + hugetlb.<hugepagesize>.numa_stat + Similar to memory.numa_stat, it shows the numa information of the + hugetlb pages of <hugepagesize> in this cgroup. Only active in + use hugetlb pages are included. The per-node values are in bytes. + Misc ----
include/linux/hugetlb_cgroup.h renamed from include/linux/hugetlb_cgroup.h~hugetlb-add-hugetlbnuma_stat-file +7 -0
@@ -36,6 +36,11 @@ enum hugetlb_memory_event { HUGETLB_NR_MEMORY_EVENTS, }; +struct hugetlb_cgroup_per_node { +/* hugetlb usage in pages over all hstates. */ +unsigned long usage[HUGE_MAX_HSTATE ]; +}; +struct hugetlb_cgroup {struct cgroup_subsys_state css;
@@ -57,6 +62,8 @@ struct hugetlb_cgroup {/* Handle for "hugetlb.events.local" */ struct cgroup_file events_local_file[HUGE_MAX_HSTATE ]; + +struct hugetlb_cgroup_per_node *nodeinfo[]; }; static inline struct hugetlb_cgroup *
include/linux/hugetlb.h renamed from include/linux/hugetlb.h~hugetlb-add-hugetlbnuma_stat-file +2 -2
@@ -622,8 +622,8 @@ struct hstate {#endif #ifdef CONFIG_CGROUP_HUGETLB /* cgroup control files */ -struct cftype cgroup_files_dfl[7 ]; -struct cftype cgroup_files_legacy[9 ]; +struct cftype cgroup_files_dfl[8 ]; +struct cftype cgroup_files_legacy[10 ];#endif char name[HSTATE_NAME_LEN ]; };
mm/hugetlb_cgroup.c renamed from mm/hugetlb_cgroup.c~hugetlb-add-hugetlbnuma_stat-file +123 -10
189 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 89/146 mm, hugepages: make memory size variable in hugepage-mremap selftest
In reply to Andrew MortonFrom: Yosry Ahmed <[email protected]>
Subject: mm, hugepages: make memory size variable in hugepage-mremap selftest
The hugetlb vma mremap() test currently maps 1GB of memory to trigger pmd sharing and make sure that 'unshare' path in mremap code works. The test originally only mapped 10MB of memory (as specified by the header comment) but was later modified to 1GB to tackle this case.
However, not all machines will have 1GB of memory to spare for this test. Adding a mapping size arg will allow run_vmtest.sh to pass an adequate mapping size, while allowing users to run the test independently with arbitrary size mappings.
---
tools/testing/selftests/vm/hugepage-mremap.c | 46 +++++++++++------ tools/testing/selftests/vm/run_vmtests.sh | 2 2 files changed, 31 insertions(+), 17 deletions(-)
tools/testing/selftests/vm/hugepage-mremap.c renamed from tools/testing/selftests/vm/hugepage-mremap.c~mm-hugepages-make-memory-size-variable-in-hugepage-mremap-selftest +30 -16
@@ -4,7 +4,11 @@* * Example of remapping huge page memory in a user application using the * mremap system call. Code assumes a hugetlbfs filesystem is mounted -* at './huge'. The code will use 10MB worth of huge pages. +* at './huge'. The amount of memory used by this test is decided by a command +* line argument in MBs. If missing, the default amount is 10MB. +* +* To make sure the test triggers pmd sharing and goes through the 'unshare' +* path in the mremap code use 1GB (1024) or more. */ #define _GNU_SOURCE
@@ -18,8 +22,10 @@#include <linux/userfaultfd.h> #include <sys/ioctl.h> -#define LENGTH (1UL * 1024 * 1024 * 1024) +#define DEFAULT_LENGTH_MB 10UL +#define MB_TO_BYTES (x) (x * 1024 * 1024) +#define FILE_NAME "huge/hugepagefile"#define PROTECTION (PROT_READ | PROT_WRITE | PROT_EXEC)#define FLAGS (MAP_SHARED | MAP_ANONYMOUS)
@@ -28,20 +34,20 @@ static void check_bytes(char *addr)printf ("First hex is %x\n" , *((unsigned int *)addr)); } -static void write_bytes (char *addr) +static void write_bytes (char *addr,size_t len) {unsigned long i; -for (i =0 ; i <LENGTH ; i++) +for (i =0 ; i < len; i++) *(addr + i) = (char )i; } -static int read_bytes (char *addr) +static int read_bytes (char *addr,size_t len) {unsigned long i;check_bytes (addr); -for (i =0 ; i <LENGTH ; i++) +for (i =0 ; i < len; i++)if (*(addr + i) != (char )i) {printf ("Mismatch at %lu\n" , i);return 1 ;
@@ -99,11 +105,19 @@ static void register_region_with_uffd(ch } } -int main (void ) +int main (int argc,char *argv[]) { +/* Read memory length as the first arg if valid, otherwise fallback to +* the default length. Any additional args are ignored. +*/ +size_t length = argc >1 ? (size_t )atoi (argv[1 ]) :0UL ; + + length = length >0 ? length :DEFAULT_LENGTH_MB ; + length =MB_TO_BYTES (length); +int ret =0 ; -int fd =open ("/huge/test" ,O_CREAT |O_RDWR ,0755 ); +int fd =open (FILE_NAME ,O_CREAT |O_RDWR ,0755 );if (fd <0 ) {perror ("Open failed" );
@@ -112,7 +126,7 @@ int main(void)/* mmap to a PUD aligned address to hopefully trigger pmd sharing. */ unsigned long suggested_addr =0x7eaa40000000 ; -void *haddr =mmap ((void *)suggested_addr,LENGTH ,PROTECTION , +void *haddr =mmap ((void *)suggested_addr, length,PROTECTION ,MAP_HUGETLB |MAP_SHARED |MAP_POPULATE , fd,0 );printf ("Map haddr: Returned address is %p\n" , haddr);if (haddr ==MAP_FAILED ) {
@@ -122,7 +136,7 @@ int main(void)/* mmap again to a dummy address to hopefully trigger pmd sharing. */ suggested_addr =0x7daa40000000 ; -void *daddr =mmap ((void *)suggested_addr,LENGTH ,PROTECTION , +void *daddr =mmap ((void *)suggested_addr, length,PROTECTION ,MAP_HUGETLB |MAP_SHARED |MAP_POPULATE , fd,0 );printf ("Map daddr: Returned address is %p\n" , daddr);if (daddr ==MAP_FAILED ) {
@@ -132,16 +146,16 @@ int main(void) suggested_addr =0x7faa40000000 ;void *vaddr = -mmap ((void *)suggested_addr,LENGTH ,PROTECTION ,FLAGS ,-1 ,0 ); +mmap ((void *)suggested_addr, length,PROTECTION ,FLAGS ,-1 ,0 );printf ("Map vaddr: Returned address is %p\n" , vaddr);if (vaddr ==MAP_FAILED ) {perror ("mmap2" );exit (1 ); } -register_region_with_uffd (haddr,LENGTH ); +register_region_with_uffd (haddr, length); -void *addr =mremap (haddr,LENGTH ,LENGTH , +void *addr =mremap (haddr, length, length,MREMAP_MAYMOVE |MREMAP_FIXED , vaddr);if (addr ==MAP_FAILED ) {perror ("mremap" );
@@ -150,10 +164,10 @@ int main(void)printf ("Mremap: Returned address is %p\n" , addr);check_bytes (addr); -write_bytes (addr); - ret =read_bytes (addr); +write_bytes (addr, length); + ret =read_bytes (addr, length); -munmap (addr,LENGTH ); +munmap (addr, length);return ret; }
tools/testing/selftests/vm/run_vmtests.sh renamed from tools/testing/selftests/vm/run_vmtests.sh~mm-hugepages-make-memory-size-variable-in-hugepage-mremap-selftest +1 -1
@@ -111,7 +111,7 @@ fi echo "-----------------------" echo "running hugepage-mremap" echo "-----------------------" -./hugepage-mremap +./hugepage-mremap 256 if [ $? -ne 0 ]; then echo "[FAIL]" exitcode=1
_
Andrew Morton
maintainerPATCH 90/146 mm/vmstat: add events for THP max_ptes_* exceeds
In reply to Andrew MortonFrom: Yang Yang <[email protected]>
Subject: mm/vmstat: add events for THP max_ptes_* exceeds
There are interfaces to adjust max_ptes_none, max_ptes_swap, max_ptes_shared values, see /sys/kernel/mm/transparent_hugepage/khugepaged/.
But system administrator maynot know which value is the best. So Add those events to support adjusting max_ptes_* to suitable values.
For example, if default max_ptes_swap value causes too much failures, and system uses zram whose IO is fast, administrator could increase max_ptes_swap until THP_SCAN_EXCEED_SWAP_PTE not increase anymore.
---
include/linux/vm_event_item.h | 3 +++ mm/khugepaged.c | 7 +++++++ mm/vmstat.c | 3 +++ 3 files changed, 13 insertions(+)
include/linux/vm_event_item.h renamed from include/linux/vm_event_item.h~mm-vmstat-add-events-for-thp-max_ptes_-exceeds +3 -0
@@ -98,6 +98,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSTHP_SPLIT_PAGE_FAILED ,THP_DEFERRED_SPLIT_PAGE ,THP_SPLIT_PMD , +THP_SCAN_EXCEED_NONE_PTE , +THP_SCAN_EXCEED_SWAP_PTE , +THP_SCAN_EXCEED_SHARED_PTE , #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD THP_SPLIT_PUD, #endif
mm/khugepaged.c renamed from mm/khugepaged.c~mm-vmstat-add-events-for-thp-max_ptes_-exceeds +7 -0
@@ -618,6 +618,7 @@ static int __collapse_huge_page_isolate(continue ; }else { result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); goto out; } }
@@ -636,6 +637,7 @@ static int __collapse_huge_page_isolate(if (page_mapcount (page) >1 && ++shared > khugepaged_max_ptes_shared) { result =SCAN_EXCEED_SHARED_PTE ; +count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); goto out; }
@@ -1253,6 +1255,7 @@ static int khugepaged_scan_pmd(struct mmcontinue ; }else { result =SCAN_EXCEED_SWAP_PTE ; +count_vm_event (THP_SCAN_EXCEED_SWAP_PTE ); goto out_unmap; } }
@@ -1262,6 +1265,7 @@ static int khugepaged_scan_pmd(struct mmcontinue ; }else { result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); goto out_unmap; } }
@@ -1290,6 +1294,7 @@ static int khugepaged_scan_pmd(struct mmif (page_mapcount (page) >1 && ++shared > khugepaged_max_ptes_shared) { result =SCAN_EXCEED_SHARED_PTE ; +count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); goto out_unmap; }
@@ -2000,6 +2005,7 @@ static void khugepaged_scan_file(structif (xa_is_value (page)) {if (++swap > khugepaged_max_ptes_swap) { result =SCAN_EXCEED_SWAP_PTE ; +count_vm_event (THP_SCAN_EXCEED_SWAP_PTE );break ; }continue ;
@@ -2046,6 +2052,7 @@ static void khugepaged_scan_file(structif (result ==SCAN_SUCCEED ) {if (present <HPAGE_PMD_NR - khugepaged_max_ptes_none) { result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); }else { node =khugepaged_find_target_node ();collapse_file (mm, file, start, hpage, node);
mm/vmstat.c renamed from mm/vmstat.c~mm-vmstat-add-events-for-thp-max_ptes_-exceeds +3 -0
@@ -1353,6 +1353,9 @@ const char * const vmstat_text[] = {"thp_split_page_failed" ,"thp_deferred_split_page" ,"thp_split_pmd" , +"thp_scan_exceed_none_pte" , +"thp_scan_exceed_swap_pte" , +"thp_scan_exceed_share_pte" , #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD "thp_split_pud", #endif
_
Andrew Morton
maintainerPATCH 91/146 selftests/vm: make charge_reserved_hugetlb.sh work with existing cgroup setting
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: selftests/vm: make charge_reserved_hugetlb.sh work with existing cgroup setting
The hugetlb cgroup reservation test charge_reserved_hugetlb.sh assume that no cgroup filesystems are mounted before running the test. That is not true in many cases. As a result, the test fails to run. Fix that by querying the current cgroup mount setting and using the existing cgroup setup instead before attempting to freshly mount a cgroup filesystem.
Similar change is also made for hugetlb_reparenting_test.sh as well, though it still has problem if cgroup v2 isn't used.
The patched test scripts were run on a centos 8 based system to verify that they ran properly.
---
tools/testing/selftests/vm/charge_reserved_hugetlb.sh | 34 +++++----- tools/testing/selftests/vm/hugetlb_reparenting_test.sh | 21 +++--- tools/testing/selftests/vm/write_hugetlb_memory.sh | 2 3 files changed, 34 insertions(+), 23 deletions(-)
tools/testing/selftests/vm/charge_reserved_hugetlb.sh renamed from tools/testing/selftests/vm/charge_reserved_hugetlb.sh~selftests-vm-make-charge_reserved_hugetlbsh-work-with-existing-cgroup-setting +20 -14
@@ -24,19 +24,23 @@ if [[ "$1" == "-cgroup-v2" ]]; then reservation_usage_file=rsvd.current fi -cgroup_path=/dev/cgroup/memory -if [[ ! -e $cgroup_path ]]; then - mkdir -p $cgroup_path - if [[ $cgroup2 ]]; then +if [[ $cgroup2 ]]; then + cgroup_path=$(mount -t cgroup2 | head -1 | awk -e '{print $3}') + if [[ -z "$cgroup_path" ]]; then + cgroup_path=/dev/cgroup/memory mount -t cgroup2 none $cgroup_path - else + do_umount=1 + fi + echo "+hugetlb" >$cgroup_path/cgroup.subtree_control +else + cgroup_path=$(mount -t cgroup | grep ",hugetlb" | awk -e '{print $3}') + if [[ -z "$cgroup_path" ]]; then + cgroup_path=/dev/cgroup/memory mount -t cgroup memory,hugetlb $cgroup_path + do_umount=1 fi fi - -if [[ $cgroup2 ]]; then - echo "+hugetlb" >/dev/cgroup/memory/cgroup.subtree_control -fi +export cgroup_path function cleanup() { if [[ $cgroup2 ]]; then
@@ -108,7 +112,7 @@ function setup_cgroup() { function wait_for_hugetlb_memory_to_get_depleted() { local cgroup="$1" - local path="/dev/cgroup/memory/$cgroup/hugetlb.${MB}MB.$reservation_usage_file" + local path="$cgroup_path/$cgroup/hugetlb.${MB}MB.$reservation_usage_file" # Wait for hugetlbfs memory to get depleted. while [ $(cat $path) != 0 ]; do echo Waiting for hugetlb memory to get depleted.
@@ -121,7 +125,7 @@ function wait_for_hugetlb_memory_to_get_ local cgroup="$1" local size="$2" - local path="/dev/cgroup/memory/$cgroup/hugetlb.${MB}MB.$reservation_usage_file" + local path="$cgroup_path/$cgroup/hugetlb.${MB}MB.$reservation_usage_file" # Wait for hugetlbfs memory to get written. while [ $(cat $path) != $size ]; do echo Waiting for hugetlb memory reservation to reach size $size.
@@ -134,7 +138,7 @@ function wait_for_hugetlb_memory_to_get_ local cgroup="$1" local size="$2" - local path="/dev/cgroup/memory/$cgroup/hugetlb.${MB}MB.$fault_usage_file" + local path="$cgroup_path/$cgroup/hugetlb.${MB}MB.$fault_usage_file" # Wait for hugetlbfs memory to get written. while [ $(cat $path) != $size ]; do echo Waiting for hugetlb memory to reach size $size.
@@ -574,5 +578,7 @@ for populate in "" "-o"; do done # populate done # method -umount $cgroup_path -rmdir $cgroup_path +if [[ $do_umount ]]; then + umount $cgroup_path + rmdir $cgroup_path +fi
tools/testing/selftests/vm/hugetlb_reparenting_test.sh renamed from tools/testing/selftests/vm/hugetlb_reparenting_test.sh~selftests-vm-make-charge_reserved_hugetlbsh-work-with-existing-cgroup-setting +13 -8
@@ -18,19 +18,24 @@ if [[ "$1" == "-cgroup-v2" ]]; then usage_file=current fi -CGROUP_ROOT='/dev/cgroup/memory' -MNT='/mnt/huge/' -if [[ ! -e $CGROUP_ROOT ]]; then - mkdir -p $CGROUP_ROOT - if [[ $cgroup2 ]]; then +if [[ $cgroup2 ]]; then + CGROUP_ROOT=$(mount -t cgroup2 | head -1 | awk -e '{print $3}') + if [[ -z "$CGROUP_ROOT" ]]; then + CGROUP_ROOT=/dev/cgroup/memory mount -t cgroup2 none $CGROUP_ROOT - sleep 1 - echo "+hugetlb +memory" >$CGROUP_ROOT/cgroup.subtree_control - else + do_umount=1 + fi + echo "+hugetlb +memory" >$CGROUP_ROOT/cgroup.subtree_control +else + CGROUP_ROOT=$(mount -t cgroup | grep ",hugetlb" | awk -e '{print $3}') + if [[ -z "$CGROUP_ROOT" ]]; then + CGROUP_ROOT=/dev/cgroup/memory mount -t cgroup memory,hugetlb $CGROUP_ROOT + do_umount=1 fi fi +MNT='/mnt/huge/' function get_machine_hugepage_size() { hpz=$(grep -i hugepagesize /proc/meminfo)
tools/testing/selftests/vm/write_hugetlb_memory.sh renamed from tools/testing/selftests/vm/write_hugetlb_memory.sh~selftests-vm-make-charge_reserved_hugetlbsh-work-with-existing-cgroup-setting +1 -1
@@ -14,7 +14,7 @@ want_sleep=$8 reserve=$9 echo "Putting task in cgroup '$cgroup'" -echo $$ > /dev/cgroup/memory/"$cgroup"/cgroup.procs +echo $$ > ${cgroup_path:-/dev/cgroup/memory}/"$cgroup"/cgroup.procs echo "Method is $method"
_
Andrew Morton
maintainerPATCH 92/146 selftests/uffd: allow EINTR/EAGAIN
In reply to Andrew MortonFrom: Peter Xu <[email protected]>
Subject: selftests/uffd: allow EINTR/EAGAIN
This allow test to continue with interruptions like gdb.
---
tools/testing/selftests/vm/userfaultfd.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/testing/selftests/vm/userfaultfd.c renamed from tools/testing/selftests/vm/userfaultfd.c~selftests-uffd-allow-eintr-eagain +5 -2
@@ -648,7 +648,7 @@ static int uffd_read_msg(int ufd, structif (ret !=sizeof (*msg)) {if (ret <0 ) { -if (errno ==EAGAIN ) +if (errno ==EAGAIN || errno ==EINTR )return 1 ;err ("blocking read error" ); }else {
@@ -724,8 +724,11 @@ static void *uffd_poll_thread(void *arg)for (;;) { ret =poll (pollfd,2 ,-1 ); -if (ret <=0 ) +if (ret <=0 ) { +if (errno ==EINTR || errno ==EAGAIN ) +continue ;err ("poll error: %d" , ret); + }if (pollfd[1 ].revents &POLLIN ) {if (read (pollfd[1 ].fd, &tmp_chr,1 ) !=1 )err ("read pipefd error" );
_
Andrew Morton
maintainerPATCH 93/146 userfaultfd/selftests: clean up hugetlb allocation code
In reply to Andrew MortonFrom: Mike Kravetz <[email protected]>
Subject: userfaultfd/selftests: clean up hugetlb allocation code
The message for commit f5c73297181c ("userfaultfd/selftests: fix hugetlb area allocations") says there is no need to create a hugetlb file in the non-shared testing case. However, the commit did not actually change the code to prevent creation of the file.
While it is technically true that there is no need to create and use a hugetlb file in the case of non-shared-testing, it is useful. This is because 'hole punching' of a hugetlb file has the potentially incorrect side effect of also removing pages from private mappings. The userfaultfd test relies on this side effect for removing pages from the destination buffer during rounds of stress testing.
Remove the incomplete code that was added to deal with no hugetlb file. Just keep the code that prevents reserves from being created for the destination area.
---
tools/testing/selftests/vm/userfaultfd.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)
tools/testing/selftests/vm/userfaultfd.c renamed from tools/testing/selftests/vm/userfaultfd.c~userfaultfd-selftests-clean-up-hugetlb-allocation-code +6 -9
@@ -87,7 +87,7 @@ static bool test_uffdio_minor = false;static bool map_shared;static int shm_fd; -static int huge_fd =-1 ;/* only used for hugetlb_shared test */ +static int huge_fd;static char *huge_fd_off0;static unsigned long long *count_verify;static int uffd =-1 ;
@@ -223,9 +223,6 @@ static void noop_alias_mapping(__u64 *ststatic void hugetlb_release_pages (char *rel_area) { -if (huge_fd ==-1 ) -return ; -if (fallocate (huge_fd,FALLOC_FL_PUNCH_HOLE |FALLOC_FL_KEEP_SIZE , rel_area == huge_fd_off0 ?0 : nr_pages * page_size, nr_pages * page_size))
@@ -238,17 +235,17 @@ static void hugetlb_allocate_area(void *char **alloc_area_alias; *alloc_area =mmap (NULL , nr_pages * page_size,PROT_READ |PROT_WRITE , - map_shared ?MAP_SHARED : -MAP_PRIVATE |MAP_HUGETLB | + (map_shared ?MAP_SHARED :MAP_PRIVATE ) | +MAP_HUGETLB | (*alloc_area == area_src ?0 :MAP_NORESERVE ), - huge_fd, - *alloc_area == area_src ?0 : nr_pages * page_size); + huge_fd, *alloc_area == area_src ?0 : + nr_pages * page_size);if (*alloc_area ==MAP_FAILED )err ("mmap of hugetlbfs file failed" );if (map_shared) { area_alias =mmap (NULL , nr_pages * page_size,PROT_READ |PROT_WRITE , -MAP_SHARED , +MAP_SHARED |MAP_HUGETLB , huge_fd, *alloc_area == area_src ?0 : nr_pages * page_size);if (area_alias ==MAP_FAILED )
_
Andrew Morton
maintainerPATCH 94/146 vmscan: make drop_slab_node static
In reply to Andrew MortonFrom: Gang Li <[email protected]>
Subject: vmscan: make drop_slab_node static
drop_slab_node is only used in drop_slab. So remove it's declaration from header file and add keyword static for it's definition.
---
include/linux/mm.h | 1 - mm/vmscan.c | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~vmscan-make-drop_slab_node-static +0 -1
@@ -3122,7 +3122,6 @@ int drop_caches_sysctl_handler(struct ct#endif void drop_slab (void ); -void drop_slab_node (int nid);#ifndef CONFIG_MMU #define randomize_va_space 0
mm/vmscan.c renamed from mm/vmscan.c~vmscan-make-drop_slab_node-static +1 -1
@@ -951,7 +951,7 @@ out:return freed; } -void drop_slab_node (int nid) +static void drop_slab_node (int nid) {unsigned long freed;int shift =0 ;
_
Andrew Morton
maintainerPATCH 95/146 mm/page_isolation: unset migratetype directly for non Buddy page
In reply to Andrew MortonFrom: Chen Wandun <[email protected]>
Subject: mm/page_isolation: unset migratetype directly for non Buddy page
In unset_migratetype_isolate(), we can bypass the call to move_freepages_block() for non-buddy pages.
It will save a few cpu cycles for some situations such as cma and hugetlb when allocating continue pages, in these situation function alloc_contig_pages will be called.
alloc_contig_pages
__alloc_contig_migrate_range isolate_freepages_range ==> pages has been remove from buddy undo_isolate_page_range unset_migratetype_isolate ==> can directly set migratetype
---
mm/page_isolation.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_isolation.c renamed from mm/page_isolation.c~mm-page_isolation-unset-migratetype-directly-for-non-buddy-page +1 -1
@@ -115,7 +115,7 @@ static void unset_migratetype_isolate(st* onlining - just onlined memory won't immediately be considered for * allocation. */ -if (!isolated_page) { +if (!isolated_page &&PageBuddy (page)) { nr_pages =move_freepages_block (zone, page, migratetype,NULL );__mod_zone_freepage_state (zone, nr_pages, migratetype); }
_
Andrew Morton
maintainerPATCH 96/146 mm/mempolicy: use policy_node helper with MPOL_PREFERRED_MANY
In reply to Andrew MortonFrom: "Aneesh Kumar K.V" <[email protected]>
Subject: mm/mempolicy: use policy_node helper with MPOL_PREFERRED_MANY
Patch series "mm: add new syscall set_mempolicy_home_node", v6.
This patch (of 3):
A followup patch will enable setting a home node with MPOL_PREFERRED_MANY memory policy. To facilitate that switch to using policy_node helper. There is no functional change in this patch.
---
mm/mempolicy.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/mempolicy.c renamed from mm/mempolicy.c~mm-mempolicy-use-policy_node-helper-with-mpol_preferred_many +3 -2
@@ -2062,7 +2062,7 @@ static struct page *alloc_pages_preferre preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); page =__alloc_pages (preferred_gfp, order, nid, &pol->nodes);if (!page) - page =__alloc_pages (gfp, order,numa_node_id (),NULL ); + page =__alloc_pages (gfp, order, nid,NULL );return page; }
@@ -2104,6 +2104,7 @@ struct page *alloc_pages_vma(gfp_t gfp, }if (pol->mode ==MPOL_PREFERRED_MANY ) { + node =policy_node (gfp, pol, node); page =alloc_pages_preferred_many (gfp, order, node, pol);mpol_cond_put (pol); goto out;
@@ -2187,7 +2188,7 @@ struct page *alloc_pages(gfp_t gfp, unsi page =alloc_page_interleave (gfp, order,interleave_nodes (pol)); else if (pol->mode == MPOL_PREFERRED_MANY) page =alloc_pages_preferred_many (gfp, order, -numa_node_id (), pol); +policy_node (gfp, pol,numa_node_id ()), pol); else page = __alloc_pages(gfp, order, policy_node(gfp, pol, numa_node_id()),
_
Andrew Morton
maintainerPATCH 97/146 mm/mempolicy: add set_mempolicy_home_node syscall
In reply to Andrew MortonFrom: "Aneesh Kumar K.V" <[email protected]>
Subject: mm/mempolicy: add set_mempolicy_home_node syscall
This syscall can be used to set a home node for the MPOL_BIND and MPOL_PREFERRED_MANY memory policy. Users should use this syscall after setting up a memory policy for the specified range as shown below.
mbind(p, nr_pages * page_size, MPOL_BIND, new_nodes->maskp,
new_nodes->size + 1, 0); sys_set_mempolicy_home_node((unsigned long)p, nr_pages * page_size, home_node, 0);
The syscall allows specifying a home node/preferred node from which kernel will fulfill memory allocation requests first.
For address range with MPOL_BIND memory policy, if nodemask specifies more than one node, page allocations will come from the node in the nodemask with sufficient free memory that is closest to the home node/preferred node.
For MPOL_PREFERRED_MANY if the nodemask specifies more than one node, page allocation will come from the node in the nodemask with sufficient free memory that is closest to the home node/preferred node. If there is not enough memory in all the nodes specified in the nodemask, the allocation will be attempted from the closest numa node to the home node in the system.
This helps applications to hint at a memory allocation preference node and fallback to _only_ a set of nodes if the memory is not available on the preferred node. Fallback allocation is attempted from the node which is nearest to the preferred node.
This helps applications to have control on memory allocation numa nodes and avoids default fallback to slow memory NUMA nodes. For example a system with NUMA nodes 1,2 and 3 with DRAM memory and 10, 11 and 12 of slow memory
new_nodes = numa_bitmask_alloc(nr_nodes);
numa_bitmask_setbit(new_nodes, 1);
numa_bitmask_setbit(new_nodes, 2);
numa_bitmask_setbit(new_nodes, 3);
p = mmap(NULL, nr_pages * page_size, protflag, mapflag, -1, 0); mbind(p, nr_pages * page_size, MPOL_BIND, new_nodes->maskp, new_nodes->size + 1, 0);
sys_set_mempolicy_home_node(p, nr_pages * page_size, 2, 0);
This will allocate from nodes closer to node 2 and will make sure the kernel will only allocate from nodes 1, 2, and 3. Memory will not be allocated from slow memory nodes 10, 11, and 12. This differs from default MPOL_BIND behavior in that with default MPOL_BIND the allocation will be attempted from node closer to the local node. One of the reasons to specify a home node is to allow allocations from cpu less NUMA node and its nearby NUMA nodes.
With MPOL_PREFERRED_MANY on the other hand will first try to allocate from the closest node to node 2 from the node list 1, 2 and 3. If those nodes don't have enough memory, kernel will allocate from slow memory node 10, 11 and 12 which ever is closer to node 2.
---
Documentation/admin-guide/mm/numa_memory_policy.rst | 16 +- include/linux/mempolicy.h | 1 mm/mempolicy.c | 79 ++++++++++ 3 files changed, 95 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/numa_memory_policy.rst renamed from Documentation/admin-guide/mm/numa_memory_policy.rst~mm-mempolicy-add-set_mempolicy_home_node-syscall +15 -1
@@ -408,7 +408,7 @@ follows: Memory Policy APIs ================== -Linux supports 3 system calls for controlling memory policy. These APIS +Linux supports 4 system calls for controlling memory policy. These APIS always affect only the calling task, the calling task's address space, or some shared object mapped into the calling task's address space.
@@ -460,6 +460,20 @@ requested via the 'flags' argument. See the mbind(2) man page for more details. +Set home node for a Range of Task's Address Spacec:: + + long sys_set_mempolicy_home_node(unsigned long start, unsigned long len, + unsigned long home_node, + unsigned long flags); + +sys_set_mempolicy_home_node set the home node for a VMA policy present in the +task's address range. The system call updates the home node only for the existing +mempolicy range. Other address ranges are ignored. A home node is the NUMA node +closest to which page allocation will come from. Specifying the home node override +the default allocation policy to allocate memory close to the local node for an +executing CPU. + + Memory Policy Command Line Interface ====================================
include/linux/mempolicy.h renamed from include/linux/mempolicy.h~mm-mempolicy-add-set_mempolicy_home_node-syscall +1 -0
@@ -46,6 +46,7 @@ struct mempolicy {unsigned short mode;/* See MPOL_* above */ unsigned short flags;/* See set_mempolicy() MPOL_F_* above */ nodemask_t nodes;/* interleave/bind/perfer */ +int home_node;/* Home node to use for MPOL_BIND and MPOL_PREFERRED_MANY */ union {nodemask_t cpuset_mems_allowed;/* relative to these nodes */
mm/mempolicy.c renamed from mm/mempolicy.c~mm-mempolicy-add-set_mempolicy_home_node-syscall +79 -0
103 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 98/146 mm/mempolicy: wire up syscall set_mempolicy_home_node
In reply to Andrew MortonFrom: "Aneesh Kumar K.V" <[email protected]>
Subject: mm/mempolicy: wire up syscall set_mempolicy_home_node
---
arch/alpha/kernel/syscalls/syscall.tbl | 1 + arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 ++ arch/ia64/kernel/syscalls/syscall.tbl | 1 + arch/m68k/kernel/syscalls/syscall.tbl | 1 + arch/microblaze/kernel/syscalls/syscall.tbl | 1 + arch/mips/kernel/syscalls/syscall_n32.tbl | 1 + arch/mips/kernel/syscalls/syscall_n64.tbl | 1 + arch/mips/kernel/syscalls/syscall_o32.tbl | 1 + arch/parisc/kernel/syscalls/syscall.tbl | 1 + arch/powerpc/kernel/syscalls/syscall.tbl | 1 + arch/s390/kernel/syscalls/syscall.tbl | 1 + arch/sh/kernel/syscalls/syscall.tbl | 1 + arch/sparc/kernel/syscalls/syscall.tbl | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/xtensa/kernel/syscalls/syscall.tbl | 1 + include/linux/syscalls.h | 3 +++ include/uapi/asm-generic/unistd.h | 5 ++++- kernel/sys_ni.c | 1 + 21 files changed, 27 insertions(+), 2 deletions(-)
arch/alpha/kernel/syscalls/syscall.tbl renamed from arch/alpha/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -489,3 +489,4 @@ # 557 reserved for memfd_secret 558 common process_mrelease sys_process_mrelease 559 common futex_waitv sys_futex_waitv +560 common set_mempolicy_home_node sys_ni_syscall
arch/arm64/include/asm/unistd32.h renamed from arch/arm64/include/asm/unistd32.h~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +2 -0
@@ -905,6 +905,8 @@ __SYSCALL(__NR_landlock_restrict_self, s __SYSCALL(__NR_process_mrelease, sys_process_mrelease)#define __NR_futex_waitv 449 __SYSCALL(__NR_futex_waitv, sys_futex_waitv) +#define __NR_set_mempolicy_home_node 450 +__SYSCALL (__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node)/* * Please add new compat syscalls above this comment and update
arch/arm64/include/asm/unistd.h renamed from arch/arm64/include/asm/unistd.h~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -1
@@ -38,7 +38,7 @@#define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5)#define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800) -#define __NR_compat_syscalls 450 +#define __NR_compat_syscalls 451#endif #define __ARCH_WANT_SYS_CLONE
arch/arm/tools/syscall.tbl renamed from arch/arm/tools/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -463,3 +463,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/ia64/kernel/syscalls/syscall.tbl renamed from arch/ia64/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -370,3 +370,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/m68k/kernel/syscalls/syscall.tbl renamed from arch/m68k/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -449,3 +449,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/microblaze/kernel/syscalls/syscall.tbl renamed from arch/microblaze/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -455,3 +455,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/mips/kernel/syscalls/syscall_n32.tbl renamed from arch/mips/kernel/syscalls/syscall_n32.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -388,3 +388,4 @@ # 447 reserved for memfd_secret 448 n32 process_mrelease sys_process_mrelease 449 n32 futex_waitv sys_futex_waitv +450 n32 set_mempolicy_home_node sys_set_mempolicy_home_node
arch/mips/kernel/syscalls/syscall_n64.tbl renamed from arch/mips/kernel/syscalls/syscall_n64.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -364,3 +364,4 @@ # 447 reserved for memfd_secret 448 n64 process_mrelease sys_process_mrelease 449 n64 futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/mips/kernel/syscalls/syscall_o32.tbl renamed from arch/mips/kernel/syscalls/syscall_o32.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -437,3 +437,4 @@ # 447 reserved for memfd_secret 448 o32 process_mrelease sys_process_mrelease 449 o32 futex_waitv sys_futex_waitv +450 o32 set_mempolicy_home_node sys_set_mempolicy_home_node
arch/parisc/kernel/syscalls/syscall.tbl renamed from arch/parisc/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -447,3 +447,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/powerpc/kernel/syscalls/syscall.tbl renamed from arch/powerpc/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -529,3 +529,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 nospu set_mempolicy_home_node sys_set_mempolicy_home_node
arch/s390/kernel/syscalls/syscall.tbl renamed from arch/s390/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -452,3 +452,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node sys_set_mempolicy_home_node
arch/sh/kernel/syscalls/syscall.tbl renamed from arch/sh/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -452,3 +452,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/sparc/kernel/syscalls/syscall.tbl renamed from arch/sparc/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -495,3 +495,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/x86/entry/syscalls/syscall_32.tbl renamed from arch/x86/entry/syscalls/syscall_32.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -454,3 +454,4 @@ 447 i386 memfd_secret sys_memfd_secret 448 i386 process_mrelease sys_process_mrelease 449 i386 futex_waitv sys_futex_waitv +450 i386 set_mempolicy_home_node sys_set_mempolicy_home_node
arch/x86/entry/syscalls/syscall_64.tbl renamed from arch/x86/entry/syscalls/syscall_64.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -371,6 +371,7 @@ 447 common memfd_secret sys_memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node # # Due to a historical design error, certain syscalls are numbered differently
arch/xtensa/kernel/syscalls/syscall.tbl renamed from arch/xtensa/kernel/syscalls/syscall.tbl~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -420,3 +420,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
include/linux/syscalls.h renamed from include/linux/syscalls.h~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +3 -0
@@ -1057,6 +1057,9 @@ asmlinkage long sys_landlock_add_rule(in const void __user *rule_attr, __u32 flags);asmlinkage long sys_landlock_restrict_self (int ruleset_fd,__u32 flags);asmlinkage long sys_memfd_secret (unsigned int flags); +asmlinkage long sys_set_mempolicy_home_node (unsigned long start,unsigned long len, +unsigned long home_node, +unsigned long flags);/* * Architecture-specific system calls
include/uapi/asm-generic/unistd.h renamed from include/uapi/asm-generic/unistd.h~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +4 -1
@@ -883,8 +883,11 @@ __SYSCALL(__NR_process_mrelease, sys_pro#define __NR_futex_waitv 449 __SYSCALL(__NR_futex_waitv, sys_futex_waitv) +#define __NR_set_mempolicy_home_node 450 +__SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node) +#undef __NR_syscalls -#define __NR_syscalls 450 +#define __NR_syscalls 451/* * 32 bit systems traditionally used different
kernel/sys_ni.c renamed from kernel/sys_ni.c~mm-mempolicy-wire-up-syscall-set_mempolicy_home_node +1 -0
@@ -297,6 +297,7 @@ COND_SYSCALL(get_mempolicy);COND_SYSCALL (set_mempolicy);COND_SYSCALL (migrate_pages);COND_SYSCALL (move_pages); +COND_SYSCALL (set_mempolicy_home_node);COND_SYSCALL (perf_event_open);COND_SYSCALL (accept4);
_
Andrew Morton
maintainerPATCH 99/146 mm/mempolicy: fix all kernel-doc warnings
In reply to Andrew MortonFrom: Randy Dunlap <[email protected]>
Subject: mm/mempolicy: fix all kernel-doc warnings
Fix kernel-doc warnings in mempolicy.c:
mempolicy.c:139: warning: No description found for return value of 'numa_map_to_online_node' mempolicy.c:2165: warning: Excess function parameter 'node' description in 'alloc_pages_vma' mempolicy.c:2973: warning: No description found for return value of 'mpol_parse_str'
---
mm/mempolicy.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/mempolicy.c renamed from mm/mempolicy.c~mm-mempolicy-fix-all-kernel-doc-warnings +3 -2
@@ -134,6 +134,8 @@ static struct mempolicy preferred_node_p* @node: Node id to start the search * * Lookup the next closest node by distance if @nid is not online. +* +* Return: this @node if it is online, otherwise the closest node by distance */ int numa_map_to_online_node (int node) {
@@ -2150,7 +2152,6 @@ static struct page *alloc_pages_preferre* @order: Order of the GFP allocation. * @vma: Pointer to VMA or NULL if not available. * @addr: Virtual address of the allocation. Must be inside @vma. -* @node: Which node to prefer for allocation (modulo policy). * @hugepage: For hugepages try only the preferred node if possible. * * Allocate a page for a specific address in @vma, using the appropriate
@@ -2966,7 +2967,7 @@ static const char * const policy_modes[]* Format of input: * <mode>[=<flags>][:<nodelist>] * -* On success, returns 0, else 1 +* Return: %0 on success, else %1 */ int mpol_parse_str (char *str,struct mempolicy **mpol) {
_
Andrew Morton
maintainerPATCH 100/146 mm, oom: OOM sysrq should always kill a process
In reply to Andrew MortonFrom: Jann Horn <[email protected]>
Subject: mm, oom: OOM sysrq should always kill a process
The OOM kill sysrq (alt+sysrq+F) should allow the user to kill the process with the highest OOM badness with a single execution.
However, at the moment, the OOM kill can bail out if an OOM notifier (e.g. the i915 one) says that it reclaimed a tiny amount of memory from somewhere. That's probably not what the user wants, so skip the bailout if the OOM was triggered via sysrq.
---
mm/oom_kill.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/oom_kill.c renamed from mm/oom_kill.c~mm-oom-oom-sysrq-should-always-kill-a-process +1 -1
@@ -1058,7 +1058,7 @@ bool out_of_memory(struct oom_control *oif (!is_memcg_oom (oc)) {blocking_notifier_call_chain (&oom_notify_list,0 , &freed); -if (freed >0 ) +if (freed >0 && !is_sysrq_oom (oc))/* Got some memory back in the last second. */ return true; }
_
Andrew Morton
maintainerPATCH 101/146 hugetlbfs: fix off-by-one error in hugetlb_vmdelete_list()
In reply to Andrew MortonFrom: Sean Christopherson <[email protected]>
Subject: hugetlbfs: fix off-by-one error in hugetlb_vmdelete_list()
Pass "end - 1" instead of "end" when walking the interval tree in hugetlb_vmdelete_list() to fix an inclusive vs. exclusive bug. The two callers that pass a non-zero "end" treat it as exclusive, whereas the interval tree iterator expects an inclusive "last". E.g. punching a hole in a file that precisely matches the size of a single hugepage, with a vma starting right on the boundary, will result in unmap_hugepage_range() being called twice, with the second call having start==end.
The off-by-one error doesn't cause functional problems as __unmap_hugepage_range() turns into a massive nop due to short-circuiting its for-loop on "address < end". But, the mmu_notifier invocations to invalid_range_{start,end}() are passed a bogus zero-sized range, which may be unexpected behavior for secondary MMUs.
The bug was exposed by commit ed922739c919 ("KVM: Use interval tree to do fast hva lookup in memslots"), currently queued in the KVM tree for 5.17, which added a WARN to detect ranges with start==end.
---
fs/hugetlbfs/inode.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
fs/hugetlbfs/inode.c renamed from fs/hugetlbfs/inode.c~hugetlbfs-fix-off-by-one-error-in-hugetlb_vmdelete_list +4 -3
@@ -409,10 +409,11 @@ hugetlb_vmdelete_list(struct rb_root_cacstruct vm_area_struct *vma;/* -* end == 0 indicates that the entire range after -* start should be unmapped. +* end == 0 indicates that the entire range after start should be +* unmapped. Note, end is exclusive, whereas the interval tree takes +* an inclusive "last". */ - vma_interval_tree_foreach(vma, root, start, end ? end : ULONG_MAX) { + vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) {unsigned long v_offset;unsigned long v_end;
_
Andrew Morton
maintainerPATCH 102/146 mm: migrate: fix the return value of migrate_pages()
In reply to Andrew MortonFrom: Baolin Wang <[email protected]>
Subject: mm: migrate: fix the return value of migrate_pages()
Patch series "Improve the migration stats".
According to talk with Zi Yan [1], this patch set changes the return value of migrate_pages() to avoid returning a number which is larger than the number of pages the users tried to migrate by move_pages() syscall. Also fix the hugetlb migration stats and migration stats in trace_mm_compaction_migratepages().
[1] https://lore.kernel.org/linux-mm/[email protected]/
This patch (of 3):
As Zi Yan pointed out, the syscall move_pages() can return a non-migrated number larger than the number of pages the users tried to migrate, when a THP page is failed to migrate. This is confusing for users.
Since other migration scenarios do not care about the actual non-migrated number of pages except the memory compaction migration which will fix in following patch. Thus we can change the return value to return the number of {normal page, THP, hugetlb} instead to avoid this issue, and the number of THP splits will be considered as the number of non-migrated THP, no matter how many subpages of the THP are migrated successfully. Meanwhile we should still keep the migration counters using the number of normal pages.
---
mm/migrate.c | 63 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 47 insertions(+), 16 deletions(-)
mm/migrate.c renamed from mm/migrate.c~mm-migrate-fix-the-return-value-of-migrate_pages +47 -16
@@ -1421,7 +1421,7 @@ static inline int try_split_thp(struct p* @mode: The migration mode that specifies the constraints for * page migration, if any. * @reason: The reason for page migration. -* @ret_succeeded: Set to the number of pages migrated successfully if +* @ret_succeeded: Set to the number of normal pages migrated successfully if * the caller passes a non-NULL pointer. * * The function returns after 10 attempts or if no pages are movable any more
@@ -1429,7 +1429,9 @@ static inline int try_split_thp(struct p* It is caller's responsibility to call putback_movable_pages() to return pages * to the LRU or free list only if ret != 0. * -* Returns the number of pages that were not migrated, or an error code. +* Returns the number of {normal page, THP} that were not migrated, or an error code. +* The number of THP splits will be considered as the number of non-migrated THP, +* no matter how many subpages of the THP are migrated successfully. */ int migrate_pages(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private,
@@ -1438,6 +1440,7 @@ int migrate_pages(struct list_head *fromint retry =1 ;int thp_retry =1 ;int nr_failed =0 ; +int nr_failed_pages =0 ;int nr_succeeded =0 ;int nr_thp_succeeded =0 ;int nr_thp_failed =0 ;
@@ -1449,13 +1452,16 @@ int migrate_pages(struct list_head *fromint swapwrite = current->flags &PF_SWAPWRITE ;int rc, nr_subpages;LIST_HEAD (ret_pages); +LIST_HEAD (thp_split_pages);bool nosplit = (reason ==MR_NUMA_MISPLACED ); +bool no_subpage_counting = false;trace_mm_migrate_pages_start (mode, reason);if (!swapwrite) current->flags |=PF_SWAPWRITE ; +thp_subpage_migration:for (pass =0 ; pass <10 && (retry || thp_retry); pass++) { retry =0 ; thp_retry =0 ;
@@ -1504,18 +1510,20 @@ retry:case -ENOSYS :/* THP migration is unsupported */ if (is_thp) { -if (!try_split_thp (page, &page2, from)) { + nr_thp_failed++; +if (!try_split_thp (page, &page2, &thp_split_pages)) { nr_thp_split++; goto retry; } - nr_thp_failed++; - nr_failed += nr_subpages; + nr_failed_pages += nr_subpages;break ; }/* Hugetlb migration is unsupported */ - nr_failed++; +if (!no_subpage_counting) + nr_failed++; + nr_failed_pages++;break ;case -ENOMEM :/*
@@ -1524,16 +1532,19 @@ retry:* THP NUMA faulting doesn't split THP to retry. */ if (is_thp && !nosplit) { -if (!try_split_thp (page, &page2, from)) { + nr_thp_failed++; +if (!try_split_thp (page, &page2, &thp_split_pages)) { nr_thp_split++; goto retry; } - nr_thp_failed++; - nr_failed += nr_subpages; + nr_failed_pages += nr_subpages; goto out; } - nr_failed++; + +if (!no_subpage_counting) + nr_failed++; + nr_failed_pages++; goto out;case -EAGAIN :if (is_thp) {
@@ -1559,17 +1570,37 @@ retry:*/ if (is_thp) { nr_thp_failed++; - nr_failed += nr_subpages; + nr_failed_pages += nr_subpages;break ; } - nr_failed++; + +if (!no_subpage_counting) + nr_failed++; + nr_failed_pages++;break ; } } } - nr_failed += retry + thp_retry; + nr_failed += retry; nr_thp_failed += thp_retry; - rc = nr_failed; +/* +* Try to migrate subpages of fail-to-migrate THPs, no nr_failed +* counting in this round, since all subpages of a THP is counted +* as 1 failure in the first round. +*/ +if (!list_empty (&thp_split_pages)) { +/* +* Move non-migrated pages (after 10 retries) to ret_pages +* to avoid migrating them again. +*/ +list_splice_init (from, &ret_pages); +list_splice_init (&thp_split_pages, from); + no_subpage_counting = true; + retry =1 ; + goto thp_subpage_migration; + } + + rc = nr_failed + nr_thp_failed; out:/* * Put the permanent failure page back to migration list, they
@@ -1578,11 +1609,11 @@ out:list_splice (&ret_pages, from);count_vm_events (PGMIGRATE_SUCCESS , nr_succeeded); -count_vm_events (PGMIGRATE_FAIL , nr_failed); +count_vm_events (PGMIGRATE_FAIL , nr_failed_pages);count_vm_events (THP_MIGRATION_SUCCESS , nr_thp_succeeded);count_vm_events (THP_MIGRATION_FAIL , nr_thp_failed);count_vm_events (THP_MIGRATION_SPLIT , nr_thp_split); -trace_mm_migrate_pages (nr_succeeded, nr_failed, nr_thp_succeeded, +trace_mm_migrate_pages (nr_succeeded, nr_failed_pages, nr_thp_succeeded, nr_thp_failed, nr_thp_split, mode, reason);if (!swapwrite)
_
Andrew Morton
maintainerPATCH 103/146 mm: migrate: correct the hugetlb migration stats
In reply to Andrew MortonFrom: Baolin Wang <[email protected]>
Subject: mm: migrate: correct the hugetlb migration stats
Correct the migration stats for hugetlb with using compound_nr() instead of thp_nr_pages(), meanwhile change 'nr_failed_pages' to record the number of normal pages failed to migrate, including THP and hugetlb, and 'nr_succeeded' will record the number of normal pages migrated successfully.
---
Documentation/vm/page_migration.rst | 12 ++++++------ mm/migrate.c | 17 ++++++++--------- 2 files changed, 14 insertions(+), 15 deletions(-)
Documentation/vm/page_migration.rst renamed from Documentation/vm/page_migration.rst~mm-migrate-correct-the-hugetlb-migration-stats +6 -6
@@ -263,15 +263,15 @@ Monitoring Migration The following events (counters) can be used to monitor page migration. 1. PGMIGRATE_SUCCESS: Normal page migration success. Each count means that a - page was migrated. If the page was a non-THP page, then this counter is - increased by one. If the page was a THP, then this counter is increased by - the number of THP subpages. For example, migration of a single 2MB THP that - has 4KB-size base pages (subpages) will cause this counter to increase by - 512. + page was migrated. If the page was a non-THP and non-hugetlb page, then + this counter is increased by one. If the page was a THP or hugetlb, then + this counter is increased by the number of THP or hugetlb subpages. + For example, migration of a single 2MB THP that has 4KB-size base pages + (subpages) will cause this counter to increase by 512. 2. PGMIGRATE_FAIL: Normal page migration failure. Same counting rules as for PGMIGRATE_SUCCESS, above: this will be increased by the number of subpages, - if it was a THP. + if it was a THP or hugetlb. 3. THP_MIGRATION_SUCCESS: A THP was migrated without being split.
mm/migrate.c renamed from mm/migrate.c~mm-migrate-correct-the-hugetlb-migration-stats +8 -9
@@ -1429,9 +1429,9 @@ static inline int try_split_thp(struct p* It is caller's responsibility to call putback_movable_pages() to return pages * to the LRU or free list only if ret != 0. * -* Returns the number of {normal page, THP} that were not migrated, or an error code. -* The number of THP splits will be considered as the number of non-migrated THP, -* no matter how many subpages of the THP are migrated successfully. +* Returns the number of {normal page, THP, hugetlb} that were not migrated, or +* an error code. The number of THP splits will be considered as the number of +* non-migrated THP, no matter how many subpages of the THP are migrated successfully. */ int migrate_pages(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private,
@@ -1474,7 +1474,7 @@ retry:* during migration. */ is_thp =PageTransHuge (page) && !PageHuge (page); - nr_subpages =thp_nr_pages (page); + nr_subpages =compound_nr (page);cond_resched ();if (PageHuge (page))
@@ -1523,7 +1523,7 @@ retry:/* Hugetlb migration is unsupported */ if (!no_subpage_counting) nr_failed++; - nr_failed_pages++; + nr_failed_pages += nr_subpages;break ;case -ENOMEM :/*
@@ -1544,7 +1544,7 @@ retry:if (!no_subpage_counting) nr_failed++; - nr_failed_pages++; + nr_failed_pages += nr_subpages; goto out;case -EAGAIN :if (is_thp) {
@@ -1554,12 +1554,11 @@ retry: retry++;break ;case MIGRATEPAGE_SUCCESS : + nr_succeeded += nr_subpages;if (is_thp) { nr_thp_succeeded++; - nr_succeeded += nr_subpages;break ; } - nr_succeeded++;break ;default :/*
@@ -1576,7 +1575,7 @@ retry:if (!no_subpage_counting) nr_failed++; - nr_failed_pages++; + nr_failed_pages += nr_subpages;break ; } }
_
Andrew Morton
maintainerPATCH 104/146 mm: compaction: fix the migration stats in trace_mm_compaction_migratepages()
In reply to Andrew MortonFrom: Baolin Wang <[email protected]>
Subject: mm: compaction: fix the migration stats in trace_mm_compaction_migratepages()
Now the migrate_pages() has changed to return the number of {normal page, THP, hugetlb} instead, thus we should not use the return value to calculate the number of pages migrated successfully. Instead we can just use the 'nr_succeeded' which indicates the number of normal pages migrated successfully to calculate the non-migrated pages in trace_mm_compaction_migratepages().
---
include/trace/events/compaction.h | 24 ++++-------------------- mm/compaction.c | 7 ++++--- 2 files changed, 8 insertions(+), 23 deletions(-)
include/trace/events/compaction.h renamed from include/trace/events/compaction.h~mm-compaction-fix-the-migration-stats-in-trace_mm_compaction_migratepages +4 -20
@@ -68,10 +68,9 @@ DEFINE_EVENT(mm_compaction_isolate_templ TRACE_EVENT(mm_compaction_migratepages, TP_PROTO(unsigned long nr_all, - int migrate_rc, - struct list_head *migratepages), + unsigned int nr_succeeded), - TP_ARGS(nr_all, migrate_rc, migratepages), + TP_ARGS(nr_all, nr_succeeded), TP_STRUCT__entry( __field(unsigned long, nr_migrated)
@@ -79,23 +78,8 @@ TRACE_EVENT(mm_compaction_migratepages, ), TP_fast_assign( - unsigned long nr_failed = 0; -struct list_head *page_lru; - -/* -* migrate_pages() returns either a non-negative number -* with the number of pages that failed migration, or an -* error code, in which case we need to count the remaining -* pages manually -*/ -if (migrate_rc >=0 ) - nr_failed = migrate_rc; -else -list_for_each (page_lru, migratepages) - nr_failed++; - - __entry->nr_migrated = nr_all - nr_failed; - __entry->nr_failed = nr_failed; + __entry->nr_migrated = nr_succeeded; + __entry->nr_failed = nr_all - nr_succeeded; ), TP_printk("nr_migrated=%lu nr_failed=%lu",
mm/compaction.c renamed from mm/compaction.c~mm-compaction-fix-the-migration-stats-in-trace_mm_compaction_migratepages +4 -3
@@ -2280,6 +2280,7 @@ compact_zone(struct compact_control *cc,unsigned long last_migrated_pfn;const bool sync = cc->mode !=MIGRATE_ASYNC ;bool update_cached; +unsigned int nr_succeeded =0 ;/* * These counters track activities during zone compaction. Initialize
@@ -2398,10 +2399,10 @@ compact_zone(struct compact_control *cc, err =migrate_pages (&cc->migratepages, compaction_alloc, compaction_free, (unsigned long )cc, cc->mode, -MR_COMPACTION ,NULL ); +MR_COMPACTION , &nr_succeeded); -trace_mm_compaction_migratepages (cc->nr_migratepages, err, - &cc->migratepages); +trace_mm_compaction_migratepages (cc->nr_migratepages, + nr_succeeded);/* All pages were either migrated or will be released */ cc->nr_migratepages =0 ;
_
Andrew Morton
maintainerPATCH 105/146 mm: migrate: support multiple target nodes demotion
In reply to Andrew MortonFrom: Baolin Wang <[email protected]>
Subject: mm: migrate: support multiple target nodes demotion
We have some machines with multiple memory types like below, which have one fast (DRAM) memory node and two slow (persistent memory) memory nodes. According to current node demotion policy, if node 0 fills up, its memory should be migrated to node 1, when node 1 fills up, its memory will be migrated to node 2: node 0 -> node 1 -> node 2 ->stop.
But this is not efficient and suitbale memory migration route for our machine with multiple slow memory nodes. Since the distance between node 0 to node 1 and node 0 to node 2 is equal, and memory migration between slow memory nodes will increase persistent memory bandwidth greatly, which will hurt the whole system's performance.
Thus for this case, we can treat the slow memory node 1 and node 2 as a whole slow memory region, and we should migrate memory from node 0 to node 1 and node 2 if node 0 fills up.
This patch changes the node_demotion data structure to support multiple target nodes, and establishes the migration path to support multiple target nodes with validating if the node distance is the best or not.
available: 3 nodes (0-2)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 node 0 size: 62153 MB
node 0 free: 55135 MB
node 1 cpus:
node 1 size: 127007 MB
node 1 free: 126930 MB
node 2 cpus:
node 2 size: 126968 MB
node 2 free: 126878 MB
node distances:
node 0 1 2
0: 10 20 20 1: 20 10 20 2: 20 20 10
---
mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 129 insertions(+), 35 deletions(-)
mm/migrate.c renamed from mm/migrate.c~mm-migrate-support-multiple-target-nodes-demotion +129 -35
@@ -50,6 +50,7 @@#include <linux/ptrace.h> #include <linux/oom.h> #include <linux/memory.h> +#include <linux/random.h> #include <asm/tlbflush.h>
@@ -1118,12 +1119,25 @@ out:* * This is represented in the node_demotion[] like this: * -* { 1, // Node 0 migrates to 1 -* 2, // Node 1 migrates to 2 -* -1, // Node 2 does not migrate -* 4, // Node 3 migrates to 4 -* 5, // Node 4 migrates to 5 -* -1} // Node 5 does not migrate +* { nr=1, nodes[0]=1 }, // Node 0 migrates to 1 +* { nr=1, nodes[0]=2 }, // Node 1 migrates to 2 +* { nr=0, nodes[0]=-1 }, // Node 2 does not migrate +* { nr=1, nodes[0]=4 }, // Node 3 migrates to 4 +* { nr=1, nodes[0]=5 }, // Node 4 migrates to 5 +* { nr=0, nodes[0]=-1 }, // Node 5 does not migrate +* +* Moreover some systems may have multiple slow memory nodes. +* Suppose a system has one socket with 3 memory nodes, node 0 +* is fast memory type, and node 1/2 both are slow memory +* type, and the distance between fast memory node and slow +* memory node is same. So the migration path should be: +* +* 0 -> 1/2 -> stop +* +* This is represented in the node_demotion[] like this: +* { nr=2, {nodes[0]=1, nodes[1]=2} }, // Node 0 migrates to node 1 and node 2 +* { nr=0, nodes[0]=-1, }, // Node 1 dose not migrate +* { nr=0, nodes[0]=-1, }, // Node 2 does not migrate */ /*
@@ -1134,8 +1148,20 @@ out:* must be held over all reads to ensure that no cycles are * observed. */ -static int node_demotion[MAX_NUMNODES ]__read_mostly = - {[0 ...MAX_NUMNODES -1 ] =NUMA_NO_NODE }; +#define DEFAULT_DEMOTION_TARGET_NODES 15 + +#if MAX_NUMNODES <DEFAULT_DEMOTION_TARGET_NODES +#define DEMOTION_TARGET_NODES (MAX_NUMNODES - 1) +#else +#define DEMOTION_TARGET_NODES DEFAULT_DEMOTION_TARGET_NODES +#endif + +struct demotion_nodes { +unsigned short nr; +short nodes[DEMOTION_TARGET_NODES ]; +}; + +static struct demotion_nodes *node_demotion__read_mostly ;/** * next_demotion_node() - Get the next node in the demotion path
@@ -1148,8 +1174,15 @@ static int node_demotion[MAX_NUMNODES] _*/ int next_demotion_node (int node) { +struct demotion_nodes *nd; +unsigned short target_nr, index;int target; +if (!node_demotion) +return NUMA_NO_NODE ; + + nd = &node_demotion[node]; +/* * node_demotion[] is updated without excluding this * function from running. RCU doesn't provide any
@@ -1160,9 +1193,28 @@ int next_demotion_node(int node)* node_demotion[] reads need to be consistent. */ rcu_read_lock (); - target =READ_ONCE (node_demotion[node]); -rcu_read_unlock (); + target_nr =READ_ONCE (nd->nr); +switch (target_nr) { +case 0 : + target =NUMA_NO_NODE ; + goto out; +case 1 : + index =0 ; +break ; +default : +/* +* If there are multiple target nodes, just select one +* target node randomly. +*/ + index =get_random_int () % target_nr; +break ; + } + + target =READ_ONCE (nd->nodes[index]); + +out: +rcu_read_unlock ();return target; }
@@ -3003,10 +3055,16 @@ EXPORT_SYMBOL(migrate_vma_finalize);/* Disable reclaim-based migration. */ static void __disable_all_migrate_targets (void ) { -int node; +int node, i; -for_each_online_node (node) - node_demotion[node] =NUMA_NO_NODE ; +if (!node_demotion) +return ; + +for_each_online_node (node) { + node_demotion[node].nr =0 ; +for (i =0 ; i <DEMOTION_TARGET_NODES ; i++) + node_demotion[node].nodes[i] =NUMA_NO_NODE ; + } }static void disable_all_migrate_targets (void )
@@ -3033,26 +3091,40 @@ static void disable_all_migrate_targets(* Failing here is OK. It might just indicate * being at the end of a chain. */ -static int establish_migrate_target (int node,nodemask_t *used) +static int establish_migrate_target (int node,nodemask_t *used, +int best_distance) { -int migration_target; +int migration_target, index, val; +struct demotion_nodes *nd; -/* -* Can not set a migration target on a -* node with it already set. -* -* No need for READ_ONCE() here since this -* in the write path for node_demotion[]. -* This should be the only thread writing. -*/ -if (node_demotion[node] !=NUMA_NO_NODE ) +if (!node_demotion)return NUMA_NO_NODE ; + nd = &node_demotion[node]; + migration_target =find_next_best_node (node, used);if (migration_target ==NUMA_NO_NODE )return NUMA_NO_NODE ; - node_demotion[node] = migration_target; +/* +* If the node has been set a migration target node before, +* which means it's the best distance between them. Still +* check if this node can be demoted to other target nodes +* if they have a same best distance. +*/ +if (best_distance !=-1 ) { + val =node_distance (node, migration_target); +if (val > best_distance) +return NUMA_NO_NODE ; + } + + index = nd->nr; +if (WARN_ONCE (index >=DEMOTION_TARGET_NODES , +"Exceeds maximum demotion target nodes\n" )) +return NUMA_NO_NODE ; + + nd->nodes[index] = migration_target; + nd->nr++;return migration_target; }
@@ -3068,7 +3140,9 @@ static int establish_migrate_target(int* * The difference here is that cycles must be avoided. If * node0 migrates to node1, then neither node1, nor anything -* node1 migrates to can migrate to node0. +* node1 migrates to can migrate to node0. Also one node can +* be migrated to multiple nodes if the target nodes all have +* a same best-distance against the source node. * * This function can run simultaneously with readers of * node_demotion[]. However, it can not run simultaneously
@@ -3080,7 +3154,7 @@ static void __set_migration_target_nodesnodemask_t next_pass =NODE_MASK_NONE ;nodemask_t this_pass =NODE_MASK_NONE ;nodemask_t used_targets =NODE_MASK_NONE ; -int node; +int node, best_distance;/* * Avoid any oddities like cycles that could occur
@@ -3109,18 +3183,33 @@ again:* multiple source nodes to share a destination. */ nodes_or (used_targets, used_targets, this_pass); -for_each_node_mask (node, this_pass) { -int target_node =establish_migrate_target (node, &used_targets); -if (target_node ==NUMA_NO_NODE ) -continue ; +for_each_node_mask (node, this_pass) { + best_distance =-1 ;/* -* Visit targets from this pass in the next pass. -* Eventually, every node will have been part of -* a pass, and will become set in 'used_targets'. +* Try to set up the migration path for the node, and the target +* migration nodes can be multiple, so doing a loop to find all +* the target nodes if they all have a best node distance. */ -node_set (target_node, next_pass); +do { +int target_node = +establish_migrate_target (node, &used_targets, + best_distance); + +if (target_node ==NUMA_NO_NODE ) +break ; + +if (best_distance ==-1 ) + best_distance =node_distance (node, target_node); + +/* +* Visit targets from this pass in the next pass. +* Eventually, every node will have been part of +* a pass, and will become set in 'used_targets'. +*/ +node_set (target_node, next_pass); + }while (1 ); }/* * 'next_pass' contains nodes which became migration
@@ -3221,6 +3310,11 @@ static int __init migrate_on_reclaim_ini {int ret; + node_demotion =kmalloc_array (nr_node_ids, +sizeof (struct demotion_nodes ), +GFP_KERNEL ); +WARN_ON (!node_demotion); + ret =cpuhp_setup_state_nocalls (CPUHP_MM_DEMOTION_DEAD ,"mm/demotion:offline" ,NULL , migration_offline_cpu);/*
_