incoming
Subsystems (109)
Andrew Morton
maintainer192 patches, based on 7cf3dead1ad70c72edb03e2d98e1f3dcd332cdb2.
Subsystems affected by this patch series:
mm/gup
mm/pagealloc
kthread
ia64
scripts
ntfs
squashfs
ocfs2
z
kernel/watchdog
mm/slab
mm/slub
mm/kmemleak
mm/dax
mm/debug
mm/pagecache
mm/gup
mm/swap
mm/memcg
mm/pagemap
mm/mprotect
mm/bootmem
mm/dma
mm/tracing
mm/vmalloc
mm/kasan
mm/initialization
mm/pagealloc
mm/memory-failure
Subsystem: mm/gup
Jann Horn <[email protected]>:
mm/gup: fix try_grab_compound_head() race with split_huge_page()
Subsystem: mm/pagealloc
Mike Rapoport <[email protected]>:
mm/page_alloc: fix memory map initialization for descending nodes
Mel Gorman <[email protected]>:
mm/page_alloc: correct return value of populated elements if bulk array is populated
Subsystem: kthread
Jonathan Neuschäfer <[email protected]>:
kthread: switch to new kerneldoc syntax for named variable macro argument
Petr Mladek <[email protected]>:
kthread_worker: fix return value when kthread_mod_delayed_work() races with kthread_cancel_delayed_work_sync()
Subsystem: ia64
Randy Dunlap <[email protected]>:
ia64: headers: drop duplicated words
Arnd Bergmann <[email protected]>:
ia64: mca_drv: fix incorrect array size calculation
Subsystem: scripts
"Steven Rostedt (VMware)" <[email protected]>:
Patch series "streamline_config.pl: Fix Perl spacing":
streamline_config.pl: make spacing consistent streamline_config.pl: add softtabstop=4 for vim users
Colin Ian King <[email protected]>:
scripts/spelling.txt: add more spellings to spelling.txt
Subsystem: ntfs
Desmond Cheong Zhi Xi <[email protected]>:
ntfs: fix validity check for file name attribute
Subsystem: squashfs
Vincent Whitchurch <[email protected]>:
squashfs: add option to panic on errors
Subsystem: ocfs2
Yang Yingliang <[email protected]>:
ocfs2: remove unnecessary INIT_LIST_HEAD()
Subsystem: z
Dan Carpenter <[email protected]>:
ocfs2: fix snprintf() checking
Colin Ian King <[email protected]>:
ocfs2: remove redundant assignment to pointer queue
Wan Jiabing <[email protected]>:
ocfs2: remove repeated uptodate check for buffer
Chen Huang <[email protected]>:
ocfs2: replace simple_strtoull() with kstrtoull()
Colin Ian King <[email protected]>:
ocfs2: remove redundant initialization of variable ret
Subsystem: kernel/watchdog
Wang Qing <[email protected]>:
kernel: watchdog: modify the explanation related to watchdog thread
doc: watchdog: modify the explanation related to watchdog thread
doc: watchdog: modify the doc related to "watchdog/%u"
Subsystem: mm/slab
gumingtao <[email protected]>:
slab: use __func__ to trace function name
Subsystem: mm/slub
Vlastimil Babka <[email protected]>:
kunit: make test->lock irq safe
Oliver Glitta <[email protected]>:
mm/slub, kunit: add a KUnit test for SLUB debugging functionality
slub: remove resiliency_test() function
Hyeonggon Yoo <[email protected]>:
mm, slub: change run-time assertion in kmalloc_index() to compile-time
Stephen Boyd <[email protected]>:
slub: restore slub_debug=- behavior
slub: actually use 'message' in restore_bytes()
Joe Perches <[email protected]>:
slub: indicate slab_fix() uses printf formats
Stephen Boyd <[email protected]>:
slub: force on no_hash_pointers when slub_debug is enabled
Faiyaz Mohammed <[email protected]>:
mm: slub: move sysfs slab alloc/free interfaces to debugfs
Georgi Djakov <[email protected]>:
mm/slub: add taint after the errors are printed
Subsystem: mm/kmemleak
Yanfei Xu <[email protected]>:
mm/kmemleak: fix possible wrong memory scanning period
Subsystem: mm/dax
Jan Kara <[email protected]>:
dax: fix ENOMEM handling in grab_mapping_entry()
Subsystem: mm/debug
Tang Bin <[email protected]>:
tools/vm/page_owner_sort.c: check malloc() return
Anshuman Khandual <[email protected]>:
mm/debug_vm_pgtable: ensure THP availability via has_transparent_hugepage()
Nicolas Saenz Julienne <[email protected]>:
mm: mmap_lock: use local locks instead of disabling preemption
Gavin Shan <[email protected]>:
Patch series "mm/page_reporting: Make page reporting work on arm64 with 64KB page size", v4:
mm/page_reporting: fix code style in __page_reporting_request() mm/page_reporting: export reporting order as module parameter mm/page_reporting: allow driver to specify reporting order
virtio_balloon: specify page reporting order if needed
Subsystem: mm/pagecache
Kefeng Wang <[email protected]>:
mm: page-writeback: kill get_writeback_state() comments
Chi Wu <[email protected]>:
mm/page-writeback: Fix performance when BDI's share of ratio is 0. mm/page-writeback: update the comment of Dirty position control mm/page-writeback: use __this_cpu_inc() in account_page_dirtied()
Roman Gushchin <[email protected]>:
Patch series "cgroup, blkcg: prevent dirty inodes to pin dying memory cgroups", v9:
writeback, cgroup: do not switch inodes with I_WILL_FREE flag writeback, cgroup: add smp_mb() to cgroup_writeback_umount() writeback, cgroup: increment isw_nr_in_flight before grabbing an inode writeback, cgroup: switch to rcu_work API in inode_switch_wbs() writeback, cgroup: keep list of inodes attached to bdi_writeback writeback, cgroup: split out the functional part of inode_switch_wbs_work_fn() writeback, cgroup: support switching multiple inodes at once writeback, cgroup: release dying cgwbs by switching attached inodes
Christoph Hellwig <[email protected]>:
Patch series "remove the implicit .set_page_dirty default":
fs: unexport __set_page_dirty
fs: move ramfs_aops to libfs
mm: require ->set_page_dirty to be explicitly wired up
"Matthew Wilcox (Oracle)" <[email protected]>:
Patch series "Further set_page_dirty cleanups":
mm/writeback: move __set_page_dirty() to core mm mm/writeback: use __set_page_dirty in __set_page_dirty_nobuffers
iomap: use __set_page_dirty_nobuffers
fs: remove anon_set_page_dirty()
fs: remove noop_set_page_dirty()
mm: move page dirtying prototypes from mm.h
Subsystem: mm/gup
Peter Xu <[email protected]>:
Patch series "mm/gup: Fix pin page write cache bouncing on has_pinned", v2:
mm/gup_benchmark: support threading
Andrea Arcangeli <[email protected]>:
mm: gup: allow FOLL_PIN to scale in SMP
mm: gup: pack has_pinned in MMF_HAS_PINNED
Christophe Leroy <[email protected]>:
mm: pagewalk: fix walk for hugepage tables
Subsystem: mm/swap
Miaohe Lin <[email protected]>:
Patch series "close various race windows for swap", v6:
mm/swapfile: use percpu_ref to serialize against concurrent swapoff
swap: fix do_swap_page() race with swapoff
mm/swap: remove confusing checking for non_swap_entry() in swap_ra_info() mm/shmem: fix shmem_swapin() race with swapoff
Patch series "Cleanups for swap", v2:
mm/swapfile: move get_swap_page_of_type() under CONFIG_HIBERNATION mm/swap: remove unused local variable nr_shadows mm/swap_slots.c: delete meaningless forward declarations
Huang Ying <[email protected]>:
mm, swap: remove unnecessary smp_rmb() in swap_type_to_swap_info()
mm: free idle swap cache page after COW
swap: check mapping_empty() for swap cache before being freed
Subsystem: mm/memcg
Waiman Long <[email protected]>:
Patch series "mm/memcg: Reduce kmemcache memory accounting overhead", v6:
mm/memcg: move mod_objcg_state() to memcontrol.c
mm/memcg: cache vmstat data in percpu memcg_stock_pcp mm/memcg: improve refill_obj_stock() performance
mm/memcg: optimize user context object stock access
Patch series "mm: memcg/slab: Fix objcg pointer array handling problem", v4:
mm: memcg/slab: properly set up gfp flags for objcg pointer array
mm: memcg/slab: create a new set of kmalloc-cg-<n> caches
mm: memcg/slab: disable cache merging for KMALLOC_NORMAL caches
Muchun Song <[email protected]>:
mm: memcontrol: fix root_mem_cgroup charging
Patch series "memcontrol code cleanup and simplification", v3:
mm: memcontrol: fix page charging in page replacement
mm: memcontrol: bail out early when !mm in get_mem_cgroup_from_mm
mm: memcontrol: remove the pgdata parameter of mem_cgroup_page_lruvec
mm: memcontrol: simplify lruvec_holds_page_lru_lock
mm: memcontrol: rename lruvec_holds_page_lru_lock to page_matches_lruvec
mm: memcontrol: simplify the logic of objcg pinning memcg
mm: memcontrol: move obj_cgroup_uncharge_pages() out of css_set_lock
mm: vmscan: remove noinline_for_stack
wenhuizhang <[email protected]>:
memcontrol: use flexible-array member
Dan Schatzberg <[email protected]>:
Patch series "Charge loop device i/o to issuing cgroup", v14:
loop: use worker per cgroup instead of kworker
mm: charge active memcg when no mm is set
loop: charge i/o to mem and blk cg
Huilong Deng <[email protected]>:
mm: memcontrol: remove trailing semicolon in macros
Subsystem: mm/pagemap
David Hildenbrand <[email protected]>:
Patch series "perf/binfmt/mm: remove in-tree usage of MAP_EXECUTABLE":
perf: MAP_EXECUTABLE does not indicate VM_MAYEXEC
binfmt: remove in-tree usage of MAP_EXECUTABLE
mm: ignore MAP_EXECUTABLE in ksys_mmap_pgoff()
Gonzalo Matias Juarez Tello <[email protected]>:
mm/mmap.c: logic of find_vma_intersection repeated in __do_munmap
Liam Howlett <[email protected]>:
mm/mmap: introduce unlock_range() for code cleanup mm/mmap: use find_vma_intersection() in do_mmap() for overlap
Liu Xiang <[email protected]>:
mm/memory.c: fix comment of finish_mkwrite_fault()
Liam Howlett <[email protected]>:
Patch series "mm: Add vma_lookup()", v2:
mm: add vma_lookup(), update find_vma_intersection() comments drm/i915/selftests: use vma_lookup() in __igt_mmap() arch/arc/kernel/troubleshoot: use vma_lookup() instead of find_vma() arch/arm64/kvm: use vma_lookup() instead of find_vma_intersection() arch/powerpc/kvm/book3s_hv_uvmem: use vma_lookup() instead of find_vma_intersection() arch/powerpc/kvm/book3s: use vma_lookup() in kvmppc_hv_setup_htab_rma() arch/mips/kernel/traps: use vma_lookup() instead of find_vma() arch/m68k/kernel/sys_m68k: use vma_lookup() in sys_cacheflush() x86/sgx: use vma_lookup() in sgx_encl_find()
virt/kvm: use vma_lookup() instead of find_vma_intersection()
vfio: use vma_lookup() instead of find_vma_intersection() net/ipv5/tcp: use vma_lookup() in tcp_zerocopy_receive()
drm/amdgpu: use vma_lookup() in amdgpu_ttm_tt_get_user_pages()
media: videobuf2: use vma_lookup() in get_vaddr_frames() misc/sgi-gru/grufault: use vma_lookup() in gru_find_vma() kernel/events/uprobes: use vma_lookup() in find_active_uprobe() lib/test_hmm: use vma_lookup() in dmirror_migrate()
mm/ksm: use vma_lookup() in find_mergeable_vma()
mm/migrate: use vma_lookup() in do_pages_stat_array()
mm/mremap: use vma_lookup() in vma_to_resize()
mm/memory.c: use vma_lookup() in __access_remote_vm()
mm/mempolicy: use vma_lookup() in __access_remote_vm()
Chen Li <[email protected]>:
mm: update legacy flush_tlb_* to use vma
Subsystem: mm/mprotect
Peter Collingbourne <[email protected]>:
mm: improve mprotect(R|W) efficiency on pages referenced once
Subsystem: mm/bootmem
Souptick Joarder <[email protected]>:
h8300: remove unused variable
Subsystem: mm/dma
YueHaibing <[email protected]>:
mm/dmapool: use DEVICE_ATTR_RO macro
Subsystem: mm/tracing
Vincent Whitchurch <[email protected]>:
mm, tracing: unify PFN format strings
Subsystem: mm/vmalloc
"Uladzislau Rezki (Sony)" <[email protected]>:
Patch series "vmalloc() vs bulk allocator", v2:
mm/page_alloc: add an alloc_pages_bulk_array_node() helper mm/vmalloc: switch to bulk allocator in __vmalloc_area_node() mm/vmalloc: print a warning message first on failure mm/vmalloc: remove quoted strings split across lines
Uladzislau Rezki <[email protected]>:
mm/vmalloc: fallback to a single page allocator
Rafael Aquini <[email protected]>:
mm: vmalloc: add cond_resched() in __vunmap()
Subsystem: mm/kasan
Alexander Potapenko <[email protected]>:
printk: introduce dump_stack_lvl()
kasan: use dump_stack_lvl(KERN_ERR) to print stacks
David Gow <[email protected]>:
kasan: test: improve failure message in KUNIT_EXPECT_KASAN_FAIL()
Daniel Axtens <[email protected]>:
Patch series "KASAN core changes for ppc64 radix KASAN", v16:
kasan: allow an architecture to disable inline instrumentation
kasan: allow architectures to provide an outline readiness check
mm: define default MAX_PTRS_PER_* in include/pgtable.h
kasan: use MAX_PTRS_PER_* for early shadow tables
Kuan-Ying Lee <[email protected]>:
Patch series "kasan: add memory corruption identification support for hw tag-based kasan", v4:
kasan: rename CONFIG_KASAN_SW_TAGS_IDENTIFY to CONFIG_KASAN_TAGS_IDENTIFY
kasan: integrate the common part of two KASAN tag-based modes
kasan: add memory corruption identification support for hardware tag-based mode
Subsystem: mm/initialization
Jungseung Lee <[email protected]>:
mm: report which part of mem is being freed on initmem case
Subsystem: mm/pagealloc
Mike Rapoport <[email protected]>:
mm/mmzone.h: simplify is_highmem_idx()
"Matthew Wilcox (Oracle)" <[email protected]>:
Patch series "Constify struct page arguments":
mm: make __dump_page static
Aaron Tomlin <[email protected]>:
mm/page_alloc: bail out on fatal signal during reclaim/compaction retry attempt
"Matthew Wilcox (Oracle)" <[email protected]>:
mm/debug: factor PagePoisoned out of __dump_page mm/page_owner: constify dump_page_owner
mm: make compound_head const-preserving
mm: constify get_pfnblock_flags_mask and get_pfnblock_migratetype
mm: constify page_count and page_ref_count
mm: optimise nth_page for contiguous memmap
Heiner Kallweit <[email protected]>:
mm/page_alloc: switch to pr_debug
Andrii Nakryiko <[email protected]>:
kbuild: skip per-CPU BTF generation for pahole v1.18-v1.21
Mel Gorman <[email protected]>:
mm/page_alloc: split per cpu page lists and zone stats mm/page_alloc: convert per-cpu list protection to local_lock mm/vmstat: convert NUMA statistics to basic NUMA counters mm/vmstat: inline NUMA event counter updates mm/page_alloc: batch the accounting updates in the bulk allocator mm/page_alloc: reduce duration that IRQs are disabled for VM counters mm/page_alloc: explicitly acquire the zone lock in __free_pages_ok mm/page_alloc: avoid conflating IRQs disabled with zone->lock mm/page_alloc: update PGFREE outside the zone lock in __free_pages_ok
Minchan Kim <[email protected]>:
mm: page_alloc: dump migrate-failed pages only at -EBUSY
Mel Gorman <[email protected]>:
Patch series "Calculate pcp->high based on zone sizes and active CPUs", v2:
mm/page_alloc: delete vm.percpu_pagelist_fraction mm/page_alloc: disassociate the pcp->high from pcp->batch mm/page_alloc: adjust pcp->high after CPU hotplug events mm/page_alloc: scale the number of pages that are batch freed mm/page_alloc: limit the number of pages on PCP lists when reclaim is active mm/page_alloc: introduce vm.percpu_pagelist_high_fraction
Dong Aisheng <[email protected]>:
mm: drop SECTION_SHIFT in code comments
mm/page_alloc: improve memmap_pages dbg msg
Liu Shixin <[email protected]>:
mm/page_alloc: fix counting of managed_pages
Mel Gorman <[email protected]>:
Patch series "Allow high order pages to be stored on PCP", v2:
mm/page_alloc: move free_the_page
Mike Rapoport <[email protected]>:
Patch series "Remove DISCONTIGMEM memory model", v3:
alpha: remove DISCONTIGMEM and NUMA
arc: update comment about HIGHMEM implementation
arc: remove support for DISCONTIGMEM
m68k: remove support for DISCONTIGMEM
mm: remove CONFIG_DISCONTIGMEM
arch, mm: remove stale mentions of DISCONIGMEM
docs: remove description of DISCONTIGMEM
mm: replace CONFIG_NEED_MULTIPLE_NODES with CONFIG_NUMA
mm: replace CONFIG_FLAT_NODE_MEM_MAP with CONFIG_FLATMEM
Mel Gorman <[email protected]>:
mm/page_alloc: allow high-order pages to be stored on the per-cpu lists mm/page_alloc: split pcp->high across all online CPUs for cpuless nodes
Subsystem: mm/memory-failure
Naoya Horiguchi <[email protected]>:
mm,hwpoison: send SIGBUS with error virutal address mm,hwpoison: make get_hwpoison_page() call get_any_page()
Documentation/admin-guide/kernel-parameters.txt | 6 Documentation/admin-guide/lockup-watchdogs.rst | 4 Documentation/admin-guide/sysctl/kernel.rst | 10 Documentation/admin-guide/sysctl/vm.rst | 52 - Documentation/dev-tools/kasan.rst | 9 Documentation/vm/memory-model.rst | 45 arch/alpha/Kconfig | 22 arch/alpha/include/asm/machvec.h | 6 arch/alpha/include/asm/mmzone.h | 100 -- arch/alpha/include/asm/pgtable.h | 4 arch/alpha/include/asm/topology.h | 39 arch/alpha/kernel/core_marvel.c | 53 - arch/alpha/kernel/core_wildfire.c | 29 arch/alpha/kernel/pci_iommu.c | 29 arch/alpha/kernel/proto.h | 8 arch/alpha/kernel/setup.c | 16 arch/alpha/kernel/sys_marvel.c | 5 arch/alpha/kernel/sys_wildfire.c | 5 arch/alpha/mm/Makefile | 2 arch/alpha/mm/init.c | 3 arch/alpha/mm/numa.c | 223 ---- arch/arc/Kconfig | 13 arch/arc/include/asm/mmzone.h | 40 arch/arc/kernel/troubleshoot.c | 8 arch/arc/mm/init.c | 21 arch/arm/include/asm/tlbflush.h | 13 arch/arm/mm/tlb-v6.S | 2 arch/arm/mm/tlb-v7.S | 2 arch/arm64/Kconfig | 2 arch/arm64/kvm/mmu.c | 2 arch/h8300/kernel/setup.c | 2 arch/ia64/Kconfig | 2 arch/ia64/include/asm/pal.h | 2 arch/ia64/include/asm/spinlock.h | 2 arch/ia64/include/asm/uv/uv_hub.h | 2 arch/ia64/kernel/efi_stub.S | 2 arch/ia64/kernel/mca_drv.c | 2 arch/ia64/kernel/topology.c | 5 arch/ia64/mm/numa.c | 5 arch/m68k/Kconfig.cpu | 10 arch/m68k/include/asm/mmzone.h | 10 arch/m68k/include/asm/page.h | 2 arch/m68k/include/asm/page_mm.h | 35 arch/m68k/include/asm/tlbflush.h | 2 arch/m68k/kernel/sys_m68k.c | 4 arch/m68k/mm/init.c | 20 arch/mips/Kconfig | 2 arch/mips/include/asm/mmzone.h | 8 arch/mips/include/asm/page.h | 2 arch/mips/kernel/traps.c | 4 arch/mips/mm/init.c | 7 arch/nds32/include/asm/memory.h | 6 arch/openrisc/include/asm/tlbflush.h | 2 arch/powerpc/Kconfig | 2 arch/powerpc/include/asm/mmzone.h | 4 arch/powerpc/kernel/setup_64.c | 2 arch/powerpc/kernel/smp.c | 2 arch/powerpc/kexec/core.c | 4 arch/powerpc/kvm/book3s_hv.c | 4 arch/powerpc/kvm/book3s_hv_uvmem.c | 2 arch/powerpc/mm/Makefile | 2 arch/powerpc/mm/mem.c | 4 arch/riscv/Kconfig | 2 arch/s390/Kconfig | 2 arch/s390/include/asm/pgtable.h | 2 arch/sh/include/asm/mmzone.h | 4 arch/sh/kernel/topology.c | 2 arch/sh/mm/Kconfig | 2 arch/sh/mm/init.c | 2 arch/sparc/Kconfig | 2 arch/sparc/include/asm/mmzone.h | 4 arch/sparc/kernel/smp_64.c | 2 arch/sparc/mm/init_64.c | 12 arch/x86/Kconfig | 2 arch/x86/ia32/ia32_aout.c | 4 arch/x86/kernel/cpu/mce/core.c | 13 arch/x86/kernel/cpu/sgx/encl.h | 4 arch/x86/kernel/setup_percpu.c | 6 arch/x86/mm/init_32.c | 4 arch/xtensa/include/asm/page.h | 4 arch/xtensa/include/asm/tlbflush.h | 4 drivers/base/node.c | 18 drivers/block/loop.c | 270 ++++- drivers/block/loop.h | 15 drivers/dax/device.c | 2 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c | 2 drivers/media/common/videobuf2/frame_vector.c | 2 drivers/misc/sgi-gru/grufault.c | 4 drivers/vfio/vfio_iommu_type1.c | 2 drivers/virtio/virtio_balloon.c | 17 fs/adfs/inode.c | 1 fs/affs/file.c | 2 fs/bfs/file.c | 1 fs/binfmt_aout.c | 4 fs/binfmt_elf.c | 2 fs/binfmt_elf_fdpic.c | 11 fs/binfmt_flat.c | 2 fs/block_dev.c | 1 fs/buffer.c | 25 fs/configfs/inode.c | 8 fs/dax.c | 3 fs/ecryptfs/mmap.c | 13 fs/exfat/inode.c | 1 fs/ext2/inode.c | 4 fs/ext4/inode.c | 2 fs/fat/inode.c | 1 fs/fs-writeback.c | 366 +++++--- fs/fuse/dax.c | 3 fs/gfs2/aops.c | 2 fs/gfs2/meta_io.c | 2 fs/hfs/inode.c | 2 fs/hfsplus/inode.c | 2 fs/hpfs/file.c | 1 fs/iomap/buffered-io.c | 27 fs/jfs/inode.c | 1 fs/kernfs/inode.c | 8 fs/libfs.c | 44 fs/minix/inode.c | 1 fs/nilfs2/mdt.c | 1 fs/ntfs/inode.c | 2 fs/ocfs2/aops.c | 4 fs/ocfs2/cluster/heartbeat.c | 7 fs/ocfs2/cluster/nodemanager.c | 2 fs/ocfs2/dlm/dlmmaster.c | 2 fs/ocfs2/filecheck.c | 6 fs/ocfs2/stackglue.c | 8 fs/omfs/file.c | 1 fs/proc/task_mmu.c | 2 fs/ramfs/inode.c | 9 fs/squashfs/block.c | 5 fs/squashfs/squashfs_fs_sb.h | 1 fs/squashfs/super.c | 86 + fs/sysv/itree.c | 1 fs/udf/file.c | 1 fs/udf/inode.c | 1 fs/ufs/inode.c | 1 fs/xfs/xfs_aops.c | 4 fs/zonefs/super.c | 4 include/asm-generic/memory_model.h | 37 include/asm-generic/pgtable-nop4d.h | 1 include/asm-generic/topology.h | 2 include/kunit/test.h | 5 include/linux/backing-dev-defs.h | 20 include/linux/cpuhotplug.h | 2 include/linux/fs.h | 6 include/linux/gfp.h | 13 include/linux/iomap.h | 1 include/linux/kasan.h | 7 include/linux/kernel.h | 2 include/linux/kthread.h | 2 include/linux/memblock.h | 6 include/linux/memcontrol.h | 60 - include/linux/mm.h | 53 - include/linux/mm_types.h | 10 include/linux/mman.h | 2 include/linux/mmdebug.h | 3 include/linux/mmzone.h | 96 +- include/linux/page-flags.h | 10 include/linux/page_owner.h | 6 include/linux/page_ref.h | 4 include/linux/page_reporting.h | 3 include/linux/pageblock-flags.h | 2 include/linux/pagemap.h | 4 include/linux/pgtable.h | 22 include/linux/printk.h | 5 include/linux/sched/coredump.h | 8 include/linux/slab.h | 59 + include/linux/swap.h | 19 include/linux/swapops.h | 5 include/linux/vmstat.h | 69 - include/linux/writeback.h | 1 include/trace/events/cma.h | 4 include/trace/events/filemap.h | 2 include/trace/events/kmem.h | 12 include/trace/events/page_pool.h | 4 include/trace/events/pagemap.h | 4 include/trace/events/vmscan.h | 2 kernel/cgroup/cgroup.c | 1 kernel/crash_core.c | 4 kernel/events/core.c | 2 kernel/events/uprobes.c | 4 kernel/fork.c | 1 kernel/kthread.c | 19 kernel/sysctl.c | 16 kernel/watchdog.c | 12 lib/Kconfig.debug | 15 lib/Kconfig.kasan | 16 lib/Makefile | 1 lib/dump_stack.c | 20 lib/kunit/test.c | 18 lib/slub_kunit.c | 152 +++ lib/test_hmm.c | 5 lib/test_kasan.c | 11 lib/vsprintf.c | 2 mm/Kconfig | 38 mm/backing-dev.c | 66 + mm/compaction.c | 2 mm/debug.c | 27 mm/debug_vm_pgtable.c | 63 + mm/dmapool.c | 5 mm/filemap.c | 2 mm/gup.c | 81 + mm/hugetlb.c | 2 mm/internal.h | 9 mm/kasan/Makefile | 4 mm/kasan/common.c | 6 mm/kasan/generic.c | 3 mm/kasan/hw_tags.c | 22 mm/kasan/init.c | 6 mm/kasan/kasan.h | 12 mm/kasan/report.c | 6 mm/kasan/report_hw_tags.c | 5 mm/kasan/report_sw_tags.c | 45 mm/kasan/report_tags.c | 51 + mm/kasan/shadow.c | 6 mm/kasan/sw_tags.c | 45 mm/kasan/tags.c | 59 + mm/kfence/kfence_test.c | 5 mm/kmemleak.c | 18 mm/ksm.c | 6 mm/memblock.c | 8 mm/memcontrol.c | 385 ++++++-- mm/memory-failure.c | 344 +++++-- mm/memory.c | 22 mm/memory_hotplug.c | 6 mm/mempolicy.c | 4 mm/migrate.c | 4 mm/mmap.c | 54 - mm/mmap_lock.c | 33 mm/mprotect.c | 52 + mm/mremap.c | 5 mm/nommu.c | 2 mm/page-writeback.c | 89 + mm/page_alloc.c | 950 +++++++++++++-------- mm/page_ext.c | 2 mm/page_owner.c | 2 mm/page_reporting.c | 19 mm/page_reporting.h | 5 mm/pagewalk.c | 58 + mm/shmem.c | 18 mm/slab.h | 24 mm/slab_common.c | 60 - mm/slub.c | 420 +++++---- mm/sparse.c | 2 mm/swap.c | 4 mm/swap_slots.c | 2 mm/swap_state.c | 20 mm/swapfile.c | 177 +-- mm/vmalloc.c | 181 ++-- mm/vmscan.c | 43 mm/vmstat.c | 282 ++---- mm/workingset.c | 2 net/ipv4/tcp.c | 4 scripts/kconfig/streamline_config.pl | 76 - scripts/link-vmlinux.sh | 4 scripts/spelling.txt | 16 tools/testing/selftests/vm/gup_test.c | 96 +- tools/vm/page_owner_sort.c | 4 virt/kvm/kvm_main.c | 2 260 files changed, 3989 insertions(+), 2996 deletions(-)
Andrew Morton
maintainerPATCH 1/192 mm/gup: fix try_grab_compound_head() race with split_huge_page()
From: Jann Horn <[email protected]>
Subject: mm/gup: fix try_grab_compound_head() race with split_huge_page()
try_grab_compound_head() is used to grab a reference to a page from get_user_pages_fast(), which is only protected against concurrent freeing of page tables (via local_irq_save()), but not against concurrent TLB flushes, freeing of data pages, or splitting of compound pages.
Because no reference is held to the page when try_grab_compound_head() is called, the page may have been freed and reallocated by the time its refcount has been elevated; therefore, once we're holding a stable reference to the page, the caller re-checks whether the PTE still points to the same page (with the same access rights).
The problem is that try_grab_compound_head() has to grab a reference on the head page; but between the time we look up what the head page is and the time we actually grab a reference on the head page, the compound page may have been split up (either explicitly through split_huge_page() or by freeing the compound page to the buddy allocator and then allocating its individual order-0 pages). If that happens, get_user_pages_fast() may end up returning the right page but lifting the refcount on a now-unrelated page, leading to use-after-free of pages.
To fix it: Re-check whether the pages still belong together after lifting the refcount on the head page. Move anything else that checks compound_head(page) below the refcount increment.
This can't actually happen on bare-metal x86 (because there, disabling IRQs locks out remote TLB flushes), but it can happen on virtualized x86 (e.g. under KVM) and probably also on arm64. The race window is pretty narrow, and constantly allocating and shattering hugepages isn't exactly fast; for now I've only managed to reproduce this in an x86 KVM guest with an artificially widened timing window (by adding a loop that repeatedly calls inl(0x3f8 + 5) in try_get_compound_head() to force VM exits, so that PV TLB flushes are used instead of IPIs).
As requested on the list, also replace the existing VM_BUG_ON_PAGE() with a warning and bailout. Since the existing code only performed the BUG_ON check on DEBUG_VM kernels, ensure that the new code also only performs the check under that configuration - I don't want to mix two logically separate changes together too much. The macro VM_WARN_ON_ONCE_PAGE() doesn't return a value on !DEBUG_VM, so wrap the whole check in an #ifdef block. An alternative would be to change the VM_WARN_ON_ONCE_PAGE() definition for !DEBUG_VM such that it always returns false, but since that would differ from the behavior of the normal WARN macros, it might be too confusing for readers.
---
mm/gup.c | 58 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 15 deletions(-)
mm/gup.c renamed from mm/gup.c~mm-gup-fix-try_grab_compound_head-race-with-split_huge_page +43 -15
@@ -44,6 +44,23 @@ static void hpage_pincount_sub(struct paatomic_sub (refs,compound_pincount_ptr (page)); } +/* Equivalent to calling put_page() @refs times. */ +static void put_page_refs (struct page *page,int refs) +{ +#ifdef CONFIG_DEBUG_VM +if (VM_WARN_ON_ONCE_PAGE (page_ref_count (page) < refs, page)) +return ; +#endif + +/* +* Calling put_page() for each ref is unnecessarily slow. Only the last +* ref needs a put_page(). +*/ +if (refs >1 ) +page_ref_sub (page, refs -1 ); +put_page (page); +} +/* * Return the compound head page with ref appropriately incremented, * or NULL if that failed.
@@ -56,6 +73,21 @@ static inline struct page *try_get_comporeturn NULL ;if (unlikely (!page_cache_add_speculative (head, refs)))return NULL ; + +/* +* At this point we have a stable reference to the head page; but it +* could be that between the compound_head() lookup and the refcount +* increment, the compound page was split, in which case we'd end up +* holding a reference on a page that has nothing to do with the page +* we were given anymore. +* So now that the head page is stable, recheck that the pages still +* belong together. +*/ +if (unlikely (compound_head (page) != head)) { +put_page_refs (head, refs); +return NULL ; + } +return head; }
@@ -96,6 +128,14 @@ __maybe_unused struct page *try_grab_comreturn NULL ;/* +* CAUTION: Don't use compound_head() on the page before this +* point, the result won't be stable. +*/ + page =try_get_compound_head (page, refs); +if (!page) +return NULL ; + +/* * When pinning a compound page of order > 1 (which is what * hpage_pincount_available() checks for), use an exact count to * track it, via hpage_pincount_add/_sub().
@@ -103,15 +143,10 @@ __maybe_unused struct page *try_grab_com* However, be sure to *also* increment the normal page refcount * field at least once, so that the page really is pinned. */ -if (!hpage_pincount_available (page)) - refs *=GUP_PIN_COUNTING_BIAS ; - - page =try_get_compound_head (page, refs); -if (!page) -return NULL ; -if (hpage_pincount_available (page))hpage_pincount_add (page, refs); +else +page_ref_add (page, refs * (GUP_PIN_COUNTING_BIAS -1 ));mod_node_page_state (page_pgdat (page),NR_FOLL_PIN_ACQUIRED , orig_refs);
@@ -135,14 +170,7 @@ static void put_compound_head(struct pag refs *=GUP_PIN_COUNTING_BIAS ; } -VM_BUG_ON_PAGE (page_ref_count (page) < refs, page); -/* -* Calling put_page() for each ref is unnecessarily slow. Only the last -* ref needs a put_page(). -*/ -if (refs >1 ) -page_ref_sub (page, refs -1 ); -put_page (page); +put_page_refs (page, refs); }/**
_
Andrew Morton
maintainerPATCH 2/192 mm/page_alloc: fix memory map initialization for descending nodes
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm/page_alloc: fix memory map initialization for descending nodes
On systems with memory nodes sorted in descending order, for instance Dell Precision WorkStation T5500, the struct pages for higher PFNs and respectively lower nodes, could be overwritten by the initialization of struct pages corresponding to the holes in the memory sections.
For example for the below memory layout
[ 0.245624] Early memory node ranges [ 0.248496] node 1: [mem 0x0000000000001000-0x0000000000090fff] [ 0.251376] node 1: [mem 0x0000000000100000-0x00000000dbdf8fff] [ 0.254256] node 1: [mem 0x0000000100000000-0x0000001423ffffff] [ 0.257144] node 0: [mem 0x0000001424000000-0x0000002023ffffff]
the range 0x1424000000 - 0x1428000000 in the beginning of node 0 starts in the middle of a section and will be considered as a hole during the initialization of the last section in node 1.
The wrong initialization of the memory map causes panic on boot when CONFIG_DEBUG_VM is enabled.
Reorder loop order of the memory map initialization so that the outer loop will always iterate over populated memory regions in the ascending order and the inner loop will select the zone corresponding to the PFN range.
This way initialization of the struct pages for the memory holes will be always done for the ranges that are actually not populated.
---
include/linux/mm.h | 1 mm/page_alloc.c | 96 ++++++++++++++++++++++++++----------------- 2 files changed, 59 insertions(+), 38 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-page_alloc-fix-memory-map-initialization-for-descending-nodes +0 -1
@@ -2474,7 +2474,6 @@ extern void set_dma_reserve(unsigned lonextern void memmap_init_range (unsigned long ,int ,unsigned long ,unsigned long ,unsigned long ,enum meminit_context ,struct vmem_altmap *,int migratetype); -extern void memmap_init_zone (struct zone *zone);extern void setup_per_zone_wmarks (void );extern int __meminit init_per_zone_wmark_min (void );extern void mem_init (void );
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-fix-memory-map-initialization-for-descending-nodes +59 -37
151 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 3/192 mm/page_alloc: correct return value of populated elements if bulk array is populated
In reply to Andrew MortonFrom: Mel Gorman <[email protected]>
Subject: mm/page_alloc: correct return value of populated elements if bulk array is populated
Dave Jones reported the following
This made it into 5.13 final, and completely breaks NFSD for me (Serving tcp v3 mounts). Existing mounts on clients hang, as do new mounts from new clients. Rebooting the server back to rc7 everything recovers.
The commit b3b64ebd3822 ("mm/page_alloc: do bulk array bounds check after checking populated elements") returns the wrong value if the array is already populated which is interpreted as an allocation failure. Dave reported this fixes his problem and it also passed a test running dbench over NFS.
---
mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-correct-return-value-of-populated-elements-if-bulk-array-is-populated +1 -1
@@ -5058,7 +5058,7 @@ unsigned long __alloc_pages_bulk(gfp_t g/* Already populated array? */ if (unlikely (page_array && nr_pages - nr_populated ==0 )) -return 0 ; +return nr_populated;/* Use the single page allocator for one page. */ if (nr_pages - nr_populated ==1 )
_
Andrew Morton
maintainerPATCH 4/192 kthread: switch to new kerneldoc syntax for named variable macro argument
In reply to Andrew MortonFrom: Jonathan Neuschäfer <[email protected]>
Subject: kthread: switch to new kerneldoc syntax for named variable macro argument
The syntax without dots is available since commit 43756e347f21 ("scripts/kernel-doc: Add support for named variable macro arguments").
The same HTML output is produced with and without this patch.
---
include/linux/kthread.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/kthread.h renamed from include/linux/kthread.h~kthread-switch-to-new-kerneldoc-syntax-for-named-variable-macro-argument +1 -1
@@ -18,7 +18,7 @@ struct task_struct *kthread_create_on_no* @threadfn: the function to run in the thread * @data: data pointer for @threadfn() * @namefmt: printf-style format string for the thread name -* @arg...: arguments for @namefmt. +* @arg: arguments for @namefmt. * * This macro will create a kthread on the current node, leaving it in * the stopped state. This is just a helper for kthread_create_on_node();
_
Andrew Morton
maintainerPATCH 5/192 kthread_worker: fix return value when kthread_mod_delayed_work() races with kthread_cancel_delayed_work_sync()
In reply to Andrew MortonFrom: Petr Mladek <[email protected]>
Subject: kthread_worker: fix return value when kthread_mod_delayed_work() races with kthread_cancel_delayed_work_sync()
kthread_mod_delayed_work() might race with kthread_cancel_delayed_work_sync() or another kthread_mod_delayed_work() call. The function lets the other operation win when it sees work->canceling counter set. And it returns @false.
But it should return @true as it is done by the related workqueue API, see mod_delayed_work_on().
The reason is that the return value might be used for reference counting. It has to distinguish the case when the number of queued works has changed or stayed the same.
The change is safe. kthread_mod_delayed_work() return value is not checked anywhere at the moment.
---
kernel/kthread.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
kernel/kthread.c renamed from kernel/kthread.c~kthread_worker-fix-return-value-when-kthread_mod_delayed_work-races-with-kthread_cancel_delayed_work_sync +12 -7
@@ -1156,14 +1156,14 @@ static bool __kthread_cancel_work(struct* modify @dwork's timer so that it expires after @delay. If @delay is zero, * @work is guaranteed to be queued immediately. * -* Return: %true if @dwork was pending and its timer was modified, -* %false otherwise. +* Return: %false if @dwork was idle and queued, %true otherwise. * * A special case is when the work is being canceled in parallel. * It might be caused either by the real kthread_cancel_delayed_work_sync() * or yet another kthread_mod_delayed_work() call. We let the other command -* win and return %false here. The caller is supposed to synchronize these -* operations a reasonable way. +* win and return %true here. The return value can be used for reference +* counting and the number of queued works stays the same. Anyway, the caller +* is supposed to synchronize these operations a reasonable way. * * This function is safe to call from any context including IRQ handler. * See __kthread_cancel_work() and kthread_delayed_work_timer_fn()
@@ -1175,13 +1175,15 @@ bool kthread_mod_delayed_work(struct kth {struct kthread_work *work = &dwork->work;unsigned long flags; -int ret = false; +int ret;raw_spin_lock_irqsave (&worker->lock, flags);/* Do not bother with canceling when never queued. */ -if (!work->worker) +if (!work->worker) { + ret = false; goto fast_queue; + }/* Work must not be used with >1 worker, see kthread_queue_work() */ WARN_ON_ONCE (work->worker != worker);
@@ -1199,8 +1201,11 @@ bool kthread_mod_delayed_work(struct kth* be used for reference counting. */ kthread_cancel_delayed_work_timer (work, &flags); -if (work->canceling) +if (work->canceling) { +/* The number of works in the queue does not change. */ + ret = true; goto out; + } ret =__kthread_cancel_work (work); fast_queue:
_
Andrew Morton
maintainerPATCH 6/192 ia64: headers: drop duplicated words
In reply to Andrew MortonFrom: Randy Dunlap <[email protected]>
Subject: ia64: headers: drop duplicated words
Delete the repeated words "to" and "the".
---
arch/ia64/include/asm/pal.h | 2 +- arch/ia64/include/asm/spinlock.h | 2 +- arch/ia64/include/asm/uv/uv_hub.h | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
arch/ia64/include/asm/pal.h renamed from arch/ia64/include/asm/pal.h~ia64-headers-drop-duplicated-words +1 -1
@@ -1086,7 +1086,7 @@ static inline long ia64_pal_freq_base(un/* * Get the ratios for processor frequency, bus frequency and interval timer to -* to base frequency of the platform +* the base frequency of the platform */ static inline s64 ia64_pal_freq_ratios (struct pal_freq_ratio *proc_ratio, struct pal_freq_ratio *bus_ratio,
arch/ia64/include/asm/spinlock.h renamed from arch/ia64/include/asm/spinlock.h~ia64-headers-drop-duplicated-words +1 -1
@@ -26,7 +26,7 @@* the queue, and the other indicating the current tail. The lock is acquired * by atomically noting the tail and incrementing it by one (thus adding * ourself to the queue and noting our position), then waiting until the head -* becomes equal to the the initial value of the tail. +* becomes equal to the initial value of the tail. * The pad bits in the middle are used to prevent the next_ticket number * overflowing into the now_serving number. *
arch/ia64/include/asm/uv/uv_hub.h renamed from arch/ia64/include/asm/uv/uv_hub.h~ia64-headers-drop-duplicated-words +1 -1
@@ -257,7 +257,7 @@ static inline int uv_numa_blade_id(void)return 0 ; } -/* Convert a cpu number to the the UV blade number */ +/* Convert a cpu number to the UV blade number */ static inline int uv_cpu_to_blade_id (int cpu) {return 0 ;
_
Andrew Morton
maintainerPATCH 7/192 ia64: mca_drv: fix incorrect array size calculation
In reply to Andrew MortonFrom: Arnd Bergmann <[email protected]>
Subject: ia64: mca_drv: fix incorrect array size calculation
gcc points out a mistake in the mca driver that goes back to before the git history:
arch/ia64/kernel/mca_drv.c: In function 'init_record_index_pools':arch/ia64/kernel/mca_drv.c:346:54: error: expression does not compute the number of elements in this array; element typ e is 'int', not 'size_t' {aka 'long unsigned int'} [-Werror=sizeof-array-div]
346 | for (i = 1; i < sizeof sal_log_sect_min_sizes/sizeof(size_t); i++)
| ^
This is the same as sizeof(size_t), which is two shorter than the actual array. Use the ARRAY_SIZE() macro to get the correct calculation instead.
---
arch/ia64/kernel/mca_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/ia64/kernel/mca_drv.c renamed from arch/ia64/kernel/mca_drv.c~ia64-mca_drv-fix-incorrect-array-size-calculation +1 -1
@@ -343,7 +343,7 @@ init_record_index_pools(void)/* - 2 - */ sect_min_size = sal_log_sect_min_sizes[0 ]; -for (i =1 ; i <sizeof sal_log_sect_min_sizes/sizeof (size_t ); i++) +for (i =1 ; i <ARRAY_SIZE (sal_log_sect_min_sizes); i++)if (sect_min_size > sal_log_sect_min_sizes[i]) sect_min_size = sal_log_sect_min_sizes[i];
_
Andrew Morton
maintainerPATCH 8/192 streamline_config.pl: make spacing consistent
In reply to Andrew MortonFrom: "Steven Rostedt (VMware)" <[email protected]>
Subject: streamline_config.pl: make spacing consistent
Patch series "streamline_config.pl: Fix Perl spacing".
Talking with John Hawley about how vim and emacs deal with Perl files with respect to tabs and spaces, I found that some of my Perl code in the kernel had inconsistent spacing. The way emacs handles Perl by default is to use 4 spaces per indent, but make all 8 spaces into a single tab. Vim does not do this by default. But if you add the vim variable control:
# vim: softtabstop=4
to a perl file, it makes vim behave the same way as emacs.
The first patch is to change all 8 spaces into a single tab (mostly from people editing the file with vim). The next patch adds the softtabstop variable to make vim act like emacs by default.
This patch (of 2):
As Perl code tends to have 4 space indentation, but uses tabs for every 8 spaces, make that consistent in the streamline_config.pl code. Replace all 8 spaces with a single tab.
---
scripts/kconfig/streamline_config.pl | 74 ++++++++++++------------- 1 file changed, 37 insertions(+), 37 deletions(-)
scripts/kconfig/streamline_config.pl renamed from scripts/kconfig/streamline_config.pl~streamline_configpl-make-spacing-consistent +37 -37
@@ -601,12 +601,12 @@ if (defined($ENV{'LMC_KEEP'})) { sub in_preserved_kconfigs { my $kconfig = $config2kfile{$_[0]}; if (!defined($kconfig)) { - return 0; + return 0; } foreach my $excl (@preserved_kconfigs) { - if($kconfig =~ /^$excl/) { - return 1; - } + if($kconfig =~ /^$excl/) { + return 1; + } } return 0; }
@@ -629,52 +629,52 @@ foreach my $line (@config_file) { } if (/CONFIG_MODULE_SIG_KEY="(.+)"/) { - my $orig_cert = $1; - my $default_cert = "certs/signing_key.pem"; + my $orig_cert = $1; + my $default_cert = "certs/signing_key.pem"; - # Check that the logic in this script still matches the one in Kconfig - if (!defined($depends{"MODULE_SIG_KEY"}) || - $depends{"MODULE_SIG_KEY"} !~ /"\Q$default_cert\E"/) { - print STDERR "WARNING: MODULE_SIG_KEY assertion failure, ", - "update needed to ", __FILE__, " line ", __LINE__, "\n"; - print; - } elsif ($orig_cert ne $default_cert && ! -f $orig_cert) { - print STDERR "Module signature verification enabled but ", - "module signing key \"$orig_cert\" not found. Resetting ", - "signing key to default value.\n"; - print "CONFIG_MODULE_SIG_KEY=\"$default_cert\"\n"; - } else { - print; - } - next; + # Check that the logic in this script still matches the one in Kconfig + if (!defined($depends{"MODULE_SIG_KEY"}) || + $depends{"MODULE_SIG_KEY"} !~ /"\Q$default_cert\E"/) { + print STDERR "WARNING: MODULE_SIG_KEY assertion failure, ", + "update needed to ", __FILE__, " line ", __LINE__, "\n"; + print; + } elsif ($orig_cert ne $default_cert && ! -f $orig_cert) { + print STDERR "Module signature verification enabled but ", + "module signing key \"$orig_cert\" not found. Resetting ", + "signing key to default value.\n"; + print "CONFIG_MODULE_SIG_KEY=\"$default_cert\"\n"; + } else { + print; + } + next; } if (/CONFIG_SYSTEM_TRUSTED_KEYS="(.+)"/) { - my $orig_keys = $1; + my $orig_keys = $1; - if (! -f $orig_keys) { - print STDERR "System keyring enabled but keys \"$orig_keys\" ", - "not found. Resetting keys to default value.\n"; - print "CONFIG_SYSTEM_TRUSTED_KEYS=\"\"\n"; - } else { - print; - } - next; + if (! -f $orig_keys) { + print STDERR "System keyring enabled but keys \"$orig_keys\" ", + "not found. Resetting keys to default value.\n"; + print "CONFIG_SYSTEM_TRUSTED_KEYS=\"\"\n"; + } else { + print; + } + next; } if (/^(CONFIG.*)=(m|y)/) { - if (in_preserved_kconfigs($1)) { - dprint "Preserve config $1"; - print; - next; - } + if (in_preserved_kconfigs($1)) { + dprint "Preserve config $1"; + print; + next; + } if (defined($configs{$1})) { if ($localyesconfig) { - $setconfigs{$1} = 'y'; + $setconfigs{$1} = 'y'; print "$1=y\n"; next; } else { - $setconfigs{$1} = $2; + $setconfigs{$1} = $2; } } elsif ($2 eq "m") { print "# $1 is not set\n";
_
Andrew Morton
maintainerPATCH 9/192 streamline_config.pl: add softtabstop=4 for vim users
In reply to Andrew MortonFrom: "Steven Rostedt (VMware)" <[email protected]>
Subject: streamline_config.pl: add softtabstop=4 for vim users
The tab stop for Perl files is by default (at least in emacs) to be 4 spaces, where a tab is used for all 8 spaces. Add a local variable comment to make vim do the same by default, and this will help keep the file consistent in the future when others edit it via vim and not emacs.
---
scripts/kconfig/streamline_config.pl | 2 ++ 1 file changed, 2 insertions(+)
scripts/kconfig/streamline_config.pl renamed from scripts/kconfig/streamline_config.pl~streamline_configpl-add-softtabstop=4-for-vim-users +2 -0
@@ -702,3 +702,5 @@ foreach my $module (keys(%modules)) { print STDERR "\n"; } } + +# vim: softtabstop=4
_
Andrew Morton
maintainerPATCH 10/192 scripts/spelling.txt: add more spellings to spelling.txt
In reply to Andrew MortonFrom: Colin Ian King <[email protected]>
Subject: scripts/spelling.txt: add more spellings to spelling.txt
Here are some of the more common spelling mistakes and typos that I've found while fixing up spelling mistakes in the kernel in the past few months.
---
scripts/spelling.txt | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
scripts/spelling.txt renamed from scripts/spelling.txt~scripts-spellingtxt-add-more-spellings-to-spellingtxt +16 -0
@@ -22,6 +22,7 @@ absolut||absolute absoulte||absolute acccess||access acceess||access +accelaration||acceleration acceleratoin||acceleration accelleration||acceleration accesing||accessing
@@ -264,6 +265,7 @@ calucate||calculate calulate||calculate cancelation||cancellation cancle||cancel +canot||cannot capabilites||capabilities capabilties||capabilities capabilty||capability
@@ -494,7 +496,10 @@ digial||digital dimention||dimension dimesions||dimensions diconnected||disconnected +disabed||disabled +disble||disable disgest||digest +disired||desired dispalying||displaying diplay||display directon||direction
@@ -710,6 +715,7 @@ havind||having heirarchically||hierarchically heirarchy||hierarchy helpfull||helpful +hearbeat||heartbeat heterogenous||heterogeneous hexdecimal||hexadecimal hybernate||hibernate
@@ -989,6 +995,7 @@ notications||notifications notifcations||notifications notifed||notified notity||notify +nubmer||number numebr||number numner||number obtaion||obtain
@@ -1014,8 +1021,10 @@ ommiting||omitting ommitted||omitted onself||oneself ony||only +openning||opening operatione||operation opertaions||operations +opportunies||opportunities optionnal||optional optmizations||optimizations orientatied||orientated
@@ -1111,6 +1120,7 @@ prefitler||prefilter preform||perform premption||preemption prepaired||prepared +prepate||prepare preperation||preparation preprare||prepare pressre||pressure
@@ -1123,6 +1133,7 @@ privilaged||privileged privilage||privilege priviledge||privilege priviledges||privileges +privleges||privileges probaly||probably procceed||proceed proccesors||processors
@@ -1167,6 +1178,7 @@ promixity||proximity psudo||pseudo psuedo||pseudo psychadelic||psychedelic +purgable||purgeable pwoer||power queing||queuing quering||querying
@@ -1180,6 +1192,7 @@ receieve||receive recepient||recipient recevied||received receving||receiving +recievd||received recieved||received recieve||receive reciever||receiver
@@ -1228,6 +1241,7 @@ reponse||response representaion||representation reqeust||request reqister||register +requed||requeued requestied||requested requiere||require requirment||requirement
@@ -1332,6 +1346,7 @@ singal||signal singed||signed sleeped||slept sliped||slipped +softwade||software softwares||software soley||solely souce||source
@@ -1510,6 +1525,7 @@ unintialized||uninitialized unitialized||uninitialized unkmown||unknown unknonw||unknown +unknouwn||unknown unknow||unknown unkown||unknown unamed||unnamed
_
Andrew Morton
maintainerPATCH 11/192 ntfs: fix validity check for file name attribute
In reply to Andrew MortonFrom: Desmond Cheong Zhi Xi <[email protected]>
Subject: ntfs: fix validity check for file name attribute
When checking the file name attribute, we want to ensure that it fits within the bounds of ATTR_RECORD. To do this, we should check that (attr record + file name offset + file name length) < (attr record + attr record length).
However, the original check did not include the file name offset in the calculation. This means that corrupted on-disk metadata might not caught by the incorrect file name check, and lead to an invalid memory access.
An example can be seen in the crash report of a memory corruption error found by Syzbot:
https://syzkaller.appspot.com/bug?id=a1a1e379b225812688566745c3e2f7242bffc246
Adding the file name offset to the validity check fixes this error and passes the Syzbot reproducer test.
---
fs/ntfs/inode.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ntfs/inode.c renamed from fs/ntfs/inode.c~ntfs-fix-validity-check-for-file-name-attribute +1 -1
@@ -477,7 +477,7 @@ err_corrupt_attr: } file_name_attr = (FILE_NAME_ATTR *)((u8 *)attr +le16_to_cpu (attr->data.resident.value_offset)); - p2 = (u8 *)attr +le32_to_cpu (attr->data.resident.value_length); + p2 = (u8 *)file_name_attr +le32_to_cpu (attr->data.resident.value_length);if (p2 < (u8 *)attr || p2 > p) goto err_corrupt_attr;/* This attribute is ok, but is it in the $Extend directory? */
_
Andrew Morton
maintainerPATCH 12/192 squashfs: add option to panic on errors
In reply to Andrew MortonFrom: Vincent Whitchurch <[email protected]>
Subject: squashfs: add option to panic on errors
Add an errors=panic mount option to make squashfs trigger a panic when errors are encountered, similar to several other filesystems. This allows a kernel dump to be saved using which the corruption can be analysed and debugged.
Inspired by a pre-fs_context patch by Anton Eliasson.
---
fs/squashfs/block.c | 5 + fs/squashfs/squashfs_fs_sb.h | 1 fs/squashfs/super.c | 86 +++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+), 1 deletion(-)
fs/squashfs/block.c renamed from fs/squashfs/block.c~squashfs-add-option-to-panic-on-errors +4 -1
@@ -226,8 +226,11 @@ out_free_bio:bio_free_pages (bio);bio_put (bio); out: -if (res <0 ) +if (res <0 ) {ERROR ("Failed to read block 0x%llx: %d\n" , index, res); +if (msblk->panic_on_errors) +panic ("squashfs read failed" ); + }return res; }
fs/squashfs/squashfs_fs_sb.h renamed from fs/squashfs/squashfs_fs_sb.h~squashfs-add-option-to-panic-on-errors +1 -0
@@ -65,5 +65,6 @@ struct squashfs_sb_info {unsigned int fragments;int xattr_ids;unsigned int ids; + bool panic_on_errors; };#endif
fs/squashfs/super.c renamed from fs/squashfs/super.c~squashfs-add-option-to-panic-on-errors +86 -0
143 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 13/192 ocfs2: remove unnecessary INIT_LIST_HEAD()
In reply to Andrew MortonFrom: Yang Yingliang <[email protected]>
Subject: ocfs2: remove unnecessary INIT_LIST_HEAD()
The list_head o2hb_node_events is initialized statically. It is unnecessary to initialize by INIT_LIST_HEAD().
---
fs/ocfs2/cluster/heartbeat.c | 2 -- 1 file changed, 2 deletions(-)
fs/ocfs2/cluster/heartbeat.c renamed from fs/ocfs2/cluster/heartbeat.c~ocfs2-remove-unnecessary-init_list_head +0 -2
@@ -1442,8 +1442,6 @@ void o2hb_init(void)for (i =0 ; i <ARRAY_SIZE (o2hb_live_slots); i++)INIT_LIST_HEAD (&o2hb_live_slots[i]); -INIT_LIST_HEAD (&o2hb_node_events); -memset (o2hb_live_node_bitmap,0 ,sizeof (o2hb_live_node_bitmap));memset (o2hb_region_bitmap,0 ,sizeof (o2hb_region_bitmap));memset (o2hb_live_region_bitmap,0 ,sizeof (o2hb_live_region_bitmap));
_
Andrew Morton
maintainerPATCH 14/192 ocfs2: fix snprintf() checking
In reply to Andrew MortonFrom: Dan Carpenter <[email protected]>
Subject: ocfs2: fix snprintf() checking
The snprintf() function returns the number of bytes which would have been printed if the buffer was large enough. In other words it can return ">= remain" but this code assumes it returns "== remain".
The run time impact of this bug is not very severe. The next iteration through the loop would trigger a WARN() when we pass a negative limit to snprintf(). We would then return success instead of -E2BIG.
The kernel implementation of snprintf() will never return negatives so there is no need to check and I have deleted that dead code.
---
fs/ocfs2/filecheck.c | 6 +----- fs/ocfs2/stackglue.c | 8 ++------ 2 files changed, 3 insertions(+), 11 deletions(-)
fs/ocfs2/filecheck.c renamed from fs/ocfs2/filecheck.c~ocfs2-fix-snprintf-checking +1 -5
@@ -326,11 +326,7 @@ static ssize_t ocfs2_filecheck_attr_show ret =snprintf (buf + total, remain,"%lu\t\t%u\t%s\n" , p->fe_ino, p->fe_done,ocfs2_filecheck_error (p->fe_status)); -if (ret <0 ) { - total = ret; -break ; - } -if (ret == remain) { +if (ret >= remain) {/* snprintf() didn't fit */ total = -E2BIG ;break ;
fs/ocfs2/stackglue.c renamed from fs/ocfs2/stackglue.c~ocfs2-fix-snprintf-checking +2 -6
@@ -500,11 +500,7 @@ static ssize_t ocfs2_loaded_cluster_pluglist_for_each_entry (p, &ocfs2_stack_list, sp_list) { ret =snprintf (buf, remain,"%s\n" , p->sp_name); -if (ret <0 ) { - total = ret; -break ; - } -if (ret == remain) { +if (ret >= remain) {/* snprintf() didn't fit */ total = -E2BIG ;break ;
@@ -531,7 +527,7 @@ static ssize_t ocfs2_active_cluster_plugif (active_stack) { ret =snprintf (buf,PAGE_SIZE ,"%s\n" , active_stack->sp_name); -if (ret ==PAGE_SIZE ) +if (ret >=PAGE_SIZE ) ret = -E2BIG ; }spin_unlock (&ocfs2_stack_lock);
_
Andrew Morton
maintainerPATCH 15/192 ocfs2: remove redundant assignment to pointer queue
In reply to Andrew MortonFrom: Colin Ian King <[email protected]>
Subject: ocfs2: remove redundant assignment to pointer queue
The pointer queue is being initialized with a value that is never read and it is being updated later with a new value. The initialization is redundant and can be removed.
Addresses-Coverity: ("Unused value")
---
fs/ocfs2/dlm/dlmmaster.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ocfs2/dlm/dlmmaster.c renamed from fs/ocfs2/dlm/dlmmaster.c~ocfs2-remove-redundant-assignment-to-pointer-queue +1 -1
@@ -2977,7 +2977,7 @@ static u8 dlm_pick_migration_target(stru struct dlm_lock_resource *res) {enum dlm_lockres_list idx; -struct list_head *queue = &res->granted; +struct list_head *queue;struct dlm_lock *lock;int noderef;u8 nodenum =O2NM_MAX_NODES ;
_
Andrew Morton
maintainerPATCH 16/192 ocfs2: remove repeated uptodate check for buffer
In reply to Andrew MortonFrom: Wan Jiabing <[email protected]>
Subject: ocfs2: remove repeated uptodate check for buffer
In commit 60f91826ca62 ("buffer: Avoid setting buffer bits that are already set"), function set_buffer_##name was added a test_bit() to check buffer, which is the same as function buffer_##name. The !buffer_uptodate(bh) here is a repeated check. Remove it.
---
fs/ocfs2/aops.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/ocfs2/aops.c renamed from fs/ocfs2/aops.c~ocfs2-remove-repeated-uptodate-check-for-buffer +1 -2
@@ -632,8 +632,7 @@ int ocfs2_map_page_blocks(struct page *p }if (PageUptodate (page)) { -if (!buffer_uptodate (bh)) -set_buffer_uptodate (bh); +set_buffer_uptodate (bh); } else if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_new(bh) && ocfs2_should_read_blk(inode, page, block_start) &&
_
Andrew Morton
maintainerPATCH 17/192 ocfs2: replace simple_strtoull() with kstrtoull()
In reply to Andrew MortonFrom: Chen Huang <[email protected]>
Subject: ocfs2: replace simple_strtoull() with kstrtoull()
simple_strtoull() is deprecated in some situation since it does not check for the range overflow, use kstrtoull() instead.
---
fs/ocfs2/cluster/heartbeat.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/ocfs2/cluster/heartbeat.c renamed from fs/ocfs2/cluster/heartbeat.c~ocfs2-replaced-simple_strtoull-with-kstrtoull +3 -2
@@ -1596,12 +1596,13 @@ static ssize_t o2hb_region_start_block_sstruct o2hb_region *reg =to_o2hb_region (item);unsigned long long tmp;char *p = (char *)page; +ssize_t ret;if (reg->hr_bdev)return -EINVAL ; - tmp =simple_strtoull (p, &p,0 ); -if (!p || (*p && (*p !='\n' ))) + ret =kstrtoull (p,0 , &tmp); +if (ret)return -EINVAL ; reg->hr_start_block = tmp;
_
Andrew Morton
maintainerPATCH 18/192 ocfs2: remove redundant initialization of variable ret
In reply to Andrew MortonFrom: Colin Ian King <[email protected]>
Subject: ocfs2: remove redundant initialization of variable ret
The variable ret is being initialized with a value that is never read, the assignment is redundant and can be removed.
Addresses-Coverity: ("Unused value")
---
fs/ocfs2/cluster/nodemanager.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ocfs2/cluster/nodemanager.c renamed from fs/ocfs2/cluster/nodemanager.c~ocfs2-remove-redundant-initialization-of-variable-ret +1 -1
@@ -824,7 +824,7 @@ static void __exit exit_o2nm(void)static int __init init_o2nm (void ) { -int ret =-1 ; +int ret;o2hb_init ();
_
Andrew Morton
maintainerPATCH 19/192 kernel: watchdog: modify the explanation related to watchdog thread
In reply to Andrew MortonFrom: Wang Qing <[email protected]>
Subject: kernel: watchdog: modify the explanation related to watchdog thread
The watchdog thread has been replaced by cpu_stop_work, modify the explanation related.
---
kernel/watchdog.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
kernel/watchdog.c renamed from kernel/watchdog.c~kernel-watchdog-modify-the-explanation-related-to-watchdog-thread +4 -8
@@ -92,7 +92,7 @@ __setup("nmi_watchdog=", hardlockup_pani* own hardlockup detector. * * watchdog_nmi_enable/disable can be implemented to start and stop when -* softlockup watchdog threads start and stop. The arch must select the +* softlockup watchdog start and stop. The arch must select the * SOFTLOCKUP_DETECTOR Kconfig. */ int __weak watchdog_nmi_enable (unsigned int cpu)
@@ -335,7 +335,7 @@ static DEFINE_PER_CPU(struct completion, static DEFINE_PER_CPU(struct cpu_stop_work, softlockup_stop_work);/* -* The watchdog thread function - touches the timestamp. +* The watchdog feed function - touches the timestamp. * * It only runs once every sample_period seconds (4 seconds by * default) to reset the softlockup timestamp. If this gets delayed
@@ -558,11 +558,7 @@ static void lockup_detector_reconfigure( }/* -* Create the watchdog thread infrastructure and configure the detector(s). -* -* The threads are not unparked as watchdog_allowed_mask is empty. When -* the threads are successfully initialized, take the proper locks and -* unpark the threads in the watchdog_cpumask if the watchdog is enabled. +* Create the watchdog infrastructure and configure the detector(s). */ static __init void lockup_detector_setup (void ) {
@@ -628,7 +624,7 @@ void lockup_detector_soft_poweroff(void)#ifdef CONFIG_SYSCTL -/* Propagate any changes to the watchdog threads */ +/* Propagate any changes to the watchdog infrastructure */ static void proc_watchdog_update (void ) {/* Remove impossible cpus to keep sysctl output clean. */
_
Andrew Morton
maintainerPATCH 20/192 doc: watchdog: modify the explanation related to watchdog thread
In reply to Andrew MortonFrom: Wang Qing <[email protected]>
Subject: doc: watchdog: modify the explanation related to watchdog thread
"watchdog/%u" threads has be replaced by cpu_stop_work. The current description is extremely misleading.
---
Documentation/admin-guide/lockup-watchdogs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/admin-guide/lockup-watchdogs.rst renamed from Documentation/admin-guide/lockup-watchdogs.rst~doc-watchdog-modify-the-explanation-related-to-watchdog-thread +2 -2
@@ -39,7 +39,7 @@ in principle, they should work in any ar subsystems are present. A periodic hrtimer runs to generate interrupts and kick the watchdog -task. An NMI perf event is generated every "watchdog_thresh" +job. An NMI perf event is generated every "watchdog_thresh" (compile-time initialized to 10 and configurable through sysctl of the same name) seconds to check for hardlockups. If any CPU in the system does not receive any hrtimer interrupt during that time the
@@ -47,7 +47,7 @@ does not receive any hrtimer interrupt d generate a kernel warning or call panic, depending on the configuration. -The watchdog task is a high priority kernel thread that updates a +The watchdog job runs in a stop scheduling thread that updates a timestamp every time it is scheduled. If that timestamp is not updated for 2*watchdog_thresh seconds (the softlockup threshold) the 'softlockup detector' (coded inside the hrtimer callback function)
_
Andrew Morton
maintainerPATCH 21/192 doc: watchdog: modify the doc related to "watchdog/%u"
In reply to Andrew MortonFrom: Wang Qing <[email protected]>
Subject: doc: watchdog: modify the doc related to "watchdog/%u"
"watchdog/%u" threads has be replaced by cpu_stop_work. The current description is extremely misleading.
---
Documentation/admin-guide/sysctl/kernel.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
Documentation/admin-guide/sysctl/kernel.rst renamed from Documentation/admin-guide/sysctl/kernel.rst~doc-watchdog-modify-the-doc-related-to-watchdog-%u +5 -5
@@ -1283,11 +1283,11 @@ This parameter can be used to control th = ================================= The soft lockup detector monitors CPUs for threads that are hogging the CPUs -without rescheduling voluntarily, and thus prevent the 'watchdog/N' threads -from running. The mechanism depends on the CPUs ability to respond to timer -interrupts which are needed for the 'watchdog/N' threads to be woken up by -the watchdog timer function, otherwise the NMI watchdog — if enabled — can -detect a hard lockup condition. +without rescheduling voluntarily, and thus prevent the 'migration/N' threads +from running, causing the watchdog work fail to execute. The mechanism depends +on the CPUs ability to respond to timer interrupts which are needed for the +watchdog work to be queued by the watchdog timer function, otherwise the NMI +watchdog — if enabled — can detect a hard lockup condition. stack_erasing
_
Andrew Morton
maintainerPATCH 22/192 slab: use __func__ to trace function name
In reply to Andrew MortonFrom: gumingtao <[email protected]>
Subject: slab: use __func__ to trace function name
It is better to use __func__ to trace function name.
---
mm/slab_common.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
mm/slab_common.c renamed from mm/slab_common.c~slab-use-__func__-to-trace-function-name +6 -6
@@ -377,11 +377,11 @@ out_unlock:if (err) {if (flags &SLAB_PANIC ) -panic ("kmem_cache_create: Failed to create slab '%s'. Error %d\n" , - name, err); +panic ("%s: Failed to create slab '%s'. Error %d\n" , + __func__, name, err);else { -pr_warn ("kmem_cache_create(%s) failed with error %d\n" , - name, err); +pr_warn ("%s(%s) failed with error %d\n" , + __func__, name, err);dump_stack (); }return NULL ;
@@ -508,8 +508,8 @@ void kmem_cache_destroy(struct kmem_cach err =shutdown_cache (s);if (err) { -pr_err ("kmem_cache_destroy %s: Slab cache still has objects\n" , - s->name); +pr_err ("%s %s: Slab cache still has objects\n" , + __func__, s->name);dump_stack (); } out_unlock:
_
Andrew Morton
maintainerPATCH 23/192 kunit: make test->lock irq safe
In reply to Andrew MortonFrom: Vlastimil Babka <[email protected]>
Subject: kunit: make test->lock irq safe
The upcoming SLUB kunit test will be calling kunit_find_named_resource() from a context with disabled interrupts. That means kunit's test->lock needs to be IRQ safe to avoid potential deadlocks and lockdep splats.
This patch therefore changes the test->lock usage to spin_lock_irqsave() and spin_unlock_irqrestore().
---
include/kunit/test.h | 5 +++-- lib/kunit/test.c | 18 +++++++++++------- 2 files changed, 14 insertions(+), 9 deletions(-)
include/kunit/test.h renamed from include/kunit/test.h~kunit-make-test-lock-irq-safe +3 -2
@@ -515,8 +515,9 @@ kunit_find_resource(struct kunit *test, void *match_data) {struct kunit_resource *res, *found =NULL ; +unsigned long flags; -spin_lock (&test->lock); +spin_lock_irqsave (&test->lock, flags);list_for_each_entry_reverse (res, &test->resources, node) {if (match (test, res, (void *)match_data)) {
@@ -526,7 +527,7 @@ kunit_find_resource(struct kunit *test, } } -spin_unlock (&test->lock); +spin_unlock_irqrestore (&test->lock, flags);return found; }
lib/kunit/test.c renamed from lib/kunit/test.c~kunit-make-test-lock-irq-safe +11 -7
@@ -475,6 +475,7 @@ int kunit_add_resource(struct kunit *tes void *data) {int ret =0 ; +unsigned long flags; res->free = free;kref_init (&res->refcount);
@@ -487,10 +488,10 @@ int kunit_add_resource(struct kunit *tes res->data = data; } -spin_lock (&test->lock); +spin_lock_irqsave (&test->lock, flags);list_add_tail (&res->node, &test->resources);/* refcount for list is established by kref_init() */ -spin_unlock (&test->lock); +spin_unlock_irqrestore (&test->lock, flags);return ret; }
@@ -548,9 +549,11 @@ EXPORT_SYMBOL_GPL(kunit_alloc_and_get_revoid kunit_remove_resource (struct kunit *test,struct kunit_resource *res) { -spin_lock (&test->lock); +unsigned long flags; + +spin_lock_irqsave (&test->lock, flags);list_del (&res->node); -spin_unlock (&test->lock); +spin_unlock_irqrestore (&test->lock, flags);kunit_put_resource (res); }EXPORT_SYMBOL_GPL (kunit_remove_resource);
@@ -630,6 +633,7 @@ EXPORT_SYMBOL_GPL(kunit_kfree);void kunit_cleanup (struct kunit *test) {struct kunit_resource *res; +unsigned long flags;/* * test->resources is a stack - each allocation must be freed in the
@@ -641,9 +645,9 @@ void kunit_cleanup(struct kunit *test)* protect against the current node being deleted, not the next. */ while (true) { -spin_lock (&test->lock); +spin_lock_irqsave (&test->lock, flags);if (list_empty (&test->resources)) { -spin_unlock (&test->lock); +spin_unlock_irqrestore (&test->lock, flags);break ; } res = list_last_entry(&test->resources,
@@ -654,7 +658,7 @@ void kunit_cleanup(struct kunit *test)* resource, and this can't happen if the test->lock * is held. */ -spin_unlock (&test->lock); +spin_unlock_irqrestore (&test->lock, flags);kunit_remove_resource (test, res); } current->kunit_test =NULL ;
_
Andrew Morton
maintainerPATCH 24/192 mm/slub, kunit: add a KUnit test for SLUB debugging functionality
In reply to Andrew MortonFrom: Oliver Glitta <[email protected]>
Subject: mm/slub, kunit: add a KUnit test for SLUB debugging functionality
SLUB has resiliency_test() function which is hidden behind #ifdef SLUB_RESILIENCY_TEST that is not part of Kconfig, so nobody runs it. KUnit should be a proper replacement for it.
Try changing byte in redzone after allocation and changing pointer to next free node, first byte, 50th byte and redzone byte. Check if validation finds errors.
There are several differences from the original resiliency test: Tests create own caches with known state instead of corrupting shared kmalloc caches.
The corruption of freepointer uses correct offset, the original resiliency test got broken with freepointer changes.
Scratch changing random byte test, because it does not have meaning in this form where we need deterministic results.
Add new option CONFIG_SLUB_KUNIT_TEST in Kconfig. Tests next_pointer, first_word and clobber_50th_byte do not run with KASAN option on. Because the test deliberately modifies non-allocated objects.
Use kunit_resource to count errors in cache and silence bug reports. Count error whenever slab_bug() or slab_fix() is called or when the count of pages is wrong.
---
lib/Kconfig.debug | 12 +++ lib/Makefile | 1 lib/slub_kunit.c | 152 ++++++++++++++++++++++++++++++++++++++++++++ mm/kasan/common.c | 3 mm/slab.h | 1 mm/slub.c | 46 ++++++++++++- 6 files changed, 212 insertions(+), 3 deletions(-)
lib/Kconfig.debug renamed from lib/Kconfig.debug~mm-slub-kunit-add-a-kunit-test-for-slub-debugging-functionality +12 -0
@@ -2429,6 +2429,18 @@ config BITS_TEST If unsure, say N. +config SLUB_KUNIT_TEST + tristate "KUnit test for SLUB cache error detection" if !KUNIT_ALL_TESTS + depends on SLUB_DEBUG && KUNIT + default KUNIT_ALL_TESTS + help + This builds SLUB allocator unit test. + Tests SLUB cache debugging functionality. + For more information on KUnit and unit tests in general please refer + to the KUnit documentation in Documentation/dev-tools/kunit/. + + If unsure, say N. + config TEST_UDELAY tristate "udelay test driver" help
lib/Makefile renamed from lib/Makefile~mm-slub-kunit-add-a-kunit-test-for-slub-debugging-functionality +1 -0
@@ -354,5 +354,6 @@ obj-$(CONFIG_LIST_KUNIT_TEST) += list-te obj-$(CONFIG_LINEAR_RANGES_TEST) += test_linear_ranges.o obj-$(CONFIG_BITS_TEST) += test_bits.o obj-$(CONFIG_CMDLINE_KUNIT_TEST) += cmdline_kunit.o +obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o obj-$(CONFIG_GENERIC_LIB_DEVMEM_IS_ALLOWED) += devmem_is_allowed.o
lib/slub_kunit.c new file +152 -0
152 lines not on this page · raw message
mm/kasan/common.c renamed from mm/kasan/common.c~mm-slub-kunit-add-a-kunit-test-for-slub-debugging-functionality +3 -0
@@ -51,11 +51,14 @@ void kasan_enable_current(void) { current->kasan_depth++; } +EXPORT_SYMBOL (kasan_enable_current);void kasan_disable_current (void ) { current->kasan_depth--; } +EXPORT_SYMBOL (kasan_disable_current); +#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */ void __kasan_unpoison_range (const void *address,size_t size)
mm/slab.h renamed from mm/slab.h~mm-slub-kunit-add-a-kunit-test-for-slub-debugging-functionality +1 -0
@@ -215,6 +215,7 @@ DECLARE_STATIC_KEY_TRUE(slub_debug_enablDECLARE_STATIC_KEY_FALSE (slub_debug_enabled);#endif extern void print_tracking (struct kmem_cache *s,void *object); +long validate_slab_cache (struct kmem_cache *s);#else static inline void print_tracking (struct kmem_cache *s,void *object) {
mm/slub.c renamed from mm/slub.c~mm-slub-kunit-add-a-kunit-test-for-slub-debugging-functionality +43 -3
116 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 25/192 slub: remove resiliency_test() function
In reply to Andrew MortonFrom: Oliver Glitta <[email protected]>
Subject: slub: remove resiliency_test() function
Function resiliency_test() is hidden behind #ifdef SLUB_RESILIENCY_TEST that is not part of Kconfig, so nobody runs it.
This function is replaced with KUnit test for SLUB added by the previous patch "selftests: add a KUnit test for SLUB debugging functionality".
---
mm/slub.c | 64 ---------------------------------------------------- 1 file changed, 64 deletions(-)
mm/slub.c renamed from mm/slub.c~slub-remove-resiliency_test-function +0 -64
@@ -155,9 +155,6 @@ static inline bool kmem_cache_has_cpu_pa* - Variable sizing of the per node arrays */ -/* Enable to test recovery from slab corruption on boot */ -#undef SLUB_RESILIENCY_TEST -/* Enable to log cmpxchg failures */ #undef SLUB_DEBUG_CMPXCHG
@@ -4938,66 +4935,6 @@ static int list_locations(struct kmem_ca }#endif /* CONFIG_SLUB_DEBUG */ -#ifdef SLUB_RESILIENCY_TEST -static void __init resiliency_test (void ) -{ -u8 *p; -int type =KMALLOC_NORMAL ; - -BUILD_BUG_ON (KMALLOC_MIN_SIZE >16 ||KMALLOC_SHIFT_HIGH <10 ); - -pr_err ("SLUB resiliency testing\n" ); -pr_err ("-----------------------\n" ); -pr_err ("A. Corruption after allocation\n" ); - - p =kzalloc (16 ,GFP_KERNEL ); - p[16 ] =0x12 ; -pr_err ("\n1. kmalloc-16: Clobber Redzone/next pointer 0x12->0x%p\n\n" , - p +16 ); - -validate_slab_cache (kmalloc_caches[type][4 ]); - -/* Hmmm... The next two are dangerous */ - p =kzalloc (32 ,GFP_KERNEL ); - p[32 +sizeof (void *)] =0x34 ; -pr_err ("\n2. kmalloc-32: Clobber next pointer/next slab 0x34 -> -0x%p\n" , - p); -pr_err ("If allocated object is overwritten then not detectable\n\n" ); - -validate_slab_cache (kmalloc_caches[type][5 ]); - p =kzalloc (64 ,GFP_KERNEL ); - p +=64 + (get_cycles () &0xff ) *sizeof (void *); - *p =0x56 ; -pr_err ("\n3. kmalloc-64: corrupting random byte 0x56->0x%p\n" , - p); -pr_err ("If allocated object is overwritten then not detectable\n\n" ); -validate_slab_cache (kmalloc_caches[type][6 ]); - -pr_err ("\nB. Corruption after free\n" ); - p =kzalloc (128 ,GFP_KERNEL ); -kfree (p); - *p =0x78 ; -pr_err ("1. kmalloc-128: Clobber first word 0x78->0x%p\n\n" , p); -validate_slab_cache (kmalloc_caches[type][7 ]); - - p =kzalloc (256 ,GFP_KERNEL ); -kfree (p); - p[50 ] =0x9a ; -pr_err ("\n2. kmalloc-256: Clobber 50th byte 0x9a->0x%p\n\n" , p); -validate_slab_cache (kmalloc_caches[type][8 ]); - - p =kzalloc (512 ,GFP_KERNEL ); -kfree (p); - p[512 ] =0xab ; -pr_err ("\n3. kmalloc-512: Clobber redzone 0xab->0x%p\n\n" , p); -validate_slab_cache (kmalloc_caches[type][9 ]); -} -#else -#ifdef CONFIG_SYSFS -static void resiliency_test (void ) {}; -#endif -#endif /* SLUB_RESILIENCY_TEST */ -#ifdef CONFIG_SYSFS enum slab_stat_type { SL_ALL, /* All slabs */
@@ -5846,7 +5783,6 @@ static int __init slab_sysfs_init(void) }mutex_unlock (&slab_mutex); -resiliency_test ();return 0 ; }
_
Andrew Morton
maintainerPATCH 26/192 mm, slub: change run-time assertion in kmalloc_index() to compile-time
In reply to Andrew MortonFrom: Hyeonggon Yoo <[email protected]>
Subject: mm, slub: change run-time assertion in kmalloc_index() to compile-time
Currently when size is not supported by kmalloc_index, compiler will generate a run-time BUG() while compile-time error is also possible, and better. So change BUG to BUILD_BUG_ON_MSG to make compile-time check possible.
Also remove code that allocates more than 32MB because current implementation supports only up to 32MB.
---
include/linux/slab.h | 17 ++++++++++++++--- mm/kfence/kfence_test.c | 5 +++-- mm/slab_common.c | 7 +++---- 3 files changed, 20 insertions(+), 9 deletions(-)
include/linux/slab.h renamed from include/linux/slab.h~mm-slub-change-run-time-assertion-in-kmalloc_index-to-compile-time +14 -3
@@ -346,8 +346,14 @@ static __always_inline enum kmalloc_cach* 1 = 65 .. 96 bytes * 2 = 129 .. 192 bytes * n = 2^(n-1)+1 .. 2^n +* +* Note: __kmalloc_index() is compile-time optimized, and not runtime optimized; +* typical usage is via kmalloc_index() and therefore evaluated at compile-time. +* Callers where !size_is_constant should only be test modules, where runtime +* overheads of __kmalloc_index() can be tolerated. Also see kmalloc_slab(). */ -static __always_inline unsigned int kmalloc_index(size_t size) +static __always_inline unsigned int __kmalloc_index(size_t size, +bool size_is_constant) {if (!size)return 0 ;
@@ -382,12 +388,17 @@ static __always_inline unsigned int kmalif (size <=8 *1024 *1024 )return 23 ;if (size <=16 *1024 *1024 )return 24 ;if (size <=32 *1024 *1024 )return 25 ; -if (size <=64 *1024 *1024 )return 26 ; -BUG (); + +if ((IS_ENABLED (CONFIG_CC_IS_GCC ) ||CONFIG_CLANG_VERSION >=110000 ) + && !IS_ENABLED (CONFIG_PROFILE_ALL_BRANCHES ) && size_is_constant) +BUILD_BUG_ON_MSG (1 ,"unexpected size in kmalloc_index()" ); +else +BUG ();/* Will never be reached. Needed because the compiler may complain */ return -1; } +#define kmalloc_index (s) __kmalloc_index(s, true)#endif /* !CONFIG_SLOB */ void *__kmalloc (size_t size,gfp_t flags) __assume_kmalloc_alignment __malloc;
mm/kfence/kfence_test.c renamed from mm/kfence/kfence_test.c~mm-slub-change-run-time-assertion-in-kmalloc_index-to-compile-time +3 -2
@@ -197,7 +197,7 @@ static void test_cache_destroy(void)static inline size_t kmalloc_cache_alignment (size_t size) { -return kmalloc_caches[kmalloc_type (GFP_KERNEL )][kmalloc_index (size)]->align; +return kmalloc_caches[kmalloc_type (GFP_KERNEL )][__kmalloc_index (size, false)]->align; }/* Must always inline to match stack trace against caller. */
@@ -267,7 +267,8 @@ static void *test_alloc(struct kunit *teif (is_kfence_address (alloc)) {struct page *page =virt_to_head_page (alloc); -struct kmem_cache *s = test_cache ?: kmalloc_caches[kmalloc_type (GFP_KERNEL )][kmalloc_index (size)]; +struct kmem_cache *s = test_cache ?: + kmalloc_caches[kmalloc_type (GFP_KERNEL )][__kmalloc_index (size, false)];/* * Verify that various helpers return the right values
mm/slab_common.c renamed from mm/slab_common.c~mm-slub-change-run-time-assertion-in-kmalloc_index-to-compile-time +3 -4
@@ -754,8 +754,8 @@ struct kmem_cache *kmalloc_slab(size_t s/* * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time. -* kmalloc_index() supports up to 2^26=64MB, so the final entry of the table is -* kmalloc-67108864. +* kmalloc_index() supports up to 2^25=32MB, so the final entry of the table is +* kmalloc-32M. */ const struct kmalloc_info_struct kmalloc_info[]__initconst = {INIT_KMALLOC_INFO (0 ,0 ),
@@ -783,8 +783,7 @@ const struct kmalloc_info_struct kmalloc INIT_KMALLOC_INFO(4194304, 4M), INIT_KMALLOC_INFO(8388608, 8M), INIT_KMALLOC_INFO(16777216, 16M), - INIT_KMALLOC_INFO(33554432, 32M), - INIT_KMALLOC_INFO(67108864, 64M) + INIT_KMALLOC_INFO(33554432, 32M) };/*
_
Andrew Morton
maintainerPATCH 27/192 slub: restore slub_debug=- behavior
In reply to Andrew MortonFrom: Stephen Boyd <[email protected]>
Subject: slub: restore slub_debug=- behavior
Petch series "slub: Print non-hashed pointers in slub debugging", v3.
I was doing some debugging recently and noticed that my pointers were being hashed while slub_debug was on the kernel commandline. Let's force on the no hash pointer option when slub_debug is on the kernel commandline so that the prints are more meaningful.
The first two patches are something else I noticed while looking at the code. The message argument is never used so the debugging messages are not as clear as they could be and the slub_debug=- behavior seems to be busted. Then there's a printf fixup from Joe and the final patch is the one that force disables pointer hashing.
This patch (of 4):
Passing slub_debug=- on the kernel commandline is supposed to disable slub debugging. This is especially useful with CONFIG_SLUB_DEBUG_ON where the default is to have slub debugging enabled in the build. Due to some code reorganization this behavior was dropped, but the code to make it work mostly stuck around. Restore the previous behavior by disabling the static key when we parse the commandline and see that we're trying to disable slub debugging.
---
mm/slub.c | 2 ++ 1 file changed, 2 insertions(+)
mm/slub.c renamed from mm/slub.c~slub-restore-slub_debug=-behavior +2 -0
@@ -1429,6 +1429,8 @@ static int __init setup_slub_debug(char out:if (slub_debug !=0 || slub_debug_string)static_branch_enable (&slub_debug_enabled); +else +static_branch_disable (&slub_debug_enabled);if ((static_branch_unlikely (&init_on_alloc) ||static_branch_unlikely (&init_on_free)) && (slub_debug &SLAB_POISON ))
_
Andrew Morton
maintainerPATCH 28/192 slub: actually use 'message' in restore_bytes()
In reply to Andrew MortonFrom: Stephen Boyd <[email protected]>
Subject: slub: actually use 'message' in restore_bytes()
The message argument isn't used here. Let's pass the string to the printk message so that the developer can figure out what's happening, instead of guessing that a redzone is being restored, etc.
---
mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/slub.c renamed from mm/slub.c~slub-actually-use-message-in-restore_bytes +1 -1
@@ -806,7 +806,7 @@ static void init_object(struct kmem_cachstatic void restore_bytes (struct kmem_cache *s,char *message,u8 data,void *from,void *to) { -slab_fix (s,"Restoring 0x%p-0x%p=0x%x\n" , from, to -1 , data); +slab_fix (s,"Restoring %s 0x%p-0x%p=0x%x\n" , message, from, to -1 , data);memset (from, data, to - from); }
_
Andrew Morton
maintainerPATCH 29/192 slub: indicate slab_fix() uses printf formats
In reply to Andrew MortonFrom: Joe Perches <[email protected]>
Subject: slub: indicate slab_fix() uses printf formats
Ideally, slab_fix() would be marked with __printf and the format here would not use \n as that's emitted by the slab_fix(). Make these changes.
---
mm/slub.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
mm/slub.c renamed from mm/slub.c~slub-indicate-slab_fix-uses-printf-formats +4 -3
@@ -692,6 +692,7 @@ static void slab_bug(struct kmem_cache *va_end (args); } +__printf(2, 3)static void slab_fix (struct kmem_cache *s,char *fmt, ...) {struct va_format vaf;
@@ -806,7 +807,7 @@ static void init_object(struct kmem_cachstatic void restore_bytes (struct kmem_cache *s,char *message,u8 data,void *from,void *to) { -slab_fix (s,"Restoring %s 0x%p-0x%p=0x%x\n" , message, from, to -1 , data); +slab_fix (s,"Restoring %s 0x%p-0x%p=0x%x" , message, from, to -1 , data);memset (from, data, to - from); }
@@ -1059,13 +1060,13 @@ static int on_freelist(struct kmem_cacheslab_err (s, page,"Wrong number of objects. Found %d but should be %d" , page->objects, max_objects); page->objects = max_objects; -slab_fix (s,"Number of objects adjusted." ); +slab_fix (s,"Number of objects adjusted" ); }if (page->inuse != page->objects - nr) {slab_err (s, page,"Wrong object count. Counter is %d but counted were %d" , page->inuse, page->objects - nr); page->inuse = page->objects - nr; -slab_fix (s,"Object count adjusted." ); +slab_fix (s,"Object count adjusted" ); }return search ==NULL ; }
_
Andrew Morton
maintainerPATCH 30/192 slub: force on no_hash_pointers when slub_debug is enabled
In reply to Andrew MortonFrom: Stephen Boyd <[email protected]>
Subject: slub: force on no_hash_pointers when slub_debug is enabled
Obscuring the pointers that slub shows when debugging makes for some confusing slub debug messages:
Padding overwritten. 0x0000000079f0674a-0x000000000d4dce17
Those addresses are hashed for kernel security reasons. If we're trying to be secure with slub_debug on the commandline we have some big problems given that we dump whole chunks of kernel memory to the kernel logs. Let's force on the no_hash_pointers commandline flag when slub_debug is on the commandline. This makes slub debug messages more meaningful and if by chance a kernel address is in some slub debug object dump we will have a better chance of figuring out what went wrong.
Note that we don't use %px in the slub code because we want to reduce the number of places that %px is used in the kernel. This also nicely prints a big fat warning at kernel boot if slub_debug is on the commandline so that we know that this kernel shouldn't be used on production systems.
---
include/linux/kernel.h | 2 ++ lib/vsprintf.c | 2 +- mm/slub.c | 20 +++++++++++++++++++- 3 files changed, 22 insertions(+), 2 deletions(-)
include/linux/kernel.h renamed from include/linux/kernel.h~slub-force-on-no_hash_pointers-when-slub_debug-is-enabled +2 -0
@@ -357,6 +357,8 @@ int sscanf(const char *, const char *, . extern __scanf(2, 0) int vsscanf(const char *, const char *, va_list); +extern int no_hash_pointers_enable(char *str); + extern int get_option(char **str, int *pint); extern char *get_options(const char *str, int nints, int *ints); extern unsigned long long memparse(const char *ptr, char **retptr);
lib/vsprintf.c renamed from lib/vsprintf.c~slub-force-on-no_hash_pointers-when-slub_debug-is-enabled +1 -1
@@ -2186,7 +2186,7 @@ char *fwnode_string(char *buf, char *endbool no_hash_pointers__ro_after_init ;EXPORT_SYMBOL_GPL (no_hash_pointers); -static int __init no_hash_pointers_enable (char *str) +int __init no_hash_pointers_enable (char *str) {if (no_hash_pointers)return 0 ;
mm/slub.c renamed from mm/slub.c~slub-force-on-no_hash_pointers-when-slub_debug-is-enabled +19 -1
@@ -118,12 +118,26 @@*/ #ifdef CONFIG_SLUB_DEBUG +#ifdef CONFIG_SLUB_DEBUG_ON DEFINE_STATIC_KEY_TRUE (slub_debug_enabled);#else DEFINE_STATIC_KEY_FALSE (slub_debug_enabled);#endif -#endif + +static inline bool __slub_debug_enabled (void ) +{ +return static_branch_unlikely (&slub_debug_enabled); +} + +#else /* CONFIG_SLUB_DEBUG */ + +static inline bool __slub_debug_enabled (void ) +{ +return false; +} + +#endif /* CONFIG_SLUB_DEBUG */ static inline bool kmem_cache_debug (struct kmem_cache *s) {
@@ -4487,6 +4501,10 @@ void __init kmem_cache_init(void)if (debug_guardpage_minorder ()) slub_max_order =0 ; +/* Print slub debugging pointers without hashing */ +if (__slub_debug_enabled ()) +no_hash_pointers_enable (NULL ); + kmem_cache_node = &boot_kmem_cache_node; kmem_cache = &boot_kmem_cache;
_
Andrew Morton
maintainerPATCH 31/192 mm: slub: move sysfs slab alloc/free interfaces to debugfs
In reply to Andrew MortonFrom: Faiyaz Mohammed <[email protected]>
Subject: mm: slub: move sysfs slab alloc/free interfaces to debugfs
alloc_calls and free_calls implementation in sysfs have two issues, one is PAGE_SIZE limitation of sysfs and other is it does not adhere to "one value per file" rule.
To overcome this issues, move the alloc_calls and free_calls implementation to debugfs.
Debugfs cache will be created if SLAB_STORE_USER flag is set.
Rename the alloc_calls/free_calls to alloc_traces/free_traces, to be inline with what it does.
---
mm/slab.h | 6 mm/slab_common.c | 2 mm/slub.c | 274 +++++++++++++++++++++++++++++---------------- 3 files changed, 189 insertions(+), 93 deletions(-)
mm/slab_common.c renamed from mm/slab_common.c~mm-slub-move-sysfs-slab-alloc-free-interfaces-to-debugfs +2 -0
@@ -448,6 +448,7 @@ static void slab_caches_to_rcu_destroy_wrcu_barrier ();list_for_each_entry_safe (s, s2, &to_destroy, list) { +debugfs_slab_release (s);kfence_shutdown_cache (s);#ifdef SLAB_SUPPORTS_SYSFS sysfs_slab_release (s);
@@ -475,6 +476,7 @@ static int shutdown_cache(struct kmem_caschedule_work (&slab_caches_to_rcu_destroy_work); }else {kfence_shutdown_cache (s); +debugfs_slab_release (s);#ifdef SLAB_SUPPORTS_SYSFS sysfs_slab_unlink (s);sysfs_slab_release (s);
mm/slab.h renamed from mm/slab.h~mm-slub-move-sysfs-slab-alloc-free-interfaces-to-debugfs +6 -0
@@ -631,6 +631,12 @@ static inline bool slab_want_init_on_frereturn false; } +#if defined(CONFIG_DEBUG_FS ) && defined(CONFIG_SLUB_DEBUG ) +void debugfs_slab_release (struct kmem_cache *); +#else +static inline void debugfs_slab_release (struct kmem_cache *s) { } +#endif +#ifdef CONFIG_PRINTK #define KS_ADDRS_COUNT 16 struct kmem_obj_info {
mm/slub.c renamed from mm/slub.c~mm-slub-move-sysfs-slab-alloc-free-interfaces-to-debugfs +181 -93
328 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 32/192 mm/slub: add taint after the errors are printed
In reply to Andrew MortonFrom: Georgi Djakov <[email protected]>
Subject: mm/slub: add taint after the errors are printed
When running the kernel with panic_on_taint, the usual slub debug error messages are not being printed when object corruption happens. That's because we panic in add_taint(), which is called before printing the additional information. This is a bit unfortunate as the error messages are actually very useful, especially before a panic. Let's fix this by moving add_taint() after the errors are printed on the console.
---
mm/slub.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/slub.c renamed from mm/slub.c~mm-slub-add-taint-after-the-errors-are-printed +3 -2
@@ -708,8 +708,6 @@ static void slab_bug(struct kmem_cache *pr_err ("=============================================================================\n" );pr_err ("BUG %s (%s): %pV\n" , s->name,print_tainted (), &vaf);pr_err ("-----------------------------------------------------------------------------\n\n" ); - -add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE );va_end (args); }
@@ -790,6 +788,7 @@ void object_err(struct kmem_cache *s, stslab_bug (s,"%s" , reason);print_trailer (s, page, object); +add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } static __printf(3, 4) void slab_err(struct kmem_cache *s, struct page *page,
@@ -807,6 +806,7 @@ static __printf(3, 4) void slab_err(struslab_bug (s,"%s" , buf);print_page_info (page);dump_stack (); +add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); }static void init_object (struct kmem_cache *s,void *object,u8 val)
@@ -858,6 +858,7 @@ static int check_bytes_and_report(struct fault, end -1 , fault - addr, fault[0], value);print_trailer (s, page, object); +add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); skip_bug_print:restore_bytes (s, what, value, fault, end);
_
Andrew Morton
maintainerPATCH 33/192 mm/kmemleak: fix possible wrong memory scanning period
In reply to Andrew MortonFrom: Yanfei Xu <[email protected]>
Subject: mm/kmemleak: fix possible wrong memory scanning period
This commit contains 3 modifications:
1. Convert the type of jiffies_scan_wait to "unsigned long".
2. Use READ/WRITE_ONCE() for accessing "jiffies_scan_wait".
3. Fix the possible wrong memory scanning period. If you set a large memory scanning period like blow, then the "secs" variable will be non-zero, however the value of "jiffies_scan_wait" will be zero.
echo "scan=0x10000000" > /sys/kernel/debug/kmemleak
It is because the type of the msecs_to_jiffies()'s parameter is "unsigned int", and the "secs * 1000" is larger than its max value. This in turn leads a unexpected jiffies_scan_wait, maybe zero. We corret it by replacing kstrtoul() with kstrtouint(), and check the msecs to prevent it larger than UINT_MAX.
---
mm/kmemleak.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
mm/kmemleak.c renamed from mm/kmemleak.c~mm-kmemleak-fix-the-possible-wrong-memory-scanning-period +12 -6
@@ -219,7 +219,7 @@ static struct task_struct *scan_thread;static unsigned long jiffies_min_age;static unsigned long jiffies_last_scan;/* delay between automatic memory scannings */ -static signed long jiffies_scan_wait; +static unsigned long jiffies_scan_wait;/* enables or disables the task stacks scanning */ static int kmemleak_stack_scan =1 ;/* protects the memory scanning, parameters and debug/kmemleak file access */
@@ -1567,7 +1567,7 @@ static int kmemleak_scan_thread(void *ar }while (!kthread_should_stop ()) { -signed long timeout = jiffies_scan_wait; +signed long timeout =READ_ONCE (jiffies_scan_wait);mutex_lock (&scan_mutex);kmemleak_scan ();
@@ -1807,14 +1807,20 @@ static ssize_t kmemleak_write(struct filelse if (strncmp (buf,"scan=off" ,8 ) ==0 )stop_scan_thread ();else if (strncmp (buf,"scan=" ,5 ) ==0 ) { -unsigned long secs; +unsigned secs; +unsigned long msecs; - ret =kstrtoul (buf +5 ,0 , &secs); + ret =kstrtouint (buf +5 ,0 , &secs);if (ret <0 ) goto out; + + msecs = secs *MSEC_PER_SEC ; +if (msecs >UINT_MAX ) + msecs =UINT_MAX ; +stop_scan_thread (); -if (secs) { - jiffies_scan_wait =msecs_to_jiffies (secs *1000 ); +if (msecs) { +WRITE_ONCE (jiffies_scan_wait,msecs_to_jiffies (msecs));start_scan_thread (); } }else if (strncmp (buf,"scan" ,4 ) ==0 )
_
Andrew Morton
maintainerPATCH 34/192 dax: fix ENOMEM handling in grab_mapping_entry()
In reply to Andrew MortonFrom: Jan Kara <[email protected]>
Subject: dax: fix ENOMEM handling in grab_mapping_entry()
grab_mapping_entry() has a bug in handling of ENOMEM condition. Suppose we have a PMD entry at index i which we are downgrading to a PTE entry. grab_mapping_entry() will set pmd_downgrade to true, lock the entry, clear the entry in xarray, and decrement mapping->nrpages. The it will call:
entry = dax_make_entry(pfn_to_pfn_t(0), flags); dax_lock_entry(xas, entry);
which inserts new PTE entry into xarray. However this may fail allocating the new node. We handle this by:
if (xas_nomem(xas, mapping_gfp_mask(mapping) & ~__GFP_HIGHMEM)) goto retry;
however pmd_downgrade stays set to true even though 'entry' returned from get_unlocked_entry() will be NULL now. And we will go again through the downgrade branch. This is mostly harmless except that mapping->nrpages is decremented again and we temporarily have an invalid entry stored in xarray. Fix the problem by setting pmd_downgrade to false each time we lookup the entry we work with so that it matches the entry we found.
---
fs/dax.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/dax.c renamed from fs/dax.c~dax-fix-enomem-handling-in-grab_mapping_entry +2 -1
@@ -488,10 +488,11 @@ static void *grab_mapping_entry(struct x struct address_space *mapping, unsigned int order) {unsigned long index = xas->xa_index; -bool pmd_downgrade = false;/* splitting PMD entry into PTE entries? */ +bool pmd_downgrade;/* splitting PMD entry into PTE entries? */ void *entry; retry: + pmd_downgrade = false;xas_lock_irq (xas); entry =get_unlocked_entry (xas, order);
_
Andrew Morton
maintainerPATCH 35/192 tools/vm/page_owner_sort.c: check malloc() return
In reply to Andrew MortonFrom: Tang Bin <[email protected]>
Subject: tools/vm/page_owner_sort.c: check malloc() return
---
tools/vm/page_owner_sort.c | 4 ++++ 1 file changed, 4 insertions(+)
tools/vm/page_owner_sort.c renamed from tools/vm/page_owner_sort.c~tools-vm-page_owner_sortc-fix-the-potential-stack-overflow-risk +4 -0
@@ -132,6 +132,10 @@ int main(int argc, char **argv)qsort (list, list_size,sizeof (list[0 ]), compare_txt); list2 =malloc (sizeof (*list) * list_size); +if (!list2) { +printf ("Out of memory\n" ); +exit (1 ); + }printf ("culling\n" );
_
Andrew Morton
maintainerPATCH 36/192 mm/debug_vm_pgtable: ensure THP availability via has_transparent_hugepage()
In reply to Andrew MortonFrom: Anshuman Khandual <[email protected]>
Subject: mm/debug_vm_pgtable: ensure THP availability via has_transparent_hugepage()
On certain platforms, THP support could not just be validated via the build option CONFIG_TRANSPARENT_HUGEPAGE. Instead has_transparent_hugepage() also needs to be called upon to verify THP runtime support. Otherwise the debug test will just run into unusable THP helpers like in the case of a 4K hash config on powerpc platform [1]. This just moves all pfn_pmd() and pfn_pud() after THP runtime validation with has_transparent_hugepage() which prevents the mentioned problem.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=213069
---
mm/debug_vm_pgtable.c | 63 ++++++++++++++++++++++++++++++++-------- 1 file changed, 51 insertions(+), 12 deletions(-)
mm/debug_vm_pgtable.c renamed from mm/debug_vm_pgtable.c~mm-debug_vm_pgtable-ensure-thp-availability-via-has_transparent_hugepage +51 -12
@@ -146,13 +146,14 @@ static void __init pte_savedwrite_tests(static void __init pmd_basic_tests (unsigned long pfn,int idx) {pgprot_t prot = protection_map[idx]; -pmd_t pmd =pfn_pmd (pfn, prot);unsigned long val = idx, *ptr = &val; +pmd_t pmd;if (!has_transparent_hugepage ())return ;pr_debug ("Validating PMD basic (%pGv)\n" , ptr); + pmd =pfn_pmd (pfn, prot);/* * This test needs to be executed after the given page table entry
@@ -185,7 +186,7 @@ static void __init pmd_advanced_tests(st unsigned long pfn, unsigned long vaddr, pgprot_t prot, pgtable_t pgtable) { - pmd_t pmd = pfn_pmd(pfn, prot); + pmd_t pmd;if (!has_transparent_hugepage ())return ;
@@ -232,9 +233,14 @@ static void __init pmd_advanced_tests(ststatic void __init pmd_leaf_tests (unsigned long pfn,pgprot_t prot) { -pmd_t pmd =pfn_pmd (pfn, prot); +pmd_t pmd; + +if (!has_transparent_hugepage ()) +return ;pr_debug ("Validating PMD leaf\n" ); + pmd =pfn_pmd (pfn, prot); +/* * PMD based THP is a leaf entry. */
@@ -267,12 +273,16 @@ static void __init pmd_huge_tests(pmd_tstatic void __init pmd_savedwrite_tests (unsigned long pfn,pgprot_t prot) { -pmd_t pmd =pfn_pmd (pfn, prot); +pmd_t pmd;if (!IS_ENABLED (CONFIG_NUMA_BALANCING ))return ; +if (!has_transparent_hugepage ()) +return ; +pr_debug ("Validating PMD saved write\n" ); + pmd =pfn_pmd (pfn, prot);WARN_ON (!pmd_savedwrite (pmd_mk_savedwrite (pmd_clear_savedwrite (pmd))));WARN_ON (pmd_savedwrite (pmd_clear_savedwrite (pmd_mk_savedwrite (pmd)))); }
@@ -281,13 +291,14 @@ static void __init pmd_savedwrite_tests(static void __init pud_basic_tests (struct mm_struct *mm,unsigned long pfn,int idx) {pgprot_t prot = protection_map[idx]; -pud_t pud =pfn_pud (pfn, prot);unsigned long val = idx, *ptr = &val; +pud_t pud;if (!has_transparent_hugepage ())return ;pr_debug ("Validating PUD basic (%pGv)\n" , ptr); + pud =pfn_pud (pfn, prot);/* * This test needs to be executed after the given page table entry
@@ -323,7 +334,7 @@ static void __init pud_advanced_tests(st unsigned long pfn, unsigned long vaddr, pgprot_t prot) { - pud_t pud = pfn_pud(pfn, prot); + pud_t pud;if (!has_transparent_hugepage ())return ;
@@ -332,6 +343,7 @@ static void __init pud_advanced_tests(st/* Align the address wrt HPAGE_PUD_SIZE */ vaddr &=HPAGE_PUD_MASK ; + pud =pfn_pud (pfn, prot);set_pud_at (mm, vaddr, pudp, pud);pudp_set_wrprotect (mm, vaddr, pudp); pud =READ_ONCE (*pudp);
@@ -370,9 +382,13 @@ static void __init pud_advanced_tests(ststatic void __init pud_leaf_tests (unsigned long pfn,pgprot_t prot) { -pud_t pud =pfn_pud (pfn, prot); +pud_t pud; + +if (!has_transparent_hugepage ()) +return ;pr_debug ("Validating PUD leaf\n" ); + pud =pfn_pud (pfn, prot);/* * PUD based THP is a leaf entry. */
@@ -654,12 +670,16 @@ static void __init pte_protnone_tests(un#ifdef CONFIG_TRANSPARENT_HUGEPAGE static void __init pmd_protnone_tests (unsigned long pfn,pgprot_t prot) { -pmd_t pmd =pmd_mkhuge (pfn_pmd (pfn, prot)); +pmd_t pmd;if (!IS_ENABLED (CONFIG_NUMA_BALANCING ))return ; +if (!has_transparent_hugepage ()) +return ; +pr_debug ("Validating PMD protnone\n" ); + pmd =pmd_mkhuge (pfn_pmd (pfn, prot));WARN_ON (!pmd_protnone (pmd));WARN_ON (!pmd_present (pmd)); }
@@ -679,18 +699,26 @@ static void __init pte_devmap_tests(unsi#ifdef CONFIG_TRANSPARENT_HUGEPAGE static void __init pmd_devmap_tests (unsigned long pfn,pgprot_t prot) { -pmd_t pmd =pfn_pmd (pfn, prot); +pmd_t pmd; + +if (!has_transparent_hugepage ()) +return ;pr_debug ("Validating PMD devmap\n" ); + pmd =pfn_pmd (pfn, prot);WARN_ON (!pmd_devmap (pmd_mkdevmap (pmd))); }#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD static void __init pud_devmap_tests (unsigned long pfn,pgprot_t prot) { -pud_t pud =pfn_pud (pfn, prot); +pud_t pud; + +if (!has_transparent_hugepage ()) +return ;pr_debug ("Validating PUD devmap\n" ); + pud =pfn_pud (pfn, prot);WARN_ON (!pud_devmap (pud_mkdevmap (pud))); }#else /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */
@@ -733,25 +761,33 @@ static void __init pte_swap_soft_dirty_t#ifdef CONFIG_TRANSPARENT_HUGEPAGE static void __init pmd_soft_dirty_tests (unsigned long pfn,pgprot_t prot) { -pmd_t pmd =pfn_pmd (pfn, prot); +pmd_t pmd;if (!IS_ENABLED (CONFIG_MEM_SOFT_DIRTY ))return ; +if (!has_transparent_hugepage ()) +return ; +pr_debug ("Validating PMD soft dirty\n" ); + pmd =pfn_pmd (pfn, prot);WARN_ON (!pmd_soft_dirty (pmd_mksoft_dirty (pmd)));WARN_ON (pmd_soft_dirty (pmd_clear_soft_dirty (pmd))); }static void __init pmd_swap_soft_dirty_tests (unsigned long pfn,pgprot_t prot) { -pmd_t pmd =pfn_pmd (pfn, prot); +pmd_t pmd;if (!IS_ENABLED (CONFIG_MEM_SOFT_DIRTY ) || !IS_ENABLED (CONFIG_ARCH_ENABLE_THP_MIGRATION ))return ; +if (!has_transparent_hugepage ()) +return ; +pr_debug ("Validating PMD swap soft dirty\n" ); + pmd =pfn_pmd (pfn, prot);WARN_ON (!pmd_swp_soft_dirty (pmd_swp_mksoft_dirty (pmd)));WARN_ON (pmd_swp_soft_dirty (pmd_swp_clear_soft_dirty (pmd))); }
@@ -780,6 +816,9 @@ static void __init pmd_swap_tests(unsignswp_entry_t swp;pmd_t pmd; +if (!has_transparent_hugepage ()) +return ; +pr_debug ("Validating PMD swap\n" ); pmd =pfn_pmd (pfn, prot); swp =__pmd_to_swp_entry (pmd);
_
Andrew Morton
maintainerPATCH 37/192 mm: mmap_lock: use local locks instead of disabling preemption
In reply to Andrew MortonFrom: Nicolas Saenz Julienne <[email protected]>
Subject: mm: mmap_lock: use local locks instead of disabling preemption
mmap_lock will explicitly disable/enable preemption upon manipulating its local CPU variables. This is to be expected, but in this case, it doesn't play well with PREEMPT_RT. The preemption disabled code section also takes a spin-lock. Spin-locks in RT systems will try to schedule, which is exactly what we're trying to avoid.
To mitigate this, convert the explicit preemption handling to local_locks. Which are RT aware, and will disable migration instead of preemption when PREEMPT_RT=y.
The faulty call trace looks like the following:
__mmap_lock_do_trace_*()
preempt_disable()
get_mm_memcg_path()
cgroup_path()
kernfs_path_from_node()
spin_lock_irqsave() /* Scheduling while atomic! */---
mm/mmap_lock.c | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-)
mm/mmap_lock.c renamed from mm/mmap_lock.c~mm-mmap_lock-use-local-locks-instead-of-disabling-preemption +22 -11
@@ -11,6 +11,7 @@#include <linux/rcupdate.h> #include <linux/smp.h> #include <linux/trace_events.h> +#include <linux/local_lock.h> EXPORT_TRACEPOINT_SYMBOL (mmap_lock_start_locking);EXPORT_TRACEPOINT_SYMBOL (mmap_lock_acquire_returned);
@@ -39,21 +40,30 @@ static int reg_refcount; /* Protected by*/ #define CONTEXT_COUNT 4 -static DEFINE_PER_CPU(char __rcu *, memcg_path_buf); +struct memcg_path { +local_lock_t lock; +char __rcu *buf; +local_t buf_idx; +}; +static DEFINE_PER_CPU(struct memcg_path, memcg_paths) = { + .lock =INIT_LOCAL_LOCK (lock), + .buf_idx =LOCAL_INIT (0 ), +}; +static char **tmp_bufs; -static DEFINE_PER_CPU(int, memcg_path_buf_idx);/* Called with reg_lock held. */ static void free_memcg_path_bufs (void ) { +struct memcg_path *memcg_path;int cpu;char **old = tmp_bufs;for_each_possible_cpu (cpu) { - *(old++) =rcu_dereference_protected ( -per_cpu (memcg_path_buf, cpu), + memcg_path =per_cpu_ptr (&memcg_paths, cpu); + *(old++) =rcu_dereference_protected (memcg_path->buf,lockdep_is_held (®_lock)); -rcu_assign_pointer (per_cpu (memcg_path_buf, cpu),NULL ); +rcu_assign_pointer (memcg_path->buf,NULL ); }/* Wait for inflight memcg_path_buf users to finish. */
@@ -88,7 +98,7 @@ int trace_mmap_lock_reg(void) new =kmalloc (MEMCG_PATH_BUF_SIZE *CONTEXT_COUNT ,GFP_KERNEL );if (new ==NULL ) goto out_fail_free; -rcu_assign_pointer (per_cpu (memcg_path_buf, cpu), new); +rcu_assign_pointer (per_cpu_ptr (&memcg_paths, cpu)->buf, new);/* Don't need to wait for inflights, they'd have gotten NULL. */ }
@@ -122,23 +132,24 @@ out:static inline char *get_memcg_path_buf (void ) { +struct memcg_path *memcg_path =this_cpu_ptr (&memcg_paths);char *buf;int idx;rcu_read_lock (); - buf =rcu_dereference (*this_cpu_ptr (&memcg_path_buf)); + buf =rcu_dereference (memcg_path->buf);if (buf ==NULL ) {rcu_read_unlock ();return NULL ; } - idx =this_cpu_add_return (memcg_path_buf_idx,MEMCG_PATH_BUF_SIZE ) - + idx =local_add_return (MEMCG_PATH_BUF_SIZE , &memcg_path->buf_idx) -MEMCG_PATH_BUF_SIZE ;return &buf[idx]; }static inline void put_memcg_path_buf (void ) { -this_cpu_sub (memcg_path_buf_idx,MEMCG_PATH_BUF_SIZE ); +local_sub (MEMCG_PATH_BUF_SIZE , &this_cpu_ptr (&memcg_paths)->buf_idx);rcu_read_unlock (); }
@@ -179,14 +190,14 @@ out:#define TRACE_MMAP_LOCK_EVENT (type, mm, ...) \ do { \ const char *memcg_path; \ - preempt_disable(); \ + local_lock(&memcg_paths.lock); \ memcg_path = get_mm_memcg_path(mm); \ trace_mmap_lock_##type(mm, \ memcg_path != NULL ? memcg_path : "", \ ##__VA_ARGS__); \ if (likely(memcg_path != NULL)) \ put_memcg_path_buf(); \ - preempt_enable(); \ + local_unlock(&memcg_paths.lock); \ } while (0)#else /* !CONFIG_MEMCG */
_
Andrew Morton
maintainerPATCH 38/192 mm/page_reporting: fix code style in __page_reporting_request()
In reply to Andrew MortonFrom: Gavin Shan <[email protected]>
Subject: mm/page_reporting: fix code style in __page_reporting_request()
Patch series "mm/page_reporting: Make page reporting work on arm64 with 64KB page size", v4.
The page reporting threshold is currently equal to @pageblock_order, which is 13 and 512MB on arm64 with 64KB base page size selected. The page reporting won't be triggered if the freeing page can't come up with a free area like that huge. The condition is hard to be met, especially when the system memory becomes fragmented.
This series intends to solve the issue by having page reporting threshold as 5 (2MB) on arm64 with 64KB base page size. The patches are organized as:
PATCH[1/4] Fix some coding style in __page_reporting_request(). PATCH[2/4] Represents page reporting order with variable so that it can
be exported as module parameter.
PATCH[3/4] Allows the device driver (e.g. virtio_balloon) to specify
the page reporting order when the device info is registered.
PATCH[4/4] Specifies the page reporting order to 5, corresponding to
2MB in size on ARM64 when 64KB base page size is used.
This patch (of 4):
The lines of comments would be starting with one, instead two space. This corrects the style.
---
mm/page_reporting.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/page_reporting.c renamed from mm/page_reporting.c~mm-page_reporting-fix-code-style-in-__page_reporting_request +2 -2
@@ -31,8 +31,8 @@ __page_reporting_request(struct page_repreturn ;/* -* If reporting is already active there is nothing we need to do. -* Test against 0 as that represents PAGE_REPORTING_IDLE. +* If reporting is already active there is nothing we need to do. +* Test against 0 as that represents PAGE_REPORTING_IDLE. */ state =atomic_xchg (&prdev->state,PAGE_REPORTING_REQUESTED );if (state !=PAGE_REPORTING_IDLE )
_
Andrew Morton
maintainerPATCH 39/192 mm/page_reporting: export reporting order as module parameter
In reply to Andrew MortonFrom: Gavin Shan <[email protected]>
Subject: mm/page_reporting: export reporting order as module parameter
The macro PAGE_REPORTING_MIN_ORDER is defined as the page reporting threshold. It can't be adjusted at runtime.
This introduces a variable (@page_reporting_order) to replace the marcro (PAGE_REPORTING_MIN_ORDER). MAX_ORDER is assigned to it initially, meaning the page reporting is disabled. It will be specified by driver if valid one is provided. Otherwise, it will fall back to @pageblock_order. It's also exported so that the page reporting order can be adjusted at runtime.
---
Documentation/admin-guide/kernel-parameters.txt | 6 ++++++ mm/page_reporting.c | 9 +++++++-- mm/page_reporting.h | 5 ++--- 3 files changed, 15 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt renamed from Documentation/admin-guide/kernel-parameters.txt~mm-page_reporting-export-reporting-order-as-module-parameter +6 -0
@@ -3566,6 +3566,12 @@ off: turn off poisoning (default) on: turn on poisoning + page_reporting.page_reporting_order= + [KNL] Minimal page reporting order + Format: <integer> + Adjust the minimal page reporting order. The page + reporting is disabled when it exceeds (MAX_ORDER-1). + panic= [KNL] Kernel behaviour on panic: delay <timeout> timeout > 0: seconds before rebooting timeout = 0: wait forever
mm/page_reporting.c renamed from mm/page_reporting.c~mm-page_reporting-export-reporting-order-as-module-parameter +7 -2
@@ -4,12 +4,17 @@#include <linux/page_reporting.h> #include <linux/gfp.h> #include <linux/export.h> +#include <linux/module.h> #include <linux/delay.h> #include <linux/scatterlist.h> #include "page_reporting.h" #include "internal.h" +unsigned int page_reporting_order =MAX_ORDER ; +module_param (page_reporting_order, uint,0644 ); +MODULE_PARM_DESC (page_reporting_order,"Set page reporting order" ); +#define PAGE_REPORTING_DELAY (2 * HZ)static struct page_reporting_dev_info __rcu *pr_dev_info__read_mostly ;
@@ -229,7 +234,7 @@ page_reporting_process_zone(struct page_/* Generate minimum watermark to be able to guarantee progress */ watermark =low_wmark_pages (zone) + - (PAGE_REPORTING_CAPACITY <<PAGE_REPORTING_MIN_ORDER ); + (PAGE_REPORTING_CAPACITY << page_reporting_order);/* * Cancel request if insufficient free memory or if we failed
@@ -239,7 +244,7 @@ page_reporting_process_zone(struct page_return err;/* Process each free list starting from lowest order/mt */ -for (order =PAGE_REPORTING_MIN_ORDER ; order <MAX_ORDER ; order++) { +for (order = page_reporting_order; order <MAX_ORDER ; order++) {for (mt =0 ; mt <MIGRATE_TYPES ; mt++) {/* We do not pull pages from the isolate free list */ if (is_migrate_isolate (mt))
mm/page_reporting.h renamed from mm/page_reporting.h~mm-page_reporting-export-reporting-order-as-module-parameter +2 -3
@@ -10,10 +10,9 @@#include <linux/pgtable.h> #include <linux/scatterlist.h> -#define PAGE_REPORTING_MIN_ORDER pageblock_order -#ifdef CONFIG_PAGE_REPORTING DECLARE_STATIC_KEY_FALSE (page_reporting_enabled); +extern unsigned int page_reporting_order;void __page_reporting_notify (void );static inline bool page_reported (struct page *page)
@@ -38,7 +37,7 @@ static inline void page_reporting_notifyreturn ;/* Determine if we have crossed reporting threshold */ -if (order <PAGE_REPORTING_MIN_ORDER ) +if (order < page_reporting_order)return ;/* This will add a few cycles, but should be called infrequently */
_
Andrew Morton
maintainerPATCH 40/192 mm/page_reporting: allow driver to specify reporting order
In reply to Andrew MortonFrom: Gavin Shan <[email protected]>
Subject: mm/page_reporting: allow driver to specify reporting order
The page reporting order (threshold) is sticky to @pageblock_order by default. The page reporting can never be triggered because the freeing page can't come up with a free area like that huge. The situation becomes worse when the system memory becomes heavily fragmented.
For example, the following configurations are used on ARM64 when 64KB base page size is enabled. In this specific case, the page reporting won't be triggered until the freeing page comes up with a 512MB free area. That's hard to be met, especially when the system memory becomes heavily fragmented.
PAGE_SIZE: 64KB HPAGE_SIZE: 512MB pageblock_order: 13 (512MB) MAX_ORDER: 14
This allows the drivers to specify the page reporting order when the page reporting device is registered. It falls back to @pageblock_order if it's not specified by the driver. The existing users (hv_balloon and virtio_balloon) don't specify it and @pageblock_order is still taken as their page reporting order. So this shouldn't introduce any functional changes.
---
include/linux/page_reporting.h | 3 +++ mm/page_reporting.c | 6 ++++++ 2 files changed, 9 insertions(+)
include/linux/page_reporting.h renamed from include/linux/page_reporting.h~mm-page_reporting-allow-driver-to-specify-reporting-order +3 -0
@@ -18,6 +18,9 @@ struct page_reporting_dev_info {/* Current state of page reporting */ atomic_t state; + +/* Minimal order of page reporting */ +unsigned int order; };/* Tear-down and bring-up for page reporting devices */
mm/page_reporting.c renamed from mm/page_reporting.c~mm-page_reporting-allow-driver-to-specify-reporting-order +6 -0
@@ -329,6 +329,12 @@ int page_reporting_register(struct page_ goto err_out; } +/* +* Update the page reporting order if it's specified by driver. +* Otherwise, it falls back to @pageblock_order. +*/ + page_reporting_order = prdev->order ? : pageblock_order; +/* initialize state and work structures */ atomic_set (&prdev->state,PAGE_REPORTING_IDLE );INIT_DELAYED_WORK (&prdev->work, &page_reporting_process);
_
Andrew Morton
maintainerPATCH 41/192 virtio_balloon: specify page reporting order if needed
In reply to Andrew MortonFrom: Gavin Shan <[email protected]>
Subject: virtio_balloon: specify page reporting order if needed
The page reporting won't be triggered if the freeing page can't come up with a free area, whose size is equal or bigger than the threshold (page reporting order). The default page reporting order, equal to @pageblock_order, is too huge on some architectures to trigger page reporting. One example is ARM64 when 64KB base page size is used.
PAGE_SIZE: 64KB
pageblock_order: 13 (512MB)
MAX_ORDER: 14This specifies the page reporting order to 5 (2MB) for this specific case so that page reporting can be triggered.
---
drivers/virtio/virtio_balloon.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
drivers/virtio/virtio_balloon.c renamed from drivers/virtio/virtio_balloon.c~virtio_balloon-specify-page-reporting-order-if-needed +17 -0
@@ -993,6 +993,23 @@ static int virtballoon_probe(struct virt goto out_unregister_oom; } +/* +* The default page reporting order is @pageblock_order, which +* corresponds to 512MB in size on ARM64 when 64KB base page +* size is used. The page reporting won't be triggered if the +* freeing page can't come up with a free area like that huge. +* So we specify the page reporting order to 5, corresponding +* to 2MB. It helps to avoid THP splitting if 4KB base page +* size is used by host. +* +* Ideally, the page reporting order is selected based on the +* host's base page size. However, it needs more work to report +* that value. The hard-coded order would be fine currently. +*/ +#if defined(CONFIG_ARM64 ) && defined(CONFIG_ARM64_64K_PAGES ) + vb->pr_dev_info.order =5 ; +#endif + err =page_reporting_register (&vb->pr_dev_info);if (err) goto out_unregister_oom;
_
Andrew Morton
maintainerPATCH 42/192 mm: page-writeback: kill get_writeback_state() comments
In reply to Andrew MortonFrom: Kefeng Wang <[email protected]>
Subject: mm: page-writeback: kill get_writeback_state() comments
The get_writeback_state() has gone since 2006, kill related comments.
---
mm/page-writeback.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
mm/page-writeback.c renamed from mm/page-writeback.c~mm-page-writeback-kill-get_writeback_state-comments +3 -6
@@ -1869,10 +1869,9 @@ DEFINE_PER_CPU(int, dirty_throttle_leaks* which was newly dirtied. The function will periodically check the system's * dirty state and will initiate writeback if needed. * -* On really big machines, get_writeback_state is expensive, so try to avoid -* calling it too often (ratelimiting). But once we're over the dirty memory -* limit we decrease the ratelimiting by a lot, to prevent individual processes -* from overshooting the limit by (ratelimit_pages) each. +* Once we're over the dirty memory limit we decrease the ratelimiting +* by a lot, to prevent individual processes from overshooting the limit +* by (ratelimit_pages) each. */ void balance_dirty_pages_ratelimited (struct address_space *mapping) {
@@ -2045,8 +2044,6 @@ void laptop_sync_completion(void)/* * If ratelimit_pages is too high then we can get into dirty-data overload * if a large number of processes all perform writes at the same time. -* If it is too low then SMP machines will call the (expensive) -* get_writeback_state too often. * * Here we set ratelimit_pages to a level which ensures that when all CPUs are * dirtying in parallel, we cannot go more than 3% (1/32) over the dirty memory
_
Andrew Morton
maintainerPATCH 43/192 mm/page-writeback: Fix performance when BDI's share of ratio is 0.
In reply to Andrew MortonFrom: Chi Wu <[email protected]>
Subject: mm/page-writeback: Fix performance when BDI's share of ratio is 0.
Fix performance when BDI's share of ratio is 0.
The issue is similar to commit 74d369443325 ("writeback: Fix performance regression in wb_over_bg_thresh()").
Balance_dirty_pages and the writeback worker will also disagree on whether writeback when a BDI uses BDI_CAP_STRICTLIMIT and BDI's share of the thresh ratio is zero.
For example, A thread on cpu0 writes 32 pages and then balance_dirty_pages, it will wake up background writeback and pauses because wb_dirty > wb->wb_thresh = 0 (share of thresh ratio is zero). A thread may runs on cpu0 again because scheduler prefers pre_cpu. Then writeback worker may runs on other cpus(1,2..) which causes the value of wb_stat(wb, WB_RECLAIMABLE) in wb_over_bg_thresh is 0 and does not writeback and returns.
Thus, balance_dirty_pages keeps looping, sleeping and then waking up the worker who will do nothing. It remains stuck in this state until the writeback worker hit the right dirty cpu or the dirty pages expire.
The fix that we should get the wb_stat_sum radically when thresh is low.
---
mm/page-writeback.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
mm/page-writeback.c renamed from mm/page-writeback.c~mm-page-writeback-fix-performance-when-bdis-share-of-ratio-is-0 +16 -4
@@ -1944,6 +1944,8 @@ bool wb_over_bg_thresh(struct bdi_writebstruct dirty_throttle_control *const gdtc = &gdtc_stor;struct dirty_throttle_control *const mdtc =mdtc_valid (&mdtc_stor) ? &mdtc_stor :NULL ; +unsigned long reclaimable; +unsigned long thresh;/* * Similar to balance_dirty_pages() but ignores pages being written
@@ -1956,8 +1958,13 @@ bool wb_over_bg_thresh(struct bdi_writebif (gdtc->dirty > gdtc->bg_thresh)return true; -if (wb_stat (wb,WB_RECLAIMABLE ) > -wb_calc_thresh (gdtc->wb, gdtc->bg_thresh)) + thresh =wb_calc_thresh (gdtc->wb, gdtc->bg_thresh); +if (thresh <2 *wb_stat_error ()) + reclaimable =wb_stat_sum (wb,WB_RECLAIMABLE ); +else + reclaimable =wb_stat (wb,WB_RECLAIMABLE ); + +if (reclaimable > thresh)return true;if (mdtc) {
@@ -1971,8 +1978,13 @@ bool wb_over_bg_thresh(struct bdi_writebif (mdtc->dirty > mdtc->bg_thresh)return true; -if (wb_stat (wb,WB_RECLAIMABLE ) > -wb_calc_thresh (mdtc->wb, mdtc->bg_thresh)) + thresh =wb_calc_thresh (mdtc->wb, mdtc->bg_thresh); +if (thresh <2 *wb_stat_error ()) + reclaimable =wb_stat_sum (wb,WB_RECLAIMABLE ); +else + reclaimable =wb_stat (wb,WB_RECLAIMABLE ); + +if (reclaimable > thresh)return true; }
_
Andrew Morton
maintainerPATCH 44/192 mm/page-writeback: update the comment of Dirty position control
In reply to Andrew MortonFrom: Chi Wu <[email protected]>
Subject: mm/page-writeback: update the comment of Dirty position control
As the value of pos_ratio_polynom() clamp between 0 and 2LL << RATELIMIT_CALC_SHIFT, the global control line should be consistent with it.
---
mm/page-writeback.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page-writeback.c renamed from mm/page-writeback.c~mm-page-writeback-update-the-comment-of-dirty-position-control +1 -1
@@ -845,7 +845,7 @@ static long long pos_ratio_polynom(unsig* ^ pos_ratio * | * | |<===== global dirty control scope ======>| -* 2.0 .............* +* 2.0 * * * * * * * * | .* * | . * * | . *
_
Andrew Morton
maintainerPATCH 45/192 mm/page-writeback: use __this_cpu_inc() in account_page_dirtied()
In reply to Andrew MortonFrom: Chi Wu <[email protected]>
Subject: mm/page-writeback: use __this_cpu_inc() in account_page_dirtied()
As account_page_dirtied() was always protected by xa_lock_irqsave(), so using __this_cpu_inc() is better.
---
mm/page-writeback.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page-writeback.c renamed from mm/page-writeback.c~mm-page-writeback-use-__this_cpu_inc-in-account_page_dirtied +1 -1
@@ -2445,7 +2445,7 @@ void account_page_dirtied(struct page *pinc_wb_stat (wb,WB_DIRTIED );task_io_account_write (PAGE_SIZE ); current->nr_dirtied++; -this_cpu_inc (bdp_ratelimits); +__this_cpu_inc (bdp_ratelimits);mem_cgroup_track_foreign_dirty (page, wb); }
_
Andrew Morton
maintainerPATCH 46/192 writeback, cgroup: do not switch inodes with I_WILL_FREE flag
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: do not switch inodes with I_WILL_FREE flag
Patch series "cgroup, blkcg: prevent dirty inodes to pin dying memory cgroups", v9.
When an inode is getting dirty for the first time it's associated with a wb structure (see __inode_attach_wb()). It can later be switched to another wb (if e.g. some other cgroup is writing a lot of data to the same inode), but otherwise stays attached to the original wb until being reclaimed.
The problem is that the wb structure holds a reference to the original memory and blkcg cgroups. So if an inode has been dirty once and later is actively used in read-only mode, it has a good chance to pin down the original memory and blkcg cgroups forever. This is often the case with services bringing data for other services, e.g. updating some rpm packages.
In the real life it becomes a problem due to a large size of the memcg structure, which can easily be 1000x larger than an inode. Also a really large number of dying cgroups can raise different scalability issues, e.g. making the memory reclaim costly and less effective.
To solve the problem inodes should be eventually detached from the corresponding writeback structure. It's inefficient to do it after every writeback completion. Instead it can be done whenever the original memory cgroup is offlined and writeback structure is getting killed. Scanning over a (potentially long) list of inodes and detach them from the writeback structure can take quite some time. To avoid scanning all inodes, attached inodes are kept on a new list (b_attached). To make it less noticeable to a user, the scanning and switching is performed from a work context.
Big thanks to Jan Kara, Dennis Zhou, Hillf Danton and Tejun Heo for their ideas and contribution to this patchset.
This patch (of 8):
If an inode's state has I_WILL_FREE flag set, the inode will be freed soon, so there is no point in trying to switch the inode to a different cgwb.
I_WILL_FREE was ignored since the introduction of the inode switching, so it looks like it doesn't lead to any noticeable issues for a user. This is why the patch is not intended for a stable backport.
---
fs/fs-writeback.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-do-not-switch-inodes-with-i_will_free-flag +4 -4
@@ -389,10 +389,10 @@ static void inode_switch_wbs_work_fn(strxa_lock_irq (&mapping->i_pages);/* -* Once I_FREEING is visible under i_lock, the eviction path owns -* the inode and we shouldn't modify ->i_io_list. +* Once I_FREEING or I_WILL_FREE are visible under i_lock, the eviction +* path owns the inode and we shouldn't modify ->i_io_list. */ -if (unlikely (inode->i_state &I_FREEING )) +if (unlikely (inode->i_state & (I_FREEING |I_WILL_FREE ))) goto skip_switch;trace_inode_switch_wbs (inode, old_wb, new_wb);
@@ -517,7 +517,7 @@ static void inode_switch_wbs(struct inod/* while holding I_WB_SWITCH, no one else can update the association */ spin_lock (&inode->i_lock);if (!(inode->i_sb->s_flags &SB_ACTIVE ) || - inode->i_state & (I_WB_SWITCH |I_FREEING ) || + inode->i_state & (I_WB_SWITCH |I_FREEING |I_WILL_FREE ) ||inode_to_wb (inode) == isw->new_wb) {spin_unlock (&inode->i_lock); goto out_free;
_
Andrew Morton
maintainerPATCH 47/192 writeback, cgroup: add smp_mb() to cgroup_writeback_umount()
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: add smp_mb() to cgroup_writeback_umount()
A full memory barrier is required between clearing SB_ACTIVE flag in generic_shutdown_super() and checking isw_nr_in_flight in cgroup_writeback_umount(), otherwise a new switch operation might be scheduled after atomic_read(&isw_nr_in_flight) returned 0. This would result in a non-flushed isw_wq, and a potential crash.
The problem hasn't yet been seen in the real life and was discovered by Jan Kara by looking into the code.
---
fs/fs-writeback.c | 6 ++++++ 1 file changed, 6 insertions(+)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-add-smp_mb-to-cgroup_writeback_umount +6 -0
@@ -1000,6 +1000,12 @@ out_bdi_put:*/ void cgroup_writeback_umount (void ) { +/* +* SB_ACTIVE should be reliably cleared before checking +* isw_nr_in_flight, see generic_shutdown_super(). +*/ +smp_mb (); +if (atomic_read (&isw_nr_in_flight)) {/* * Use rcu_barrier() to wait for all pending callbacks to
_
Andrew Morton
maintainerPATCH 48/192 writeback, cgroup: increment isw_nr_in_flight before grabbing an inode
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: increment isw_nr_in_flight before grabbing an inode
isw_nr_in_flight is used to determine whether the inode switch queue should be flushed from the umount path. Currently it's increased after grabbing an inode and even scheduling the switch work. It means the umount path can walk past cleanup_offline_cgwb() with active inode references, which can result in a "Busy inodes after unmount." message and use-after-free issues (with inode->i_sb which gets freed).
Fix it by incrementing isw_nr_in_flight before doing anything with the inode and decrementing in the case when switching wasn't scheduled.
The problem hasn't yet been seen in the real life and was discovered by Jan Kara by looking into the code.
---
fs/fs-writeback.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-increment-isw_nr_in_flight-before-grabbing-an-inode +3 -2
@@ -505,6 +505,8 @@ static void inode_switch_wbs(struct inodif (!isw)return ; +atomic_inc (&isw_nr_in_flight); +/* find and pin the new wb */ rcu_read_lock (); memcg_css =css_from_id (new_wb_id, &memory_cgrp_subsys);
@@ -535,11 +537,10 @@ static void inode_switch_wbs(struct inod* Let's continue after I_WB_SWITCH is guaranteed to be visible. */ call_rcu (&isw->rcu_head, inode_switch_wbs_rcu_fn); - -atomic_inc (&isw_nr_in_flight);return ; out_free: +atomic_dec (&isw_nr_in_flight);if (isw->new_wb)wb_put (isw->new_wb);kfree (isw);
_
Andrew Morton
maintainerPATCH 49/192 writeback, cgroup: switch to rcu_work API in inode_switch_wbs()
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: switch to rcu_work API in inode_switch_wbs()
Inode's wb switching requires two steps divided by an RCU grace period. It's currently implemented as an RCU callback inode_switch_wbs_rcu_fn(), which schedules inode_switch_wbs_work_fn() as a work.
Switching to the rcu_work API allows to do the same in a cleaner and slightly shorter form.
---
fs/fs-writeback.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-switch-to-rcu_work-api-in-inode_switch_wbs +4 -14
@@ -335,8 +335,7 @@ struct inode_switch_wbs_context {struct inode *inode;struct bdi_writeback *new_wb; -struct rcu_head rcu_head; -struct work_struct work; +struct rcu_work work; };static void bdi_down_write_wb_switch_rwsem (struct backing_dev_info *bdi)
@@ -352,7 +351,7 @@ static void bdi_up_write_wb_switch_rwsemstatic void inode_switch_wbs_work_fn (struct work_struct *work) {struct inode_switch_wbs_context *isw = - container_of(work, struct inode_switch_wbs_context, work); + container_of(to_rcu_work(work), struct inode_switch_wbs_context, work);struct inode *inode = isw->inode;struct backing_dev_info *bdi =inode_to_bdi (inode);struct address_space *mapping = inode->i_mapping;
@@ -469,16 +468,6 @@ skip_switch:atomic_dec (&isw_nr_in_flight); } -static void inode_switch_wbs_rcu_fn (struct rcu_head *rcu_head) -{ -struct inode_switch_wbs_context *isw =container_of (rcu_head, - struct inode_switch_wbs_context, rcu_head); - -/* needs to grab bh-unsafe locks, bounce to work item */ -INIT_WORK (&isw->work, inode_switch_wbs_work_fn); -queue_work (isw_wq, &isw->work); -} -/** * inode_switch_wbs - change the wb association of an inode * @inode: target inode
@@ -536,7 +525,8 @@ static void inode_switch_wbs(struct inod* lock so that stat transfer can synchronize against them. * Let's continue after I_WB_SWITCH is guaranteed to be visible. */ -call_rcu (&isw->rcu_head, inode_switch_wbs_rcu_fn); +INIT_RCU_WORK (&isw->work, inode_switch_wbs_work_fn); +queue_rcu_work (isw_wq, &isw->work);return ; out_free:
_
Andrew Morton
maintainerPATCH 50/192 writeback, cgroup: keep list of inodes attached to bdi_writeback
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: keep list of inodes attached to bdi_writeback
Currently there is no way to iterate over inodes attached to a specific cgwb structure. It limits the ability to efficiently reclaim the writeback structure itself and associated memory and block cgroup structures without scanning all inodes belonging to a sb, which can be prohibitively expensive.
While dirty/in-active-writeback an inode belongs to one of the bdi_writeback's io lists: b_dirty, b_io, b_more_io and b_dirty_time. Once cleaned up, it's removed from all io lists. So the inode->i_io_list can be reused to maintain the list of inodes, attached to a bdi_writeback structure.
This patch introduces a new wb->b_attached list, which contains all inodes which were dirty at least once and are attached to the given cgwb. Inodes attached to the root bdi_writeback structures are never placed on such list. The following patch will use this list to try to release cgwbs structures more efficiently.
---
fs/fs-writeback.c | 93 ++++++++++++++++++----------- include/linux/backing-dev-defs.h | 1 mm/backing-dev.c | 2 3 files changed, 62 insertions(+), 34 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-keep-list-of-inodes-attached-to-bdi_writeback +59 -34
@@ -131,25 +131,6 @@ static bool inode_io_list_move_locked(streturn false; } -/** -* inode_io_list_del_locked - remove an inode from its bdi_writeback IO list -* @inode: inode to be removed -* @wb: bdi_writeback @inode is being removed from -* -* Remove @inode which may be on one of @wb->b_{dirty|io|more_io} lists and -* clear %WB_has_dirty_io if all are empty afterwards. -*/ -static void inode_io_list_del_locked (struct inode *inode, -struct bdi_writeback *wb) -{ -assert_spin_locked (&wb->list_lock); -assert_spin_locked (&inode->i_lock); - - inode->i_state &= ~I_SYNC_QUEUED ; -list_del_init (&inode->i_io_list); -wb_io_lists_depopulated (wb); -} -static void wb_wakeup (struct bdi_writeback *wb) {spin_lock_bh (&wb->work_lock);
@@ -279,6 +260,28 @@ void __inode_attach_wb(struct inode *inoEXPORT_SYMBOL_GPL (__inode_attach_wb);/** +* inode_cgwb_move_to_attached - put the inode onto wb->b_attached list +* @inode: inode of interest with i_lock held +* @wb: target bdi_writeback +* +* Remove the inode from wb's io lists and if necessarily put onto b_attached +* list. Only inodes attached to cgwb's are kept on this list. +*/ +static void inode_cgwb_move_to_attached (struct inode *inode, +struct bdi_writeback *wb) +{ +assert_spin_locked (&wb->list_lock); +assert_spin_locked (&inode->i_lock); + + inode->i_state &= ~I_SYNC_QUEUED ; +if (wb != &wb->bdi->wb) +list_move (&inode->i_io_list, &wb->b_attached); +else +list_del_init (&inode->i_io_list); +wb_io_lists_depopulated (wb); +} + +/** * locked_inode_to_wb_and_lock_list - determine a locked inode's wb and lock it * @inode: inode of interest with i_lock held *
@@ -418,21 +421,28 @@ static void inode_switch_wbs_work_fn(strwb_get (new_wb);/* -* Transfer to @new_wb's IO list if necessary. The specific list -* @inode was on is ignored and the inode is put on ->b_dirty which -* is always correct including from ->b_dirty_time. The transfer -* preserves @inode->dirtied_when ordering. +* Transfer to @new_wb's IO list if necessary. If the @inode is dirty, +* the specific list @inode was on is ignored and the @inode is put on +* ->b_dirty which is always correct including from ->b_dirty_time. +* The transfer preserves @inode->dirtied_when ordering. If the @inode +* was clean, it means it was on the b_attached list, so move it onto +* the b_attached list of @new_wb. */ if (!list_empty (&inode->i_io_list)) { -struct inode *pos; - -inode_io_list_del_locked (inode, old_wb); inode->i_wb = new_wb; -list_for_each_entry (pos, &new_wb->b_dirty, i_io_list) -if (time_after_eq (inode->dirtied_when, - pos->dirtied_when)) -break ; -inode_io_list_move_locked (inode, new_wb, pos->i_io_list.prev); + +if (inode->i_state &I_DIRTY_ALL ) { +struct inode *pos; + +list_for_each_entry (pos, &new_wb->b_dirty, i_io_list) +if (time_after_eq (inode->dirtied_when, + pos->dirtied_when)) +break ; +inode_io_list_move_locked (inode, new_wb, + pos->i_io_list.prev); + }else { +inode_cgwb_move_to_attached (inode, new_wb); + } }else { inode->i_wb = new_wb; }
@@ -1021,6 +1031,17 @@ fs_initcall(cgroup_writeback_init);static void bdi_down_write_wb_switch_rwsem (struct backing_dev_info *bdi) { }static void bdi_up_write_wb_switch_rwsem (struct backing_dev_info *bdi) { } +static void inode_cgwb_move_to_attached (struct inode *inode, +struct bdi_writeback *wb) +{ +assert_spin_locked (&wb->list_lock); +assert_spin_locked (&inode->i_lock); + + inode->i_state &= ~I_SYNC_QUEUED ; +list_del_init (&inode->i_io_list); +wb_io_lists_depopulated (wb); +} +static struct bdi_writeback *locked_inode_to_wb_and_lock_list (struct inode *inode) __releases(&inode->i_lock)
@@ -1121,7 +1142,11 @@ void inode_io_list_del(struct inode *ino wb =inode_to_wb_and_lock_list (inode);spin_lock (&inode->i_lock); -inode_io_list_del_locked (inode, wb); + + inode->i_state &= ~I_SYNC_QUEUED ; +list_del_init (&inode->i_io_list); +wb_io_lists_depopulated (wb); +spin_unlock (&inode->i_lock);spin_unlock (&wb->list_lock); }
@@ -1434,7 +1459,7 @@ static void requeue_inode(struct inode * inode->i_state &= ~I_SYNC_QUEUED ; }else {/* The inode is clean. Remove from writeback lists. */ -inode_io_list_del_locked (inode, wb); +inode_cgwb_move_to_attached (inode, wb); } }
@@ -1586,7 +1611,7 @@ static int writeback_single_inode(struct* responsible for the writeback lists. */ if (!(inode->i_state &I_DIRTY_ALL )) -inode_io_list_del_locked (inode, wb); +inode_cgwb_move_to_attached (inode, wb);spin_unlock (&wb->list_lock);inode_sync_complete (inode); out:
include/linux/backing-dev-defs.h renamed from include/linux/backing-dev-defs.h~writeback-cgroup-keep-list-of-inodes-attached-to-bdi_writeback +1 -0
@@ -154,6 +154,7 @@ struct bdi_writeback {struct cgroup_subsys_state *blkcg_css;/* and blkcg */ struct list_head memcg_node;/* anchored at memcg->cgwb_list */ struct list_head blkcg_node;/* anchored at blkcg->cgwb_list */ +struct list_head b_attached;/* attached inodes, protected by list_lock */ union {struct work_struct release_work;
mm/backing-dev.c renamed from mm/backing-dev.c~writeback-cgroup-keep-list-of-inodes-attached-to-bdi_writeback +2 -0
@@ -396,6 +396,7 @@ static void cgwb_release_workfn(struct wfprop_local_destroy_percpu (&wb->memcg_completions);percpu_ref_exit (&wb->refcnt);wb_exit (wb); +WARN_ON_ONCE (!list_empty (&wb->b_attached));kfree_rcu (wb, rcu); }
@@ -472,6 +473,7 @@ static int cgwb_create(struct backing_de wb->memcg_css = memcg_css; wb->blkcg_css = blkcg_css; +INIT_LIST_HEAD (&wb->b_attached);INIT_WORK (&wb->release_work, cgwb_release_workfn);set_bit (WB_registered, &wb->state);
_
Andrew Morton
maintainerPATCH 51/192 writeback, cgroup: split out the functional part of inode_switch_wbs_work_fn()
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: split out the functional part of inode_switch_wbs_work_fn()
Split out the functional part of the inode_switch_wbs_work_fn() function as inode_do switch_wbs() to reuse it later for switching inodes attached to dying cgwbs.
This commit doesn't bring any functional changes.
---
fs/fs-writeback.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-split-out-the-functional-part-of-inode_switch_wbs_work_fn +11 -8
@@ -351,15 +351,12 @@ static void bdi_up_write_wb_switch_rwsemup_write (&bdi->wb_switch_rwsem); } -static void inode_switch_wbs_work_fn (struct work_struct *work) +static void inode_do_switch_wbs (struct inode *inode, +struct bdi_writeback *new_wb) { -struct inode_switch_wbs_context *isw = - container_of(to_rcu_work(work), struct inode_switch_wbs_context, work); -struct inode *inode = isw->inode;struct backing_dev_info *bdi =inode_to_bdi (inode);struct address_space *mapping = inode->i_mapping;struct bdi_writeback *old_wb = inode->i_wb; -struct bdi_writeback *new_wb = isw->new_wb;XA_STATE (xas, &mapping->i_pages,0 );struct page *page;bool switched = false;
@@ -470,11 +467,17 @@ skip_switch:wb_wakeup (new_wb);wb_put (old_wb); } -wb_put (new_wb); +} -iput (inode); -kfree (isw); +static void inode_switch_wbs_work_fn (struct work_struct *work) +{ +struct inode_switch_wbs_context *isw = + container_of(to_rcu_work(work), struct inode_switch_wbs_context, work); +inode_do_switch_wbs (isw->inode, isw->new_wb); +wb_put (isw->new_wb); +iput (isw->inode); +kfree (isw);atomic_dec (&isw_nr_in_flight); }
_
Andrew Morton
maintainerPATCH 52/192 writeback, cgroup: support switching multiple inodes at once
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: support switching multiple inodes at once
Currently only a single inode can be switched to another writeback structure at once. That means to switch an inode a separate inode_switch_wbs_context structure must be allocated, and a separate rcu callback and work must be scheduled.
It's fine for the existing ad-hoc switching, which is not happening that often, but sub-optimal for massive switching required in order to release a writeback structure. To prepare for it, let's add a support for switching multiple inodes at once.
Instead of containing a single inode pointer, inode_switch_wbs_context will contain a NULL-terminated array of inode pointers. inode_do_switch_wbs() will be called for each inode.
To optimize the locking bdi->wb_switch_rwsem, old_wb's and new_wb's list_locks will be acquired and released only once altogether for all inodes. wb_wakeup() will be also be called only once. Instead of calling wb_put(old_wb) after each successful switch, wb_put_many() is introduced and used.
---
fs/fs-writeback.c | 106 +++++++++++++++++------------ include/linux/backing-dev-defs.h | 18 ++++ 2 files changed, 80 insertions(+), 44 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-support-switching-multiple-inodes-at-once +64 -42
@@ -335,10 +335,18 @@ static struct bdi_writeback *inode_to_wb }struct inode_switch_wbs_context { -struct inode *inode; -struct bdi_writeback *new_wb; -struct rcu_work work; + +/* +* Multiple inodes can be switched at once. The switching procedure +* consists of two parts, separated by a RCU grace period. To make +* sure that the second part is executed for each inode gone through +* the first part, all inode pointers are placed into a NULL-terminated +* array embedded into struct inode_switch_wbs_context. Otherwise +* an inode could be left in a non-consistent state. +*/ +struct bdi_writeback *new_wb; +struct inode *inodes[]; };static void bdi_down_write_wb_switch_rwsem (struct backing_dev_info *bdi)
@@ -351,39 +359,15 @@ static void bdi_up_write_wb_switch_rwsemup_write (&bdi->wb_switch_rwsem); } -static void inode_do_switch_wbs (struct inode *inode, +static bool inode_do_switch_wbs (struct inode *inode, +struct bdi_writeback *old_wb,struct bdi_writeback *new_wb) { -struct backing_dev_info *bdi =inode_to_bdi (inode);struct address_space *mapping = inode->i_mapping; -struct bdi_writeback *old_wb = inode->i_wb;XA_STATE (xas, &mapping->i_pages,0 );struct page *page;bool switched = false; -/* -* If @inode switches cgwb membership while sync_inodes_sb() is -* being issued, sync_inodes_sb() might miss it. Synchronize. -*/ -down_read (&bdi->wb_switch_rwsem); - -/* -* By the time control reaches here, RCU grace period has passed -* since I_WB_SWITCH assertion and all wb stat update transactions -* between unlocked_inode_to_wb_begin/end() are guaranteed to be -* synchronizing against the i_pages lock. -* -* Grabbing old_wb->list_lock, inode->i_lock and the i_pages lock -* gives us exclusion against all wb related operations on @inode -* including IO list manipulations and stat updates. -*/ -if (old_wb < new_wb) { -spin_lock (&old_wb->list_lock); -spin_lock_nested (&new_wb->list_lock,SINGLE_DEPTH_NESTING ); - }else { -spin_lock (&new_wb->list_lock); -spin_lock_nested (&old_wb->list_lock,SINGLE_DEPTH_NESTING ); - }spin_lock (&inode->i_lock);xa_lock_irq (&mapping->i_pages);
@@ -458,25 +442,63 @@ skip_switch:xa_unlock_irq (&mapping->i_pages);spin_unlock (&inode->i_lock); -spin_unlock (&new_wb->list_lock); -spin_unlock (&old_wb->list_lock); - -up_read (&bdi->wb_switch_rwsem); -if (switched) { -wb_wakeup (new_wb); -wb_put (old_wb); - } +return switched; }static void inode_switch_wbs_work_fn (struct work_struct *work) {struct inode_switch_wbs_context *isw = container_of(to_rcu_work(work), struct inode_switch_wbs_context, work); +struct backing_dev_info *bdi =inode_to_bdi (isw->inodes[0 ]); +struct bdi_writeback *old_wb = isw->inodes[0 ]->i_wb; +struct bdi_writeback *new_wb = isw->new_wb; +unsigned long nr_switched =0 ; +struct inode **inodep; + +/* +* If @inode switches cgwb membership while sync_inodes_sb() is +* being issued, sync_inodes_sb() might miss it. Synchronize. +*/ +down_read (&bdi->wb_switch_rwsem); + +/* +* By the time control reaches here, RCU grace period has passed +* since I_WB_SWITCH assertion and all wb stat update transactions +* between unlocked_inode_to_wb_begin/end() are guaranteed to be +* synchronizing against the i_pages lock. +* +* Grabbing old_wb->list_lock, inode->i_lock and the i_pages lock +* gives us exclusion against all wb related operations on @inode +* including IO list manipulations and stat updates. +*/ +if (old_wb < new_wb) { +spin_lock (&old_wb->list_lock); +spin_lock_nested (&new_wb->list_lock,SINGLE_DEPTH_NESTING ); + }else { +spin_lock (&new_wb->list_lock); +spin_lock_nested (&old_wb->list_lock,SINGLE_DEPTH_NESTING ); + } + +for (inodep = isw->inodes; *inodep; inodep++) { +WARN_ON_ONCE ((*inodep)->i_wb != old_wb); +if (inode_do_switch_wbs (*inodep, old_wb, new_wb)) + nr_switched++; + } + +spin_unlock (&new_wb->list_lock); +spin_unlock (&old_wb->list_lock); + +up_read (&bdi->wb_switch_rwsem); + +if (nr_switched) { +wb_wakeup (new_wb); +wb_put_many (old_wb, nr_switched); + } -inode_do_switch_wbs (isw->inode, isw->new_wb); -wb_put (isw->new_wb); -iput (isw->inode); +for (inodep = isw->inodes; *inodep; inodep++) +iput (*inodep); +wb_put (new_wb);kfree (isw);atomic_dec (&isw_nr_in_flight); }
@@ -503,7 +525,7 @@ static void inode_switch_wbs(struct inodif (atomic_read (&isw_nr_in_flight) >WB_FRN_MAX_IN_FLIGHT )return ; - isw =kzalloc (sizeof (*isw),GFP_ATOMIC ); + isw =kzalloc (sizeof (*isw) +2 *sizeof (struct inode *),GFP_ATOMIC );if (!isw)return ;
@@ -530,7 +552,7 @@ static void inode_switch_wbs(struct inod__iget (inode);spin_unlock (&inode->i_lock); - isw->inode = inode; + isw->inodes[0 ] = inode;/* * In addition to synchronizing among switchers, I_WB_SWITCH tells
include/linux/backing-dev-defs.h renamed from include/linux/backing-dev-defs.h~writeback-cgroup-support-switching-multiple-inodes-at-once +16 -2
@@ -240,8 +240,9 @@ static inline void wb_get(struct bdi_wri/** * wb_put - decrement a wb's refcount * @wb: bdi_writeback to put +* @nr: number of references to put */ -static inline void wb_put (struct bdi_writeback *wb) +static inline void wb_put_many (struct bdi_writeback *wb,unsigned long nr) {if (WARN_ON_ONCE (!wb->bdi)) {/*
@@ -252,7 +253,16 @@ static inline void wb_put(struct bdi_wri }if (wb != &wb->bdi->wb) -percpu_ref_put (&wb->refcnt); +percpu_ref_put_many (&wb->refcnt, nr); +} + +/** +* wb_put - decrement a wb's refcount +* @wb: bdi_writeback to put +*/ +static inline void wb_put (struct bdi_writeback *wb) +{ +wb_put_many (wb,1 ); }/**
@@ -281,6 +291,10 @@ static inline void wb_put(struct bdi_wri { } +static inline void wb_put_many (struct bdi_writeback *wb,unsigned long nr) +{ +} +static inline bool wb_dying (struct bdi_writeback *wb) {return false;
_
Andrew Morton
maintainerPATCH 53/192 writeback, cgroup: release dying cgwbs by switching attached inodes
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: writeback, cgroup: release dying cgwbs by switching attached inodes
Asynchronously try to release dying cgwbs by switching attached inodes to the nearest living ancestor wb. It helps to get rid of per-cgroup writeback structures themselves and of pinned memory and block cgroups, which are significantly larger structures (mostly due to large per-cpu statistics data). This prevents memory waste and helps to avoid different scalability problems caused by large piles of dying cgroups.
Reuse the existing mechanism of inode switching used for foreign inode detection. To speed things up batch up to 115 inode switching in a single operation (the maximum number is selected so that the resulting struct inode_switch_wbs_context can fit into 1024 bytes). Because every switching consists of two steps divided by an RCU grace period, it would be too slow without batching. Please note that the whole batch counts as a single operation (when increasing/decreasing isw_nr_in_flight). This allows to keep umounting working (flush the switching queue), however prevents cleanups from consuming the whole switching quota and effectively blocking the frn switching.
A cgwb cleanup operation can fail due to different reasons (e.g. not enough memory, the cgwb has an in-flight/pending io, an attached inode in a wrong state, etc). In this case the next scheduled cleanup will make a new attempt. An attempt is made each time a new cgwb is offlined (in other words a memcg and/or a blkcg is deleted by a user). In the future an additional attempt scheduled by a timer can be implemented.
---
fs/fs-writeback.c | 111 ++++++++++++++++++++++++++--- include/linux/backing-dev-defs.h | 1 include/linux/writeback.h | 1 mm/backing-dev.c | 64 ++++++++++++++++ 4 files changed, 165 insertions(+), 12 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~writeback-cgroup-release-dying-cgwbs-by-switching-attached-inodes +101 -10
@@ -225,6 +225,13 @@ void wb_wait_for_completion(struct wb_co/* one round can affect upto 5 slots */ #define WB_FRN_MAX_IN_FLIGHT 1024/* don't queue too many concurrently */ +/* +* Maximum inodes per isw. A specific value has been chosen to make +* struct inode_switch_wbs_context fit into 1024 bytes kmalloc. +*/ +#define WB_MAX_INODES_PER_ISW ((1024UL - sizeof(struct inode_switch_wbs_context)) \ + / sizeof(struct inode *)) +static atomic_t isw_nr_in_flight =ATOMIC_INIT (0 );static struct workqueue_struct *isw_wq;
@@ -503,6 +510,32 @@ static void inode_switch_wbs_work_fn(stratomic_dec (&isw_nr_in_flight); } +static bool inode_prepare_wbs_switch (struct inode *inode, +struct bdi_writeback *new_wb) +{ +/* +* Paired with smp_mb() in cgroup_writeback_umount(). +* isw_nr_in_flight must be increased before checking SB_ACTIVE and +* grabbing an inode, otherwise isw_nr_in_flight can be observed as 0 +* in cgroup_writeback_umount() and the isw_wq will be not flushed. +*/ +smp_mb (); + +/* while holding I_WB_SWITCH, no one else can update the association */ +spin_lock (&inode->i_lock); +if (!(inode->i_sb->s_flags &SB_ACTIVE ) || + inode->i_state & (I_WB_SWITCH |I_FREEING |I_WILL_FREE ) || +inode_to_wb (inode) == new_wb) { +spin_unlock (&inode->i_lock); +return false; + } + inode->i_state |=I_WB_SWITCH ; +__iget (inode); +spin_unlock (&inode->i_lock); + +return true; +} +/** * inode_switch_wbs - change the wb association of an inode * @inode: target inode
@@ -540,17 +573,8 @@ static void inode_switch_wbs(struct inodif (!isw->new_wb) goto out_free; -/* while holding I_WB_SWITCH, no one else can update the association */ -spin_lock (&inode->i_lock); -if (!(inode->i_sb->s_flags &SB_ACTIVE ) || - inode->i_state & (I_WB_SWITCH |I_FREEING |I_WILL_FREE ) || -inode_to_wb (inode) == isw->new_wb) { -spin_unlock (&inode->i_lock); +if (!inode_prepare_wbs_switch (inode, isw->new_wb)) goto out_free; - } - inode->i_state |=I_WB_SWITCH ; -__iget (inode); -spin_unlock (&inode->i_lock); isw->inodes[0 ] = inode;
@@ -572,6 +596,73 @@ out_free: }/** +* cleanup_offline_cgwb - detach associated inodes +* @wb: target wb +* +* Switch all inodes attached to @wb to a nearest living ancestor's wb in order +* to eventually release the dying @wb. Returns %true if not all inodes were +* switched and the function has to be restarted. +*/ +bool cleanup_offline_cgwb (struct bdi_writeback *wb) +{ +struct cgroup_subsys_state *memcg_css; +struct inode_switch_wbs_context *isw; +struct inode *inode; +int nr; +bool restart = false; + + isw =kzalloc (sizeof (*isw) +WB_MAX_INODES_PER_ISW * +sizeof (struct inode *),GFP_KERNEL ); +if (!isw) +return restart; + +atomic_inc (&isw_nr_in_flight); + +for (memcg_css = wb->memcg_css->parent; memcg_css; + memcg_css = memcg_css->parent) { + isw->new_wb =wb_get_create (wb->bdi, memcg_css,GFP_KERNEL ); +if (isw->new_wb) +break ; + } +if (unlikely (!isw->new_wb)) + isw->new_wb = &wb->bdi->wb;/* wb_get() is noop for bdi's wb */ + + nr =0 ; +spin_lock (&wb->list_lock); +list_for_each_entry (inode, &wb->b_attached, i_io_list) { +if (!inode_prepare_wbs_switch (inode, isw->new_wb)) +continue ; + + isw->inodes[nr++] = inode; + +if (nr >=WB_MAX_INODES_PER_ISW -1 ) { + restart = true; +break ; + } + } +spin_unlock (&wb->list_lock); + +/* no attached inodes? bail out */ +if (nr ==0 ) { +atomic_dec (&isw_nr_in_flight); +wb_put (isw->new_wb); +kfree (isw); +return restart; + } + +/* +* In addition to synchronizing among switchers, I_WB_SWITCH tells +* the RCU protected stat update paths to grab the i_page +* lock so that stat transfer can synchronize against them. +* Let's continue after I_WB_SWITCH is guaranteed to be visible. +*/ +INIT_RCU_WORK (&isw->work, inode_switch_wbs_work_fn); +queue_rcu_work (isw_wq, &isw->work); + +return restart; +} + +/** * wbc_attach_and_unlock_inode - associate wbc with target inode and unlock it * @wbc: writeback_control of interest * @inode: target inode
include/linux/backing-dev-defs.h renamed from include/linux/backing-dev-defs.h~writeback-cgroup-release-dying-cgwbs-by-switching-attached-inodes +1 -0
@@ -155,6 +155,7 @@ struct bdi_writeback {struct list_head memcg_node;/* anchored at memcg->cgwb_list */ struct list_head blkcg_node;/* anchored at blkcg->cgwb_list */ struct list_head b_attached;/* attached inodes, protected by list_lock */ +struct list_head offline_node;/* anchored at offline_cgwbs */ union {struct work_struct release_work;
include/linux/writeback.h renamed from include/linux/writeback.h~writeback-cgroup-release-dying-cgwbs-by-switching-attached-inodes +1 -0
@@ -221,6 +221,7 @@ void wbc_account_cgroup_owner(struct wriint cgroup_writeback_by_id (u64 bdi_id,int memcg_id,unsigned long nr_pages,enum wb_reason reason,struct wb_completion *done);void cgroup_writeback_umount (void ); +bool cleanup_offline_cgwb (struct bdi_writeback *wb);/** * inode_attach_wb - associate an inode with its wb
mm/backing-dev.c renamed from mm/backing-dev.c~writeback-cgroup-release-dying-cgwbs-by-switching-attached-inodes +62 -2
98 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 54/192 fs: unexport __set_page_dirty
In reply to Andrew MortonFrom: Christoph Hellwig <[email protected]>
Subject: fs: unexport __set_page_dirty
Patch series "remove the implicit .set_page_dirty default".
This series cleans up a few lose ends around ->set_page_dirty, most importantly removes the default to the buffer head based on if no method is wired up.
This patch (of 3):
__set_page_dirty is only used by built-in code.
---
fs/buffer.c | 1 - 1 file changed, 1 deletion(-)
fs/buffer.c renamed from fs/buffer.c~fs-unexport-__set_page_dirty +0 -1
@@ -611,7 +611,6 @@ void __set_page_dirty(struct page *page, }xa_unlock_irqrestore (&mapping->i_pages, flags); } -EXPORT_SYMBOL_GPL (__set_page_dirty);/* * Add a page to the dirty page list.
_
Andrew Morton
maintainerPATCH 55/192 fs: move ramfs_aops to libfs
In reply to Andrew MortonFrom: Christoph Hellwig <[email protected]>
Subject: fs: move ramfs_aops to libfs
Move the ramfs aops to libfs and reuse them for kernfs and configfs. Thosw two did not wire up ->set_page_dirty before and now get __set_page_dirty_no_writeback, which is the right one for no-writeback address_space usage.
Drop the now unused exports of the libfs helpers only used for ramfs-style pagecache usage.
---
fs/configfs/inode.c | 8 +------- fs/kernfs/inode.c | 8 +------- fs/libfs.c | 17 +++++++++++++---- fs/ramfs/inode.c | 9 +-------- include/linux/fs.h | 5 +---- 5 files changed, 17 insertions(+), 30 deletions(-)
fs/configfs/inode.c renamed from fs/configfs/inode.c~fs-move-ramfs_aops-to-libfs +1 -7
@@ -28,12 +28,6 @@static struct lock_class_key default_group_class[MAX_LOCK_DEPTH ];#endif -static const struct address_space_operations configfs_aops = { - .readpage = simple_readpage, - .write_begin = simple_write_begin, - .write_end = simple_write_end, -}; -static const struct inode_operations configfs_inode_operations ={ .setattr = configfs_setattr, };
@@ -114,7 +108,7 @@ struct inode *configfs_new_inode(umode_tstruct inode * inode =new_inode (s);if (inode) { inode->i_ino =get_next_ino (); - inode->i_mapping->a_ops = &configfs_aops; + inode->i_mapping->a_ops = &ram_aops; inode->i_op = &configfs_inode_operations;if (sd->s_iattr) {
fs/kernfs/inode.c renamed from fs/kernfs/inode.c~fs-move-ramfs_aops-to-libfs +1 -7
@@ -17,12 +17,6 @@#include "kernfs-internal.h" -static const struct address_space_operations kernfs_aops = { - .readpage = simple_readpage, - .write_begin = simple_write_begin, - .write_end = simple_write_end, -}; -static const struct inode_operations kernfs_iops = { .permission = kernfs_iop_permission, .setattr = kernfs_iop_setattr,
@@ -203,7 +197,7 @@ static void kernfs_init_inode(struct ker {kernfs_get (kn); inode->i_private = kn; - inode->i_mapping->a_ops = &kernfs_aops; + inode->i_mapping->a_ops = &ram_aops; inode->i_op = &kernfs_iops; inode->i_generation =kernfs_gen (kn);
fs/libfs.c renamed from fs/libfs.c~fs-move-ramfs_aops-to-libfs +13 -4
@@ -512,7 +512,7 @@ int simple_setattr(struct user_namespace }EXPORT_SYMBOL (simple_setattr); -int simple_readpage (struct file *file,struct page *page) +static int simple_readpage (struct file *file,struct page *page) {clear_highpage (page);flush_dcache_page (page);
@@ -520,7 +520,6 @@ int simple_readpage(struct file *file, sunlock_page (page);return 0 ; } -EXPORT_SYMBOL (simple_readpage); int simple_write_begin(struct file *file, struct address_space *mapping, loff_t pos, unsigned len, unsigned flags,
@@ -568,7 +567,7 @@ EXPORT_SYMBOL(simple_write_begin);* * Use *ONLY* with simple_readpage() */ -int simple_write_end (struct file *file,struct address_space *mapping, +static int simple_write_end (struct file *file,struct address_space *mapping,loff_t pos,unsigned len,unsigned copied,struct page *page,void *fsdata) {
@@ -597,7 +596,17 @@ int simple_write_end(struct file *file,return copied; } -EXPORT_SYMBOL (simple_write_end); + +/* +* Provides ramfs-style behavior: data in the pagecache, but no writeback. +*/ +const struct address_space_operations ram_aops = { + .readpage = simple_readpage, + .write_begin = simple_write_begin, + .write_end = simple_write_end, + .set_page_dirty = __set_page_dirty_no_writeback, +}; +EXPORT_SYMBOL (ram_aops);/* * the inodes created here are not hashed. If you use iunique to generate
fs/ramfs/inode.c renamed from fs/ramfs/inode.c~fs-move-ramfs_aops-to-libfs +1 -8
@@ -53,13 +53,6 @@ struct ramfs_fs_info {static const struct super_operations ramfs_ops;static const struct inode_operations ramfs_dir_inode_operations; -static const struct address_space_operations ramfs_aops = { - .readpage = simple_readpage, - .write_begin = simple_write_begin, - .write_end = simple_write_end, - .set_page_dirty = __set_page_dirty_no_writeback, -}; -struct inode *ramfs_get_inode (struct super_block *sb,const struct inode *dir,umode_t mode,dev_t dev) {
@@ -68,7 +61,7 @@ struct inode *ramfs_get_inode(struct supif (inode) { inode->i_ino =get_next_ino ();inode_init_owner (&init_user_ns, inode, dir, mode); - inode->i_mapping->a_ops = &ramfs_aops; + inode->i_mapping->a_ops = &ram_aops;mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER );mapping_set_unevictable (inode->i_mapping); inode->i_atime = inode->i_mtime = inode->i_ctime =current_time (inode);
include/linux/fs.h renamed from include/linux/fs.h~fs-move-ramfs_aops-to-libfs +1 -4
@@ -3422,13 +3422,10 @@ extern void noop_invalidatepage(struct p unsigned int length);extern ssize_t noop_direct_IO (struct kiocb *iocb,struct iov_iter *iter);extern int simple_empty (struct dentry *); -extern int simple_readpage (struct file *file,struct page *page);extern int simple_write_begin (struct file *file,struct address_space *mapping,loff_t pos,unsigned len,unsigned flags,struct page **pagep,void **fsdata); -extern int simple_write_end (struct file *file,struct address_space *mapping, -loff_t pos,unsigned len,unsigned copied, -struct page *page,void *fsdata); +extern const struct address_space_operations ram_aops;extern int always_delete_dentry (const struct dentry *);extern struct inode *alloc_anon_inode (struct super_block *);extern int simple_nosetlease (struct file *,long ,struct file_lock **,void **);
_
Andrew Morton
maintainerPATCH 56/192 mm: require ->set_page_dirty to be explicitly wired up
In reply to Andrew MortonFrom: Christoph Hellwig <[email protected]>
Subject: mm: require ->set_page_dirty to be explicitly wired up
Remove the CONFIG_BLOCK default to __set_page_dirty_buffers and just wire that method up for the missing instances.
---
fs/adfs/inode.c | 1 + fs/affs/file.c | 2 ++ fs/bfs/file.c | 1 + fs/block_dev.c | 1 + fs/ecryptfs/mmap.c | 13 +++++++++++++ fs/exfat/inode.c | 1 + fs/ext2/inode.c | 2 ++ fs/fat/inode.c | 1 + fs/gfs2/meta_io.c | 2 ++ fs/hfs/inode.c | 2 ++ fs/hfsplus/inode.c | 2 ++ fs/hpfs/file.c | 1 + fs/jfs/inode.c | 1 + fs/minix/inode.c | 1 + fs/nilfs2/mdt.c | 1 + fs/ocfs2/aops.c | 1 + fs/omfs/file.c | 1 + fs/sysv/itree.c | 1 + fs/udf/file.c | 1 + fs/udf/inode.c | 1 + fs/ufs/inode.c | 1 + mm/page-writeback.c | 18 ++++-------------- 22 files changed, 42 insertions(+), 14 deletions(-)
fs/adfs/inode.c renamed from fs/adfs/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -73,6 +73,7 @@ static sector_t _adfs_bmap(struct addres }static const struct address_space_operations adfs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = adfs_readpage, .writepage = adfs_writepage, .write_begin = adfs_write_begin,
fs/affs/file.c renamed from fs/affs/file.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +2 -0
@@ -453,6 +453,7 @@ static sector_t _affs_bmap(struct addres }const struct address_space_operations affs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = affs_readpage, .writepage = affs_writepage, .write_begin = affs_write_begin,
@@ -833,6 +834,7 @@ err_bh: }const struct address_space_operations affs_aops_ofs = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = affs_readpage_ofs,//.writepage = affs_writepage_ofs, .write_begin = affs_write_begin_ofs,
fs/bfs/file.c renamed from fs/bfs/file.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -188,6 +188,7 @@ static sector_t bfs_bmap(struct address_ }const struct address_space_operations bfs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = bfs_readpage, .writepage = bfs_writepage, .write_begin = bfs_write_begin,
fs/block_dev.c renamed from fs/block_dev.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -1754,6 +1754,7 @@ static int blkdev_writepages(struct addr }static const struct address_space_operations def_blk_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = blkdev_readpage, .readahead = blkdev_readahead, .writepage = blkdev_writepage,
fs/ecryptfs/mmap.c renamed from fs/ecryptfs/mmap.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +13 -0
@@ -533,7 +533,20 @@ static sector_t ecryptfs_bmap(struct addreturn block; } +#include <linux/buffer_head.h> +const struct address_space_operations ecryptfs_aops = { +/* +* XXX: This is pretty broken for multiple reasons: ecryptfs does not +* actually use buffer_heads, and ecryptfs will crash without +* CONFIG_BLOCK. But it matches the behavior before the default for +* address_space_operations without the ->set_page_dirty method was +* cleaned up, so this is the best we can do without maintainer +* feedback. +*/ +#ifdef CONFIG_BLOCK + .set_page_dirty = __set_page_dirty_buffers, +#endif .writepage = ecryptfs_writepage, .readpage = ecryptfs_readpage, .write_begin = ecryptfs_write_begin,
fs/exfat/inode.c renamed from fs/exfat/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -491,6 +491,7 @@ int exfat_block_truncate_page(struct ino }static const struct address_space_operations exfat_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = exfat_readpage, .readahead = exfat_readahead, .writepage = exfat_writepage,
fs/ext2/inode.c renamed from fs/ext2/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +2 -0
@@ -961,6 +961,7 @@ ext2_dax_writepages(struct address_space }const struct address_space_operations ext2_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = ext2_readpage, .readahead = ext2_readahead, .writepage = ext2_writepage,
@@ -975,6 +976,7 @@ const struct address_space_operations ex };const struct address_space_operations ext2_nobh_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = ext2_readpage, .readahead = ext2_readahead, .writepage = ext2_nobh_writepage,
fs/fat/inode.c renamed from fs/fat/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -342,6 +342,7 @@ int fat_block_truncate_page(struct inode }static const struct address_space_operations fat_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = fat_readpage, .readahead = fat_readahead, .writepage = fat_writepage,
fs/gfs2/meta_io.c renamed from fs/gfs2/meta_io.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +2 -0
@@ -89,11 +89,13 @@ static int gfs2_aspace_writepage(struct }const struct address_space_operations gfs2_meta_aops = { + .set_page_dirty = __set_page_dirty_buffers, .writepage = gfs2_aspace_writepage, .releasepage = gfs2_releasepage, };const struct address_space_operations gfs2_rgrp_aops = { + .set_page_dirty = __set_page_dirty_buffers, .writepage = gfs2_aspace_writepage, .releasepage = gfs2_releasepage, };
fs/hfs/inode.c renamed from fs/hfs/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +2 -0
@@ -159,6 +159,7 @@ static int hfs_writepages(struct address }const struct address_space_operations hfs_btree_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = hfs_readpage, .writepage = hfs_writepage, .write_begin = hfs_write_begin,
@@ -168,6 +169,7 @@ const struct address_space_operations hf };const struct address_space_operations hfs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = hfs_readpage, .writepage = hfs_writepage, .write_begin = hfs_write_begin,
fs/hfsplus/inode.c renamed from fs/hfsplus/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +2 -0
@@ -156,6 +156,7 @@ static int hfsplus_writepages(struct add }const struct address_space_operations hfsplus_btree_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = hfsplus_readpage, .writepage = hfsplus_writepage, .write_begin = hfsplus_write_begin,
@@ -165,6 +166,7 @@ const struct address_space_operations hf };const struct address_space_operations hfsplus_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = hfsplus_readpage, .writepage = hfsplus_writepage, .write_begin = hfsplus_write_begin,
fs/hpfs/file.c renamed from fs/hpfs/file.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -196,6 +196,7 @@ static int hpfs_fiemap(struct inode *ino }const struct address_space_operations hpfs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = hpfs_readpage, .writepage = hpfs_writepage, .readahead = hpfs_readahead,
fs/jfs/inode.c renamed from fs/jfs/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -356,6 +356,7 @@ static ssize_t jfs_direct_IO(struct kioc }const struct address_space_operations jfs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = jfs_readpage, .readahead = jfs_readahead, .writepage = jfs_writepage,
fs/minix/inode.c renamed from fs/minix/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -442,6 +442,7 @@ static sector_t minix_bmap(struct addres }static const struct address_space_operations minix_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = minix_readpage, .writepage = minix_writepage, .write_begin = minix_write_begin,
fs/nilfs2/mdt.c renamed from fs/nilfs2/mdt.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -434,6 +434,7 @@ nilfs_mdt_write_page(struct page *page,static const struct address_space_operations def_mdt_aops = { + .set_page_dirty = __set_page_dirty_buffers, .writepage = nilfs_mdt_write_page, };
fs/ocfs2/aops.c renamed from fs/ocfs2/aops.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -2453,6 +2453,7 @@ static ssize_t ocfs2_direct_IO(struct ki }const struct address_space_operations ocfs2_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = ocfs2_readpage, .readahead = ocfs2_readahead, .writepage = ocfs2_writepage,
fs/omfs/file.c renamed from fs/omfs/file.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -372,6 +372,7 @@ const struct inode_operations omfs_file_ };const struct address_space_operations omfs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = omfs_readpage, .readahead = omfs_readahead, .writepage = omfs_writepage,
fs/sysv/itree.c renamed from fs/sysv/itree.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -495,6 +495,7 @@ static sector_t sysv_bmap(struct address }const struct address_space_operations sysv_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = sysv_readpage, .writepage = sysv_writepage, .write_begin = sysv_write_begin,
fs/udf/file.c renamed from fs/udf/file.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -125,6 +125,7 @@ static int udf_adinicb_write_end(struct }const struct address_space_operations udf_adinicb_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = udf_adinicb_readpage, .writepage = udf_adinicb_writepage, .write_begin = udf_adinicb_write_begin,
fs/udf/inode.c renamed from fs/udf/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -235,6 +235,7 @@ static sector_t udf_bmap(struct address_ }const struct address_space_operations udf_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = udf_readpage, .readahead = udf_readahead, .writepage = udf_writepage,
fs/ufs/inode.c renamed from fs/ufs/inode.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +1 -0
@@ -526,6 +526,7 @@ static sector_t ufs_bmap(struct address_ }const struct address_space_operations ufs_aops = { + .set_page_dirty = __set_page_dirty_buffers, .readpage = ufs_readpage, .writepage = ufs_writepage, .write_begin = ufs_write_begin,
mm/page-writeback.c renamed from mm/page-writeback.c~mm-require-set_page_dirty-to-be-explicitly-wire-up +4 -14
@@ -32,7 +32,6 @@#include <linux/sysctl.h> #include <linux/cpu.h> #include <linux/syscalls.h> -#include <linux/buffer_head.h> /* __set_page_dirty_buffers */ #include <linux/pagevec.h> #include <linux/timer.h> #include <linux/sched/rt.h>
@@ -2555,13 +2554,9 @@ EXPORT_SYMBOL(redirty_page_for_writepage/* * Dirty a page. * -* For pages with a mapping this should be done under the page lock -* for the benefit of asynchronous memory errors who prefer a consistent -* dirty state. This rule can be broken in some special cases, -* but should be better not to. -* -* If the mapping doesn't provide a set_page_dirty a_op, then -* just fall through and assume that it wants buffer_heads. +* For pages with a mapping this should be done under the page lock for the +* benefit of asynchronous memory errors who prefer a consistent dirty state. +* This rule can be broken in some special cases, but should be better not to. */ int set_page_dirty (struct page *page) {
@@ -2569,7 +2564,6 @@ int set_page_dirty(struct page *page) page =compound_head (page);if (likely (mapping)) { -int (*spd)(struct page *) = mapping->a_ops->set_page_dirty;/* * readahead/lru_deactivate_page could remain * PG_readahead/PG_reclaim due to race with end_page_writeback
@@ -2582,11 +2576,7 @@ int set_page_dirty(struct page *page)*/ if (PageReclaim (page))ClearPageReclaim (page); -#ifdef CONFIG_BLOCK -if (!spd) - spd = __set_page_dirty_buffers; -#endif -return (*spd)(page); +return mapping->a_ops->set_page_dirty (page); }if (!PageDirty (page)) {if (!TestSetPageDirty (page))
_
Andrew Morton
maintainerPATCH 57/192 mm/writeback: move __set_page_dirty() to core mm
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm/writeback: move __set_page_dirty() to core mm
Patch series "Further set_page_dirty cleanups".
Prompted by Christoph's recent patches, here are some more patches to improve the state of set_page_dirty(). They're all from the folio tree, so they've been tested to a certain extent.
This patch (of 6):
Nothing in __set_page_dirty() is specific to buffer_head, so move it to mm/page-writeback.c. That removes the only caller of account_page_dirtied() outside of page-writeback.c, so make it static.
---
fs/buffer.c | 24 ------------------------ include/linux/mm.h | 1 - mm/page-writeback.c | 27 ++++++++++++++++++++++++++- 3 files changed, 26 insertions(+), 26 deletions(-)
fs/buffer.c renamed from fs/buffer.c~mm-writeback-move-__set_page_dirty-to-core-mm +0 -24
@@ -589,30 +589,6 @@ void mark_buffer_dirty_inode(struct buffEXPORT_SYMBOL (mark_buffer_dirty_inode);/* -* Mark the page dirty, and set it dirty in the page cache, and mark the inode -* dirty. -* -* If warn is true, then emit a warning if the page is not uptodate and has -* not been truncated. -* -* The caller must hold lock_page_memcg(). -*/ -void __set_page_dirty (struct page *page,struct address_space *mapping, -int warn) -{ -unsigned long flags; - -xa_lock_irqsave (&mapping->i_pages, flags); -if (page->mapping) {/* Race with truncate? */ -WARN_ON_ONCE (warn && !PageUptodate (page)); -account_page_dirtied (page, mapping); -__xa_set_mark (&mapping->i_pages,page_index (page), -PAGECACHE_TAG_DIRTY ); - } -xa_unlock_irqrestore (&mapping->i_pages, flags); -} - -/* * Add a page to the dirty page list. * * It is a sad fact of life that this function is called from several places
include/linux/mm.h renamed from include/linux/mm.h~mm-writeback-move-__set_page_dirty-to-core-mm +0 -1
@@ -1855,7 +1855,6 @@ int __set_page_dirty_nobuffers(struct paint __set_page_dirty_no_writeback (struct page *page);int redirty_page_for_writepage (struct writeback_control *wbc,struct page *page); -void account_page_dirtied (struct page *page,struct address_space *mapping);void account_page_cleaned (struct page *page,struct address_space *mapping,struct bdi_writeback *wb);int set_page_dirty (struct page *page);
mm/page-writeback.c renamed from mm/page-writeback.c~mm-writeback-move-__set_page_dirty-to-core-mm +26 -1
@@ -2425,7 +2425,8 @@ int __set_page_dirty_no_writeback(struct* * NOTE: This relies on being atomic wrt interrupts. */ -void account_page_dirtied (struct page *page,struct address_space *mapping) +static void account_page_dirtied (struct page *page, +struct address_space *mapping) {struct inode *inode = mapping->host;
@@ -2467,6 +2468,30 @@ void account_page_cleaned(struct page *p }/* +* Mark the page dirty, and set it dirty in the page cache, and mark the inode +* dirty. +* +* If warn is true, then emit a warning if the page is not uptodate and has +* not been truncated. +* +* The caller must hold lock_page_memcg(). +*/ +void __set_page_dirty (struct page *page,struct address_space *mapping, +int warn) +{ +unsigned long flags; + +xa_lock_irqsave (&mapping->i_pages, flags); +if (page->mapping) {/* Race with truncate? */ +WARN_ON_ONCE (warn && !PageUptodate (page)); +account_page_dirtied (page, mapping); +__xa_set_mark (&mapping->i_pages,page_index (page), +PAGECACHE_TAG_DIRTY ); + } +xa_unlock_irqrestore (&mapping->i_pages, flags); +} + +/* * For address_spaces which do not use buffers. Just tag the page as dirty in * the xarray. *
_
Andrew Morton
maintainerPATCH 58/192 mm/writeback: use __set_page_dirty in __set_page_dirty_nobuffers
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm/writeback: use __set_page_dirty in __set_page_dirty_nobuffers
This is fundamentally the same code, so just call it instead of duplicating it.
---
mm/page-writeback.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)
mm/page-writeback.c renamed from mm/page-writeback.c~mm-writeback-use-__set_page_dirty-in-__set_page_dirty_nobuffers +1 -9
@@ -2508,20 +2508,12 @@ int __set_page_dirty_nobuffers(struct palock_page_memcg (page);if (!TestSetPageDirty (page)) {struct address_space *mapping =page_mapping (page); -unsigned long flags;if (!mapping) {unlock_page_memcg (page);return 1 ; } - -xa_lock_irqsave (&mapping->i_pages, flags); -BUG_ON (page_mapping (page) != mapping); -WARN_ON_ONCE (!PagePrivate (page) && !PageUptodate (page)); -account_page_dirtied (page, mapping); -__xa_set_mark (&mapping->i_pages,page_index (page), -PAGECACHE_TAG_DIRTY ); -xa_unlock_irqrestore (&mapping->i_pages, flags); +__set_page_dirty (page, mapping, !PagePrivate (page));unlock_page_memcg (page);if (mapping->host) {
_
Andrew Morton
maintainerPATCH 59/192 iomap: use __set_page_dirty_nobuffers
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: iomap: use __set_page_dirty_nobuffers
The only difference between iomap_set_page_dirty() and __set_page_dirty_nobuffers() is that the latter includes a debugging check that a !Uptodate page has private data.
---
fs/gfs2/aops.c | 2 +- fs/iomap/buffered-io.c | 27 +-------------------------- fs/xfs/xfs_aops.c | 2 +- fs/zonefs/super.c | 2 +- include/linux/iomap.h | 1 - 5 files changed, 4 insertions(+), 30 deletions(-)
fs/gfs2/aops.c renamed from fs/gfs2/aops.c~iomap-use-__set_page_dirty_nobuffers +1 -1
@@ -784,7 +784,7 @@ static const struct address_space_operat .writepages = gfs2_writepages, .readpage = gfs2_readpage, .readahead = gfs2_readahead, - .set_page_dirty = iomap_set_page_dirty, + .set_page_dirty = __set_page_dirty_nobuffers, .releasepage = iomap_releasepage, .invalidatepage = iomap_invalidatepage, .bmap = gfs2_bmap,
fs/iomap/buffered-io.c renamed from fs/iomap/buffered-io.c~iomap-use-__set_page_dirty_nobuffers +1 -26
@@ -640,31 +640,6 @@ out_no_page:return status; } -int -iomap_set_page_dirty (struct page *page) -{ -struct address_space *mapping =page_mapping (page); -int newly_dirty; - -if (unlikely (!mapping)) -return !TestSetPageDirty (page); - -/* -* Lock out page's memcg migration to keep PageDirty -* synchronized with per-memcg dirty page counters. -*/ -lock_page_memcg (page); - newly_dirty = !TestSetPageDirty (page); -if (newly_dirty) -__set_page_dirty (page, mapping,0 ); -unlock_page_memcg (page); - -if (newly_dirty) -__mark_inode_dirty (mapping->host,I_DIRTY_PAGES ); -return newly_dirty; -} -EXPORT_SYMBOL_GPL (iomap_set_page_dirty); -static size_t __iomap_write_end (struct inode *inode,loff_t pos,size_t len,size_t copied,struct page *page) {
@@ -684,7 +659,7 @@ static size_t __iomap_write_end(struct iif (unlikely (copied < len && !PageUptodate (page)))return 0 ;iomap_set_range_uptodate (page,offset_in_page (pos), len); -iomap_set_page_dirty (page); +__set_page_dirty_nobuffers (page);return copied; }
fs/xfs/xfs_aops.c renamed from fs/xfs/xfs_aops.c~iomap-use-__set_page_dirty_nobuffers +1 -1
@@ -561,7 +561,7 @@ const struct address_space_operations xf .readahead = xfs_vm_readahead, .writepage = xfs_vm_writepage, .writepages = xfs_vm_writepages, - .set_page_dirty = iomap_set_page_dirty, + .set_page_dirty = __set_page_dirty_nobuffers, .releasepage = iomap_releasepage, .invalidatepage = iomap_invalidatepage, .bmap = xfs_vm_bmap,
fs/zonefs/super.c renamed from fs/zonefs/super.c~iomap-use-__set_page_dirty_nobuffers +1 -1
@@ -185,7 +185,7 @@ static const struct address_space_operat .readahead = zonefs_readahead, .writepage = zonefs_writepage, .writepages = zonefs_writepages, - .set_page_dirty = iomap_set_page_dirty, + .set_page_dirty = __set_page_dirty_nobuffers, .releasepage = iomap_releasepage, .invalidatepage = iomap_invalidatepage, .migratepage = iomap_migrate_page,
include/linux/iomap.h renamed from include/linux/iomap.h~iomap-use-__set_page_dirty_nobuffers +0 -1
@@ -159,7 +159,6 @@ ssize_t iomap_file_buffered_write(struct const struct iomap_ops *ops);int iomap_readpage (struct page *page,const struct iomap_ops *ops);void iomap_readahead (struct readahead_control *,const struct iomap_ops *ops); -int iomap_set_page_dirty (struct page *page);int iomap_is_partially_uptodate (struct page *page,unsigned long from,unsigned long count);int iomap_releasepage (struct page *page,gfp_t gfp_mask);
_
Andrew Morton
maintainerPATCH 60/192 fs: remove anon_set_page_dirty()
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: fs: remove anon_set_page_dirty()
Use __set_page_dirty_no_writeback() instead. This will set the dirty bit on the page, which will be used to avoid calling set_page_dirty() in the future. It will have no effect on actually writing the page back, as the pages are not on any LRU lists.
---
fs/libfs.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
fs/libfs.c renamed from fs/libfs.c~fs-remove-anon_set_page_dirty +1 -10
@@ -1217,19 +1217,10 @@ void kfree_link(void *p) }EXPORT_SYMBOL (kfree_link); -/* -* nop .set_page_dirty method so that people can use .page_mkwrite on -* anon inodes. -*/ -static int anon_set_page_dirty (struct page *page) -{ -return 0 ; -}; -struct inode *alloc_anon_inode (struct super_block *s) {static const struct address_space_operations anon_aops = { - .set_page_dirty = anon_set_page_dirty, + .set_page_dirty = __set_page_dirty_no_writeback, };struct inode *inode =new_inode_pseudo (s);
_
Andrew Morton
maintainerPATCH 61/192 fs: remove noop_set_page_dirty()
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: fs: remove noop_set_page_dirty()
Use __set_page_dirty_no_writeback() instead. This will set the dirty bit on the page, which will be used to avoid calling set_page_dirty() in the future. It will have no effect on actually writing the page back, as the pages are not on any LRU lists.
---
drivers/dax/device.c | 2 +- fs/ext2/inode.c | 2 +- fs/ext4/inode.c | 2 +- fs/fuse/dax.c | 2 +- fs/libfs.c | 16 ---------------- fs/xfs/xfs_aops.c | 2 +- include/linux/fs.h | 1 - mm/page-writeback.c | 1 + 8 files changed, 6 insertions(+), 22 deletions(-)
drivers/dax/device.c renamed from drivers/dax/device.c~fs-remove-noop_set_page_dirty +1 -1
@@ -337,7 +337,7 @@ static unsigned long dax_get_unmapped_ar }static const struct address_space_operations dev_dax_aops = { - .set_page_dirty = noop_set_page_dirty, + .set_page_dirty = __set_page_dirty_no_writeback, .invalidatepage = noop_invalidatepage, };
fs/ext2/inode.c renamed from fs/ext2/inode.c~fs-remove-noop_set_page_dirty +1 -1
@@ -992,7 +992,7 @@ const struct address_space_operations exstatic const struct address_space_operations ext2_dax_aops = { .writepages = ext2_dax_writepages, .direct_IO = noop_direct_IO, - .set_page_dirty = noop_set_page_dirty, + .set_page_dirty = __set_page_dirty_no_writeback, .invalidatepage = noop_invalidatepage, };
fs/ext4/inode.c renamed from fs/ext4/inode.c~fs-remove-noop_set_page_dirty +1 -1
@@ -3701,7 +3701,7 @@ static const struct address_space_operatstatic const struct address_space_operations ext4_dax_aops = { .writepages = ext4_dax_writepages, .direct_IO = noop_direct_IO, - .set_page_dirty = noop_set_page_dirty, + .set_page_dirty = __set_page_dirty_no_writeback, .bmap = ext4_bmap, .invalidatepage = noop_invalidatepage, .swap_activate = ext4_iomap_swap_activate,
fs/fuse/dax.c renamed from fs/fuse/dax.c~fs-remove-noop_set_page_dirty +1 -1
@@ -1329,7 +1329,7 @@ bool fuse_dax_inode_alloc(struct super_bstatic const struct address_space_operations fuse_dax_file_aops = { .writepages = fuse_dax_writepages, .direct_IO = noop_direct_IO, - .set_page_dirty = noop_set_page_dirty, + .set_page_dirty = __set_page_dirty_no_writeback, .invalidatepage = noop_invalidatepage, };
fs/libfs.c renamed from fs/libfs.c~fs-remove-noop_set_page_dirty +0 -16
@@ -1171,22 +1171,6 @@ int noop_fsync(struct file *file, loff_t }EXPORT_SYMBOL (noop_fsync); -int noop_set_page_dirty (struct page *page) -{ -/* -* Unlike __set_page_dirty_no_writeback that handles dirty page -* tracking in the page object, dax does all dirty tracking in -* the inode address_space in response to mkwrite faults. In the -* dax case we only need to worry about potentially dirty CPU -* caches, not dirty page cache pages to write back. -* -* This callback is defined to prevent fallback to -* __set_page_dirty_buffers() in set_page_dirty(). -*/ -return 0 ; -} -EXPORT_SYMBOL_GPL (noop_set_page_dirty); -void noop_invalidatepage (struct page *page,unsigned int offset,unsigned int length) {
fs/xfs/xfs_aops.c renamed from fs/xfs/xfs_aops.c~fs-remove-noop_set_page_dirty +1 -1
@@ -575,7 +575,7 @@ const struct address_space_operations xfconst struct address_space_operations xfs_dax_aops = { .writepages = xfs_dax_writepages, .direct_IO = noop_direct_IO, - .set_page_dirty = noop_set_page_dirty, + .set_page_dirty = __set_page_dirty_no_writeback, .invalidatepage = noop_invalidatepage, .swap_activate = xfs_iomap_swapfile_activate, };
include/linux/fs.h renamed from include/linux/fs.h~fs-remove-noop_set_page_dirty +0 -1
@@ -3417,7 +3417,6 @@ extern int simple_rename(struct user_namextern void simple_recursive_removal (struct dentry *,void (*callback)(struct dentry *));extern int noop_fsync (struct file *,loff_t ,loff_t ,int ); -extern int noop_set_page_dirty (struct page *page);extern void noop_invalidatepage (struct page *page,unsigned int offset,unsigned int length);extern ssize_t noop_direct_IO (struct kiocb *iocb,struct iov_iter *iter);
mm/page-writeback.c renamed from mm/page-writeback.c~fs-remove-noop_set_page_dirty +1 -0
@@ -2417,6 +2417,7 @@ int __set_page_dirty_no_writeback(structreturn !TestSetPageDirty (page);return 0 ; } +EXPORT_SYMBOL (__set_page_dirty_no_writeback);/* * Helper function for set_page_dirty family.
_
Andrew Morton
maintainerPATCH 62/192 mm: move page dirtying prototypes from mm.h
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm: move page dirtying prototypes from mm.h
These functions implement the address_space ->set_page_dirty operation and should live in pagemap.h, not mm.h so that the rest of the kernel doesn't get funny ideas about calling them directly.
---
fs/fuse/dax.c | 1 + fs/zonefs/super.c | 2 +- include/linux/mm.h | 3 --- include/linux/pagemap.h | 4 ++++ 4 files changed, 6 insertions(+), 4 deletions(-)
fs/fuse/dax.c renamed from fs/fuse/dax.c~mm-move-page-dirtying-prototypes-from-mmh +1 -0
@@ -9,6 +9,7 @@#include <linux/delay.h> #include <linux/dax.h> #include <linux/uio.h> +#include <linux/pagemap.h> #include <linux/pfn_t.h> #include <linux/iomap.h> #include <linux/interval_tree.h>
fs/zonefs/super.c renamed from fs/zonefs/super.c~mm-move-page-dirtying-prototypes-from-mmh +1 -1
@@ -5,7 +5,7 @@* Copyright (C) 2019 Western Digital Corporation or its affiliates. */ #include <linux/module.h> -#include <linux/fs.h> +#include <linux/pagemap.h> #include <linux/magic.h> #include <linux/iomap.h> #include <linux/init.h>
include/linux/mm.h renamed from include/linux/mm.h~mm-move-page-dirtying-prototypes-from-mmh +0 -3
@@ -1850,9 +1850,6 @@ extern int try_to_release_page(struct paextern void do_invalidatepage (struct page *page,unsigned int offset,unsigned int length); -void __set_page_dirty (struct page *,struct address_space *,int warn); -int __set_page_dirty_nobuffers (struct page *page); -int __set_page_dirty_no_writeback (struct page *page);int redirty_page_for_writepage (struct writeback_control *wbc,struct page *page); void account_page_cleaned(struct page *page, struct address_space *mapping,
include/linux/pagemap.h renamed from include/linux/pagemap.h~mm-move-page-dirtying-prototypes-from-mmh +4 -0
@@ -702,6 +702,10 @@ int wait_on_page_writeback_killable(struextern void end_page_writeback (struct page *page);void wait_for_stable_page (struct page *page); +void __set_page_dirty (struct page *,struct address_space *,int warn); +int __set_page_dirty_nobuffers (struct page *page); +int __set_page_dirty_no_writeback (struct page *page); +void page_endio (struct page *page,bool is_write,int err);/**
_
Andrew Morton
maintainerPATCH 63/192 mm/gup_benchmark: support threading
In reply to Andrew MortonFrom: Peter Xu <[email protected]>
Subject: mm/gup_benchmark: support threading
Patch series "mm/gup: Fix pin page write cache bouncing on has_pinned", v2.
This series contains 3 patches, the 1st one enables threading for gup_benchmark in the kselftest. The latter two patches are collected from Andrea's local branch which can fix write cache bouncing issue with pinning fast-gup.
To be explicit on the latter two patches:
- the 2nd patch fixes the perf degrade when introducing has_pinned, then
- the last patch tries to remove the has_pinned with a bit in mm->flags
For patch 3: originally I think we had a plan to reuse has_pinned into a counter very soon, however that's not happening at least until today, so maybe it proves that we can remove it until we really want such a counter for whatever reason. As the commit message stated, it saves 4 bytes for each mm without observable regressions.
Regarding testing: we can reference to the commit message of patch 2 for some detailed testing with will-is-scale. Meanwhile I did patch 1 just because then we can even easily verify the patchset using the existing kselftest facilities or even regress test it in the future with the repo if we want.
Below numbers are extra verification tests that I did besides commit message of patch 2 using the new gup_benchmark and 256 cpus. Below test is done on 40 cpus host with Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, and I can get similar result (of course the write cache bouncing get severe with even more cores).
After patch 1 applied (only test patch, so using old kernel):
$ sudo chrt -f 1 ./gup_test -a -m 512 -j 40 PIN_FAST_BENCHMARK: Time: get:459632 put:5990 us PIN_FAST_BENCHMARK: Time: get:461967 put:5840 us PIN_FAST_BENCHMARK: Time: get:464521 put:6140 us PIN_FAST_BENCHMARK: Time: get:465176 put:7100 us PIN_FAST_BENCHMARK: Time: get:465960 put:6733 us PIN_FAST_BENCHMARK: Time: get:465324 put:6781 us PIN_FAST_BENCHMARK: Time: get:466018 put:7130 us PIN_FAST_BENCHMARK: Time: get:466362 put:7118 us PIN_FAST_BENCHMARK: Time: get:465118 put:6975 us PIN_FAST_BENCHMARK: Time: get:466422 put:6602 us PIN_FAST_BENCHMARK: Time: get:465791 put:6818 us PIN_FAST_BENCHMARK: Time: get:467091 put:6298 us PIN_FAST_BENCHMARK: Time: get:467694 put:5432 us PIN_FAST_BENCHMARK: Time: get:469575 put:5581 us PIN_FAST_BENCHMARK: Time: get:468124 put:6055 us PIN_FAST_BENCHMARK: Time: get:468877 put:6720 us PIN_FAST_BENCHMARK: Time: get:467212 put:4961 us PIN_FAST_BENCHMARK: Time: get:467834 put:6697 us PIN_FAST_BENCHMARK: Time: get:470778 put:6398 us PIN_FAST_BENCHMARK: Time: get:469788 put:6310 us PIN_FAST_BENCHMARK: Time: get:488277 put:7113 us PIN_FAST_BENCHMARK: Time: get:486613 put:7085 us PIN_FAST_BENCHMARK: Time: get:486940 put:7202 us PIN_FAST_BENCHMARK: Time: get:488728 put:7101 us PIN_FAST_BENCHMARK: Time: get:487570 put:7327 us PIN_FAST_BENCHMARK: Time: get:489260 put:7027 us PIN_FAST_BENCHMARK: Time: get:488846 put:6866 us PIN_FAST_BENCHMARK: Time: get:488521 put:6745 us PIN_FAST_BENCHMARK: Time: get:489950 put:6459 us PIN_FAST_BENCHMARK: Time: get:489777 put:6617 us PIN_FAST_BENCHMARK: Time: get:488224 put:6591 us PIN_FAST_BENCHMARK: Time: get:488644 put:6477 us PIN_FAST_BENCHMARK: Time: get:488754 put:6711 us PIN_FAST_BENCHMARK: Time: get:488875 put:6743 us PIN_FAST_BENCHMARK: Time: get:489290 put:6657 us PIN_FAST_BENCHMARK: Time: get:490264 put:6684 us PIN_FAST_BENCHMARK: Time: get:489631 put:6737 us PIN_FAST_BENCHMARK: Time: get:488434 put:6655 us PIN_FAST_BENCHMARK: Time: get:492213 put:6297 us PIN_FAST_BENCHMARK: Time: get:491124 put:6173 us
After the whole series applied (new fixed kernel):
$ sudo chrt -f 1 ./gup_test -a -m 512 -j 40 PIN_FAST_BENCHMARK: Time: get:82038 put:7041 us PIN_FAST_BENCHMARK: Time: get:82144 put:6817 us PIN_FAST_BENCHMARK: Time: get:83417 put:6674 us PIN_FAST_BENCHMARK: Time: get:82540 put:6594 us PIN_FAST_BENCHMARK: Time: get:83214 put:6681 us PIN_FAST_BENCHMARK: Time: get:83444 put:6889 us PIN_FAST_BENCHMARK: Time: get:83194 put:7499 us PIN_FAST_BENCHMARK: Time: get:84876 put:7369 us PIN_FAST_BENCHMARK: Time: get:86092 put:10289 us PIN_FAST_BENCHMARK: Time: get:86153 put:10415 us PIN_FAST_BENCHMARK: Time: get:85026 put:7751 us PIN_FAST_BENCHMARK: Time: get:85458 put:7944 us PIN_FAST_BENCHMARK: Time: get:85735 put:8154 us PIN_FAST_BENCHMARK: Time: get:85851 put:8299 us PIN_FAST_BENCHMARK: Time: get:86323 put:9617 us PIN_FAST_BENCHMARK: Time: get:86288 put:10496 us PIN_FAST_BENCHMARK: Time: get:87697 put:9346 us PIN_FAST_BENCHMARK: Time: get:87980 put:8382 us PIN_FAST_BENCHMARK: Time: get:88719 put:8400 us PIN_FAST_BENCHMARK: Time: get:87616 put:8588 us PIN_FAST_BENCHMARK: Time: get:86730 put:9563 us PIN_FAST_BENCHMARK: Time: get:88167 put:8673 us PIN_FAST_BENCHMARK: Time: get:86844 put:9777 us PIN_FAST_BENCHMARK: Time: get:88068 put:11774 us PIN_FAST_BENCHMARK: Time: get:86170 put:15676 us PIN_FAST_BENCHMARK: Time: get:87967 put:12827 us PIN_FAST_BENCHMARK: Time: get:95773 put:7652 us PIN_FAST_BENCHMARK: Time: get:87734 put:13650 us PIN_FAST_BENCHMARK: Time: get:89833 put:14237 us PIN_FAST_BENCHMARK: Time: get:96186 put:8029 us PIN_FAST_BENCHMARK: Time: get:95532 put:8886 us PIN_FAST_BENCHMARK: Time: get:95351 put:5826 us PIN_FAST_BENCHMARK: Time: get:96401 put:8407 us PIN_FAST_BENCHMARK: Time: get:96473 put:8287 us PIN_FAST_BENCHMARK: Time: get:97177 put:8430 us PIN_FAST_BENCHMARK: Time: get:98120 put:5263 us PIN_FAST_BENCHMARK: Time: get:96271 put:7757 us PIN_FAST_BENCHMARK: Time: get:99628 put:10467 us PIN_FAST_BENCHMARK: Time: get:99344 put:10045 us PIN_FAST_BENCHMARK: Time: get:94212 put:15485 us
Summary:
Old kernel: 477729.97 (+-3.79%)
New kernel: 89144.65 (+-11.76%)
This patch (of 3):
Add a new parameter "-j N" to support concurrent gup test.
---
tools/testing/selftests/vm/gup_test.c | 96 ++++++++++++++++-------- 1 file changed, 65 insertions(+), 31 deletions(-)
tools/testing/selftests/vm/gup_test.c renamed from tools/testing/selftests/vm/gup_test.c~mm-gup_benchmark-support-threading +65 -31
@@ -6,6 +6,8 @@#include <sys/mman.h> #include <sys/stat.h> #include <sys/types.h> +#include <pthread.h> +#include <assert.h> #include "../../../../mm/gup_test.h" #define MB (1UL << 20)
@@ -15,6 +17,12 @@#define FOLL_WRITE 0x01/* check pte is writable */ #define FOLL_TOUCH 0x02/* mark page accessed */ +static unsigned long cmd =GUP_FAST_BENCHMARK ; +static int gup_fd, repeats =1 ; +static unsigned long size =128 *MB ; +/* Serialize prints */ +static pthread_mutex_t print_mutex =PTHREAD_MUTEX_INITIALIZER ; +static char *cmd_to_str (unsigned long cmd) {switch (cmd) {
@@ -34,17 +42,55 @@ static char *cmd_to_str(unsigned long cmreturn "Unknown command" ; } +void *gup_thread (void *data) +{ +struct gup_test gup = *(struct gup_test *)data; +int i; + +/* Only report timing information on the *_BENCHMARK commands: */ +if ((cmd ==PIN_FAST_BENCHMARK ) || (cmd ==GUP_FAST_BENCHMARK ) || + (cmd ==PIN_LONGTERM_BENCHMARK )) { +for (i =0 ; i < repeats; i++) { + gup.size = size; +if (ioctl (gup_fd, cmd, &gup)) +perror ("ioctl" ),exit (1 ); + +pthread_mutex_lock (&print_mutex); +printf ("%s: Time: get:%lld put:%lld us" , +cmd_to_str (cmd), gup.get_delta_usec, + gup.put_delta_usec); +if (gup.size != size) +printf (", truncated (size: %lld)" , gup.size); +printf ("\n" ); +pthread_mutex_unlock (&print_mutex); + } + }else { + gup.size = size; +if (ioctl (gup_fd, cmd, &gup)) { +perror ("ioctl" ); +exit (1 ); + } + +pthread_mutex_lock (&print_mutex); +printf ("%s: done\n" ,cmd_to_str (cmd)); +if (gup.size != size) +printf ("Truncated (size: %lld)\n" , gup.size); +pthread_mutex_unlock (&print_mutex); + } + +return NULL ; +} +int main (int argc,char **argv) {struct gup_test gup = {0 }; -unsigned long size =128 *MB ; -int i, fd, filed, opt, nr_pages =1 , thp =-1 , repeats =1 , write =1 ; -unsigned long cmd =GUP_FAST_BENCHMARK ; +int filed, i, opt, nr_pages =1 , thp =-1 , write =1 , nthreads =1 , ret;int flags =MAP_PRIVATE , touch =0 ;char *file ="/dev/zero" ; +pthread_t *tid;char *p; -while ((opt =getopt (argc, argv,"m:r:n:F:f:abctTLUuwWSHpz" )) !=-1 ) { +while ((opt =getopt (argc, argv,"m:r:n:F:f:abcj:tTLUuwWSHpz" )) !=-1 ) {switch (opt) {case 'a' : cmd =PIN_FAST_BENCHMARK ;
@@ -74,6 +120,9 @@ int main(int argc, char **argv)/* strtol, so you can pass flags in hex form */ gup.gup_flags =strtol (optarg,0 ,0 );break ; +case 'j' : + nthreads =atoi (optarg); +break ;case 'm' : size =atoi (optarg) *MB ;break ;
@@ -154,8 +203,8 @@ int main(int argc, char **argv)if (write) gup.gup_flags |=FOLL_WRITE ; - fd =open ("/sys/kernel/debug/gup_test" ,O_RDWR ); -if (fd ==-1 ) { + gup_fd =open ("/sys/kernel/debug/gup_test" ,O_RDWR ); +if (gup_fd ==-1 ) {perror ("open" );exit (1 ); }
@@ -185,32 +234,17 @@ int main(int argc, char **argv) p[0 ] =0 ; } -/* Only report timing information on the *_BENCHMARK commands: */ -if ((cmd ==PIN_FAST_BENCHMARK ) || (cmd ==GUP_FAST_BENCHMARK ) || - (cmd ==PIN_LONGTERM_BENCHMARK )) { -for (i =0 ; i < repeats; i++) { - gup.size = size; -if (ioctl (fd, cmd, &gup)) -perror ("ioctl" ),exit (1 ); - -printf ("%s: Time: get:%lld put:%lld us" , -cmd_to_str (cmd), gup.get_delta_usec, - gup.put_delta_usec); -if (gup.size != size) -printf (", truncated (size: %lld)" , gup.size); -printf ("\n" ); - } - }else { - gup.size = size; -if (ioctl (fd, cmd, &gup)) { -perror ("ioctl" ); -exit (1 ); - } - -printf ("%s: done\n" ,cmd_to_str (cmd)); -if (gup.size != size) -printf ("Truncated (size: %lld)\n" , gup.size); + tid =malloc (sizeof (pthread_t) * nthreads); +assert (tid); +for (i =0 ; i < nthreads; i++) { + ret =pthread_create (&tid[i],NULL , gup_thread, &gup); +assert (ret ==0 ); + } +for (i =0 ; i < nthreads; i++) { + ret =pthread_join (tid[i],NULL ); +assert (ret ==0 ); } +free (tid);return 0 ; }
_
Andrew Morton
maintainerPATCH 64/192 mm: gup: allow FOLL_PIN to scale in SMP
In reply to Andrew MortonFrom: Andrea Arcangeli <[email protected]>
Subject: mm: gup: allow FOLL_PIN to scale in SMP
has_pinned cannot be written by each pin-fast or it won't scale in SMP. This isn't "false sharing" strictly speaking (it's more like "true non-sharing"), but it creates the same SMP scalability bottleneck of "false sharing".
To verify the improvement, below test is done on 40 cpus host with Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (must be with CONFIG_GUP_TEST=y):
$ sudo chrt -f 1 ./gup_test -a -m 512 -j 40
Where we can get (average value for 40 threads):
Old kernel: 477729.97 (+- 3.79%)
New kernel: 89144.65 (+-11.76%)
On a similar condition with 256 cpus, this commits increases the SMP scalability of pin_user_pages_fast() executed by different threads of the same process by more than 4000%.
---
mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/gup.c renamed from mm/gup.c~mm-gup-allow-foll_pin-to-scale-in-smp +2 -2
@@ -1320,7 +1320,7 @@ static __always_inline long __get_user_pBUG_ON (*locked !=1 ); } -if (flags &FOLL_PIN ) +if ((flags &FOLL_PIN ) && !atomic_read (&mm->has_pinned))atomic_set (&mm->has_pinned,1 );/*
@@ -2641,7 +2641,7 @@ static int internal_get_user_pages_fast( FOLL_FAST_ONLY)))return -EINVAL ; -if (gup_flags &FOLL_PIN ) +if ((gup_flags &FOLL_PIN ) && !atomic_read (¤t->mm->has_pinned))atomic_set (¤t->mm->has_pinned,1 );if (!(gup_flags &FOLL_FAST_ONLY ))
_
Andrew Morton
maintainerPATCH 65/192 mm: gup: pack has_pinned in MMF_HAS_PINNED
In reply to Andrew MortonFrom: Andrea Arcangeli <[email protected]>
Subject: mm: gup: pack has_pinned in MMF_HAS_PINNED
has_pinned 32bit can be packed in the MMF_HAS_PINNED bit as a noop cleanup.
Any atomic_inc/dec to the mm cacheline shared by all threads in pin-fast would reintroduce a loss of SMP scalability to pin-fast, so there's no future potential usefulness to keep an atomic in the mm for this.
set_bit(MMF_HAS_PINNED) will be theoretically a bit slower than WRITE_ONCE (atomic_set is equivalent to WRITE_ONCE), but the set_bit (just like atomic_set after this commit) has to be still issued only once per "mm", so the difference between the two will be lost in the noise.
will-it-scale "mmap2" shows no change in performance with enterprise config as expected.
will-it-scale "pin_fast" retains the > 4000% SMP scalability performance improvement against upstream as expected.
This is a noop as far as overall performance and SMP scalability are concerned.
---
fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 2 +- include/linux/mm_types.h | 10 ---------- include/linux/sched/coredump.h | 8 ++++++++ kernel/fork.c | 1 - mm/gup.c | 19 +++++++++++++++---- 6 files changed, 25 insertions(+), 17 deletions(-)
fs/proc/task_mmu.c renamed from fs/proc/task_mmu.c~mm-gup-pack-has_pinned-in-mmf_has_pinned +1 -1
@@ -1047,7 +1047,7 @@ static inline bool pte_is_pinned(structreturn false;if (!is_cow_mapping (vma->vm_flags))return false; -if (likely (!atomic_read (&vma->vm_mm->has_pinned))) +if (likely (!test_bit (MMF_HAS_PINNED , &vma->vm_mm->flags)))return false; page =vm_normal_page (vma, addr, pte);if (!page)
include/linux/mm.h renamed from include/linux/mm.h~mm-gup-pack-has_pinned-in-mmf_has_pinned +1 -1
@@ -1341,7 +1341,7 @@ static inline bool page_needs_cow_for_dmif (!is_cow_mapping (vma->vm_flags))return false; -if (!atomic_read (&vma->vm_mm->has_pinned)) +if (!test_bit (MMF_HAS_PINNED , &vma->vm_mm->flags))return false;return page_maybe_dma_pinned (page);
include/linux/mm_types.h renamed from include/linux/mm_types.h~mm-gup-pack-has_pinned-in-mmf_has_pinned +0 -10
@@ -435,16 +435,6 @@ struct mm_struct {*/ atomic_t mm_count; -/** -* @has_pinned: Whether this mm has pinned any pages. This can -* be either replaced in the future by @pinned_vm when it -* becomes stable, or grow into a counter on its own. We're -* aggresive on this bit now - even if the pinned pages were -* unpinned later on, we'll still keep this bit set for the -* lifecycle of this mm just for simplicity. -*/ -atomic_t has_pinned; -#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* PTE page table pages */ #endif
include/linux/sched/coredump.h renamed from include/linux/sched/coredump.h~mm-gup-pack-has_pinned-in-mmf_has_pinned +8 -0
@@ -73,6 +73,14 @@ static inline int get_dumpable(struct mm#define MMF_OOM_VICTIM 25/* mm is the oom victim */ #define MMF_OOM_REAP_QUEUED 26/* mm was queued for oom_reaper */ #define MMF_MULTIPROCESS 27/* mm is shared between processes */ +/* +* MMF_HAS_PINNED: Whether this mm has pinned any pages. This can be either +* replaced in the future by mm.pinned_vm when it becomes stable, or grow into +* a counter on its own. We're aggresive on this bit for now: even if the +* pinned pages were unpinned later on, we'll still keep this bit set for the +* lifecycle of this mm, just for simplicity. +*/ +#define MMF_HAS_PINNED 28/* FOLL_PIN has run, never cleared */ #define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP)#define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\
kernel/fork.c renamed from kernel/fork.c~mm-gup-pack-has_pinned-in-mmf_has_pinned +0 -1
@@ -1029,7 +1029,6 @@ static struct mm_struct *mm_init(structmm_pgtables_bytes_init (mm); mm->map_count =0 ; mm->locked_vm =0 ; -atomic_set (&mm->has_pinned,0 );atomic64_set (&mm->pinned_vm,0 );memset (&mm->rss_stat,0 ,sizeof (mm->rss_stat));spin_lock_init (&mm->page_table_lock);
mm/gup.c renamed from mm/gup.c~mm-gup-pack-has_pinned-in-mmf_has_pinned +15 -4
@@ -420,6 +420,17 @@ void unpin_user_pages(struct page **page }EXPORT_SYMBOL (unpin_user_pages); +/* +* Set the MMF_HAS_PINNED if not set yet; after set it'll be there for the mm's +* lifecycle. Avoid setting the bit unless necessary, or it might cause write +* cache bouncing on large SMP machines for concurrent pinned gups. +*/ +static inline void mm_set_has_pinned_flag (unsigned long *mm_flags) +{ +if (!test_bit (MMF_HAS_PINNED , mm_flags)) +set_bit (MMF_HAS_PINNED , mm_flags); +} +#ifdef CONFIG_MMU static struct page *no_page_table (struct vm_area_struct *vma,unsigned int flags)
@@ -1320,8 +1331,8 @@ static __always_inline long __get_user_pBUG_ON (*locked !=1 ); } -if ((flags &FOLL_PIN ) && !atomic_read (&mm->has_pinned)) -atomic_set (&mm->has_pinned,1 ); +if (flags &FOLL_PIN ) +mm_set_has_pinned_flag (&mm->flags);/* * FOLL_PIN and FOLL_GET are mutually exclusive. Traditional behavior
@@ -2641,8 +2652,8 @@ static int internal_get_user_pages_fast( FOLL_FAST_ONLY)))return -EINVAL ; -if ((gup_flags &FOLL_PIN ) && !atomic_read (¤t->mm->has_pinned)) -atomic_set (¤t->mm->has_pinned,1 ); +if (gup_flags &FOLL_PIN ) +mm_set_has_pinned_flag (¤t->mm->flags);if (!(gup_flags &FOLL_FAST_ONLY ))might_lock_read (¤t->mm->mmap_lock);
_
Andrew Morton
maintainerPATCH 66/192 mm: pagewalk: fix walk for hugepage tables
In reply to Andrew MortonFrom: Christophe Leroy <[email protected]>
Subject: mm: pagewalk: fix walk for hugepage tables
Pagewalk ignores hugepd entries and walk down the tables as if it was traditionnal entries, leading to crazy result.
Add walk_hugepd_range() and use it to walk hugepage tables.
---
mm/pagewalk.c | 58 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 5 deletions(-)
mm/pagewalk.c renamed from mm/pagewalk.c~mm-pagewalk-fix-walk-for-hugepage-tables +53 -5
@@ -58,6 +58,45 @@ static int walk_pte_range(pmd_t *pmd, unreturn err; } +#ifdef CONFIG_ARCH_HAS_HUGEPD +static int walk_hugepd_range (hugepd_t *phpd,unsigned long addr, +unsigned long end,struct mm_walk *walk,int pdshift) +{ +int err =0 ; +const struct mm_walk_ops *ops = walk->ops; +int shift =hugepd_shift (*phpd); +int page_size =1 << shift; + +if (!ops->pte_entry) +return 0 ; + +if (addr & (page_size -1 )) +return 0 ; + +for (;;) { +pte_t *pte; + +spin_lock (&walk->mm->page_table_lock); + pte =hugepte_offset (*phpd, addr, pdshift); + err = ops->pte_entry (pte, addr, addr + page_size, walk); +spin_unlock (&walk->mm->page_table_lock); + +if (err) +break ; +if (addr >= end - page_size) +break ; + addr += page_size; + } +return err; +} +#else +static int walk_hugepd_range (hugepd_t *phpd,unsigned long addr, +unsigned long end,struct mm_walk *walk,int pdshift) +{ +return 0 ; +} +#endif +static int walk_pmd_range (pud_t *pud,unsigned long addr,unsigned long end,struct mm_walk *walk) {
@@ -108,7 +147,10 @@ again: goto again; } - err =walk_pte_range (pmd, addr, next, walk); +if (is_hugepd (__hugepd (pmd_val (*pmd)))) + err =walk_hugepd_range ((hugepd_t *)pmd, addr, next, walk,PMD_SHIFT ); +else + err =walk_pte_range (pmd, addr, next, walk);if (err)break ; }while (pmd++, addr = next, addr != end);
@@ -157,7 +199,10 @@ static int walk_pud_range(p4d_t *p4d, unif (pud_none (*pud)) goto again; - err =walk_pmd_range (pud, addr, next, walk); +if (is_hugepd (__hugepd (pud_val (*pud)))) + err =walk_hugepd_range ((hugepd_t *)pud, addr, next, walk,PUD_SHIFT ); +else + err =walk_pmd_range (pud, addr, next, walk);if (err)break ; }while (pud++, addr = next, addr != end);
@@ -189,7 +234,9 @@ static int walk_p4d_range(pgd_t *pgd, unif (err)break ; } -if (ops->pud_entry || ops->pmd_entry || ops->pte_entry) +if (is_hugepd (__hugepd (p4d_val (*p4d)))) + err =walk_hugepd_range ((hugepd_t *)p4d, addr, next, walk,P4D_SHIFT ); +else if (ops->pud_entry || ops->pmd_entry || ops->pte_entry) err =walk_pud_range (p4d, addr, next, walk);if (err)break ;
@@ -224,8 +271,9 @@ static int walk_pgd_range(unsigned longif (err)break ; } -if (ops->p4d_entry || ops->pud_entry || ops->pmd_entry || - ops->pte_entry) +if (is_hugepd (__hugepd (pgd_val (*pgd)))) + err =walk_hugepd_range ((hugepd_t *)pgd, addr, next, walk,PGDIR_SHIFT ); +else if (ops->p4d_entry || ops->pud_entry || ops->pmd_entry || ops->pte_entry) err =walk_p4d_range (pgd, addr, next, walk);if (err)break ;
_
Andrew Morton
maintainerPATCH 67/192 mm/swapfile: use percpu_ref to serialize against concurrent swapoff
In reply to Andrew MortonFrom: Miaohe Lin <[email protected]>
Subject: mm/swapfile: use percpu_ref to serialize against concurrent swapoff
Patch series "close various race windows for swap", v6.
When I was investigating the swap code, I found some possible race windows. This series aims to fix all these races. But using current get/put_swap_device() to guard against concurrent swapoff for swap_readpage() looks terrible because swap_readpage() may take really long time. And to reduce the performance overhead on the hot-path as much as possible, it appears we can use the percpu_ref to close this race window(as suggested by Huang, Ying). The patch 1 adds percpu_ref support for swap and most of the remaining patches try to use this to close various race windows. More details can be found in the respective changelogs.
This patch (of 4):
Using current get/put_swap_device() to guard against concurrent swapoff for some swap ops, e.g. swap_readpage(), looks terrible because they might take really long time. This patch adds the percpu_ref support to serialize against concurrent swapoff(as suggested by Huang, Ying). Also we remove the SWP_VALID flag because it's used together with RCU solution.
---
include/linux/swap.h | 5 +- mm/swapfile.c | 79 +++++++++++++++++++++++++---------------- 2 files changed, 52 insertions(+), 32 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~mm-swapfile-use-percpu_ref-to-serialize-against-concurrent-swapoff +3 -2
@@ -177,7 +177,6 @@ enum {SWP_PAGE_DISCARD = (1 <<10 ),/* freed swap page-cluster discards */ SWP_STABLE_WRITES = (1 <<11 ),/* no overwrite PG_writeback pages */ SWP_SYNCHRONOUS_IO = (1 <<12 ),/* synchronous IO is efficient */ -SWP_VALID = (1 <<13 ),/* swap is valid to be operated on? */ /* add others here before... */ SWP_SCANNING = (1 << 14), /* refcount in scan_swap_map */ };
@@ -240,6 +239,7 @@ struct swap_cluster_list {* The in-memory structure used to track swap areas. */ struct swap_info_struct { +struct percpu_ref users;/* indicate and keep swap device valid. */ unsigned long flags;/* SWP_USED etc: see above */ signed short prio;/* swap priority of this type */ struct plist_node list;/* entry in swap_active_head */
@@ -260,6 +260,7 @@ struct swap_info_struct {struct block_device *bdev;/* swap device or bdev of swap file */ struct file *swap_file;/* seldom referenced */ unsigned int old_block_size;/* seldom referenced */ +struct completion comp;/* seldom referenced */ #ifdef CONFIG_FRONTSWAP unsigned long *frontswap_map;/* frontswap in-use, one bit per page */ atomic_t frontswap_pages;/* frontswap pages in-use counter */
@@ -511,7 +512,7 @@ sector_t swap_page_sector(struct page *pstatic inline void put_swap_device (struct swap_info_struct *si) { -rcu_read_unlock (); +percpu_ref_put (&si->users); }#else /* CONFIG_SWAP */
mm/swapfile.c renamed from mm/swapfile.c~mm-swapfile-use-percpu_ref-to-serialize-against-concurrent-swapoff +49 -30
154 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 68/192 swap: fix do_swap_page() race with swapoff
In reply to Andrew MortonFrom: Miaohe Lin <[email protected]>
Subject: swap: fix do_swap_page() race with swapoff
When I was investigating the swap code, I found the below possible race window:
CPU 1 CPU 2
----- -----do_swap_page
if (data_race(si->flags & SWP_SYNCHRONOUS_IO)
swap_readpage
if (data_race(sis->flags & SWP_FS_OPS)) {
swapoff
..
p->swap_file = NULL;
..
struct file *swap_file = sis->swap_file;
struct address_space *mapping = swap_file->f_mapping;[oops!]Note that for the pages that are swapped in through swap cache, this isn't an issue. Because the page is locked, and the swap entry will be marked with SWAP_HAS_CACHE, so swapoff() can not proceed until the page has been unlocked.
Fix this race by using get/put_swap_device() to guard against concurrent swapoff.
---
include/linux/swap.h | 9 +++++++++ mm/memory.c | 11 +++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~swap-fix-do_swap_page-race-with-swapoff +9 -0
@@ -527,6 +527,15 @@ static inline struct swap_info_struct *sreturn NULL ; } +static inline struct swap_info_struct *get_swap_device (swp_entry_t entry) +{ +return NULL ; +} + +static inline void put_swap_device (struct swap_info_struct *si) +{ +} +#define swap_address_space (entry) (NULL)#define get_nr_swap_pages () 0L#define total_swap_pages 0L
mm/memory.c renamed from mm/memory.c~swap-fix-do_swap_page-race-with-swapoff +9 -2
@@ -3353,6 +3353,7 @@ vm_fault_t do_swap_page(struct vm_fault {struct vm_area_struct *vma = vmf->vma;struct page *page =NULL , *swapcache; +struct swap_info_struct *si =NULL ;swp_entry_t entry;pte_t pte;int locked;
@@ -3380,14 +3381,16 @@ vm_fault_t do_swap_page(struct vm_fault goto out; } +/* Prevent swapoff from happening to us. */ + si =get_swap_device (entry); +if (unlikely (!si)) + goto out;delayacct_set_flag (current,DELAYACCT_PF_SWAPIN ); page =lookup_swap_cache (entry, vma, vmf->address); swapcache = page;if (!page) { -struct swap_info_struct *si =swp_swap_info (entry); -if (data_race (si->flags &SWP_SYNCHRONOUS_IO ) &&__swap_count (entry) ==1 ) {/* skip swapcache */
@@ -3556,6 +3559,8 @@ vm_fault_t do_swap_page(struct vm_fault unlock:pte_unmap_unlock (vmf->pte, vmf->ptl); out: +if (si) +put_swap_device (si);return ret; out_nomap:pte_unmap_unlock (vmf->pte, vmf->ptl);
@@ -3567,6 +3572,8 @@ out_release:unlock_page (swapcache);put_page (swapcache); } +if (si) +put_swap_device (si);return ret; }
_
Andrew Morton
maintainerPATCH 69/192 mm/swap: remove confusing checking for non_swap_entry() in swap_ra_info()
In reply to Andrew MortonFrom: Miaohe Lin <[email protected]>
Subject: mm/swap: remove confusing checking for non_swap_entry() in swap_ra_info()
The non_swap_entry() was used for working with VMA based swap readahead via commit ec560175c0b6 ("mm, swap: VMA based swap readahead"). At that time, the non_swap_entry() checking is necessary because the function is called before checking that in do_swap_page(). Then it's moved to swap_ra_info() since commit eaf649ebc3ac ("mm: swap: clean up swap readahead"). After that, the non_swap_entry() checking is unnecessary, because swap_ra_info() is called after non_swap_entry() has been checked already. The resulting code is confusing as the non_swap_entry() check looks racy now because while we released the pte lock, somebody else might have faulted in this pte. So we should check whether it's swap pte first to guard against such race or swap_type will be unexpected. But the race isn't important because it will not cause problem. We would have enough checking when we really operate the PTE entries later. So we remove the non_swap_entry() check here to avoid confusion.
---
mm/swap_state.c | 6 ------ 1 file changed, 6 deletions(-)
mm/swap_state.c renamed from mm/swap_state.c~mm-swap-remove-confusing-checking-for-non_swap_entry-in-swap_ra_info +0 -6
@@ -721,7 +721,6 @@ static void swap_ra_info(struct vm_fault {struct vm_area_struct *vma = vmf->vma;unsigned long ra_val; -swp_entry_t entry;unsigned long faddr, pfn, fpfn;unsigned long start, end;pte_t *pte, *orig_pte;
@@ -739,11 +738,6 @@ static void swap_ra_info(struct vm_fault faddr = vmf->address; orig_pte = pte =pte_offset_map (vmf->pmd, faddr); - entry =pte_to_swp_entry (*pte); -if ((unlikely (non_swap_entry (entry)))) { -pte_unmap (orig_pte); -return ; - } fpfn =PFN_DOWN (faddr); ra_val =GET_SWAP_RA_VAL (vma);
_
Andrew Morton
maintainerPATCH 70/192 mm/shmem: fix shmem_swapin() race with swapoff
In reply to Andrew MortonFrom: Miaohe Lin <[email protected]>
Subject: mm/shmem: fix shmem_swapin() race with swapoff
When I was investigating the swap code, I found the below possible race window:
CPU 1 CPU 2
----- -----shmem_swapin
swap_cluster_readahead
if (likely(si->flags & (SWP_BLKDEV | SWP_FS_OPS))) {
swapoff
..
si->swap_file = NULL;
..
struct inode *inode = si->swap_file->f_mapping->host;[oops!]Close this race window by using get/put_swap_device() to guard against concurrent swapoff.
---
mm/shmem.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
mm/shmem.c renamed from mm/shmem.c~mm-shmem-fix-shmem_swapin-race-with-swapoff +13 -1
@@ -1696,7 +1696,8 @@ static int shmem_swapin_page(struct inodstruct address_space *mapping = inode->i_mapping;struct shmem_inode_info *info =SHMEM_I (inode);struct mm_struct *charge_mm = vma ? vma->vm_mm : current->mm; -struct page *page; +struct swap_info_struct *si; +struct page *page =NULL ;swp_entry_t swap;int error;
@@ -1704,6 +1705,12 @@ static int shmem_swapin_page(struct inod swap =radix_to_swp_entry (*pagep); *pagep =NULL ; +/* Prevent swapoff from happening to us. */ + si =get_swap_device (swap); +if (!si) { + error =EINVAL ; + goto failed; + }/* Look it up and read it in.. */ page =lookup_swap_cache (swap,NULL ,0 );if (!page) {
@@ -1765,6 +1772,8 @@ static int shmem_swapin_page(struct inodswap_free (swap); *pagep = page; +if (si) +put_swap_device (si);return 0 ; failed:if (!shmem_confirm_swap (mapping, index, swap))
@@ -1775,6 +1784,9 @@ unlock:put_page (page); } +if (si) +put_swap_device (si); +return error; }
_
Andrew Morton
maintainerPATCH 71/192 mm/swapfile: move get_swap_page_of_type() under CONFIG_HIBERNATION
In reply to Andrew MortonFrom: Miaohe Lin <[email protected]>
Subject: mm/swapfile: move get_swap_page_of_type() under CONFIG_HIBERNATION
Patch series "Cleanups for swap", v2.
This series contains just cleanups to remove some unused variables, delete meaningless forward declarations and so on. More details can be found in the respective changelogs.
This patch (of 4):
We should move get_swap_page_of_type() under CONFIG_HIBERNATION since the only caller of this function is now suspend routine.
---
mm/swapfile.c | 83 +++++++++++++++++------------------------------- 1 file changed, 31 insertions(+), 52 deletions(-)
mm/swapfile.c renamed from mm/swapfile.c~mm-swapfile-move-get_swap_page_of_type-under-config_hibernation +31 -52
@@ -453,10 +453,10 @@ static void swap_cluster_schedule_discar unsigned int idx) {/* -* If scan_swap_map() can't find a free cluster, it will check +* If scan_swap_map_slots() can't find a free cluster, it will check * si->swap_map directly. To make sure the discarding cluster isn't -* taken by scan_swap_map(), mark the swap entries bad (occupied). It -* will be cleared after discard +* taken by scan_swap_map_slots(), mark the swap entries bad (occupied). +* It will be cleared after discard */ memset (si->swap_map + idx *SWAPFILE_CLUSTER ,SWAP_MAP_BAD ,SWAPFILE_CLUSTER );
@@ -589,7 +589,7 @@ static void dec_cluster_info_page(struct }/* -* It's possible scan_swap_map() uses a free cluster in the middle of free +* It's possible scan_swap_map_slots() uses a free cluster in the middle of free * cluster list. Avoiding such abuse to avoid list corruption. */ static bool
@@ -1037,21 +1037,6 @@ static void swap_free_cluster(struct swaswap_range_free (si, offset,SWAPFILE_CLUSTER ); } -static unsigned long scan_swap_map (struct swap_info_struct *si, -unsigned char usage) -{ -swp_entry_t entry; -int n_ret; - - n_ret =scan_swap_map_slots (si, usage,1 , &entry); - -if (n_ret) -return swp_offset (entry); -else -return 0 ; - -} -int get_swap_pages (int n_goal,swp_entry_t swp_entries[],int entry_size) {unsigned long size =swap_entry_size (entry_size);
@@ -1114,14 +1099,14 @@ start_over: nextsi:/* * if we got here, it's likely that si was almost full before, -* and since scan_swap_map() can drop the si->lock, multiple -* callers probably all tried to get a page from the same si -* and it filled up before we could get one; or, the si filled -* up between us dropping swap_avail_lock and taking si->lock. -* Since we dropped the swap_avail_lock, the swap_avail_head -* list may have been modified; so if next is still in the -* swap_avail_head list then try it, otherwise start over -* if we have not gotten any slots. +* and since scan_swap_map_slots() can drop the si->lock, +* multiple callers probably all tried to get a page from the +* same si and it filled up before we could get one; or, the si +* filled up between us dropping swap_avail_lock and taking +* si->lock. Since we dropped the swap_avail_lock, the +* swap_avail_head list may have been modified; so if next is +* still in the swap_avail_head list then try it, otherwise +* start over if we have not gotten any slots. */ if (plist_node_empty (&next->avail_lists[node])) goto start_over;
@@ -1137,30 +1122,6 @@ noswap:return n_ret; } -/* The only caller of this function is now suspend routine */ -swp_entry_t get_swap_page_of_type (int type) -{ -struct swap_info_struct *si =swap_type_to_swap_info (type); -pgoff_t offset; - -if (!si) - goto fail; - -spin_lock (&si->lock); -if (si->flags &SWP_WRITEOK ) { -/* This is called for allocating swap entry, not cache */ - offset =scan_swap_map (si,1 ); -if (offset) { -atomic_long_dec (&nr_swap_pages); -spin_unlock (&si->lock); -return swp_entry (type, offset); - } - } -spin_unlock (&si->lock); -fail: -return (swp_entry_t ) {0 }; -} -static struct swap_info_struct *__swap_info_get (swp_entry_t entry) {struct swap_info_struct *p;
@@ -1812,6 +1773,24 @@ int free_swap_and_cache(swp_entry_t entr }#ifdef CONFIG_HIBERNATION + +swp_entry_t get_swap_page_of_type (int type) +{ +struct swap_info_struct *si =swap_type_to_swap_info (type); +swp_entry_t entry = {0 }; + +if (!si) + goto fail; + +/* This is called for allocating swap entry, not cache */ +spin_lock (&si->lock); +if ((si->flags &SWP_WRITEOK ) &&scan_swap_map_slots (si,1 ,1 , &entry)) +atomic_long_dec (&nr_swap_pages); +spin_unlock (&si->lock); +fail: +return entry; +} +/* * Find the swap type that corresponds to given device (if any). *
@@ -2649,7 +2628,7 @@ SYSCALL_DEFINE1(swapoff, const char __usspin_lock (&p->lock);drain_mmlist (); -/* wait for anyone still in scan_swap_map */ +/* wait for anyone still in scan_swap_map_slots */ p->highest_bit =0 ;/* cuts scans short */ while (p->flags >=SWP_SCANNING ) {spin_unlock (&p->lock);
_
Andrew Morton
maintainerPATCH 72/192 mm/swap: remove unused local variable nr_shadows
In reply to Andrew MortonFrom: Miaohe Lin <[email protected]>
Subject: mm/swap: remove unused local variable nr_shadows
Since commit 55c653b71e8c ("mm: stop accounting shadow entries"), nr_shadows is not used anymore.
---
mm/swap_state.c | 5 ----- 1 file changed, 5 deletions(-)
mm/swap_state.c renamed from mm/swap_state.c~mm-swap-remove-unused-local-variable-nr_shadows +0 -5
@@ -114,8 +114,6 @@ int add_to_swap_cache(struct page *page,SetPageSwapCache (page); do { - unsigned long nr_shadows = 0; - xas_lock_irq(&xas); xas_create_range(&xas); if (xas_error(&xas))
@@ -124,7 +122,6 @@ int add_to_swap_cache(struct page *page,VM_BUG_ON_PAGE (xas.xa_index != idx + i, page); old =xas_load (&xas);if (xa_is_value (old)) { - nr_shadows++;if (shadowp) *shadowp = old; }
@@ -260,7 +257,6 @@ void clear_shadow_from_swap_cache(int tyvoid *old;for (;;) { -unsigned long nr_shadows =0 ;swp_entry_t entry =swp_entry (type, curr);struct address_space *address_space =swap_address_space (entry);XA_STATE (xas, &address_space->i_pages, curr);
@@ -270,7 +266,6 @@ void clear_shadow_from_swap_cache(int tyif (!xa_is_value (old))continue ;xas_store (&xas,NULL ); - nr_shadows++; }xa_unlock_irq (&address_space->i_pages);
_
Andrew Morton
maintainerPATCH 73/192 mm/swap_slots.c: delete meaningless forward declarations
In reply to Andrew MortonFrom: Miaohe Lin <[email protected]>
Subject: mm/swap_slots.c: delete meaningless forward declarations
deactivate_swap_slots_cache() and reactivate_swap_slots_cache() are only called below their implementations. So these forward declarations are meaningless and should be removed.
---
mm/swap_slots.c | 2 -- 1 file changed, 2 deletions(-)
mm/swap_slots.c renamed from mm/swap_slots.c~mm-swap_slotsc-delete-meaningless-forward-declarations +0 -2
@@ -43,8 +43,6 @@ static DEFINE_MUTEX(swap_slots_cache_mutstatic DEFINE_MUTEX (swap_slots_cache_enable_mutex);static void __drain_swap_slots_cache (unsigned int type); -static void deactivate_swap_slots_cache (void ); -static void reactivate_swap_slots_cache (void );#define use_swap_slot_cache (swap_slot_cache_active && swap_slot_cache_enabled)#define SLOTS_CACHE 0x1
_
Andrew Morton
maintainerPATCH 74/192 mm, swap: remove unnecessary smp_rmb() in swap_type_to_swap_info()
In reply to Andrew MortonFrom: Huang Ying <[email protected]>
Subject: mm, swap: remove unnecessary smp_rmb() in swap_type_to_swap_info()
Before commit c10d38cc8d3e ("mm, swap: bounds check swap_info array accesses to avoid NULL derefs"), the typical code to reference the swap_info[] is as follows,
type = swp_type(swp_entry);
if (type >= nr_swapfiles)
/* handle invalid swp_entry */;
p = swap_info[type];
/* access fields of *p. OOPS! p may be NULL! */Because the ordering isn't guaranteed, it's possible that swap_info[type] is read before "nr_swapfiles". And that may result in NULL pointer dereference.
So after commit c10d38cc8d3e, the code becomes,
struct swap_info_struct *swap_type_to_swap_info(int type)
{
if (type >= READ_ONCE(nr_swapfiles))
return NULL;
smp_rmb();
return READ_ONCE(swap_info[type]);}
/* users */
type = swp_type(swp_entry);
p = swap_type_to_swap_info(type);
if (!p)
/* handle invalid swp_entry */;
/* dereference p */
Where the value of swap_info[type] (that is, "p") is checked to be non-zero before being dereferenced. So, the NULL deferencing becomes impossible even if "nr_swapfiles" is read after swap_info[type]. Therefore, the "smp_rmb()" becomes unnecessary.
And, we don't even need to read "nr_swapfiles" here. Because the non-zero checking for "p" is sufficient. We just need to make sure we will not access out of the boundary of the array. With the change, nr_swapfiles will only be accessed with swap_lock held, except in swapcache_free_entries(). Where the absolute correctness of the value isn't needed, as described in the comments.
We still need to guarantee swap_info[type] is read before being dereferenced. That can be satisfied via the data dependency ordering enforced by READ_ONCE(swap_info[type]). This needs to be paired with proper write barriers. So smp_store_release() is used in alloc_swap_info() to guarantee the fields of *swap_info[type] is initialized before swap_info[type] itself being written. Note that the fields of *swap_info[type] is initialized to be 0 via kvzalloc() firstly. The assignment and deferencing of swap_info[type] is like rcu_assign_pointer() and rcu_dereference().
---
mm/swapfile.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)
mm/swapfile.c renamed from mm/swapfile.c~mm-swap-remove-unnecessary-smp_rmb-in-swap_type_to_swap_info +6 -9
@@ -100,11 +100,10 @@ atomic_t nr_rotate_swap = ATOMIC_INIT(0)static struct swap_info_struct *swap_type_to_swap_info (int type) { -if (type >=READ_ONCE (nr_swapfiles)) +if (type >=MAX_SWAPFILES )return NULL ; -smp_rmb ();/* Pairs with smp_wmb in alloc_swap_info. */ -return READ_ONCE (swap_info[type]); +return READ_ONCE (swap_info[type]);/* rcu_dereference() */ }static inline unsigned char swap_count (unsigned char ent)
@@ -2863,14 +2862,12 @@ static struct swap_info_struct *alloc_sw }if (type >= nr_swapfiles) { p->type = type; -WRITE_ONCE (swap_info[type], p);/* -* Write swap_info[type] before nr_swapfiles, in case a -* racing procfs swap_start() or swap_next() is reading them. -* (We never shrink nr_swapfiles, we never free this entry.) +* Publish the swap_info_struct after initializing it. +* Note that kvzalloc() above zeroes all its fields. */ -smp_wmb (); -WRITE_ONCE (nr_swapfiles, nr_swapfiles +1 ); +smp_store_release (&swap_info[type], p);/* rcu_assign_pointer() */ + nr_swapfiles++; }else { defer = p; p = swap_info[type];
_
Andrew Morton
maintainerPATCH 75/192 mm: free idle swap cache page after COW
In reply to Andrew MortonFrom: Huang Ying <[email protected]>
Subject: mm: free idle swap cache page after COW
With commit 09854ba94c6a ("mm: do_wp_page() simplification"), after COW, the idle swap cache page (neither the page nor the corresponding swap entry is mapped by any process) will be left in the LRU list, even if it's in the active list or the head of the inactive list. So, the page reclaimer may take quite some overhead to reclaim these actually unused pages.
To help the page reclaiming, in this patch, after COW, the idle swap cache page will be tried to be freed. To avoid to introduce much overhead to the hot COW code path,
a) there's almost zero overhead for non-swap case via checking PageSwapCache() firstly.
b) the page lock is acquired via trylock only.
To test the patch, we used pmbench memory accessing benchmark with working-set larger than available memory on a 2-socket Intel server with a NVMe SSD as swap device. Test results shows that the pmbench score increases up to 23.8% with the decreased size of swap cache and swapin throughput.
---
include/linux/swap.h | 5 +++++ mm/memory.c | 2 ++ mm/swap_state.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-)
include/linux/swap.h renamed from include/linux/swap.h~mm-free-idle-swap-cache-page-after-cow +5 -0
@@ -446,6 +446,7 @@ extern void __delete_from_swap_cache(strextern void delete_from_swap_cache (struct page *);extern void clear_shadow_from_swap_cache (int type,unsigned long begin,unsigned long end); +extern void free_swap_cache (struct page *);extern void free_page_and_swap_cache (struct page *);extern void free_pages_and_swap_cache (struct page **,int ); extern struct page *lookup_swap_cache(swp_entry_t entry,
@@ -551,6 +552,10 @@ static inline void put_swap_device(struc#define free_pages_and_swap_cache (pages, nr) \ release_pages((pages), (nr)); +static inline void free_swap_cache (struct page *page) +{ +} +static inline void show_swap_cache_info (void ) { }
mm/memory.c renamed from mm/memory.c~mm-free-idle-swap-cache-page-after-cow +2 -0
@@ -3023,6 +3023,8 @@ static vm_fault_t wp_page_copy(struct vmmunlock_vma_page (old_page);unlock_page (old_page); } +if (page_copied) +free_swap_cache (old_page);put_page (old_page); }return page_copied ?VM_FAULT_WRITE :0 ;
mm/swap_state.c renamed from mm/swap_state.c~mm-free-idle-swap-cache-page-after-cow +1 -1
@@ -286,7 +286,7 @@ void clear_shadow_from_swap_cache(int ty* try_to_free_swap() _with_ the lock. * - Marcelo */ -static inline void free_swap_cache (struct page *page) +void free_swap_cache (struct page *page) {if (PageSwapCache (page) && !page_mapped (page) &&trylock_page (page)) {try_to_free_swap (page);
_
Andrew Morton
maintainerPATCH 76/192 swap: check mapping_empty() for swap cache before being freed
In reply to Andrew MortonFrom: Huang Ying <[email protected]>
Subject: swap: check mapping_empty() for swap cache before being freed
To check whether all pages and shadow entries in swap cache has been removed before swap cache is freed.
---
mm/swap_state.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/swap_state.c renamed from mm/swap_state.c~swap-check-mapping_empty-for-swap-cache-before-being-freed +6 -1
@@ -693,7 +693,12 @@ int init_swap_address_space(unsigned intvoid exit_swap_address_space (unsigned int type) { -kvfree (swapper_spaces[type]); +int i; +struct address_space *spaces = swapper_spaces[type]; + +for (i =0 ; i < nr_swapper_spaces[type]; i++) +VM_WARN_ON_ONCE (!mapping_empty (&spaces[i])); +kvfree (spaces); nr_swapper_spaces[type] =0 ; swapper_spaces[type] =NULL ; }
_
Andrew Morton
maintainerPATCH 77/192 mm/memcg: move mod_objcg_state() to memcontrol.c
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: mm/memcg: move mod_objcg_state() to memcontrol.c
Patch series "mm/memcg: Reduce kmemcache memory accounting overhead", v6.
With the recent introduction of the new slab memory controller, we eliminate the need for having separate kmemcaches for each memory cgroup and reduce overall kernel memory usage. However, we also add additional memory accounting overhead to each call of kmem_cache_alloc() and kmem_cache_free().
For workloads that require a lot of kmemcache allocations and de-allocations, they may experience performance regression as illustrated in [1] and [2].
A simple kernel module that performs repeated loop of 100,000,000 kmem_cache_alloc() and kmem_cache_free() of either a small 32-byte object or a big 4k object at module init time with a batch size of 4 (4 kmalloc's followed by 4 kfree's) is used for benchmarking. The benchmarking tool was run on a kernel based on linux-next-20210419. The test was run on a CascadeLake server with turbo-boosting disable to reduce run-to-run variation.
The small object test exercises mainly the object stock charging and vmstat update code paths. The large object test also exercises the refill_obj_stock() and __memcg_kmem_charge()/__memcg_kmem_uncharge() code paths.
With memory accounting disabled, the run time was 3.130s with both small object big object tests.
With memory accounting enabled, both cgroup v1 and v2 showed similar results in the small object test. The performance results of the large object test, however, differed between cgroup v1 and v2.
The execution times with the application of various patches in the patchset were:
Applied patches Run time Accounting overhead %age 1 %age 2 --------------- -------- ------------------- ------ ------
Small 32-byte object:
None 11.634s 8.504s 100.0% 271.7%
1-2 9.425s 6.295s 74.0% 201.1%
1-3 9.708s 6.578s 77.4% 210.2%
1-4 8.062s 4.932s 58.0% 157.6%Large 4k object (v2):
None 22.107s 18.977s 100.0% 606.3%
1-2 20.960s 17.830s 94.0% 569.6%
1-3 14.238s 11.108s 58.5% 354.9%
1-4 11.329s 8.199s 43.2% 261.9%Large 4k object (v1):
None 36.807s 33.677s 100.0% 1075.9%
1-2 36.648s 33.518s 99.5% 1070.9%
1-3 22.345s 19.215s 57.1% 613.9%
1-4 18.662s 15.532s 46.1% 496.2% N.B. %age 1 = overhead/unpatched overhead
%age 2 = overhead/accounting disabled time
Patch 2 (vmstat data stock caching) helps in both the small object test and the large v2 object test. It doesn't help much in v1 big object test.
Patch 3 (refill_obj_stock improvement) does help the small object test but offer significant performance improvement for the large object test (both v1 and v2).
Patch 4 (eliminating irq disable/enable) helps in all test cases.
To test for the extreme case, a multi-threaded kmalloc/kfree microbenchmark was run on the 2-socket 48-core 96-thread system with 96 testing threads in the same memcg doing kmalloc+kfree of a 4k object with accounting enabled for 10s. The total number of kmalloc+kfree done in kilo operations per second (kops/s) were as follows:
Applied patches v1 kops/s v1 change v2 kops/s v2 change
--------------- --------- --------- --------- ---------
None 3,520 1.00X 6,242 1.00X
1-2 4,304 1.22X 8,478 1.36X
1-3 4,731 1.34X 418,142 66.99X
1-4 4,587 1.30X 438,838 70.30XWith memory accounting disabled, the kmalloc/kfree rate was 1,481,291 kop/s. This test shows how significant the memory accouting overhead can be in some extreme situations.
For this multithreaded test, the improvement from patch 2 mainly comes from the conditional atomic xchg of objcg->nr_charged_bytes in mod_objcg_state(). By using an unconditional xchg, the operation rates were similar to the unpatched kernel.
Patch 3 elminates the single highly contended cacheline of objcg->nr_charged_bytes for cgroup v2 leading to a huge performance improvement. Cgroup v1, however, still has another highly contended cacheline in the shared page counter &memcg->kmem. So the improvement is only modest.
Patch 4 helps in cgroup v2, but performs worse in cgroup v1 as eliminating the irq_disable/irq_enable overhead seems to aggravate the cacheline contention.
[1] https://lore.kernel.org/linux-mm/20210408193948.vfktg3azh2wrt56t@gabell/T/#u
[2] https://lore.kernel.org/lkml/20210114025151.GA22932@xsang-OptiPlex-9020/
This patch (of 4):
mod_objcg_state() is moved from mm/slab.h to mm/memcontrol.c so that further optimization can be done to it in later patches without exposing unnecessary details to other mm components.
---
mm/memcontrol.c | 13 +++++++++++++ mm/slab.h | 16 ++-------------- 2 files changed, 15 insertions(+), 14 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-move-mod_objcg_state-to-memcontrolc +13 -0
@@ -782,6 +782,19 @@ void __mod_lruvec_kmem_state(void *p, enrcu_read_unlock (); } +void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, +enum node_stat_item idx,int nr) +{ +struct mem_cgroup *memcg; +struct lruvec *lruvec; + +rcu_read_lock (); + memcg =obj_cgroup_memcg (objcg); + lruvec =mem_cgroup_lruvec (memcg, pgdat); +mod_memcg_lruvec_state (lruvec, idx, nr); +rcu_read_unlock (); +} +/** * __count_memcg_events - account VM events in a cgroup * @memcg: the memory cgroup
mm/slab.h renamed from mm/slab.h~mm-memcg-move-mod_objcg_state-to-memcontrolc +2 -14
@@ -240,6 +240,8 @@ static inline bool kmem_cache_debug_flag#ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page); +void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, +enum node_stat_item idx,int nr);static inline void memcg_free_page_obj_cgroups (struct page *page) {
@@ -284,20 +286,6 @@ static inline bool memcg_slab_pre_alloc_return true; } -static inline void mod_objcg_state (struct obj_cgroup *objcg, -struct pglist_data *pgdat, -enum node_stat_item idx,int nr) -{ -struct mem_cgroup *memcg; -struct lruvec *lruvec; - -rcu_read_lock (); - memcg =obj_cgroup_memcg (objcg); - lruvec =mem_cgroup_lruvec (memcg, pgdat); -mod_memcg_lruvec_state (lruvec, idx, nr); -rcu_read_unlock (); -} - static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, gfp_t flags, size_t size,
_
Andrew Morton
maintainerPATCH 78/192 mm/memcg: cache vmstat data in percpu memcg_stock_pcp
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: mm/memcg: cache vmstat data in percpu memcg_stock_pcp
Before the new slab memory controller with per object byte charging, charging and vmstat data update happen only when new slab pages are allocated or freed. Now they are done with every kmem_cache_alloc() and kmem_cache_free(). This causes additional overhead for workloads that generate a lot of alloc and free calls.
The memcg_stock_pcp is used to cache byte charge for a specific obj_cgroup to reduce that overhead. To further reducing it, this patch makes the vmstat data cached in the memcg_stock_pcp structure as well until it accumulates a page size worth of update or when other cached data change. Caching the vmstat data in the per-cpu stock eliminates two writes to non-hot cachelines for memcg specific as well as memcg-lruvecs specific vmstat data by a write to a hot local stock cacheline.
On a 2-socket Cascade Lake server with instrumentation enabled and this patch applied, it was found that about 20% (634400 out of 3243830) of the time when mod_objcg_state() is called leads to an actual call to __mod_objcg_state() after initial boot. When doing parallel kernel build, the figure was about 17% (24329265 out of 142512465). So caching the vmstat data reduces the number of calls to __mod_objcg_state() by more than 80%.
---
mm/memcontrol.c | 90 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 87 insertions(+), 3 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-cache-vmstat-data-in-percpu-memcg_stock_pcp +87 -3
@@ -782,8 +782,9 @@ void __mod_lruvec_kmem_state(void *p, enrcu_read_unlock (); } -void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, -enum node_stat_item idx,int nr) +static inline void mod_objcg_mlstate (struct obj_cgroup *objcg, +struct pglist_data *pgdat, +enum node_stat_item idx,int nr) {struct mem_cgroup *memcg;struct lruvec *lruvec;
@@ -791,7 +792,7 @@ void mod_objcg_state(struct obj_cgroup *rcu_read_lock (); memcg =obj_cgroup_memcg (objcg); lruvec =mem_cgroup_lruvec (memcg, pgdat); -mod_memcg_lruvec_state (lruvec, idx, nr); +__mod_memcg_lruvec_state (lruvec, idx, nr);rcu_read_unlock (); }
@@ -2059,7 +2060,10 @@ struct memcg_stock_pcp {#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cached_objcg; +struct pglist_data *cached_pgdat;unsigned int nr_bytes; +int nr_slab_reclaimable_b; +int nr_slab_unreclaimable_b;#endif struct work_struct work;
@@ -3008,6 +3012,67 @@ void __memcg_kmem_uncharge_page(struct pobj_cgroup_put (objcg); } +void mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat, +enum node_stat_item idx,int nr) +{ +struct memcg_stock_pcp *stock; +unsigned long flags; +int *bytes; + +local_irq_save (flags); + stock =this_cpu_ptr (&memcg_stock); + +/* +* Save vmstat data in stock and skip vmstat array update unless +* accumulating over a page of vmstat data or when pgdat or idx +* changes. +*/ +if (stock->cached_objcg != objcg) { +drain_obj_stock (stock); +obj_cgroup_get (objcg); + stock->nr_bytes =atomic_read (&objcg->nr_charged_bytes) + ?atomic_xchg (&objcg->nr_charged_bytes,0 ) :0 ; + stock->cached_objcg = objcg; + stock->cached_pgdat = pgdat; + }else if (stock->cached_pgdat != pgdat) { +/* Flush the existing cached vmstat data */ +if (stock->nr_slab_reclaimable_b) { +mod_objcg_mlstate (objcg, pgdat,NR_SLAB_RECLAIMABLE_B , + stock->nr_slab_reclaimable_b); + stock->nr_slab_reclaimable_b =0 ; + } +if (stock->nr_slab_unreclaimable_b) { +mod_objcg_mlstate (objcg, pgdat,NR_SLAB_UNRECLAIMABLE_B , + stock->nr_slab_unreclaimable_b); + stock->nr_slab_unreclaimable_b =0 ; + } + stock->cached_pgdat = pgdat; + } + + bytes = (idx ==NR_SLAB_RECLAIMABLE_B ) ? &stock->nr_slab_reclaimable_b + : &stock->nr_slab_unreclaimable_b; +/* +* Even for large object >= PAGE_SIZE, the vmstat data will still be +* cached locally at least once before pushing it out. +*/ +if (!*bytes) { + *bytes = nr; + nr =0 ; + }else { + *bytes += nr; +if (abs (*bytes) >PAGE_SIZE ) { + nr = *bytes; + *bytes =0 ; + }else { + nr =0 ; + } + } +if (nr) +mod_objcg_mlstate (objcg, pgdat, idx, nr); + +local_irq_restore (flags); +} +static bool consume_obj_stock (struct obj_cgroup *objcg,unsigned int nr_bytes) {struct memcg_stock_pcp *stock;
@@ -3055,6 +3120,25 @@ static void drain_obj_stock(struct memcg stock->nr_bytes =0 ; } +/* +* Flush the vmstat data in current stock +*/ +if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) { +if (stock->nr_slab_reclaimable_b) { +mod_objcg_mlstate (old, stock->cached_pgdat, +NR_SLAB_RECLAIMABLE_B , + stock->nr_slab_reclaimable_b); + stock->nr_slab_reclaimable_b =0 ; + } +if (stock->nr_slab_unreclaimable_b) { +mod_objcg_mlstate (old, stock->cached_pgdat, +NR_SLAB_UNRECLAIMABLE_B , + stock->nr_slab_unreclaimable_b); + stock->nr_slab_unreclaimable_b =0 ; + } + stock->cached_pgdat =NULL ; + } +obj_cgroup_put (old); stock->cached_objcg =NULL ; }
_
Andrew Morton
maintainerPATCH 79/192 mm/memcg: improve refill_obj_stock() performance
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: mm/memcg: improve refill_obj_stock() performance
There are two issues with the current refill_obj_stock() code. First of all, when nr_bytes reaches over PAGE_SIZE, it calls drain_obj_stock() to atomically flush out remaining bytes to obj_cgroup, clear cached_objcg and do a obj_cgroup_put(). It is likely that the same obj_cgroup will be used again which leads to another call to drain_obj_stock() and obj_cgroup_get() as well as atomically retrieve the available byte from obj_cgroup. That is costly. Instead, we should just uncharge the excess pages, reduce the stock bytes and be done with it. The drain_obj_stock() function should only be called when obj_cgroup changes.
Secondly, when charging an object of size not less than a page in obj_cgroup_charge(), it is possible that the remaining bytes to be refilled to the stock will overflow a page and cause refill_obj_stock() to uncharge 1 page. To avoid the additional uncharge in this case, a new allow_uncharge flag is added to refill_obj_stock() which will be set to false when called from obj_cgroup_charge() so that an uncharge_pages() call won't be issued right after a charge_pages() call unless the objcg changes.
A multithreaded kmalloc+kfree microbenchmark on a 2-socket 48-core 96-thread x86-64 system with 96 testing threads were run. Before this patch, the total number of kilo kmalloc+kfree operations done for a 4k large object by all the testing threads per second were 4,304 kops/s (cgroup v1) and 8,478 kops/s (cgroup v2). After applying this patch, the number were 4,731 (cgroup v1) and 418,142 (cgroup v2) respectively. This represents a performance improvement of 1.10X (cgroup v1) and 49.3X (cgroup v2).
---
mm/memcontrol.c | 48 +++++++++++++++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 13 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-improve-refill_obj_stock-performance +35 -13
@@ -3157,10 +3157,12 @@ static bool obj_stock_flush_required(strreturn false; } -static void refill_obj_stock (struct obj_cgroup *objcg,unsigned int nr_bytes) +static void refill_obj_stock (struct obj_cgroup *objcg,unsigned int nr_bytes, +bool allow_uncharge) {struct memcg_stock_pcp *stock;unsigned long flags; +unsigned int nr_pages =0 ;local_irq_save (flags);
@@ -3169,14 +3171,21 @@ static void refill_obj_stock(struct obj_drain_obj_stock (stock);obj_cgroup_get (objcg); stock->cached_objcg = objcg; - stock->nr_bytes =atomic_xchg (&objcg->nr_charged_bytes,0 ); + stock->nr_bytes =atomic_read (&objcg->nr_charged_bytes) + ?atomic_xchg (&objcg->nr_charged_bytes,0 ) :0 ; + allow_uncharge = true;/* Allow uncharge when objcg changes */ } stock->nr_bytes += nr_bytes; -if (stock->nr_bytes >PAGE_SIZE ) -drain_obj_stock (stock); +if (allow_uncharge && (stock->nr_bytes >PAGE_SIZE )) { + nr_pages = stock->nr_bytes >>PAGE_SHIFT ; + stock->nr_bytes &= (PAGE_SIZE -1 ); + }local_irq_restore (flags); + +if (nr_pages) +obj_cgroup_uncharge_pages (objcg, nr_pages); }int obj_cgroup_charge (struct obj_cgroup *objcg,gfp_t gfp,size_t size)
@@ -3188,14 +3197,27 @@ int obj_cgroup_charge(struct obj_cgroupreturn 0 ;/* -* In theory, memcg->nr_charged_bytes can have enough +* In theory, objcg->nr_charged_bytes can have enough * pre-charged bytes to satisfy the allocation. However, -* flushing memcg->nr_charged_bytes requires two atomic -* operations, and memcg->nr_charged_bytes can't be big, -* so it's better to ignore it and try grab some new pages. -* memcg->nr_charged_bytes will be flushed in -* refill_obj_stock(), called from this function or -* independently later. +* flushing objcg->nr_charged_bytes requires two atomic +* operations, and objcg->nr_charged_bytes can't be big. +* The shared objcg->nr_charged_bytes can also become a +* performance bottleneck if all tasks of the same memcg are +* trying to update it. So it's better to ignore it and try +* grab some new pages. The stock's nr_bytes will be flushed to +* objcg->nr_charged_bytes later on when objcg changes. +* +* The stock's nr_bytes may contain enough pre-charged bytes +* to allow one less page from being charged, but we can't rely +* on the pre-charged bytes not being changed outside of +* consume_obj_stock() or refill_obj_stock(). So ignore those +* pre-charged bytes as well when charging pages. To avoid a +* page uncharge right after a page charge, we set the +* allow_uncharge flag to false when calling refill_obj_stock() +* to temporarily allow the pre-charged bytes to exceed the page +* size limit. The maximum reachable value of the pre-charged +* bytes is (sizeof(object) + PAGE_SIZE - 2) if there is no data +* race. */ nr_pages = size >>PAGE_SHIFT ; nr_bytes = size & (PAGE_SIZE -1 );
@@ -3205,14 +3227,14 @@ int obj_cgroup_charge(struct obj_cgroup ret =obj_cgroup_charge_pages (objcg, gfp, nr_pages);if (!ret && nr_bytes) -refill_obj_stock (objcg,PAGE_SIZE - nr_bytes); +refill_obj_stock (objcg,PAGE_SIZE - nr_bytes, false); return ret; }void obj_cgroup_uncharge (struct obj_cgroup *objcg,size_t size) { -refill_obj_stock (objcg, size); +refill_obj_stock (objcg, size, true); }#endif /* CONFIG_MEMCG_KMEM */
_
Andrew Morton
maintainerPATCH 80/192 mm/memcg: optimize user context object stock access
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: mm/memcg: optimize user context object stock access
Most kmem_cache_alloc() calls are from user context. With instrumentation enabled, the measured amount of kmem_cache_alloc() calls from non-task context was about 0.01% of the total.
The irq disable/enable sequence used in this case to access content from object stock is slow. To optimize for user context access, there are now two sets of object stocks (in the new obj_stock structure) for task context and interrupt context access respectively.
The task context object stock can be accessed after disabling preemption which is cheap in non-preempt kernel. The interrupt context object stock can only be accessed after disabling interrupt. User context code can access interrupt object stock, but not vice versa.
The downside of this change is that there are more data stored in local object stocks and not reflected in the charge counter and the vmstat arrays. However, this is a small price to pay for better performance.
---
mm/memcontrol.c | 100 +++++++++++++++++++++++++++++++++------------- 1 file changed, 72 insertions(+), 28 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-optimize-user-context-object-stock-access +72 -28
@@ -782,6 +782,10 @@ void __mod_lruvec_kmem_state(void *p, enrcu_read_unlock (); } +/* +* mod_objcg_mlstate() may be called with irq enabled, so +* mod_memcg_lruvec_state() should be used. +*/ static inline void mod_objcg_mlstate (struct obj_cgroup *objcg,struct pglist_data *pgdat,enum node_stat_item idx,int nr)
@@ -792,7 +796,7 @@ static inline void mod_objcg_mlstate(strrcu_read_lock (); memcg =obj_cgroup_memcg (objcg); lruvec =mem_cgroup_lruvec (memcg, pgdat); -__mod_memcg_lruvec_state (lruvec, idx, nr); +mod_memcg_lruvec_state (lruvec, idx, nr);rcu_read_unlock (); }
@@ -2054,17 +2058,23 @@ void unlock_page_memcg(struct page *page }EXPORT_SYMBOL (unlock_page_memcg); -struct memcg_stock_pcp { -struct mem_cgroup *cached;/* this never be root cgroup */ -unsigned int nr_pages; - +struct obj_stock {#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cached_objcg;struct pglist_data *cached_pgdat;unsigned int nr_bytes;int nr_slab_reclaimable_b;int nr_slab_unreclaimable_b; +#else +int dummy[0 ];#endif +}; + +struct memcg_stock_pcp { +struct mem_cgroup *cached;/* this never be root cgroup */ +unsigned int nr_pages; +struct obj_stock task_obj; +struct obj_stock irq_obj;struct work_struct work;unsigned long flags;
@@ -2074,12 +2084,12 @@ static DEFINE_PER_CPU(struct memcg_stockstatic DEFINE_MUTEX (percpu_charge_mutex);#ifdef CONFIG_MEMCG_KMEM -static void drain_obj_stock (struct memcg_stock_pcp *stock); +static void drain_obj_stock (struct obj_stock *stock);static bool obj_stock_flush_required (struct memcg_stock_pcp *stock,struct mem_cgroup *root_memcg);#else -static inline void drain_obj_stock (struct memcg_stock_pcp *stock) +static inline void drain_obj_stock (struct obj_stock *stock) { } static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,
@@ -2089,6 +2099,41 @@ static bool obj_stock_flush_required(str }#endif +/* +* Most kmem_cache_alloc() calls are from user context. The irq disable/enable +* sequence used in this case to access content from object stock is slow. +* To optimize for user context access, there are now two object stocks for +* task context and interrupt context access respectively. +* +* The task context object stock can be accessed by disabling preemption only +* which is cheap in non-preempt kernel. The interrupt context object stock +* can only be accessed after disabling interrupt. User context code can +* access interrupt object stock, but not vice versa. +*/ +static inline struct obj_stock *get_obj_stock (unsigned long *pflags) +{ +struct memcg_stock_pcp *stock; + +if (likely (in_task ())) { + *pflags =0UL ; +preempt_disable (); + stock =this_cpu_ptr (&memcg_stock); +return &stock->task_obj; + } + +local_irq_save (*pflags); + stock =this_cpu_ptr (&memcg_stock); +return &stock->irq_obj; +} + +static inline void put_obj_stock (unsigned long flags) +{ +if (likely (in_task ())) +preempt_enable (); +else +local_irq_restore (flags); +} +/** * consume_stock: Try to consume stocked charge on this cpu. * @memcg: memcg to consume from.
@@ -2155,7 +2200,9 @@ static void drain_local_stock(struct worlocal_irq_save (flags); stock =this_cpu_ptr (&memcg_stock); -drain_obj_stock (stock); +drain_obj_stock (&stock->irq_obj); +if (in_task ()) +drain_obj_stock (&stock->task_obj);drain_stock (stock);clear_bit (FLUSHING_CACHED_CHARGE , &stock->flags);
@@ -3015,13 +3062,10 @@ void __memcg_kmem_uncharge_page(struct pvoid mod_objcg_state (struct obj_cgroup *objcg,struct pglist_data *pgdat,enum node_stat_item idx,int nr) { -struct memcg_stock_pcp *stock;unsigned long flags; +struct obj_stock *stock =get_obj_stock (&flags);int *bytes; -local_irq_save (flags); - stock =this_cpu_ptr (&memcg_stock); -/* * Save vmstat data in stock and skip vmstat array update unless * accumulating over a page of vmstat data or when pgdat or idx
@@ -3070,29 +3114,26 @@ void mod_objcg_state(struct obj_cgroup *if (nr)mod_objcg_mlstate (objcg, pgdat, idx, nr); -local_irq_restore (flags); +put_obj_stock (flags); }static bool consume_obj_stock (struct obj_cgroup *objcg,unsigned int nr_bytes) { -struct memcg_stock_pcp *stock;unsigned long flags; +struct obj_stock *stock =get_obj_stock (&flags);bool ret = false; -local_irq_save (flags); - - stock =this_cpu_ptr (&memcg_stock);if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; } -local_irq_restore (flags); +put_obj_stock (flags);return ret; } -static void drain_obj_stock (struct memcg_stock_pcp *stock) +static void drain_obj_stock (struct obj_stock *stock) {struct obj_cgroup *old = stock->cached_objcg;
@@ -3148,8 +3189,13 @@ static bool obj_stock_flush_required(str {struct mem_cgroup *memcg; -if (stock->cached_objcg) { - memcg =obj_cgroup_memcg (stock->cached_objcg); +if (in_task () && stock->task_obj.cached_objcg) { + memcg =obj_cgroup_memcg (stock->task_obj.cached_objcg); +if (memcg &&mem_cgroup_is_descendant (memcg, root_memcg)) +return true; + } +if (stock->irq_obj.cached_objcg) { + memcg =obj_cgroup_memcg (stock->irq_obj.cached_objcg);if (memcg &&mem_cgroup_is_descendant (memcg, root_memcg))return true; }
@@ -3160,13 +3206,10 @@ static bool obj_stock_flush_required(strstatic void refill_obj_stock (struct obj_cgroup *objcg,unsigned int nr_bytes,bool allow_uncharge) { -struct memcg_stock_pcp *stock;unsigned long flags; +struct obj_stock *stock =get_obj_stock (&flags);unsigned int nr_pages =0 ; -local_irq_save (flags); - - stock =this_cpu_ptr (&memcg_stock);if (stock->cached_objcg != objcg) {/* reset if necessary */ drain_obj_stock (stock);obj_cgroup_get (objcg);
@@ -3182,7 +3225,7 @@ static void refill_obj_stock(struct obj_ stock->nr_bytes &= (PAGE_SIZE -1 ); } -local_irq_restore (flags); +put_obj_stock (flags);if (nr_pages)obj_cgroup_uncharge_pages (objcg, nr_pages);
@@ -6790,6 +6833,7 @@ static void uncharge_page(struct page *punsigned long nr_pages;struct mem_cgroup *memcg;struct obj_cgroup *objcg; +bool use_objcg =PageMemcgKmem (page);VM_BUG_ON_PAGE (PageLRU (page), page);
@@ -6798,7 +6842,7 @@ static void uncharge_page(struct page *p* page memcg or objcg at this point, we have fully * exclusive access to the page. */ -if (PageMemcgKmem (page)) { +if (use_objcg) { objcg =__page_objcg (page);/* * This get matches the put at the end of the function and
@@ -6826,7 +6870,7 @@ static void uncharge_page(struct page *p nr_pages =compound_nr (page); -if (PageMemcgKmem (page)) { +if (use_objcg) { ug->nr_memory += nr_pages; ug->nr_kmem += nr_pages;
_
Andrew Morton
maintainerPATCH 81/192 mm: memcg/slab: properly set up gfp flags for objcg pointer array
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: mm: memcg/slab: properly set up gfp flags for objcg pointer array
Patch series "mm: memcg/slab: Fix objcg pointer array handling problem", v4.
Since the merging of the new slab memory controller in v5.9, the page structure stores a pointer to objcg pointer array for slab pages. When the slab has no used objects, it can be freed in free_slab() which will call kfree() to free the objcg pointer array in memcg_alloc_page_obj_cgroups(). If it happens that the objcg pointer array is the last used object in its slab, that slab may then be freed which may caused kfree() to be called again.
With the right workload, the slab cache may be set up in a way that allows the recursive kfree() calling loop to nest deep enough to cause a kernel stack overflow and panic the system. In fact, we have a reproducer that can cause kernel stack overflow on a s390 system involving kmalloc-rcl-256 and kmalloc-rcl-128 slabs with the following kfree() loop recursively called 74 times:
[ 285.520739] [<000000000ec432fc>] kfree+0x4bc/0x560 [ 285.520740] [<000000000ec43466>] __free_slab+0xc6/0x228 [ 285.520741] [<000000000ec41fc2>] __slab_free+0x3c2/0x3e0 [ 285.520742] [<000000000ec432fc>] kfree+0x4bc/0x560 : While investigating this issue, I
also found an issue on the allocation side. If the objcg pointer array happen to come from the same slab or a circular dependency linkage is formed with multiple slabs, those affected slabs can never be freed again.
This patch series addresses these two issues by introducing a new set of kmalloc-cg-<n> caches split from kmalloc-<n> caches. The new set will only contain non-reclaimable and non-dma objects that are accounted in memory cgroups whereas the old set are now for unaccounted objects only. By making this split, all the objcg pointer arrays will come from the kmalloc-<n> caches, but those caches will never hold any objcg pointer array. As a result, deeply nested kfree() call and the unfreeable slab problems are now gone.
This patch (of 4):
Since the merging of the new slab memory controller in v5.9, the page structure may store a pointer to obj_cgroup pointer array for slab pages. Currently, only the __GFP_ACCOUNT bit is masked off. However, the array is not readily reclaimable and doesn't need to come from the DMA buffer. So those GFP bits should be masked off as well.
Do the flag bit clearing at memcg_alloc_page_obj_cgroups() to make sure that it is consistently applied no matter where it is called.
---
mm/memcontrol.c | 8 ++++++++ mm/slab.h | 1 - 2 files changed, 8 insertions(+), 1 deletion(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-slab-properly-set-up-gfp-flags-for-objcg-pointer-array +8 -0
@@ -2803,6 +2803,13 @@ retry: }#ifdef CONFIG_MEMCG_KMEM +/* +* The allocated objcg pointers array is not accounted directly. +* Moreover, it should not come from DMA buffer and is not readily +* reclaimable. So those GFP bits should be masked off. +*/ +#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | __GFP_ACCOUNT) +int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page) {
@@ -2810,6 +2817,7 @@ int memcg_alloc_page_obj_cgroups(structunsigned long memcg_data;void *vec; + gfp &= ~OBJCGS_CLEAR_MASK ; vec =kcalloc_node (objects,sizeof (struct obj_cgroup *), gfp,page_to_nid (page));if (!vec)
mm/slab.h renamed from mm/slab.h~mm-memcg-slab-properly-set-up-gfp-flags-for-objcg-pointer-array +0 -1
@@ -298,7 +298,6 @@ static inline void memcg_slab_post_allocif (!memcg_kmem_enabled () || !objcg)return ; - flags &= ~__GFP_ACCOUNT;for (i =0 ; i < size; i++) {if (likely (p[i])) { page =virt_to_head_page (p[i]);
_
Andrew Morton
maintainerPATCH 82/192 mm: memcg/slab: create a new set of kmalloc-cg-<n> caches
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: mm: memcg/slab: create a new set of kmalloc-cg-<n> caches
There are currently two problems in the way the objcg pointer array (memcg_data) in the page structure is being allocated and freed.
On its allocation, it is possible that the allocated objcg pointer array comes from the same slab that requires memory accounting. If this happens, the slab will never become empty again as there is at least one object left (the obj_cgroup array) in the slab.
When it is freed, the objcg pointer array object may be the last one in its slab and hence causes kfree() to be called again. With the right workload, the slab cache may be set up in a way that allows the recursive kfree() calling loop to nest deep enough to cause a kernel stack overflow and panic the system.
One way to solve this problem is to split the kmalloc-<n> caches (KMALLOC_NORMAL) into two separate sets - a new set of kmalloc-<n> (KMALLOC_NORMAL) caches for unaccounted objects only and a new set of kmalloc-cg-<n> (KMALLOC_CGROUP) caches for accounted objects only. All the other caches can still allow a mix of accounted and unaccounted objects.
With this change, all the objcg pointer array objects will come from KMALLOC_NORMAL caches which won't have their objcg pointer arrays. So both the recursive kfree() problem and non-freeable slab problem are gone.
Since both the KMALLOC_NORMAL and KMALLOC_CGROUP caches no longer have mixed accounted and unaccounted objects, this will slightly reduce the number of objcg pointer arrays that need to be allocated and save a bit of memory. On the other hand, creating a new set of kmalloc caches does have the effect of reducing cache utilization. So it is properly a wash.
The new KMALLOC_CGROUP is added between KMALLOC_NORMAL and KMALLOC_RECLAIM so that the first for loop in create_kmalloc_caches() will include the newly added caches without change.
---
include/linux/slab.h | 42 ++++++++++++++++++++++++++++++++--------- mm/internal.h | 5 ++++ mm/memcontrol.c | 2 - mm/slab_common.c | 32 ++++++++++++++++++++++--------- 4 files changed, 62 insertions(+), 19 deletions(-)
include/linux/slab.h renamed from include/linux/slab.h~mm-memcg-slab-create-a-new-set-of-kmalloc-cg-n-caches +33 -9
@@ -305,9 +305,21 @@ static inline void __check_heap_object(c/* * Whenever changing this, take care of that kmalloc_type() and * create_kmalloc_caches() still work as intended. +* +* KMALLOC_NORMAL can contain only unaccounted objects whereas KMALLOC_CGROUP +* is for accounted but unreclaimable and non-dma objects. All the other +* kmem caches can have both accounted and unaccounted objects. */ enum kmalloc_cache_type {KMALLOC_NORMAL =0 , +#ifndef CONFIG_ZONE_DMA +KMALLOC_DMA =KMALLOC_NORMAL , +#endif +#ifndef CONFIG_MEMCG_KMEM +KMALLOC_CGROUP =KMALLOC_NORMAL , +#else +KMALLOC_CGROUP , +#endif KMALLOC_RECLAIM ,#ifdef CONFIG_ZONE_DMA KMALLOC_DMA ,
@@ -319,24 +331,36 @@ enum kmalloc_cache_type {extern struct kmem_cache * kmalloc_caches[NR_KMALLOC_TYPES ][KMALLOC_SHIFT_HIGH +1 ]; +/* +* Define gfp bits that should not be set for KMALLOC_NORMAL. +*/ +#define KMALLOC_NOT_NORMAL_BITS \ + (__GFP_RECLAIMABLE | \ + (IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \ + (IS_ENABLED(CONFIG_MEMCG_KMEM) ? __GFP_ACCOUNT : 0)) + static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) { -#ifdef CONFIG_ZONE_DMA /* * The most common case is KMALLOC_NORMAL, so test for it -* with a single branch for both flags. +* with a single branch for all the relevant flags. */ -if (likely ((flags & (__GFP_DMA | __GFP_RECLAIMABLE)) ==0 )) +if (likely ((flags &KMALLOC_NOT_NORMAL_BITS ) ==0 ))return KMALLOC_NORMAL ;/* -* At least one of the flags has to be set. If both are, __GFP_DMA -* is more important. +* At least one of the flags has to be set. Their priorities in +* decreasing order are: +* 1) __GFP_DMA +* 2) __GFP_RECLAIMABLE +* 3) __GFP_ACCOUNT */ -return flags & __GFP_DMA ?KMALLOC_DMA :KMALLOC_RECLAIM ; -#else -return flags & __GFP_RECLAIMABLE ?KMALLOC_RECLAIM :KMALLOC_NORMAL ; -#endif +if (IS_ENABLED (CONFIG_ZONE_DMA ) && (flags & __GFP_DMA)) +return KMALLOC_DMA ; +if (!IS_ENABLED (CONFIG_MEMCG_KMEM ) || (flags & __GFP_RECLAIMABLE)) +return KMALLOC_RECLAIM ; +else +return KMALLOC_CGROUP ; }/*
mm/internal.h renamed from mm/internal.h~mm-memcg-slab-create-a-new-set-of-kmalloc-cg-n-caches +5 -0
@@ -116,6 +116,11 @@ extern void putback_lru_page(struct pageextern pmd_t *mm_find_pmd (struct mm_struct *mm,unsigned long address);/* +* in mm/memcontrol.c: +*/ +extern bool cgroup_memory_nokmem; + +/* * in mm/page_alloc.c */
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-slab-create-a-new-set-of-kmalloc-cg-n-caches +1 -1
@@ -83,7 +83,7 @@ DEFINE_PER_CPU(struct mem_cgroup *, int_static bool cgroup_memory_nosocket;/* Kernel memory accounting disabled? */ -static bool cgroup_memory_nokmem; +bool cgroup_memory_nokmem;/* Whether the swap controller is active */ #ifdef CONFIG_MEMCG_SWAP
mm/slab_common.c renamed from mm/slab_common.c~mm-memcg-slab-create-a-new-set-of-kmalloc-cg-n-caches +23 -9
@@ -738,21 +738,25 @@ struct kmem_cache *kmalloc_slab(size_t s }#ifdef CONFIG_ZONE_DMA -#define INIT_KMALLOC_INFO (__size, __short_size) \ -{ \ - .name[KMALLOC_NORMAL] = "kmalloc-" #__short_size, \ - .name[KMALLOC_RECLAIM] = "kmalloc-rcl-" #__short_size, \ - .name[KMALLOC_DMA] = "dma-kmalloc-" #__short_size, \ - .size = __size, \ -} +#define KMALLOC_DMA_NAME (sz) .name[KMALLOC_DMA] = "dma-kmalloc-" #sz,#else +#define KMALLOC_DMA_NAME (sz) +#endif + +#ifdef CONFIG_MEMCG_KMEM +#define KMALLOC_CGROUP_NAME (sz) .name[KMALLOC_CGROUP] = "kmalloc-cg-" #sz, +#else +#define KMALLOC_CGROUP_NAME (sz) +#endif +#define INIT_KMALLOC_INFO (__size, __short_size) \ { \ .name[KMALLOC_NORMAL] = "kmalloc-" #__short_size, \ .name[KMALLOC_RECLAIM] = "kmalloc-rcl-" #__short_size, \ + KMALLOC_CGROUP_NAME(__short_size) \ + KMALLOC_DMA_NAME(__short_size) \ .size = __size, \ } -#endif /* * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time.
@@ -838,8 +842,15 @@ void __init setup_kmalloc_cache_index_tastatic void __init new_kmalloc_cache (int idx,enum kmalloc_cache_type type,slab_flags_t flags) { -if (type ==KMALLOC_RECLAIM ) +if (type ==KMALLOC_RECLAIM ) { flags |=SLAB_RECLAIM_ACCOUNT ; + }else if (IS_ENABLED (CONFIG_MEMCG_KMEM ) && (type ==KMALLOC_CGROUP )) { +if (cgroup_memory_nokmem) { + kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL ][idx]; +return ; + } + flags |=SLAB_ACCOUNT ; + } kmalloc_caches[type][idx] = create_kmalloc_cache( kmalloc_info[idx].name[type],
@@ -857,6 +868,9 @@ void __init create_kmalloc_caches(slab_fint i;enum kmalloc_cache_type type; +/* +* Including KMALLOC_CGROUP if CONFIG_MEMCG_KMEM defined +*/ for (type =KMALLOC_NORMAL ; type <=KMALLOC_RECLAIM ; type++) {for (i =KMALLOC_SHIFT_LOW ; i <=KMALLOC_SHIFT_HIGH ; i++) {if (!kmalloc_caches[type][i])
_
Andrew Morton
maintainerPATCH 83/192 mm: memcg/slab: disable cache merging for KMALLOC_NORMAL caches
In reply to Andrew MortonFrom: Waiman Long <[email protected]>
Subject: mm: memcg/slab: disable cache merging for KMALLOC_NORMAL caches
The KMALLOC_NORMAL (kmalloc-<n>) caches are for unaccounted objects only when CONFIG_MEMCG_KMEM is enabled. To make sure that this condition remains true, we will have to prevent KMALOC_NORMAL caches to merge with other kmem caches. This is now done by setting its refcount to -1 right after its creation.
---
mm/slab_common.c | 7 +++++++ 1 file changed, 7 insertions(+)
mm/slab_common.c renamed from mm/slab_common.c~mm-memcg-slab-disable-cache-merging-for-kmalloc_normal-caches +7 -0
@@ -856,6 +856,13 @@ new_kmalloc_cache(int idx, enum kmalloc_ kmalloc_info[idx].name[type], kmalloc_info[idx].size, flags,0 , kmalloc_info[idx].size); + +/* +* If CONFIG_MEMCG_KMEM is enabled, disable cache merging for +* KMALLOC_NORMAL caches. +*/ +if (IS_ENABLED (CONFIG_MEMCG_KMEM ) && (type ==KMALLOC_NORMAL )) + kmalloc_caches[type][idx]->refcount =-1 ; }/*
_
Andrew Morton
maintainerPATCH 84/192 mm: memcontrol: fix root_mem_cgroup charging
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: fix root_mem_cgroup charging
The below scenario can cause the page counters of the root_mem_cgroup to be out of balance.
CPU0: CPU1:
objcg = get_obj_cgroup_from_current()obj_cgroup_charge_pages(objcg)
memcg_reparent_objcgs()
// reparent to root_mem_cgroup
WRITE_ONCE(iter->memcg, parent)
// memcg == root_mem_cgroup
memcg = get_mem_cgroup_from_objcg(objcg)
// do not charge to the root_mem_cgroup
try_charge(memcg)obj_cgroup_uncharge_pages(objcg)
memcg = get_mem_cgroup_from_objcg(objcg)
// uncharge from the root_mem_cgroup
refill_stock(memcg)
drain_stock(memcg)
page_counter_uncharge(&memcg->memory)get_obj_cgroup_from_current() never returns a root_mem_cgroup's objcg, so we never explicitly charge the root_mem_cgroup. And it's not going to change. It's all about a race when we got an obj_cgroup pointing at some non-root memcg, but before we were able to charge it, the cgroup was gone, objcg was reparented to the root and so we're skipping the charging. Then we store the objcg pointer and later use to uncharge the root_mem_cgroup.
This can cause the page counter to be less than the actual value. Although we do not display the value (mem_cgroup_usage) so there shouldn't be any actual problem, but there is a WARN_ON_ONCE in the page_counter_cancel(). Who knows if it will trigger? So it is better to fix it.
---
mm/memcontrol.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-fix-root_mem_cgroup-charging +12 -5
@@ -2568,8 +2568,8 @@ out:css_put (&memcg->css); } -static int try_charge (struct mem_cgroup *memcg,gfp_t gfp_mask, -unsigned int nr_pages) +static int try_charge_memcg (struct mem_cgroup *memcg,gfp_t gfp_mask, +unsigned int nr_pages) {unsigned int batch =max (MEMCG_CHARGE_BATCH , nr_pages);int nr_retries =MAX_RECLAIM_RETRIES ;
@@ -2581,8 +2581,6 @@ static int try_charge(struct mem_cgroupbool drained = false;unsigned long pflags; -if (mem_cgroup_is_root (memcg)) -return 0 ; retry:if (consume_stock (memcg, nr_pages))return 0 ;
@@ -2762,6 +2760,15 @@ done_restock:return 0 ; } +static inline int try_charge (struct mem_cgroup *memcg,gfp_t gfp_mask, +unsigned int nr_pages) +{ +if (mem_cgroup_is_root (memcg)) +return 0 ; + +return try_charge_memcg (memcg, gfp_mask, nr_pages); +} +#if defined(CONFIG_MEMCG_KMEM ) || defined(CONFIG_MMU )static void cancel_charge (struct mem_cgroup *memcg,unsigned int nr_pages) {
@@ -2997,7 +3004,7 @@ static int obj_cgroup_charge_pages(struc memcg =get_mem_cgroup_from_objcg (objcg); - ret =try_charge (memcg, gfp, nr_pages); + ret =try_charge_memcg (memcg, gfp, nr_pages);if (ret) goto out;
_
Andrew Morton
maintainerPATCH 85/192 mm: memcontrol: fix page charging in page replacement
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: fix page charging in page replacement
Patch series "memcontrol code cleanup and simplification", v3.
This patch (of 8):
The pages aren't accounted at the root level, so do not charge the page to the root memcg in page replacement. Although we do not display the value (mem_cgroup_usage) so there shouldn't be any actual problem, but there is a WARN_ON_ONCE in the page_counter_cancel(). Who knows if it will trigger? So it is better to fix it.
---
mm/memcontrol.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-fix-page-charging-in-page-replacement +5 -3
@@ -6984,9 +6984,11 @@ void mem_cgroup_migrate(struct page *old/* Force-charge the new page. The old one will be freed soon */ nr_pages =thp_nr_pages (newpage); -page_counter_charge (&memcg->memory, nr_pages); -if (do_memsw_account ()) -page_counter_charge (&memcg->memsw, nr_pages); +if (!mem_cgroup_is_root (memcg)) { +page_counter_charge (&memcg->memory, nr_pages); +if (do_memsw_account ()) +page_counter_charge (&memcg->memsw, nr_pages); + }css_get (&memcg->css);commit_charge (newpage, memcg);
_
Andrew Morton
maintainerPATCH 86/192 mm: memcontrol: bail out early when !mm in get_mem_cgroup_from_mm
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: bail out early when !mm in get_mem_cgroup_from_mm
When mm is NULL, we do not need to hold rcu lock and call css_tryget for the root memcg. And we also do not need to check !mm in every loop of while. So bail out early when !mm.
---
mm/memcontrol.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-bail-out-early-when-mm-in-get_mem_cgroup_from_mm +14 -11
@@ -919,20 +919,23 @@ struct mem_cgroup *get_mem_cgroup_from_mif (mem_cgroup_disabled ())return NULL ; +/* +* Page cache insertions can happen without an +* actual mm context, e.g. during disk probing +* on boot, loopback IO, acct() writes etc. +* +* No need to css_get on root memcg as the reference +* counting is disabled on the root level in the +* cgroup core. See CSS_NO_REF. +*/ +if (unlikely (!mm)) +return root_mem_cgroup; +rcu_read_lock ();do { -/* -* Page cache insertions can happen without an -* actual mm context, e.g. during disk probing -* on boot, loopback IO, acct() writes etc. -*/ -if (unlikely (!mm)) + memcg =mem_cgroup_from_task (rcu_dereference (mm->owner)); +if (unlikely (!memcg)) memcg = root_mem_cgroup; -else { - memcg =mem_cgroup_from_task (rcu_dereference (mm->owner)); -if (unlikely (!memcg)) - memcg = root_mem_cgroup; - } }while (!css_tryget (&memcg->css));rcu_read_unlock ();return memcg;
_
Andrew Morton
maintainerPATCH 87/192 mm: memcontrol: remove the pgdata parameter of mem_cgroup_page_lruvec
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: remove the pgdata parameter of mem_cgroup_page_lruvec
All the callers of mem_cgroup_page_lruvec() just pass page_pgdat(page) as the 2nd parameter to it (except isolate_migratepages_block()). But for isolate_migratepages_block(), the page_pgdat(page) is also equal to the local variable of @pgdat. So mem_cgroup_page_lruvec() do not need the pgdat parameter. Just remove it to simplify the code.
---
include/linux/memcontrol.h | 10 +++++----- mm/compaction.c | 2 +- mm/memcontrol.c | 9 +++------ mm/swap.c | 2 +- mm/workingset.c | 2 +- 5 files changed, 11 insertions(+), 14 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-memcontrol-remove-the-pgdata-parameter-of-mem_cgroup_page_lruvec +5 -5
@@ -743,13 +743,12 @@ out:/** * mem_cgroup_page_lruvec - return lruvec for isolating/putting an LRU page * @page: the page -* @pgdat: pgdat of the page * * This function relies on page->mem_cgroup being stable. */ -static inline struct lruvec *mem_cgroup_page_lruvec (struct page *page, -struct pglist_data *pgdat) +static inline struct lruvec *mem_cgroup_page_lruvec (struct page *page) { +pg_data_t *pgdat =page_pgdat (page);struct mem_cgroup *memcg =page_memcg (page);VM_WARN_ON_ONCE_PAGE (!memcg && !mem_cgroup_disabled (), page);
@@ -1221,9 +1220,10 @@ static inline struct lruvec *mem_cgroup_return &pgdat->__lruvec; } -static inline struct lruvec *mem_cgroup_page_lruvec (struct page *page, -struct pglist_data *pgdat) +static inline struct lruvec *mem_cgroup_page_lruvec (struct page *page) { +pg_data_t *pgdat =page_pgdat (page); +return &pgdat->__lruvec; }
mm/compaction.c renamed from mm/compaction.c~mm-memcontrol-remove-the-pgdata-parameter-of-mem_cgroup_page_lruvec +1 -1
@@ -1028,7 +1028,7 @@ isolate_migratepages_block(struct compacif (!TestClearPageLRU (page)) goto isolate_fail_put; - lruvec =mem_cgroup_page_lruvec (page, pgdat); + lruvec =mem_cgroup_page_lruvec (page);/* If we already hold the lock, we can skip some rechecking */ if (lruvec != locked) {
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-remove-the-pgdata-parameter-of-mem_cgroup_page_lruvec +3 -6
@@ -1199,9 +1199,8 @@ void lruvec_memcg_debug(struct lruvec *lstruct lruvec *lock_page_lruvec (struct page *page) {struct lruvec *lruvec; -struct pglist_data *pgdat =page_pgdat (page); - lruvec =mem_cgroup_page_lruvec (page, pgdat); + lruvec =mem_cgroup_page_lruvec (page);spin_lock (&lruvec->lru_lock);lruvec_memcg_debug (lruvec, page);
@@ -1212,9 +1211,8 @@ struct lruvec *lock_page_lruvec(struct pstruct lruvec *lock_page_lruvec_irq (struct page *page) {struct lruvec *lruvec; -struct pglist_data *pgdat =page_pgdat (page); - lruvec =mem_cgroup_page_lruvec (page, pgdat); + lruvec =mem_cgroup_page_lruvec (page);spin_lock_irq (&lruvec->lru_lock);lruvec_memcg_debug (lruvec, page);
@@ -1225,9 +1223,8 @@ struct lruvec *lock_page_lruvec_irq(strustruct lruvec *lock_page_lruvec_irqsave (struct page *page,unsigned long *flags) {struct lruvec *lruvec; -struct pglist_data *pgdat =page_pgdat (page); - lruvec =mem_cgroup_page_lruvec (page, pgdat); + lruvec =mem_cgroup_page_lruvec (page);spin_lock_irqsave (&lruvec->lru_lock, *flags);lruvec_memcg_debug (lruvec, page);
mm/swap.c renamed from mm/swap.c~mm-memcontrol-remove-the-pgdata-parameter-of-mem_cgroup_page_lruvec +1 -1
@@ -313,7 +313,7 @@ void lru_note_cost(struct lruvec *lruvecvoid lru_note_cost_page (struct page *page) { -lru_note_cost (mem_cgroup_page_lruvec (page,page_pgdat (page)), +lru_note_cost (mem_cgroup_page_lruvec (page),page_is_file_lru (page),thp_nr_pages (page)); }
mm/workingset.c renamed from mm/workingset.c~mm-memcontrol-remove-the-pgdata-parameter-of-mem_cgroup_page_lruvec +1 -1
@@ -408,7 +408,7 @@ void workingset_activation(struct page * memcg =page_memcg_rcu (page);if (!mem_cgroup_disabled () && !memcg) goto out; - lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); + lruvec =mem_cgroup_page_lruvec (page);workingset_age_nonresident (lruvec,thp_nr_pages (page)); out:rcu_read_unlock ();
_
Andrew Morton
maintainerPATCH 88/192 mm: memcontrol: simplify lruvec_holds_page_lru_lock
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: simplify lruvec_holds_page_lru_lock
We already have a helper lruvec_memcg() to get the memcg from lruvec, we do not need to do it ourselves in the lruvec_holds_page_lru_lock(). So use lruvec_memcg() instead. And if mem_cgroup_disabled() returns false, the page_memcg(page) (the LRU pages) cannot be NULL. So remove the odd logic of "memcg = page_memcg(page) ? : root_mem_cgroup". And use lruvec_pgdat to simplify the code. We can have a single definition for this function that works for !CONFIG_MEMCG, CONFIG_MEMCG + mem_cgroup_disabled() and CONFIG_MEMCG.
---
include/linux/memcontrol.h | 31 +++++++------------------------ 1 file changed, 7 insertions(+), 24 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-memcontrol-simplify-lruvec_holds_page_lru_lock +7 -24
@@ -755,22 +755,6 @@ static inline struct lruvec *mem_cgroup_return mem_cgroup_lruvec (memcg, pgdat); } -static inline bool lruvec_holds_page_lru_lock (struct page *page, -struct lruvec *lruvec) -{ -pg_data_t *pgdat =page_pgdat (page); -const struct mem_cgroup *memcg; -struct mem_cgroup_per_node *mz; - -if (mem_cgroup_disabled ()) -return lruvec == &pgdat->__lruvec; - - mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - memcg =page_memcg (page) ? : root_mem_cgroup; - -return lruvec->pgdat == pgdat && mz->memcg == memcg; -} -struct mem_cgroup *mem_cgroup_from_task (struct task_struct *p);struct mem_cgroup *get_mem_cgroup_from_mm (struct mm_struct *mm);
@@ -1227,14 +1211,6 @@ static inline struct lruvec *mem_cgroup_return &pgdat->__lruvec; } -static inline bool lruvec_holds_page_lru_lock (struct page *page, -struct lruvec *lruvec) -{ -pg_data_t *pgdat =page_pgdat (page); - -return lruvec == &pgdat->__lruvec; -} -static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) { }
@@ -1516,6 +1492,13 @@ static inline void unlock_page_lruvec_irspin_unlock_irqrestore (&lruvec->lru_lock, flags); } +static inline bool lruvec_holds_page_lru_lock (struct page *page, +struct lruvec *lruvec) +{ +return lruvec_pgdat (lruvec) ==page_pgdat (page) && +lruvec_memcg (lruvec) ==page_memcg (page); +} +/* Don't lock again iff page's lruvec locked */ static inline struct lruvec *relock_page_lruvec_irq (struct page *page,struct lruvec *locked_lruvec)
_
Andrew Morton
maintainerPATCH 89/192 mm: memcontrol: rename lruvec_holds_page_lru_lock to page_matches_lruvec
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: rename lruvec_holds_page_lru_lock to page_matches_lruvec
lruvec_holds_page_lru_lock() doesn't check anything about locking and is used to check whether the page belongs to the lruvec. So rename it to page_matches_lruvec().
---
include/linux/memcontrol.h | 8 ++++---- mm/vmscan.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-memcontrol-rename-lruvec_holds_page_lru_lock-to-page_matches_lruvec +4 -4
@@ -1492,8 +1492,8 @@ static inline void unlock_page_lruvec_irspin_unlock_irqrestore (&lruvec->lru_lock, flags); } -static inline bool lruvec_holds_page_lru_lock (struct page *page, -struct lruvec *lruvec) +/* Test requires a stable page->memcg binding, see page_memcg() */ +static inline bool page_matches_lruvec (struct page *page,struct lruvec *lruvec) {return lruvec_pgdat (lruvec) ==page_pgdat (page) &&lruvec_memcg (lruvec) ==page_memcg (page);
@@ -1504,7 +1504,7 @@ static inline struct lruvec *relock_page struct lruvec *locked_lruvec) {if (locked_lruvec) { -if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +if (page_matches_lruvec (page, locked_lruvec))return locked_lruvec;unlock_page_lruvec_irq (locked_lruvec);
@@ -1518,7 +1518,7 @@ static inline struct lruvec *relock_page struct lruvec *locked_lruvec, unsigned long *flags) {if (locked_lruvec) { -if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +if (page_matches_lruvec (page, locked_lruvec))return locked_lruvec;unlock_page_lruvec_irqrestore (locked_lruvec, *flags);
mm/vmscan.c renamed from mm/vmscan.c~mm-memcontrol-rename-lruvec_holds_page_lru_lock-to-page_matches_lruvec +1 -1
@@ -2063,7 +2063,7 @@ static unsigned noinline_for_stack move_* All pages were isolated from the same lruvec (and isolation * inhibits memcg migration). */ -VM_BUG_ON_PAGE (!lruvec_holds_page_lru_lock (page, lruvec), page); +VM_BUG_ON_PAGE (!page_matches_lruvec (page, lruvec), page);add_page_to_lru_list (page, lruvec); nr_pages =thp_nr_pages (page); nr_moved += nr_pages;
_
Andrew Morton
maintainerPATCH 90/192 mm: memcontrol: simplify the logic of objcg pinning memcg
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: simplify the logic of objcg pinning memcg
The obj_cgroup_release() and memcg_reparent_objcgs() are serialized by the css_set_lock. We do not need to care about objcg->memcg being released in the process of obj_cgroup_release(). So there is no need to pin memcg before releasing objcg. Remove those pinning logic to simplfy the code.
There are only two places that modifies the objcg->memcg. One is the initialization to objcg->memcg in the memcg_online_kmem(), another is objcgs reparenting in the memcg_reparent_objcgs(). It is also impossible for the two to run in parallel. So xchg() is unnecessary and it is enough to use WRITE_ONCE().
---
mm/memcontrol.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-simplify-the-logic-of-objcg-pinning-memcg +6 -14
@@ -261,7 +261,6 @@ static void obj_cgroup_uncharge_pages(ststatic void obj_cgroup_release (struct percpu_ref *ref) { struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt); -struct mem_cgroup *memcg;unsigned int nr_bytes;unsigned int nr_pages;unsigned long flags;
@@ -291,11 +290,9 @@ static void obj_cgroup_release(struct pe nr_pages = nr_bytes >>PAGE_SHIFT ;spin_lock_irqsave (&css_set_lock, flags); - memcg =obj_cgroup_memcg (objcg);if (nr_pages)obj_cgroup_uncharge_pages (objcg, nr_pages);list_del (&objcg->list); -mem_cgroup_put (memcg);spin_unlock_irqrestore (&css_set_lock, flags);percpu_ref_exit (ref);
@@ -330,17 +327,12 @@ static void memcg_reparent_objcgs(structspin_lock_irq (&css_set_lock); -/* Move active objcg to the parent's list */ -xchg (&objcg->memcg, parent); -css_get (&parent->css); -list_add (&objcg->list, &parent->objcg_list); - -/* Move already reparented objcgs to the parent's list */ -list_for_each_entry (iter, &memcg->objcg_list, list) { -css_get (&parent->css); -xchg (&iter->memcg, parent); -css_put (&memcg->css); - } +/* 1) Ready to reparent active objcg. */ +list_add (&objcg->list, &memcg->objcg_list); +/* 2) Reparent active objcg and already reparented objcgs to parent. */ +list_for_each_entry (iter, &memcg->objcg_list, list) +WRITE_ONCE (iter->memcg, parent); +/* 3) Move already reparented objcgs to the parent's list */ list_splice (&memcg->objcg_list, &parent->objcg_list);spin_unlock_irq (&css_set_lock);
_
Andrew Morton
maintainerPATCH 91/192 mm: memcontrol: move obj_cgroup_uncharge_pages() out of css_set_lock
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: memcontrol: move obj_cgroup_uncharge_pages() out of css_set_lock
The css_set_lock is used to guard the list of inherited objcgs. So there is no need to uncharge kernel memory under css_set_lock. Just move it out of the lock.
---
mm/memcontrol.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-move-obj_cgroup_uncharge_pages-out-of-css_set_lock +2 -1
@@ -289,9 +289,10 @@ static void obj_cgroup_release(struct peWARN_ON_ONCE (nr_bytes & (PAGE_SIZE -1 )); nr_pages = nr_bytes >>PAGE_SHIFT ; -spin_lock_irqsave (&css_set_lock, flags);if (nr_pages)obj_cgroup_uncharge_pages (objcg, nr_pages); + +spin_lock_irqsave (&css_set_lock, flags);list_del (&objcg->list);spin_unlock_irqrestore (&css_set_lock, flags);
_
Andrew Morton
maintainerPATCH 92/192 mm: vmscan: remove noinline_for_stack
In reply to Andrew MortonFrom: Muchun Song <[email protected]>
Subject: mm: vmscan: remove noinline_for_stack
The noinline_for_stack is introduced by commit 666356297ec4 ("vmscan: set up pagevec as late as possible in shrink_inactive_list()"), its purpose is to delay the allocation of pagevec as late as possible to save stack memory. But the commit 2bcf88796381 ("mm: take pagevecs off reclaim stack") replace pagevecs by lists of pages_to_free. So we do not need noinline_for_stack, just remove it (let the compiler decide whether to inline).
---
mm/vmscan.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-remove-noinline_for_stack +3 -3
@@ -2015,8 +2015,8 @@ static int too_many_isolated(struct pgli* * Returns the number of pages moved to the given lruvec. */ -static unsigned noinline_for_stack move_pages_to_lru (struct lruvec *lruvec, -struct list_head *list) +static unsigned int move_pages_to_lru (struct lruvec *lruvec, +struct list_head *list) {int nr_pages, nr_moved =0 ;LIST_HEAD (pages_to_free);
@@ -2096,7 +2096,7 @@ static int current_may_throttle(void)* shrink_inactive_list() is a helper for shrink_node(). It returns the number * of reclaimed pages */ -static noinline_for_stack unsigned long +static unsigned long shrink_inactive_list (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,enum lru_list lru) {
_
Andrew Morton
maintainerPATCH 93/192 memcontrol: use flexible-array member
In reply to Andrew MortonFrom: wenhuizhang <[email protected]>
Subject: memcontrol: use flexible-array member
Change deprecated zero-length-and-one-element-arrays into flexible array member.Zero-length and one-element arrays detected by Lukas's CodeChecker. Zero/one element arrays cause undefined behaviours if sizeof() used.
---
include/linux/memcontrol.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~memcontrol-use-flexible-array-member +1 -2
@@ -349,8 +349,7 @@ struct mem_cgroup {struct deferred_split deferred_split_queue;#endif -struct mem_cgroup_per_node *nodeinfo[0 ]; -/* WARNING: nodeinfo must be the last member here */ +struct mem_cgroup_per_node *nodeinfo[]; };/*
_
Andrew Morton
maintainerPATCH 94/192 loop: use worker per cgroup instead of kworker
In reply to Andrew MortonFrom: Dan Schatzberg <[email protected]>
Subject: loop: use worker per cgroup instead of kworker
Patch series "Charge loop device i/o to issuing cgroup", v14.
The loop device runs all i/o to the backing file on a separate kworker thread which results in all i/o being charged to the root cgroup. This allows a loop device to be used to trivially bypass resource limits and other policy. This patch series fixes this gap in accounting.
A simple script to demonstrate this behavior on cgroupv2 machine:
'''
#!/bin/bash
set -e
CGROUP=/sys/fs/cgroup/test.slice
LOOP_DEV=/dev/loop0
if [[ ! -d $CGROUP ]]
then
sudo mkdir $CGROUP
fi
grep oom_kill $CGROUP/memory.events
# Set a memory limit, write more than that limit to tmpfs -> OOM kill sudo unshare -m bash -c "
echo \$\$ > $CGROUP/cgroup.procs;
echo 0 > $CGROUP/memory.swap.max;
echo 64M > $CGROUP/memory.max;
mount -t tmpfs -o size=512m tmpfs /tmp;
dd if=/dev/zero of=/tmp/file bs=1M count=256" || true
grep oom_kill $CGROUP/memory.events
# Set a memory limit, write more than that limit through loopback
# device -> no OOM kill sudo unshare -m bash -c " echo \$\$ > $CGROUP/cgroup.procs; echo 0 > $CGROUP/memory.swap.max; echo 64M > $CGROUP/memory.max; mount -t tmpfs -o size=512m tmpfs /tmp; truncate -s 512m /tmp/backing_file losetup $LOOP_DEV /tmp/backing_file dd if=/dev/zero of=$LOOP_DEV bs=1M count=256; losetup -D $LOOP_DEV" || true
grep oom_kill $CGROUP/memory.events
'''
Naively charging cgroups could result in priority inversions through the single kworker thread in the case where multiple cgroups are reading/writing to the same loop device. This patch series does some minor modification to the loop driver so that each cgroup can make forward progress independently to avoid this inversion.
With this patch series applied, the above script triggers OOM kills when writing through the loop device as expected.
This patch (of 3):
Existing uses of loop device may have multiple cgroups reading/writing to the same device. Simply charging resources for I/O to the backing file could result in priority inversion where one cgroup gets synchronously blocked, holding up all other I/O to the loop device.
In order to avoid this priority inversion, we use a single workqueue where each work item is a "struct loop_worker" which contains a queue of struct loop_cmds to issue. The loop device maintains a tree mapping blk css_id -> loop_worker. This allows each cgroup to independently make forward progress issuing I/O to the backing file.
There is also a single queue for I/O associated with the rootcg which can be used in cases of extreme memory shortage where we cannot allocate a loop_worker.
The locking for the tree and queues is fairly heavy handed - we acquire a per-loop-device spinlock any time either is accessed. The existing implementation serializes all I/O through a single thread anyways, so I don't believe this is any worse.
---
drivers/block/loop.c | 209 +++++++++++++++++++++++++++++++++++------ drivers/block/loop.h | 12 +- 2 files changed, 187 insertions(+), 34 deletions(-)
drivers/block/loop.c renamed from drivers/block/loop.c~loop-use-worker-per-cgroup-instead-of-kworker +179 -30
@@ -71,7 +71,6 @@#include <linux/writeback.h> #include <linux/completion.h> #include <linux/highmem.h> -#include <linux/kthread.h> #include <linux/splice.h> #include <linux/sysfs.h> #include <linux/miscdevice.h>
@@ -84,6 +83,8 @@#include <linux/uaccess.h> +#define LOOP_IDLE_WORKER_TIMEOUT (60 * HZ) +static DEFINE_IDR (loop_index_idr);static DEFINE_MUTEX (loop_ctl_mutex);
@@ -921,27 +922,95 @@ static void loop_config_discard(struct l q->limits.discard_alignment =0 ; } -static void loop_unprepare_queue (struct loop_device *lo) +struct loop_worker { +struct rb_node rb_node; +struct work_struct work; +struct list_head cmd_list; +struct list_head idle_list; +struct loop_device *lo; +struct cgroup_subsys_state *css; +unsigned long last_ran_at; +}; + +static void loop_workfn (struct work_struct *work); +static void loop_rootcg_workfn (struct work_struct *work); +static void loop_free_idle_workers (struct timer_list *timer); + +#ifdef CONFIG_BLK_CGROUP +static inline int queue_on_root_worker (struct cgroup_subsys_state *css) { -kthread_flush_worker (&lo->worker); -kthread_stop (lo->worker_task); +return !css || css == blkcg_root_css; } - -static int loop_kthread_worker_fn (void *worker_ptr) +#else +static inline int queue_on_root_worker (struct cgroup_subsys_state *css) { - current->flags |=PF_LOCAL_THROTTLE |PF_MEMALLOC_NOIO ; -return kthread_worker_fn (worker_ptr); +return !css; } +#endif -static int loop_prepare_queue (struct loop_device *lo) +static void loop_queue_work (struct loop_device *lo,struct loop_cmd *cmd) { -kthread_init_worker (&lo->worker); - lo->worker_task =kthread_run (loop_kthread_worker_fn, - &lo->worker,"loop%d" , lo->lo_number); -if (IS_ERR (lo->worker_task)) -return -ENOMEM ; -set_user_nice (lo->worker_task,MIN_NICE ); -return 0 ; +struct rb_node **node = &(lo->worker_tree.rb_node), *parent =NULL ; +struct loop_worker *cur_worker, *worker =NULL ; +struct work_struct *work; +struct list_head *cmd_list; + +spin_lock_irq (&lo->lo_work_lock); + +if (queue_on_root_worker (cmd->css)) + goto queue_work; + + node = &lo->worker_tree.rb_node; + +while (*node) { + parent = *node; + cur_worker = container_of(*node, struct loop_worker, rb_node); +if (cur_worker->css == cmd->css) { + worker = cur_worker; +break ; + }else if ((long )cur_worker->css < (long )cmd->css) { + node = &(*node)->rb_left; + }else { + node = &(*node)->rb_right; + } + } +if (worker) + goto queue_work; + + worker =kzalloc (sizeof (struct loop_worker ),GFP_NOWAIT | __GFP_NOWARN); +/* +* In the event we cannot allocate a worker, just queue on the +* rootcg worker +*/ +if (!worker) + goto queue_work; + + worker->css = cmd->css; +css_get (worker->css); +INIT_WORK (&worker->work, loop_workfn); +INIT_LIST_HEAD (&worker->cmd_list); +INIT_LIST_HEAD (&worker->idle_list); + worker->lo = lo; +rb_link_node (&worker->rb_node, parent, node); +rb_insert_color (&worker->rb_node, &lo->worker_tree); +queue_work: +if (worker) { +/* +* We need to remove from the idle list here while +* holding the lock so that the idle timer doesn't +* free the worker +*/ +if (!list_empty (&worker->idle_list)) +list_del_init (&worker->idle_list); + work = &worker->work; + cmd_list = &worker->cmd_list; + }else { + work = &lo->rootcg_work; + cmd_list = &lo->rootcg_cmd_list; + } +list_add_tail (&cmd->list_entry, cmd_list); +queue_work (lo->workqueue, work); +spin_unlock_irq (&lo->lo_work_lock); }static void loop_update_rotational (struct loop_device *lo)
@@ -1127,12 +1196,23 @@ static int loop_configure(struct loop_de !file->f_op->write_iter) lo->lo_flags |=LO_FLAGS_READ_ONLY ; - error =loop_prepare_queue (lo); -if (error) + lo->workqueue =alloc_workqueue ("loop%d" , +WQ_UNBOUND |WQ_FREEZABLE , +0 , + lo->lo_number); +if (!lo->workqueue) { + error = -ENOMEM ; goto out_unlock; + }set_disk_ro (lo->lo_disk, (lo->lo_flags &LO_FLAGS_READ_ONLY ) !=0 ); +INIT_WORK (&lo->rootcg_work, loop_rootcg_workfn); +INIT_LIST_HEAD (&lo->rootcg_cmd_list); +INIT_LIST_HEAD (&lo->idle_worker_list); + lo->worker_tree =RB_ROOT ; +timer_setup (&lo->timer, loop_free_idle_workers, +TIMER_DEFERRABLE ); lo->use_dio = lo->lo_flags &LO_FLAGS_DIRECT_IO ; lo->lo_device = bdev; lo->lo_backing_file = file;
@@ -1200,6 +1280,7 @@ static int __loop_clr_fd(struct loop_devint err =0 ;bool partscan = false;int lo_number; +struct loop_worker *pos, *worker;mutex_lock (&lo->lo_mutex);if (WARN_ON_ONCE (lo->lo_state != Lo_rundown)) {
@@ -1219,6 +1300,18 @@ static int __loop_clr_fd(struct loop_dev/* freeze request queue during the transition */ blk_mq_freeze_queue (lo->lo_queue); +destroy_workqueue (lo->workqueue); +spin_lock_irq (&lo->lo_work_lock); +list_for_each_entry_safe (worker, pos, &lo->idle_worker_list, + idle_list) { +list_del (&worker->idle_list); +rb_erase (&worker->rb_node, &lo->worker_tree); +css_put (worker->css); +kfree (worker); + } +spin_unlock_irq (&lo->lo_work_lock); +del_timer_sync (&lo->timer); +spin_lock_irq (&lo->lo_lock); lo->lo_backing_file =NULL ;spin_unlock_irq (&lo->lo_lock);
@@ -1255,7 +1348,6 @@ static int __loop_clr_fd(struct loop_dev partscan = lo->lo_flags &LO_FLAGS_PARTSCAN && bdev; lo_number = lo->lo_number; -loop_unprepare_queue (lo); out_unlock:mutex_unlock (&lo->lo_mutex);if (partscan) {
@@ -2015,7 +2107,7 @@ static blk_status_t loop_queue_rq(struct } else#endif cmd->css =NULL ; -kthread_queue_work (&lo->worker, &cmd->work); +loop_queue_work (lo, cmd);return BLK_STS_OK ; }
@@ -2045,26 +2137,82 @@ static void loop_handle_cmd(struct loop_ } } -static void loop_queue_work (struct kthread_work *work) +static void loop_set_timer (struct loop_device *lo) +{ +timer_reduce (&lo->timer, jiffies +LOOP_IDLE_WORKER_TIMEOUT ); +} + +static void loop_process_work (struct loop_worker *worker, +struct list_head *cmd_list,struct loop_device *lo) { -struct loop_cmd *cmd = - container_of(work, struct loop_cmd, work); +int orig_flags = current->flags; +struct loop_cmd *cmd; + + current->flags |=PF_LOCAL_THROTTLE |PF_MEMALLOC_NOIO ; +spin_lock_irq (&lo->lo_work_lock); +while (!list_empty (cmd_list)) { + cmd =container_of ( + cmd_list->next, struct loop_cmd, list_entry); +list_del (cmd_list->next); +spin_unlock_irq (&lo->lo_work_lock); + +loop_handle_cmd (cmd); +cond_resched (); + +spin_lock_irq (&lo->lo_work_lock); + } -loop_handle_cmd (cmd); +/* +* We only add to the idle list if there are no pending cmds +* *and* the worker will not run again which ensures that it +* is safe to free any worker on the idle list +*/ +if (worker && !work_pending (&worker->work)) { + worker->last_ran_at = jiffies; +list_add_tail (&worker->idle_list, &lo->idle_worker_list); +loop_set_timer (lo); + } +spin_unlock_irq (&lo->lo_work_lock); + current->flags = orig_flags; } -static int loop_init_request (struct blk_mq_tag_set *set,struct request *rq, -unsigned int hctx_idx,unsigned int numa_node) +static void loop_workfn (struct work_struct *work) { -struct loop_cmd *cmd =blk_mq_rq_to_pdu (rq); +struct loop_worker *worker = + container_of(work, struct loop_worker, work); +loop_process_work (worker, &worker->cmd_list, worker->lo); +} -kthread_init_work (&cmd->work, loop_queue_work); -return 0 ; +static void loop_rootcg_workfn (struct work_struct *work) +{ +struct loop_device *lo = + container_of(work, struct loop_device, rootcg_work); +loop_process_work (NULL , &lo->rootcg_cmd_list, lo); +} + +static void loop_free_idle_workers (struct timer_list *timer) +{ + struct loop_device *lo = container_of(timer, struct loop_device, timer); +struct loop_worker *pos, *worker; + +spin_lock_irq (&lo->lo_work_lock); +list_for_each_entry_safe (worker, pos, &lo->idle_worker_list, + idle_list) { +if (time_is_after_jiffies (worker->last_ran_at + +LOOP_IDLE_WORKER_TIMEOUT )) +break ; +list_del (&worker->idle_list); +rb_erase (&worker->rb_node, &lo->worker_tree); +css_put (worker->css); +kfree (worker); + } +if (!list_empty (&lo->idle_worker_list)) +loop_set_timer (lo); +spin_unlock_irq (&lo->lo_work_lock); }static const struct blk_mq_ops loop_mq_ops = { .queue_rq = loop_queue_rq, - .init_request = loop_init_request, .complete = lo_complete_rq, };
@@ -2153,6 +2301,7 @@ static int loop_add(struct loop_device *mutex_init (&lo->lo_mutex); lo->lo_number = i;spin_lock_init (&lo->lo_lock); +spin_lock_init (&lo->lo_work_lock); disk->major =LOOP_MAJOR ; disk->first_minor = i << part_shift; disk->fops = &lo_fops;
drivers/block/loop.h renamed from drivers/block/loop.h~loop-use-worker-per-cgroup-instead-of-kworker +8 -4
@@ -14,7 +14,6 @@#include <linux/blk-mq.h> #include <linux/spinlock.h> #include <linux/mutex.h> -#include <linux/kthread.h> #include <uapi/linux/loop.h> /* Possible states of device */
@@ -55,8 +54,13 @@ struct loop_device {spinlock_t lo_lock;int lo_state; -struct kthread_worker worker; -struct task_struct *worker_task; +spinlock_t lo_work_lock; +struct workqueue_struct *workqueue; +struct work_struct rootcg_work; +struct list_head rootcg_cmd_list; +struct list_head idle_worker_list; +struct rb_root worker_tree; +struct timer_list timer;bool use_dio;bool sysfs_inited;
@@ -67,7 +71,7 @@ struct loop_device { };struct loop_cmd { -struct kthread_work work; +struct list_head list_entry;bool use_aio;/* use AIO interface to handle I/O */ atomic_t ref;/* only for aio */ long ret;
_
Andrew Morton
maintainerPATCH 95/192 mm: charge active memcg when no mm is set
In reply to Andrew MortonFrom: Dan Schatzberg <[email protected]>
Subject: mm: charge active memcg when no mm is set
set_active_memcg() worked for kernel allocations but was silently ignored for user pages.
This patch establishes a precedence order for who gets charged:
1. If there is a memcg associated with the page already, that memcg is charged. This happens during swapin.
2. If an explicit mm is passed, mm->memcg is charged. This happens during page faults, which can be triggered in remote VMs (eg gup).
3. Otherwise consult the current process context. If there is an active_memcg, use that. Otherwise, current->mm->memcg.
Previously, if a NULL mm was passed to mem_cgroup_charge (case 3) it would always charge the root cgroup. Now it looks up the active_memcg first (falling back to charging the root cgroup if not set).
---
mm/filemap.c | 2 +- mm/memcontrol.c | 41 +++++++++++++++++++++++++++-------------- mm/shmem.c | 4 ++-- 3 files changed, 30 insertions(+), 17 deletions(-)
mm/filemap.c renamed from mm/filemap.c~mm-charge-active-memcg-when-no-mm-is-set +1 -1
@@ -872,7 +872,7 @@ noinline int __add_to_page_cache_locked( page->index = offset;if (!huge) { - error =mem_cgroup_charge (page, current->mm, gfp); + error =mem_cgroup_charge (page,NULL , gfp);if (error) goto error; charged = true;
mm/memcontrol.c renamed from mm/memcontrol.c~mm-charge-active-memcg-when-no-mm-is-set +27 -14
69 lines not on this page · raw message
mm/shmem.c renamed from mm/shmem.c~mm-charge-active-memcg-when-no-mm-is-set +2 -2
@@ -1695,7 +1695,7 @@ static int shmem_swapin_page(struct inod {struct address_space *mapping = inode->i_mapping;struct shmem_inode_info *info =SHMEM_I (inode); -struct mm_struct *charge_mm = vma ? vma->vm_mm : current->mm; +struct mm_struct *charge_mm = vma ? vma->vm_mm :NULL ;struct swap_info_struct *si;struct page *page =NULL ;swp_entry_t swap;
@@ -1828,7 +1828,7 @@ repeat: } sbinfo =SHMEM_SB (inode->i_sb); - charge_mm = vma ? vma->vm_mm : current->mm; + charge_mm = vma ? vma->vm_mm :NULL ; page =pagecache_get_page (mapping, index,FGP_ENTRY |FGP_HEAD |FGP_LOCK ,0 );
_
Andrew Morton
maintainerPATCH 96/192 loop: charge i/o to mem and blk cg
In reply to Andrew MortonFrom: Dan Schatzberg <[email protected]>
Subject: loop: charge i/o to mem and blk cg
The current code only associates with the existing blkcg when aio is used to access the backing file. This patch covers all types of i/o to the backing file and also associates the memcg so if the backing file is on tmpfs, memory is charged appropriately.
This patch also exports cgroup_get_e_css and int_active_memcg so it can be used by the loop module.
---
drivers/block/loop.c | 61 +++++++++++++++++++++++------------ drivers/block/loop.h | 3 + include/linux/memcontrol.h | 6 +++ kernel/cgroup/cgroup.c | 1 mm/memcontrol.c | 1 5 files changed, 51 insertions(+), 21 deletions(-)
drivers/block/loop.c renamed from drivers/block/loop.c~loop-charge-i-o-to-mem-and-blk-cg +41 -20
@@ -78,6 +78,7 @@#include <linux/uio.h> #include <linux/ioprio.h> #include <linux/blk-cgroup.h> +#include <linux/sched/mm.h> #include "loop.h"
@@ -516,8 +517,6 @@ static void lo_rw_aio_complete(struct ki { struct loop_cmd *cmd = container_of(iocb, struct loop_cmd, iocb); -if (cmd->css) -css_put (cmd->css); cmd->ret = ret;lo_rw_aio_do_completion (cmd); }
@@ -578,8 +577,6 @@ static int lo_rw_aio(struct loop_device cmd->iocb.ki_complete = lo_rw_aio_complete; cmd->iocb.ki_flags =IOCB_DIRECT ; cmd->iocb.ki_ioprio =IOPRIO_PRIO_VALUE (IOPRIO_CLASS_NONE ,0 ); -if (cmd->css) -kthread_associate_blkcg (cmd->css);if (rw ==WRITE ) ret =call_write_iter (file, &cmd->iocb, &iter);
@@ -587,7 +584,6 @@ static int lo_rw_aio(struct loop_device ret =call_read_iter (file, &cmd->iocb, &iter);lo_rw_aio_do_completion (cmd); -kthread_associate_blkcg (NULL );if (ret != -EIOCBQUEUED ) cmd->iocb.ki_complete (&cmd->iocb, ret,0 );
@@ -928,7 +924,7 @@ struct loop_worker {struct list_head cmd_list;struct list_head idle_list;struct loop_device *lo; -struct cgroup_subsys_state *css; +struct cgroup_subsys_state *blkcg_css;unsigned long last_ran_at; };
@@ -957,7 +953,7 @@ static void loop_queue_work(struct loop_spin_lock_irq (&lo->lo_work_lock); -if (queue_on_root_worker (cmd->css)) +if (queue_on_root_worker (cmd->blkcg_css)) goto queue_work; node = &lo->worker_tree.rb_node;
@@ -965,10 +961,10 @@ static void loop_queue_work(struct loop_while (*node) { parent = *node; cur_worker = container_of(*node, struct loop_worker, rb_node); -if (cur_worker->css == cmd->css) { +if (cur_worker->blkcg_css == cmd->blkcg_css) { worker = cur_worker;break ; - }else if ((long )cur_worker->css < (long )cmd->css) { + }else if ((long )cur_worker->blkcg_css < (long )cmd->blkcg_css) { node = &(*node)->rb_left; }else { node = &(*node)->rb_right;
@@ -980,13 +976,18 @@ static void loop_queue_work(struct loop_ worker =kzalloc (sizeof (struct loop_worker ),GFP_NOWAIT | __GFP_NOWARN);/* * In the event we cannot allocate a worker, just queue on the -* rootcg worker +* rootcg worker and issue the I/O as the rootcg */ -if (!worker) +if (!worker) { + cmd->blkcg_css =NULL ; +if (cmd->memcg_css) +css_put (cmd->memcg_css); + cmd->memcg_css =NULL ; goto queue_work; + } - worker->css = cmd->css; -css_get (worker->css); + worker->blkcg_css = cmd->blkcg_css; +css_get (worker->blkcg_css);INIT_WORK (&worker->work, loop_workfn);INIT_LIST_HEAD (&worker->cmd_list);INIT_LIST_HEAD (&worker->idle_list);
@@ -1306,7 +1307,7 @@ static int __loop_clr_fd(struct loop_dev idle_list) {list_del (&worker->idle_list);rb_erase (&worker->rb_node, &lo->worker_tree); -css_put (worker->css); +css_put (worker->blkcg_css);kfree (worker); }spin_unlock_irq (&lo->lo_work_lock);
@@ -2100,13 +2101,18 @@ static blk_status_t loop_queue_rq(struct }/* always use the first bio's css */ + cmd->blkcg_css =NULL ; + cmd->memcg_css =NULL ;#ifdef CONFIG_BLK_CGROUP -if (cmd->use_aio && rq->bio && rq->bio->bi_blkg) { - cmd->css = &bio_blkcg (rq->bio)->css; -css_get (cmd->css); - } else +if (rq->bio && rq->bio->bi_blkg) { + cmd->blkcg_css = &bio_blkcg (rq->bio)->css; +#ifdef CONFIG_MEMCG + cmd->memcg_css = +cgroup_get_e_css (cmd->blkcg_css->cgroup, + &memory_cgrp_subsys); +#endif + }#endif - cmd->css =NULL ;loop_queue_work (lo, cmd);return BLK_STS_OK ;
@@ -2118,13 +2124,28 @@ static void loop_handle_cmd(struct loop_const bool write =op_is_write (req_op (rq));struct loop_device *lo = rq->q->queuedata;int ret =0 ; +struct mem_cgroup *old_memcg =NULL ;if (write && (lo->lo_flags &LO_FLAGS_READ_ONLY )) { ret = -EIO ; goto failed; } +if (cmd->blkcg_css) +kthread_associate_blkcg (cmd->blkcg_css); +if (cmd->memcg_css) + old_memcg =set_active_memcg ( +mem_cgroup_from_css (cmd->memcg_css)); + ret =do_req_filebacked (lo, rq); + +if (cmd->blkcg_css) +kthread_associate_blkcg (NULL ); + +if (cmd->memcg_css) { +set_active_memcg (old_memcg); +css_put (cmd->memcg_css); + } failed:/* complete non-aio request */ if (!cmd->use_aio || ret) {
@@ -2203,7 +2224,7 @@ static void loop_free_idle_workers(strucbreak ;list_del (&worker->idle_list);rb_erase (&worker->rb_node, &lo->worker_tree); -css_put (worker->css); +css_put (worker->blkcg_css);kfree (worker); }if (!list_empty (&lo->idle_worker_list))
drivers/block/loop.h renamed from drivers/block/loop.h~loop-charge-i-o-to-mem-and-blk-cg +2 -1
@@ -77,7 +77,8 @@ struct loop_cmd {long ret;struct kiocb iocb;struct bio_vec *bvec; -struct cgroup_subsys_state *css; +struct cgroup_subsys_state *blkcg_css; +struct cgroup_subsys_state *memcg_css; };/* Support for loadable transfer modules */
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~loop-charge-i-o-to-mem-and-blk-cg +6 -0
@@ -1230,6 +1230,12 @@ static inline struct mem_cgroup *get_memreturn NULL ; } +static inline +struct mem_cgroup *mem_cgroup_from_css (struct cgroup_subsys_state *css) +{ +return NULL ; +} +static inline void mem_cgroup_put (struct mem_cgroup *memcg) { }
kernel/cgroup/cgroup.c renamed from kernel/cgroup/cgroup.c~loop-charge-i-o-to-mem-and-blk-cg +1 -0
@@ -577,6 +577,7 @@ out_unlock:rcu_read_unlock ();return css; } +EXPORT_SYMBOL_GPL (cgroup_get_e_css);static void cgroup_get_live (struct cgroup *cgrp) {
mm/memcontrol.c renamed from mm/memcontrol.c~loop-charge-i-o-to-mem-and-blk-cg +1 -0
@@ -78,6 +78,7 @@ struct mem_cgroup *root_mem_cgroup __rea/* Active memory cgroup to use from an interrupt context */ DEFINE_PER_CPU(struct mem_cgroup *, int_active_memcg); +EXPORT_PER_CPU_SYMBOL_GPL (int_active_memcg);/* Socket memory accounting disabled? */ static bool cgroup_memory_nosocket;
_
Andrew Morton
maintainerPATCH 97/192 mm: memcontrol: remove trailing semicolon in macros
In reply to Andrew MortonFrom: Huilong Deng <[email protected]>
Subject: mm: memcontrol: remove trailing semicolon in macros
Macros should not use a trailing semicolon.
---
include/linux/memcontrol.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-memcontrol-remove-trailing-semicolon-in-macros +1 -1
@@ -192,7 +192,7 @@ enum memcg_kmem_state {struct memcg_padding {char x[0 ]; }____cacheline_internodealigned_in_smp ; -#define MEMCG_PADDING (name) struct memcg_padding name; +#define MEMCG_PADDING (name) struct memcg_padding name#else #define MEMCG_PADDING (name)#endif
_
Andrew Morton
maintainerPATCH 98/192 perf: MAP_EXECUTABLE does not indicate VM_MAYEXEC
In reply to Andrew MortonFrom: David Hildenbrand <[email protected]>
Subject: perf: MAP_EXECUTABLE does not indicate VM_MAYEXEC
Patch series "perf/binfmt/mm: remove in-tree usage of MAP_EXECUTABLE".
Stumbling over the history of MAP_EXECUTABLE, I noticed that we still have some in-tree users that we can get rid of.
This patch (of 3):
Before commit e9714acf8c43 ("mm: kill vma flag VM_EXECUTABLE and mm->num_exe_file_vmas"), VM_EXECUTABLE indicated MAP_EXECUTABLE. MAP_EXECUTABLE is nowadays essentially ignored by the kernel and does not relate to VM_MAYEXEC.
---
kernel/events/core.c | 2 -- 1 file changed, 2 deletions(-)
kernel/events/core.c renamed from kernel/events/core.c~perf-map_executable-does-not-indicate-vm_mayexec +0 -2
@@ -8301,8 +8301,6 @@ static void perf_event_mmap_event(structif (vma->vm_flags &VM_DENYWRITE ) flags |=MAP_DENYWRITE ; -if (vma->vm_flags &VM_MAYEXEC ) - flags |=MAP_EXECUTABLE ;if (vma->vm_flags &VM_LOCKED ) flags |=MAP_LOCKED ;if (is_vm_hugetlb_page (vma))
_
Andrew Morton
maintainerPATCH 99/192 binfmt: remove in-tree usage of MAP_EXECUTABLE
In reply to Andrew MortonFrom: David Hildenbrand <[email protected]>
Subject: binfmt: remove in-tree usage of MAP_EXECUTABLE
Ever since commit e9714acf8c43 ("mm: kill vma flag VM_EXECUTABLE and mm->num_exe_file_vmas"), VM_EXECUTABLE is gone and MAP_EXECUTABLE is essentially completely ignored. Let's remove all usage of MAP_EXECUTABLE.
---
arch/x86/ia32/ia32_aout.c | 4 ++-- fs/binfmt_aout.c | 4 ++-- fs/binfmt_elf.c | 2 +- fs/binfmt_elf_fdpic.c | 11 ++--------- fs/binfmt_flat.c | 2 +- 5 files changed, 8 insertions(+), 15 deletions(-)
arch/x86/ia32/ia32_aout.c renamed from arch/x86/ia32/ia32_aout.c~binfmt-remove-in-tree-usage-of-map_executable +2 -2
@@ -203,7 +203,7 @@ static int load_aout_binary(struct linux error =vm_mmap (bprm->file,N_TXTADDR (ex), ex.a_text,PROT_READ |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE | -MAP_EXECUTABLE |MAP_32BIT , +MAP_32BIT , fd_offset);if (error !=N_TXTADDR (ex))
@@ -212,7 +212,7 @@ static int load_aout_binary(struct linux error =vm_mmap (bprm->file,N_DATADDR (ex), ex.a_data,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE | -MAP_EXECUTABLE |MAP_32BIT , +MAP_32BIT , fd_offset + ex.a_text);if (error !=N_DATADDR (ex))return error;
fs/binfmt_aout.c renamed from fs/binfmt_aout.c~binfmt-remove-in-tree-usage-of-map_executable +2 -2
@@ -222,7 +222,7 @@ static int load_aout_binary(struct linux error =vm_mmap (bprm->file,N_TXTADDR (ex), ex.a_text,PROT_READ |PROT_EXEC , -MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE |MAP_EXECUTABLE , +MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE , fd_offset);if (error !=N_TXTADDR (ex))
@@ -230,7 +230,7 @@ static int load_aout_binary(struct linux error =vm_mmap (bprm->file,N_DATADDR (ex), ex.a_data,PROT_READ |PROT_WRITE |PROT_EXEC , -MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE |MAP_EXECUTABLE , +MAP_FIXED |MAP_PRIVATE |MAP_DENYWRITE , fd_offset + ex.a_text);if (error !=N_DATADDR (ex))return error;
fs/binfmt_elf.c renamed from fs/binfmt_elf.c~binfmt-remove-in-tree-usage-of-map_executable +1 -1
@@ -1070,7 +1070,7 @@ out_free_interp: elf_prot =make_prot (elf_ppnt->p_flags, &arch_state, !!interpreter, false); - elf_flags =MAP_PRIVATE |MAP_DENYWRITE |MAP_EXECUTABLE ; + elf_flags =MAP_PRIVATE |MAP_DENYWRITE ; vaddr = elf_ppnt->p_vaddr;/*
fs/binfmt_elf_fdpic.c renamed from fs/binfmt_elf_fdpic.c~binfmt-remove-in-tree-usage-of-map_executable +2 -9
@@ -928,7 +928,7 @@ static int elf_fdpic_map_file_constdisp_ {struct elf32_fdpic_loadseg *seg;struct elf32_phdr *phdr; -unsigned long load_addr, base =ULONG_MAX , top =0 , maddr =0 , mflags; +unsigned long load_addr, base =ULONG_MAX , top =0 , maddr =0 ;int loop, ret; load_addr = params->load_addr;
@@ -948,12 +948,8 @@ static int elf_fdpic_map_file_constdisp_ }/* allocate one big anon block for everything */ - mflags =MAP_PRIVATE ; -if (params->flags &ELF_FDPIC_FLAG_EXECUTABLE ) - mflags |=MAP_EXECUTABLE ; - maddr =vm_mmap (NULL , load_addr, top - base, -PROT_READ |PROT_WRITE |PROT_EXEC , mflags,0 ); +PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_PRIVATE ,0 );if (IS_ERR_VALUE (maddr))return (int ) maddr;
@@ -1046,9 +1042,6 @@ static int elf_fdpic_map_file_by_direct_if (phdr->p_flags &PF_X ) prot |=PROT_EXEC ; flags =MAP_PRIVATE |MAP_DENYWRITE ; -if (params->flags &ELF_FDPIC_FLAG_EXECUTABLE ) - flags |=MAP_EXECUTABLE ; - maddr =0 ;switch (params->flags &ELF_FDPIC_FLAG_ARRANGEMENT ) {
fs/binfmt_flat.c renamed from fs/binfmt_flat.c~binfmt-remove-in-tree-usage-of-map_executable +1 -1
@@ -573,7 +573,7 @@ static int load_flat_file(struct linux_bpr_debug ("ROM mapping of file (we hope)\n" ); textpos =vm_mmap (bprm->file,0 , text_len,PROT_READ |PROT_EXEC , -MAP_PRIVATE |MAP_EXECUTABLE ,0 ); +MAP_PRIVATE ,0 );if (!textpos ||IS_ERR_VALUE (textpos)) { ret = textpos;if (!textpos)
_
Andrew Morton
maintainerPATCH 100/192 mm: ignore MAP_EXECUTABLE in ksys_mmap_pgoff()
In reply to Andrew MortonFrom: David Hildenbrand <[email protected]>
Subject: mm: ignore MAP_EXECUTABLE in ksys_mmap_pgoff()
Let's also remove masking off MAP_EXECUTABLE from ksys_mmap_pgoff(): the last in-tree occurrence of MAP_EXECUTABLE is now in LEGACY_MAP_MASK, which accepts the flag e.g., for MAP_SHARED_VALIDATE; however, the flag is ignored throughout the kernel now.
Add a comment to LEGACY_MAP_MASK stating that MAP_EXECUTABLE is ignored.
---
include/linux/mman.h | 2 ++ mm/mmap.c | 2 +- mm/nommu.c | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-)
include/linux/mman.h renamed from include/linux/mman.h~mm-ignore-map_executable-in-ksys_mmap_pgoff +2 -0
@@ -31,6 +31,8 @@/* * The historical set of flags that all mmap implementations implicitly * support when a ->mmap_validate() op is not provided in file_operations. +* +* MAP_EXECUTABLE is completely ignored throughout the kernel. */ #define LEGACY_MAP_MASK (MAP_SHARED \ | MAP_PRIVATE \
mm/mmap.c renamed from mm/mmap.c~mm-ignore-map_executable-in-ksys_mmap_pgoff +1 -1
@@ -1633,7 +1633,7 @@ unsigned long ksys_mmap_pgoff(unsigned lreturn PTR_ERR (file); } - flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); + flags &= ~MAP_DENYWRITE ; retval =vm_mmap_pgoff (file, addr, len, prot, flags, pgoff); out_fput:
mm/nommu.c renamed from mm/nommu.c~mm-ignore-map_executable-in-ksys_mmap_pgoff +1 -1
@@ -1296,7 +1296,7 @@ unsigned long ksys_mmap_pgoff(unsigned l goto out; } - flags &= ~(MAP_EXECUTABLE |MAP_DENYWRITE ); + flags &= ~MAP_DENYWRITE ; retval =vm_mmap_pgoff (file, addr, len, prot, flags, pgoff);
_
Andrew Morton
maintainerPATCH 101/192 mm/mmap.c: logic of find_vma_intersection repeated in __do_munmap
In reply to Andrew MortonFrom: Gonzalo Matias Juarez Tello <[email protected]>
Subject: mm/mmap.c: logic of find_vma_intersection repeated in __do_munmap
Logic of find_vma_intersection() is repeated in __do_munmap().
Also, prev is assigned a value before checking vma->vm_start >= end which might end up on a return statement making that assignment useless.
Calling find_vma_intersection() checks that condition and returns NULL if no vma is found, hence only the !vma check is needed in __do_munmap().
---
mm/mmap.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
mm/mmap.c renamed from mm/mmap.c~mm-mmapc-logic-of-find_vma_intersection-repeated-in-__do_munmap +2 -7
@@ -2828,16 +2828,11 @@ int __do_munmap(struct mm_struct *mm, un*/ arch_unmap (mm, start, end); -/* Find the first overlapping VMA */ - vma =find_vma (mm, start); +/* Find the first overlapping VMA where start < vma->vm_end */ + vma =find_vma_intersection (mm, start, end);if (!vma)return 0 ; prev = vma->vm_prev; -/* we have start < vma->vm_end */ - -/* if it doesn't overlap, we have nothing.. */ -if (vma->vm_start >= end) -return 0 ;/* * If we need to split any vma, do it now to save pain later.
_
Andrew Morton
maintainerPATCH 102/192 mm/mmap: introduce unlock_range() for code cleanup
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: mm/mmap: introduce unlock_range() for code cleanup
Both __do_munmap() and exit_mmap() unlock a range of VMAs using almost identical code blocks. Replace both blocks by a static inline function.
---
mm/mmap.c | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-)
mm/mmap.c renamed from mm/mmap.c~mm-mmap-introduce-unlock_range-for-code-cleanup +20 -19
@@ -2802,6 +2802,22 @@ int split_vma(struct mm_struct *mm, strureturn __split_vma (mm, vma, addr, new_below); } +static inline void +unlock_range (struct vm_area_struct *start,unsigned long limit) +{ +struct mm_struct *mm = start->vm_mm; +struct vm_area_struct *tmp = start; + +while (tmp && tmp->vm_start < limit) { +if (tmp->vm_flags &VM_LOCKED ) { + mm->locked_vm -=vma_pages (tmp); +munlock_vma_pages_all (tmp); + } + + tmp = tmp->vm_next; + } +} +/* Munmap is split into 2 main parts -- this part which finds * what needs doing, and the areas themselves, which do the * work. This now handles partial unmappings.
@@ -2885,17 +2901,8 @@ int __do_munmap(struct mm_struct *mm, un/* * unlock any mlock()ed ranges before detaching vmas */ -if (mm->locked_vm) { -struct vm_area_struct *tmp = vma; -while (tmp && tmp->vm_start < end) { -if (tmp->vm_flags &VM_LOCKED ) { - mm->locked_vm -=vma_pages (tmp); -munlock_vma_pages_all (tmp); - } - - tmp = tmp->vm_next; - } - } +if (mm->locked_vm) +unlock_range (vma, end);/* Detach vmas from rbtree */ if (!detach_vmas_to_be_unmapped (mm, vma, prev, end))
@@ -3180,14 +3187,8 @@ void exit_mmap(struct mm_struct *mm)mmap_write_unlock (mm); } -if (mm->locked_vm) { - vma = mm->mmap; -while (vma) { -if (vma->vm_flags &VM_LOCKED ) -munlock_vma_pages_all (vma); - vma = vma->vm_next; - } - } +if (mm->locked_vm) +unlock_range (mm->mmap,ULONG_MAX );arch_exit_mmap (mm);
_
Andrew Morton
maintainerPATCH 103/192 mm/mmap: use find_vma_intersection() in do_mmap() for overlap
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: mm/mmap: use find_vma_intersection() in do_mmap() for overlap
Using find_vma_intersection() avoids the need for a temporary variable and makes the code cleaner.
---
mm/mmap.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
mm/mmap.c renamed from mm/mmap.c~mm-mmap-use-find_vma_intersection-in-do_mmap-for-overlap +1 -3
@@ -1457,9 +1457,7 @@ unsigned long do_mmap(struct file *file,return addr;if (flags &MAP_FIXED_NOREPLACE ) { -struct vm_area_struct *vma =find_vma (mm, addr); - -if (vma && vma->vm_start < addr + len) +if (find_vma_intersection (mm, addr, addr + len))return -EEXIST ; }
_
Andrew Morton
maintainerPATCH 104/192 mm/memory.c: fix comment of finish_mkwrite_fault()
In reply to Andrew MortonFrom: Liu Xiang <[email protected]>
Subject: mm/memory.c: fix comment of finish_mkwrite_fault()
Fix the return value in comment of finish_mkwrite_fault().
---
mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/memory.c renamed from mm/memory.c~mm-memoryc-fix-comment-of-finish_mkwrite_fault +1 -1
@@ -3049,7 +3049,7 @@ oom:* The function expects the page to be locked or other protection against * concurrent faults / writeback (such as DAX radix tree locks). * -* Return: %VM_FAULT_WRITE on success, %0 when PTE got changed before +* Return: %0 on success, %VM_FAULT_NOPAGE when PTE got changed before * we acquired PTE lock. */ vm_fault_t finish_mkwrite_fault (struct vm_fault *vmf)
_
Andrew Morton
maintainerPATCH 105/192 mm: add vma_lookup(), update find_vma_intersection() comments
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: mm: add vma_lookup(), update find_vma_intersection() comments
Patch series "mm: Add vma_lookup()", v2.
Many places in the kernel use find_vma() to get a vma and then check the start address of the vma to ensure the next vma was not returned.
Other places use the find_vma_intersection() call with add, addr + 1 as the range; looking for just the vma at a specific address.
The third use of find_vma() is by developers who do not know that the function starts searching at the provided address upwards for the next vma. This results in a bug that is often overlooked for a long time.
Adding the new vma_lookup() function will allow for cleaner code by removing the find_vma() calls which check limits, making find_vma_intersection() calls of a single address to be shorter, and potentially reduce the incorrect uses of find_vma().
This patch (of 22):
Many places in the kernel use find_vma() to get a vma and then check the start address of the vma to ensure the next vma was not returned.
Other places use the find_vma_intersection() call with add, addr + 1 as the range; looking for just the vma at a specific address.
The third use of find_vma() is by developers who do not know that the function starts searching at the provided address upwards for the next vma. This results in a bug that is often overlooked for a long time.
Adding the new vma_lookup() function will allow for cleaner code by removing the find_vma() calls which check limits, making find_vma_intersection() calls of a single address to be shorter, and potentially reduce the incorrect uses of find_vma().
Also change find_vma_intersection() comments and declaration to be of the correct length and add kernel documentation style comment.
---
include/linux/mm.h | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-add-vma_lookup-update-find_vma_intersection-comments +32 -4
@@ -2676,17 +2676,45 @@ extern struct vm_area_struct * find_vma(extern struct vm_area_struct *find_vma_prev (struct mm_struct * mm,unsigned long addr,struct vm_area_struct **pprev); -/* Look up the first VMA which intersects the interval start_addr..end_addr-1, -NULL if none. Assume start_addr < end_addr. */ -static inline struct vm_area_struct *find_vma_intersection (struct mm_struct * mm,unsigned long start_addr,unsigned long end_addr) +/** +* find_vma_intersection() - Look up the first VMA which intersects the interval +* @mm: The process address space. +* @start_addr: The inclusive start user address. +* @end_addr: The exclusive end user address. +* +* Returns: The first VMA within the provided range, %NULL otherwise. Assumes +* start_addr < end_addr. +*/ +static inline +struct vm_area_struct *find_vma_intersection (struct mm_struct *mm, +unsigned long start_addr, +unsigned long end_addr) { -struct vm_area_struct * vma =find_vma (mm,start_addr); +struct vm_area_struct *vma =find_vma (mm, start_addr);if (vma && end_addr <= vma->vm_start) vma =NULL ;return vma; } +/** +* vma_lookup() - Find a VMA at a specific address +* @mm: The process address space. +* @addr: The user address. +* +* Return: The vm_area_struct at the given address, %NULL otherwise. +*/ +static inline +struct vm_area_struct *vma_lookup (struct mm_struct *mm,unsigned long addr) +{ +struct vm_area_struct *vma =find_vma (mm, addr); + +if (vma && addr < vma->vm_start) + vma =NULL ; + +return vma; +} +static inline unsigned long vm_start_gap (struct vm_area_struct *vma) {unsigned long vm_start = vma->vm_start;
_
Andrew Morton
maintainerPATCH 106/192 drm/i915/selftests: use vma_lookup() in __igt_mmap()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: drm/i915/selftests: use vma_lookup() in __igt_mmap()
vma_lookup() will look up the vma at a specific address. find_vma() will start the search for a specific address and continue upwards. This fixes an issue with the selftest as the returned vma may not be the newly created vma, but simply the vma at a higher address.
objects
---
drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c renamed from drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c~drm-i915-selftests-use-vma_lookup-in-__igt_mmap +1 -1
@@ -871,7 +871,7 @@ static int __igt_mmap(struct drm_i915_prpr_debug ("igt_mmap(%s, %d) @ %lx\n" , obj->mm.region->name, type, addr); - area =find_vma (current->mm, addr); + area =vma_lookup (current->mm, addr);if (!area) {pr_err ("%s: Did not create a vm_area_struct for the mmap\n" , obj->mm.region->name);
_
Andrew Morton
maintainerPATCH 107/192 arch/arc/kernel/troubleshoot: use vma_lookup() instead of find_vma()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: arch/arc/kernel/troubleshoot: use vma_lookup() instead of find_vma()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
arch/arc/kernel/troubleshoot.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/arc/kernel/troubleshoot.c renamed from arch/arc/kernel/troubleshoot.c~arch-arc-kernel-troubleshoot-use-vma_lookup-instead-of-find_vma +4 -4
@@ -83,12 +83,12 @@ static void show_faulting_vma(unsigned l* non-inclusive vma */ mmap_read_lock (active_mm); - vma =find_vma (active_mm, address); + vma =vma_lookup (active_mm, address); -/* check against the find_vma( ) behaviour which returns the next VMA -* if the container VMA is not found +/* Lookup the vma at the address and report if the container VMA is not +* found */ -if (vma && (vma->vm_start <= address)) { +if (vma) {char buf[ARC_PATH_MAX ];char *nm ="?" ;
_
Andrew Morton
maintainerPATCH 108/192 arch/arm64/kvm: use vma_lookup() instead of find_vma_intersection()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: arch/arm64/kvm: use vma_lookup() instead of find_vma_intersection()
vma_lookup() finds the vma of a specific address with a cleaner interface and is more readable.
---
arch/arm64/kvm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/mmu.c renamed from arch/arm64/kvm/mmu.c~arch-arm64-kvm-use-vma_lookup-instead-of-find_vma_intersection +1 -1
@@ -855,7 +855,7 @@ static int user_mem_abort(struct kvm_vcp/* Let's check if we will get back a huge page backed by hugetlbfs */ mmap_read_lock (current->mm); - vma =find_vma_intersection (current->mm, hva, hva +1 ); + vma =vma_lookup (current->mm, hva);if (unlikely (!vma)) {kvm_err ("Failed to find VMA for hva 0x%lx\n" , hva);mmap_read_unlock (current->mm);
_
Andrew Morton
maintainerPATCH 109/192 arch/powerpc/kvm/book3s_hv_uvmem: use vma_lookup() instead of find_vma_intersection()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: arch/powerpc/kvm/book3s_hv_uvmem: use vma_lookup() instead of find_vma_intersection()
vma_lookup() finds the vma of a specific address with a cleaner interface and is more readable.
---
arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/kvm/book3s_hv_uvmem.c renamed from arch/powerpc/kvm/book3s_hv_uvmem.c~arch-powerpc-kvm-book3s_hv_uvmem-use-vma_lookup-instead-of-find_vma_intersection +1 -1
@@ -614,7 +614,7 @@ void kvmppc_uvmem_drop_pages(const struc/* Fetch the VMA if addr is not in the latest fetched one */ if (!vma || addr >= vma->vm_end) { - vma =find_vma_intersection (kvm->mm, addr, addr+1 ); + vma =vma_lookup (kvm->mm, addr);if (!vma) {pr_err ("Can't find VMA for gfn:0x%lx\n" , gfn);break ;
_
Andrew Morton
maintainerPATCH 110/192 arch/powerpc/kvm/book3s: use vma_lookup() in kvmppc_hv_setup_htab_rma()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: arch/powerpc/kvm/book3s: use vma_lookup() in kvmppc_hv_setup_htab_rma()
Using vma_lookup() removes the requirement to check if the address is within the returned vma. The code is easier to understand and more compact.
---
arch/powerpc/kvm/book3s_hv.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/powerpc/kvm/book3s_hv.c renamed from arch/powerpc/kvm/book3s_hv.c~arch-powerpc-kvm-book3s-use-vma_lookup-in-kvmppc_hv_setup_htab_rma +2 -2
@@ -4758,8 +4758,8 @@ static int kvmppc_hv_setup_htab_rma(stru/* Look up the VMA for the start of this memory slot */ hva = memslot->userspace_addr;mmap_read_lock (kvm->mm); - vma =find_vma (kvm->mm, hva); -if (!vma || vma->vm_start > hva || (vma->vm_flags &VM_IO )) + vma =vma_lookup (kvm->mm, hva); +if (!vma || (vma->vm_flags &VM_IO )) goto up_out; psize =vma_kernel_pagesize (vma);
_
Andrew Morton
maintainerPATCH 111/192 arch/mips/kernel/traps: use vma_lookup() instead of find_vma()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: arch/mips/kernel/traps: use vma_lookup() instead of find_vma()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
arch/mips/kernel/traps.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/mips/kernel/traps.c renamed from arch/mips/kernel/traps.c~arch-mips-kernel-traps-use-vma_lookup-instead-of-find_vma +1 -3
@@ -784,7 +784,6 @@ void force_fcr31_sig(unsigned long fcr31int process_fpemu_return (int sig,void __user *fault_addr,unsigned long fcr31) {int si_code; -struct vm_area_struct *vma;switch (sig) {case 0 :
@@ -800,8 +799,7 @@ int process_fpemu_return(int sig, void _case SIGSEGV :mmap_read_lock (current->mm); - vma =find_vma (current->mm, (unsigned long )fault_addr); -if (vma && (vma->vm_start <= (unsigned long )fault_addr)) +if (vma_lookup (current->mm, (unsigned long )fault_addr)) si_code =SEGV_ACCERR ;else si_code =SEGV_MAPERR ;
_
Andrew Morton
maintainerPATCH 112/192 arch/m68k/kernel/sys_m68k: use vma_lookup() in sys_cacheflush()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: arch/m68k/kernel/sys_m68k: use vma_lookup() in sys_cacheflush()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
arch/m68k/kernel/sys_m68k.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/m68k/kernel/sys_m68k.c renamed from arch/m68k/kernel/sys_m68k.c~arch-m68k-kernel-sys_m68k-use-vma_lookup-in-sys_cacheflush +2 -2
@@ -402,8 +402,8 @@ sys_cacheflush (unsigned long addr, int* to this process. */ mmap_read_lock (current->mm); - vma =find_vma (current->mm, addr); -if (!vma || addr < vma->vm_start || addr + len > vma->vm_end) + vma =vma_lookup (current->mm, addr); +if (!vma || addr + len > vma->vm_end) goto out_unlock; }
_
Andrew Morton
maintainerPATCH 113/192 x86/sgx: use vma_lookup() in sgx_encl_find()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: x86/sgx: use vma_lookup() in sgx_encl_find()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
arch/x86/kernel/cpu/sgx/encl.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/sgx/encl.h renamed from arch/x86/kernel/cpu/sgx/encl.h~x86-sgx-use-vma_lookup-in-sgx_encl_find +2 -2
@@ -91,8 +91,8 @@ static inline int sgx_encl_find(struct m {struct vm_area_struct *result; - result =find_vma (mm, addr); -if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start) + result =vma_lookup (mm, addr); +if (!result || result->vm_ops != &sgx_vm_ops)return -EINVAL ; *vma = result;
_
Andrew Morton
maintainerPATCH 114/192 virt/kvm: use vma_lookup() instead of find_vma_intersection()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: virt/kvm: use vma_lookup() instead of find_vma_intersection()
vma_lookup() finds the vma of a specific address with a cleaner interface and is more readable.
---
virt/kvm/kvm_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
virt/kvm/kvm_main.c renamed from virt/kvm/kvm_main.c~virt-kvm-use-vma_lookup-instead-of-find_vma_intersection +1 -1
@@ -2170,7 +2170,7 @@ static kvm_pfn_t hva_to_pfn(unsigned lon } retry: - vma =find_vma_intersection (current->mm, addr, addr +1 ); + vma =vma_lookup (current->mm, addr);if (vma ==NULL ) pfn =KVM_PFN_ERR_FAULT ;
_
Andrew Morton
maintainerPATCH 115/192 vfio: use vma_lookup() instead of find_vma_intersection()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: vfio: use vma_lookup() instead of find_vma_intersection()
vma_lookup() finds the vma of a specific address with a cleaner interface and is more readable.
---
drivers/vfio/vfio_iommu_type1.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/vfio_iommu_type1.c renamed from drivers/vfio/vfio_iommu_type1.c~vfio-use-vma_lookup-instead-of-find_vma_intersection +1 -1
@@ -567,7 +567,7 @@ static int vaddr_get_pfns(struct mm_stru vaddr =untagged_addr (vaddr); retry: - vma =find_vma_intersection (mm, vaddr, vaddr +1 ); + vma =vma_lookup (mm, vaddr);if (vma && vma->vm_flags &VM_PFNMAP ) { ret =follow_fault_pfn (vma, mm, vaddr, pfn, prot &IOMMU_WRITE );
_
Andrew Morton
maintainerPATCH 116/192 net/ipv5/tcp: use vma_lookup() in tcp_zerocopy_receive()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: net/ipv5/tcp: use vma_lookup() in tcp_zerocopy_receive()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
net/ipv4/tcp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/ipv4/tcp.c renamed from net/ipv4/tcp.c~net-ipv5-tcp-use-vma_lookup-in-tcp_zerocopy_receive +2 -2
@@ -2095,8 +2095,8 @@ static int tcp_zerocopy_receive(struct smmap_read_lock (current->mm); - vma =find_vma (current->mm, address); -if (!vma || vma->vm_start > address || vma->vm_ops != &tcp_vm_ops) { + vma =vma_lookup (current->mm, address); +if (!vma || vma->vm_ops != &tcp_vm_ops) {mmap_read_unlock (current->mm);return -EINVAL ; }
_
Andrew Morton
maintainerPATCH 117/192 drm/amdgpu: use vma_lookup() in amdgpu_ttm_tt_get_user_pages()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: drm/amdgpu: use vma_lookup() in amdgpu_ttm_tt_get_user_pages()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c~drm-amdgpu-use-vma_lookup-in-amdgpu_ttm_tt_get_user_pages +2 -2
@@ -709,8 +709,8 @@ int amdgpu_ttm_tt_get_user_pages(struct }mmap_read_lock (mm); - vma =find_vma (mm, start); -if (unlikely (!vma || start < vma->vm_start)) { + vma =vma_lookup (mm, start); +if (unlikely (!vma)) { r = -EFAULT ; goto out_unlock; }
_
Andrew Morton
maintainerPATCH 118/192 media: videobuf2: use vma_lookup() in get_vaddr_frames()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: media: videobuf2: use vma_lookup() in get_vaddr_frames()
vma_lookup() finds the vma of a specific address with a cleaner interface and is more readable.
---
drivers/media/common/videobuf2/frame_vector.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/media/common/videobuf2/frame_vector.c renamed from drivers/media/common/videobuf2/frame_vector.c~media-videobuf2-use-vma_lookup-in-get_vaddr_frames +1 -1
@@ -64,7 +64,7 @@ int get_vaddr_frames(unsigned long start do { unsigned long *nums = frame_vector_pfns(vec); - vma = find_vma_intersection(mm, start, start + 1); + vma = vma_lookup(mm, start); if (!vma) break;
_
Andrew Morton
maintainerPATCH 119/192 misc/sgi-gru/grufault: use vma_lookup() in gru_find_vma()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: misc/sgi-gru/grufault: use vma_lookup() in gru_find_vma()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
drivers/misc/sgi-gru/grufault.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/misc/sgi-gru/grufault.c renamed from drivers/misc/sgi-gru/grufault.c~misc-sgi-gru-grufault-use-vma_lookup-in-gru_find_vma +2 -2
@@ -49,8 +49,8 @@ struct vm_area_struct *gru_find_vma(unsi {struct vm_area_struct *vma; - vma =find_vma (current->mm, vaddr); -if (vma && vma->vm_start <= vaddr && vma->vm_ops == &gru_vm_ops) + vma =vma_lookup (current->mm, vaddr); +if (vma && vma->vm_ops == &gru_vm_ops)return vma;return NULL ; }
_
Andrew Morton
maintainerPATCH 120/192 kernel/events/uprobes: use vma_lookup() in find_active_uprobe()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: kernel/events/uprobes: use vma_lookup() in find_active_uprobe()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
kernel/events/uprobes.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/events/uprobes.c renamed from kernel/events/uprobes.c~kernel-events-uprobes-use-vma_lookup-in-find_active_uprobe +2 -2
@@ -2046,8 +2046,8 @@ static struct uprobe *find_active_uprobestruct vm_area_struct *vma;mmap_read_lock (mm); - vma =find_vma (mm, bp_vaddr); -if (vma && vma->vm_start <= bp_vaddr) { + vma =vma_lookup (mm, bp_vaddr); +if (vma) {if (valid_vma (vma, false)) {struct inode *inode =file_inode (vma->vm_file);loff_t offset =vaddr_to_offset (vma, bp_vaddr);
_
Andrew Morton
maintainerPATCH 121/192 lib/test_hmm: use vma_lookup() in dmirror_migrate()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: lib/test_hmm: use vma_lookup() in dmirror_migrate()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
lib/test_hmm.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
lib/test_hmm.c renamed from lib/test_hmm.c~lib-test_hmm-use-vma_lookup-in-dmirror_migrate +2 -3
@@ -686,9 +686,8 @@ static int dmirror_migrate(struct dmirrommap_read_lock (mm);for (addr = start; addr < end; addr = next) { - vma =find_vma (mm, addr); -if (!vma || addr < vma->vm_start || - !(vma->vm_flags &VM_READ )) { + vma =vma_lookup (mm, addr); +if (!vma || !(vma->vm_flags &VM_READ )) { ret = -EINVAL ; goto out; }
_
Andrew Morton
maintainerPATCH 122/192 mm/ksm: use vma_lookup() in find_mergeable_vma()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: mm/ksm: use vma_lookup() in find_mergeable_vma()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
mm/ksm.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
mm/ksm.c renamed from mm/ksm.c~mm-ksm-use-vma_lookup-in-find_mergeable_vma +2 -4
@@ -521,10 +521,8 @@ static struct vm_area_struct *find_mergestruct vm_area_struct *vma;if (ksm_test_exit (mm))return NULL ; - vma =find_vma (mm, addr); -if (!vma || vma->vm_start > addr) -return NULL ; -if (!(vma->vm_flags &VM_MERGEABLE ) || !vma->anon_vma) + vma =vma_lookup (mm, addr); +if (!vma || !(vma->vm_flags &VM_MERGEABLE ) || !vma->anon_vma)return NULL ;return vma; }
_
Andrew Morton
maintainerPATCH 123/192 mm/migrate: use vma_lookup() in do_pages_stat_array()
In reply to Andrew MortonFrom: Liam Howlett <[email protected]>
Subject: mm/migrate: use vma_lookup() in do_pages_stat_array()
Use vma_lookup() to find the VMA at a specific address. As vma_lookup() will return NULL if the address is not within any VMA, the start address no longer needs to be validated.
---
mm/migrate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/migrate.c renamed from mm/migrate.c~mm-migrate-use-vma_lookup-in-do_pages_stat_array +2 -2
@@ -1834,8 +1834,8 @@ static void do_pages_stat_array(struct mstruct page *page;int err = -EFAULT ; - vma =find_vma (mm, addr); -if (!vma || addr < vma->vm_start) + vma =vma_lookup (mm, addr); +if (!vma) goto set_status;/* FOLL_DUMP to ignore special (like zero) pages */
_