incoming
Subsystems (62)
Andrew Morton
maintainerMore mm/ work, plenty more to come.
131 patches, based on d6f9469a03d832dcd17041ed67774ffb5f3e73b3.
Subsystems affected by this patch series:
mm/slub
mm/memcg
mm/gup
mm/kasan
mm/pagealloc
mm/hugetlb
mm/vmscan
mm/tools
mm/mempolicy
mm/memblock
mm/hugetlbfs
mm/thp
mm/mmap
mm/kconfig
Subsystem: mm/slub
Wang Hai <[email protected]>:
mm/slub: fix a memory leak in sysfs_slab_add()
Subsystem: mm/memcg
Shakeel Butt <[email protected]>:
mm/memcg: optimize memory.numa_stat like memory.stat
Subsystem: mm/gup
John Hubbard <[email protected]>:
Patch series "mm/gup, drm/i915: refactor gup_fast, convert to pin_user_pages()", v2:
mm/gup: move __get_user_pages_fast() down a few lines in gup.c mm/gup: refactor and de-duplicate gup_fast() code
mm/gup: introduce pin_user_pages_fast_only()
drm/i915: convert get_user_pages() --> pin_user_pages()
mm/gup: might_lock_read(mmap_sem) in get_user_pages_fast()
Subsystem: mm/kasan
Daniel Axtens <[email protected]>:
Patch series "Fix some incompatibilites between KASAN and FORTIFY_SOURCE", v4:
kasan: stop tests being eliminated as dead code with FORTIFY_SOURCE string.h: fix incompatibility between FORTIFY_SOURCE and KASAN
Subsystem: mm/pagealloc
Michal Hocko <[email protected]>:
mm: clarify __GFP_MEMALLOC usage
Mike Rapoport <[email protected]>:
Patch series "mm: rework free_area_init*() funcitons":
mm: memblock: replace dereferences of memblock_region.nid with API calls
mm: make early_pfn_to_nid() and related defintions close to each other
mm: remove CONFIG_HAVE_MEMBLOCK_NODE_MAP option
mm: free_area_init: use maximal zone PFNs rather than zone sizes
mm: use free_area_init() instead of free_area_init_nodes()
alpha: simplify detection of memory zone boundaries
arm: simplify detection of memory zone boundaries
arm64: simplify detection of memory zone boundaries for UMA configs
csky: simplify detection of memory zone boundaries
m68k: mm: simplify detection of memory zone boundaries
parisc: simplify detection of memory zone boundaries
sparc32: simplify detection of memory zone boundaries
unicore32: simplify detection of memory zone boundaries
xtensa: simplify detection of memory zone boundaries
Baoquan He <[email protected]>:
mm: memmap_init: iterate over memblock regions rather that check each PFN
Mike Rapoport <[email protected]>:
mm: remove early_pfn_in_nid() and CONFIG_NODES_SPAN_OTHER_NODES
mm: free_area_init: allow defining max_zone_pfn in descending order
mm: rename free_area_init_node() to free_area_init_memoryless_node()
mm: clean up free_area_init_node() and its helpers
mm: simplify find_min_pfn_with_active_regions()
docs/vm: update memory-models documentation
Wei Yang <[email protected]>:
Patch series "mm/page_alloc.c: cleanup on check page", v3:
mm/page_alloc.c: bad_[reason|flags] is not necessary when PageHWPoison mm/page_alloc.c: bad_flags is not necessary for bad_page() mm/page_alloc.c: rename free_pages_check_bad() to check_free_page_bad() mm/page_alloc.c: rename free_pages_check() to check_free_page() mm/page_alloc.c: extract check_[new|free]_page_bad() common part to page_bad_reason()
Roman Gushchin <[email protected]>:
mm,page_alloc,cma: conditionally prefer cma pageblocks for movable allocations
Baoquan He <[email protected]>:
mm/page_alloc.c: remove unused free_bootmem_with_active_regions
Patch series "improvements about lowmem_reserve and /proc/zoneinfo", v2:
mm/page_alloc.c: only tune sysctl_lowmem_reserve_ratio value once when changing it mm/page_alloc.c: clear out zone->lowmem_reserve[] if the zone is empty mm/vmstat.c: do not show lowmem reserve protection information of empty zone
Joonsoo Kim <[email protected]>:
Patch series "integrate classzone_idx and high_zoneidx", v5:
mm/page_alloc: use ac->high_zoneidx for classzone_idx mm/page_alloc: integrate classzone_idx and high_zoneidx
Wei Yang <[email protected]>:
mm/page_alloc.c: use NODE_MASK_NONE in build_zonelists()
mm: rename gfpflags_to_migratetype to gfp_migratetype for same convention
Sandipan Das <[email protected]>:
mm/page_alloc.c: reset numa stats for boot pagesets
Charan Teja Reddy <[email protected]>:
mm, page_alloc: reset the zone->watermark_boost early
Anshuman Khandual <[email protected]>:
mm/page_alloc: restrict and formalize compound_page_dtors[]
Daniel Jordan <[email protected]>:
Patch series "initialize deferred pages with interrupts enabled", v4:
mm/pagealloc.c: call touch_nmi_watchdog() on max order boundaries in deferred init
Pavel Tatashin <[email protected]>:
mm: initialize deferred pages with interrupts enabled
mm: call cond_resched() from deferred_init_memmap()
Daniel Jordan <[email protected]>:
Patch series "padata: parallelize deferred page init", v3:
padata: remove exit routine
padata: initialize earlier
padata: allocate work structures for parallel jobs from a pool
padata: add basic support for multithreaded jobs
mm: don't track number of pages during deferred initialization
mm: parallelize deferred_init_memmap()
mm: make deferred init's max threads arch-specific
padata: document multithreaded jobs
Chen Tao <[email protected]>:
mm/page_alloc.c: add missing newline
Subsystem: mm/hugetlb
"Kirill A. Shutemov" <[email protected]>:
Patch series "thp/khugepaged improvements and CoW semantics", v4:
khugepaged: add self test
khugepaged: do not stop collapse if less than half PTEs are referenced
khugepaged: drain all LRU caches before scanning pages
khugepaged: drain LRU add pagevec after swapin
khugepaged: allow to collapse a page shared across fork
khugepaged: allow to collapse PTE-mapped compound pages
thp: change CoW semantics for anon-THP
khugepaged: introduce 'max_ptes_shared' tunable
Mike Kravetz <[email protected]>:
Patch series "Clean up hugetlb boot command line processing", v4:
hugetlbfs: add arch_hugetlb_valid_size
hugetlbfs: move hugepagesz= parsing to arch independent code
hugetlbfs: remove hugetlb_add_hstate() warning for existing hstate
hugetlbfs: clean up command line processing
hugetlbfs: fix changes to command line processing
Li Xinhai <[email protected]>:
mm/hugetlb: avoid unnecessary check on pud and pmd entry in huge_pte_offset
Anshuman Khandual <[email protected]>:
Patch series "mm/hugetlb: Add some new generic fallbacks", v3:
arm64/mm: drop __HAVE_ARCH_HUGE_PTEP_GET
mm/hugetlb: define a generic fallback for is_hugepage_only_range() mm/hugetlb: define a generic fallback for arch_clear_hugepage_flags()
"Matthew Wilcox (Oracle)" <[email protected]>:
mm: simplify calling a compound page destructor
Subsystem: mm/vmscan
Wei Yang <[email protected]>:
mm/vmscan.c: use update_lru_size() in update_lru_sizes()
Jaewon Kim <[email protected]>:
mm/vmscan: count layzfree pages and fix nr_isolated_* mismatch
Maninder Singh <[email protected]>:
mm/vmscan.c: change prototype for shrink_page_list
Qiwu Chen <[email protected]>:
mm/vmscan: update the comment of should_continue_reclaim()
Johannes Weiner <[email protected]>:
Patch series "mm: memcontrol: charge swapin pages on instantiation", v2:
mm: fix NUMA node file count error in replace_page_cache()
mm: memcontrol: fix stat-corrupting race in charge moving
mm: memcontrol: drop @compound parameter from memcg charging API
mm: shmem: remove rare optimization when swapin races with hole punching
mm: memcontrol: move out cgroup swaprate throttling
mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
mm: memcontrol: prepare uncharging for removal of private page type counters
mm: memcontrol: prepare move_account for removal of private page type counters
mm: memcontrol: prepare cgroup vmstat infrastructure for native anon counters
mm: memcontrol: switch to native NR_FILE_PAGES and NR_SHMEM counters
mm: memcontrol: switch to native NR_ANON_MAPPED counter
mm: memcontrol: switch to native NR_ANON_THPS counter
mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API
mm: memcontrol: drop unused try/commit/cancel charge API
mm: memcontrol: prepare swap controller setup for integration
mm: memcontrol: make swap tracking an integral part of memory control
mm: memcontrol: charge swapin pages on instantiation
Alex Shi <[email protected]>:
mm: memcontrol: document the new swap control behavior
Johannes Weiner <[email protected]>:
mm: memcontrol: delete unused lrucare handling
mm: memcontrol: update page->mem_cgroup stability rules
mm: fix LRU balancing effect of new transparent huge pages
mm: keep separate anon and file statistics on page reclaim activity
mm: allow swappiness that prefers reclaiming anon over the file workingset
mm: fold and remove lru_cache_add_anon() and lru_cache_add_file()
mm: workingset: let cache workingset challenge anon
mm: remove use-once cache bias from LRU balancing
mm: vmscan: drop unnecessary div0 avoidance rounding in get_scan_count()
mm: base LRU balancing on an explicit cost model
mm: deactivations shouldn't bias the LRU balance
mm: only count actual rotations as LRU reclaim cost
mm: balance LRU lists based on relative thrashing
mm: vmscan: determine anon/file pressure balance at the reclaim root
mm: vmscan: reclaim writepage is IO cost
mm: vmscan: limit the range of LRU type balancing
Shakeel Butt <[email protected]>:
mm: swap: fix vmstats for huge pages
mm: swap: memcg: fix memcg stats for huge pages
Subsystem: mm/tools
Changhee Han <[email protected]>:
tools/vm/page_owner_sort.c: filter out unneeded line
Subsystem: mm/mempolicy
Michal Hocko <[email protected]>:
mm, mempolicy: fix up gup usage in lookup_node
Subsystem: mm/memblock
chenqiwu <[email protected]>:
include/linux/memblock.h: fix minor typo and unclear comment
Mike Rapoport <[email protected]>:
sparc32: register memory occupied by kernel as memblock.memory
Subsystem: mm/hugetlbfs
Shijie Hu <[email protected]>:
hugetlbfs: get unmapped area below TASK_UNMAPPED_BASE for hugetlbfs
Subsystem: mm/thp
Yang Shi <[email protected]>:
mm: thp: don't need to drain lru cache when splitting and mlocking THP
Anshuman Khandual <[email protected]>:
Patch series "mm/thp: Rename pmd_mknotpresent() as pmd_mknotvalid()", v2:
powerpc/mm: drop platform defined pmd_mknotpresent() mm/thp: rename pmd_mknotpresent() as pmd_mkinvalid()
Subsystem: mm/mmap
Scott Cheloha <[email protected]>:
drivers/base/memory.c: cache memory blocks in xarray to accelerate lookup
Subsystem: mm/kconfig
Zong Li <[email protected]>:
Patch series "Extract DEBUG_WX to shared use":
mm: add DEBUG_WX support
riscv: support DEBUG_WX
x86: mm: use ARCH_HAS_DEBUG_WX instead of arch defined
arm64: mm: use ARCH_HAS_DEBUG_WX instead of arch defined
Documentation/admin-guide/cgroup-v1/memory.rst | 19 Documentation/admin-guide/kernel-parameters.txt | 40 Documentation/admin-guide/mm/hugetlbpage.rst | 35 Documentation/admin-guide/mm/transhuge.rst | 7 Documentation/admin-guide/sysctl/vm.rst | 23 Documentation/core-api/padata.rst | 41 Documentation/features/vm/numa-memblock/arch-support.txt | 34 Documentation/vm/memory-model.rst | 9 Documentation/vm/page_owner.rst | 3 arch/alpha/mm/init.c | 16 arch/alpha/mm/numa.c | 22 arch/arc/include/asm/hugepage.h | 2 arch/arc/mm/init.c | 41 arch/arm/include/asm/hugetlb.h | 7 arch/arm/include/asm/pgtable-3level.h | 2 arch/arm/mm/init.c | 66 arch/arm64/Kconfig | 2 arch/arm64/Kconfig.debug | 29 arch/arm64/include/asm/hugetlb.h | 13 arch/arm64/include/asm/pgtable.h | 2 arch/arm64/mm/hugetlbpage.c | 48 arch/arm64/mm/init.c | 56 arch/arm64/mm/numa.c | 9 arch/c6x/mm/init.c | 8 arch/csky/kernel/setup.c | 26 arch/h8300/mm/init.c | 6 arch/hexagon/mm/init.c | 6 arch/ia64/Kconfig | 1 arch/ia64/include/asm/hugetlb.h | 5 arch/ia64/mm/contig.c | 2 arch/ia64/mm/discontig.c | 2 arch/m68k/mm/init.c | 6 arch/m68k/mm/mcfmmu.c | 9 arch/m68k/mm/motorola.c | 15 arch/m68k/mm/sun3mmu.c | 10 arch/microblaze/Kconfig | 1 arch/microblaze/mm/init.c | 2 arch/mips/Kconfig | 1 arch/mips/include/asm/hugetlb.h | 11 arch/mips/include/asm/pgtable.h | 2 arch/mips/loongson64/numa.c | 2 arch/mips/mm/init.c | 2 arch/mips/sgi-ip27/ip27-memory.c | 2 arch/nds32/mm/init.c | 11 arch/nios2/mm/init.c | 8 arch/openrisc/mm/init.c | 9 arch/parisc/include/asm/hugetlb.h | 10 arch/parisc/mm/init.c | 22 arch/powerpc/Kconfig | 10 arch/powerpc/include/asm/book3s/64/pgtable.h | 4 arch/powerpc/include/asm/hugetlb.h | 5 arch/powerpc/mm/hugetlbpage.c | 38 arch/powerpc/mm/mem.c | 2 arch/riscv/Kconfig | 2 arch/riscv/include/asm/hugetlb.h | 10 arch/riscv/include/asm/ptdump.h | 11 arch/riscv/mm/hugetlbpage.c | 44 arch/riscv/mm/init.c | 5 arch/s390/Kconfig | 1 arch/s390/include/asm/hugetlb.h | 8 arch/s390/mm/hugetlbpage.c | 34 arch/s390/mm/init.c | 2 arch/sh/Kconfig | 1 arch/sh/include/asm/hugetlb.h | 7 arch/sh/mm/init.c | 2 arch/sparc/Kconfig | 10 arch/sparc/include/asm/hugetlb.h | 10 arch/sparc/mm/init_32.c | 1 arch/sparc/mm/init_64.c | 67 arch/sparc/mm/srmmu.c | 21 arch/um/kernel/mem.c | 12 arch/unicore32/include/asm/memory.h | 2 arch/unicore32/include/mach/memory.h | 6 arch/unicore32/kernel/pci.c | 14 arch/unicore32/mm/init.c | 43 arch/x86/Kconfig | 11 arch/x86/Kconfig.debug | 27 arch/x86/include/asm/hugetlb.h | 10 arch/x86/include/asm/pgtable.h | 2 arch/x86/mm/hugetlbpage.c | 35 arch/x86/mm/init.c | 2 arch/x86/mm/init_64.c | 12 arch/x86/mm/kmmio.c | 2 arch/x86/mm/numa.c | 11 arch/xtensa/mm/init.c | 8 drivers/base/memory.c | 44 drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 22 fs/cifs/file.c | 10 fs/fuse/dev.c | 2 fs/hugetlbfs/inode.c | 67 include/asm-generic/hugetlb.h | 2 include/linux/compaction.h | 9 include/linux/gfp.h | 7 include/linux/hugetlb.h | 16 include/linux/memblock.h | 15 include/linux/memcontrol.h | 102 - include/linux/mm.h | 52 include/linux/mmzone.h | 46 include/linux/padata.h | 43 include/linux/string.h | 60 include/linux/swap.h | 17 include/linux/vm_event_item.h | 4 include/linux/vmstat.h | 2 include/trace/events/compaction.h | 22 include/trace/events/huge_memory.h | 3 include/trace/events/vmscan.h | 14 init/Kconfig | 17 init/main.c | 2 kernel/events/uprobes.c | 22 kernel/padata.c | 293 +++- kernel/sysctl.c | 3 lib/test_kasan.c | 29 mm/Kconfig | 9 mm/Kconfig.debug | 32 mm/compaction.c | 70 - mm/filemap.c | 55 mm/gup.c | 237 ++- mm/huge_memory.c | 282 ---- mm/hugetlb.c | 260 ++- mm/internal.h | 25 mm/khugepaged.c | 316 ++-- mm/memblock.c | 19 mm/memcontrol.c | 642 +++------ mm/memory.c | 103 - mm/memory_hotplug.c | 10 mm/mempolicy.c | 5 mm/migrate.c | 30 mm/oom_kill.c | 4 mm/page_alloc.c | 735 ++++------ mm/page_owner.c | 7 mm/pgtable-generic.c | 2 mm/rmap.c | 53 mm/shmem.c | 156 -- mm/slab.c | 4 mm/slub.c | 8 mm/swap.c | 199 +- mm/swap_cgroup.c | 10 mm/swap_state.c | 110 - mm/swapfile.c | 39 mm/userfaultfd.c | 15 mm/vmscan.c | 344 ++-- mm/vmstat.c | 16 mm/workingset.c | 23 tools/testing/selftests/vm/.gitignore | 1 tools/testing/selftests/vm/Makefile | 1 tools/testing/selftests/vm/khugepaged.c | 1035 +++++++++++++++ tools/vm/page_owner_sort.c | 5 147 files changed, 3876 insertions(+), 3108 deletions(-)
Andrew Morton
maintainerPATCH 1/131 mm/slub: fix a memory leak in sysfs_slab_add()
From: Wang Hai <[email protected]>
Subject: mm/slub: fix a memory leak in sysfs_slab_add()
syzkaller reports for memory leak when kobject_init_and_add() returns an error in the function sysfs_slab_add() [1]
When this happened, the function kobject_put() is not called for the corresponding kobject, which potentially leads to memory leak.
This patch fixes the issue by calling kobject_put() even if kobject_init_and_add() fails.
[1]
BUG: memory leak
unreferenced object 0xffff8880a6d4be88 (size 8):
comm "syz-executor.3", pid 946, jiffies 4295772514 (age 18.396s)
hex dump (first 8 bytes):
70 69 64 5f 33 00 ff ff pid_3...
backtrace:
[<00000000a0980095>] kstrdup+0x35/0x70 mm/util.c:60
[<00000000ef0cff3f>] kstrdup_const+0x3d/0x50 mm/util.c:82
[<00000000e2461486>] kvasprintf_const+0x112/0x170 lib/kasprintf.c:48
[<000000005d749e93>] kobject_set_name_vargs+0x55/0x130 lib/kobject.c:289
[<0000000094e31519>] kobject_add_varg lib/kobject.c:384 [inline]
[<0000000094e31519>] kobject_init_and_add+0xd8/0x170 lib/kobject.c:473
[<0000000060f13e32>] sysfs_slab_add+0x1d8/0x290 mm/slub.c:5811
[<00000000fe1d9a22>] __kmem_cache_create+0x50a/0x570 mm/slub.c:4384
[<000000006a71a1b4>] create_cache+0x113/0x1e0 mm/slab_common.c:407
[<0000000089491438>] kmem_cache_create_usercopy+0x1a1/0x260 mm/slab_common.c:505
[<000000008c992595>] kmem_cache_create+0xd/0x10 mm/slab_common.c:564
[<000000005320c4b6>] create_pid_cachep kernel/pid_namespace.c:54 [inline]
[<000000005320c4b6>] create_pid_namespace kernel/pid_namespace.c:96 [inline]
[<000000005320c4b6>] copy_pid_ns+0x77c/0x8f0 kernel/pid_namespace.c:148
[<00000000fc8e1a2b>] create_new_namespaces+0x26b/0xa30 kernel/nsproxy.c:95
[<0000000080f0c9a5>] unshare_nsproxy_namespaces+0xa7/0x1e0 kernel/nsproxy.c:229
[<0000000007e05aea>] ksys_unshare+0x3d2/0x770 kernel/fork.c:2969
[<00000000e04c8e4b>] __do_sys_unshare kernel/fork.c:3037 [inline]
[<00000000e04c8e4b>] __se_sys_unshare kernel/fork.c:3035 [inline]
[<00000000e04c8e4b>] __x64_sys_unshare+0x2d/0x40 kernel/fork.c:3035
[<000000005c4707c7>] do_syscall_64+0xa1/0x530 arch/x86/entry/common.c:295---
mm/slub.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/slub.c renamed from mm/slub.c~mm-slub-fix-a-memory-leak-in-sysfs_slab_add +3 -1
@@ -5835,8 +5835,10 @@ static int sysfs_slab_add(struct kmem_ca s->kobj.kset = kset; err =kobject_init_and_add (&s->kobj, &slab_ktype,NULL ,"%s" , name); -if (err) +if (err) { +kobject_put (&s->kobj); goto out; + } err =sysfs_create_group (&s->kobj, &slab_attr_group);if (err)
_
Andrew Morton
maintainerPATCH 2/131 mm/memcg: optimize memory.numa_stat like memory.stat
In reply to Andrew MortonFrom: Shakeel Butt <[email protected]>
Subject: mm/memcg: optimize memory.numa_stat like memory.stat
Currently reading memory.numa_stat traverses the underlying memcg tree multiple times to accumulate the stats to present the hierarchical view of the memcg tree. However the kernel already maintains the hierarchical view of the stats and use it in memory.stat. Just use the same mechanism in memory.numa_stat as well.
I ran a simple benchmark which reads root_mem_cgroup's memory.numa_stat file in the presense of 10000 memcgs. The results are:
Without the patch:
$ time cat /dev/cgroup/memory/memory.numa_stat > /dev/null
real 0m0.700s user 0m0.001s sys 0m0.697s
With the patch:
$ time cat /dev/cgroup/memory/memory.numa_stat > /dev/null
real 0m0.001s user 0m0.001s sys 0m0.000s
---
mm/memcontrol.c | 49 +++++++++++++++++++++++----------------------- 1 file changed, 25 insertions(+), 24 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~memcg-optimize-memorynuma_stat-like-memorystat +25 -24
@@ -3743,7 +3743,7 @@ static int mem_cgroup_move_charge_write(#define LRU_ALL ((1 << NR_LRU_LISTS) - 1)static unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg, -int nid,unsigned int lru_mask) +int nid,unsigned int lru_mask,bool tree) {struct lruvec *lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid));unsigned long nr =0 ;
@@ -3754,13 +3754,17 @@ static unsigned long mem_cgroup_node_nr_for_each_lru (lru) {if (!(BIT (lru) & lru_mask))continue ; - nr +=lruvec_page_state_local (lruvec,NR_LRU_BASE + lru); +if (tree) + nr +=lruvec_page_state (lruvec,NR_LRU_BASE + lru); +else + nr +=lruvec_page_state_local (lruvec,NR_LRU_BASE + lru); }return nr; }static unsigned long mem_cgroup_nr_lru_pages (struct mem_cgroup *memcg, -unsigned int lru_mask) +unsigned int lru_mask, +bool tree) {unsigned long nr =0 ;enum lru_list lru;
@@ -3768,7 +3772,10 @@ static unsigned long mem_cgroup_nr_lru_pfor_each_lru (lru) {if (!(BIT (lru) & lru_mask))continue ; - nr +=memcg_page_state_local (memcg,NR_LRU_BASE + lru); +if (tree) + nr +=memcg_page_state (memcg,NR_LRU_BASE + lru); +else + nr +=memcg_page_state_local (memcg,NR_LRU_BASE + lru); }return nr; }
@@ -3788,34 +3795,28 @@ static int memcg_numa_stat_show(struct s };const struct numa_stat *stat;int nid; -unsigned long nr;struct mem_cgroup *memcg =mem_cgroup_from_seq (m);for (stat = stats; stat < stats +ARRAY_SIZE (stats); stat++) { - nr =mem_cgroup_nr_lru_pages (memcg, stat->lru_mask); -seq_printf (m,"%s=%lu" , stat->name, nr); -for_each_node_state (nid,N_MEMORY ) { - nr =mem_cgroup_node_nr_lru_pages (memcg, nid, - stat->lru_mask); -seq_printf (m," N%d=%lu" , nid, nr); - } +seq_printf (m,"%s=%lu" , stat->name, +mem_cgroup_nr_lru_pages (memcg, stat->lru_mask, + false)); +for_each_node_state (nid,N_MEMORY ) +seq_printf (m," N%d=%lu" , nid, +mem_cgroup_node_nr_lru_pages (memcg, nid, + stat->lru_mask, false));seq_putc (m,'\n' ); }for (stat = stats; stat < stats +ARRAY_SIZE (stats); stat++) { -struct mem_cgroup *iter; - nr =0 ; -for_each_mem_cgroup_tree (iter, memcg) - nr +=mem_cgroup_nr_lru_pages (iter, stat->lru_mask); -seq_printf (m,"hierarchical_%s=%lu" , stat->name, nr); -for_each_node_state (nid,N_MEMORY ) { - nr =0 ; -for_each_mem_cgroup_tree (iter, memcg) - nr +=mem_cgroup_node_nr_lru_pages ( - iter, nid, stat->lru_mask); -seq_printf (m," N%d=%lu" , nid, nr); - } +seq_printf (m,"hierarchical_%s=%lu" , stat->name, +mem_cgroup_nr_lru_pages (memcg, stat->lru_mask, + true)); +for_each_node_state (nid,N_MEMORY ) +seq_printf (m," N%d=%lu" , nid, +mem_cgroup_node_nr_lru_pages (memcg, nid, + stat->lru_mask, true));seq_putc (m,'\n' ); }
_
Andrew Morton
maintainerPATCH 3/131 mm/gup: move __get_user_pages_fast() down a few lines in gup.c
In reply to Andrew MortonFrom: John Hubbard <[email protected]>
Subject: mm/gup: move __get_user_pages_fast() down a few lines in gup.c
Patch series "mm/gup, drm/i915: refactor gup_fast, convert to pin_user_pages()", v2.
In order to convert the drm/i915 driver from get_user_pages() to pin_user_pages(), a FOLL_PIN equivalent of __get_user_pages_fast() was required. That led to refactoring __get_user_pages_fast(), with the following goals:
1) As above: provide a pin_user_pages*() routine for drm/i915 to call, in place of __get_user_pages_fast(),
2) Get rid of the gup.c duplicate code for walking page tables with interrupts disabled. This duplicate code is a minor maintenance problem anyway.
3) Make it easy for an upcoming patch from Souptick, which aims to convert __get_user_pages_fast() to use a gup_flags argument, instead of a bool writeable arg. Also, if this series looks good, we can ask Souptick to change the name as well, to whatever the consensus is. My initial recommendation is: get_user_pages_fast_only(), to match the new pin_user_pages_only().
This patch (of 4):
This is in order to avoid a forward declaration of internal_get_user_pages_fast(), in the next patch.
This is code movement only--all generated code should be identical.
---
mm/gup.c | 132 ++++++++++++++++++++++++++--------------------------- 1 file changed, 66 insertions(+), 66 deletions(-)
mm/gup.c renamed from mm/gup.c~mm-gup-move-__get_user_pages_fast-down-a-few-lines-in-gupc +66 -66
@@ -2703,72 +2703,6 @@ static bool gup_fast_permitted(unsigned }#endif -/* -* Like get_user_pages_fast() except it's IRQ-safe in that it won't fall back to -* the regular GUP. -* Note a difference with get_user_pages_fast: this always returns the -* number of pages pinned, 0 if no pages were pinned. -* -* If the architecture does not support this function, simply return with no -* pages pinned. -* -* Careful, careful! COW breaking can go either way, so a non-write -* access can get ambiguous page results. If you call this function without -* 'write' set, you'd better be sure that you're ok with that ambiguity. -*/ -int __get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) -{ -unsigned long len, end; -unsigned long flags; -int nr_pinned =0 ; -/* -* Internally (within mm/gup.c), gup fast variants must set FOLL_GET, -* because gup fast is always a "pin with a +1 page refcount" request. -*/ -unsigned int gup_flags =FOLL_GET ; - -if (write) - gup_flags |=FOLL_WRITE ; - - start =untagged_addr (start) &PAGE_MASK ; - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; - -if (end <= start) -return 0 ; -if (unlikely (!access_ok ((void __user *)start, len))) -return 0 ; - -/* -* Disable interrupts. We use the nested form as we can already have -* interrupts disabled by get_futex_key. -* -* With interrupts disabled, we block page table pages from being -* freed from under us. See struct mmu_table_batch comments in -* include/asm-generic/tlb.h for more details. -* -* We do not adopt an rcu_read_lock(.) here as we also want to -* block IPIs that come from THPs splitting. -* -* NOTE! We allow read-only gup_fast() here, but you'd better be -* careful about possible COW pages. You'll get _a_ COW page, but -* not necessarily the one you intended to get depending on what -* COW event happens after this. COW may break the page copy in a -* random direction. -*/ - -if (IS_ENABLED (CONFIG_HAVE_FAST_GUP ) && -gup_fast_permitted (start, end)) { -local_irq_save (flags); -gup_pgd_range (start, end, gup_flags, pages, &nr_pinned); -local_irq_restore (flags); - } - -return nr_pinned; -} -EXPORT_SYMBOL_GPL (__get_user_pages_fast); -static int __gup_longterm_unlocked (unsigned long start,int nr_pages,unsigned int gup_flags,struct page **pages) {
@@ -2848,6 +2782,72 @@ static int internal_get_user_pages_fast(return ret; } +/* +* Like get_user_pages_fast() except it's IRQ-safe in that it won't fall back to +* the regular GUP. +* Note a difference with get_user_pages_fast: this always returns the +* number of pages pinned, 0 if no pages were pinned. +* +* If the architecture does not support this function, simply return with no +* pages pinned. +* +* Careful, careful! COW breaking can go either way, so a non-write +* access can get ambiguous page results. If you call this function without +* 'write' set, you'd better be sure that you're ok with that ambiguity. +*/ +int __get_user_pages_fast (unsigned long start,int nr_pages,int write, +struct page **pages) +{ +unsigned long len, end; +unsigned long flags; +int nr_pinned =0 ; +/* +* Internally (within mm/gup.c), gup fast variants must set FOLL_GET, +* because gup fast is always a "pin with a +1 page refcount" request. +*/ +unsigned int gup_flags =FOLL_GET ; + +if (write) + gup_flags |=FOLL_WRITE ; + + start =untagged_addr (start) &PAGE_MASK ; + len = (unsigned long ) nr_pages <<PAGE_SHIFT ; + end = start + len; + +if (end <= start) +return 0 ; +if (unlikely (!access_ok ((void __user *)start, len))) +return 0 ; + +/* +* Disable interrupts. We use the nested form as we can already have +* interrupts disabled by get_futex_key. +* +* With interrupts disabled, we block page table pages from being +* freed from under us. See struct mmu_table_batch comments in +* include/asm-generic/tlb.h for more details. +* +* We do not adopt an rcu_read_lock(.) here as we also want to +* block IPIs that come from THPs splitting. +* +* NOTE! We allow read-only gup_fast() here, but you'd better be +* careful about possible COW pages. You'll get _a_ COW page, but +* not necessarily the one you intended to get depending on what +* COW event happens after this. COW may break the page copy in a +* random direction. +*/ + +if (IS_ENABLED (CONFIG_HAVE_FAST_GUP ) && +gup_fast_permitted (start, end)) { +local_irq_save (flags); +gup_pgd_range (start, end, gup_flags, pages, &nr_pinned); +local_irq_restore (flags); + } + +return nr_pinned; +} +EXPORT_SYMBOL_GPL (__get_user_pages_fast); +/** * get_user_pages_fast() - pin user pages in memory * @start: starting user address
_
John Hubbard
reviewerRe: [patch 003/131] mm/gup: move __get_user_pages_fast() down a few lines in gup.c
Quoted 4 lines · Andrew Morton
From: John Hubbard <[email protected]>
Subject: mm/gup: move__get_user_pages_fast()down a few lines ingup.cPatch series "mm/gup, drm/i915: refactor
gup_fast, convert topin_user_pages()", v2.
These patches 003 through 007 (gup refactoring and pin_user_pages stuff) all look good.
Thanks for fixing up the merge conflicts with commit17839856fd58 ("gup: document and work around "COW can break either way" issue"). I wasn't aware of that commit until the -next conflict email showed up in my inbox this morning.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 198 lines
In order to convert the drm/i915 driver from
get_user_pages()topin_user_pages(), aFOLL_PINequivalent of__get_user_pages_fast()was required. That led to refactoring__get_user_pages_fast(), with the following goals:1) As above: provide a pin_user_pages*() routine for drm/i915 to call, in place of
__get_user_pages_fast(),2) Get rid of the
gup.cduplicate code for walking page tables with interrupts disabled. This duplicate code is a minor maintenance problem anyway.3) Make it easy for an upcoming patch from Souptick, which aims to convert
__get_user_pages_fast()to use agup_flagsargument, instead of a bool writeable arg. Also, if this series looks good, we can ask Souptick to change the name as well, to whatever the consensus is. My initial recommendation is:get_user_pages_fast_only(), to match the newpin_user_pages_only().This patch (of 4):
This is in order to avoid a forward declaration of
internal_get_user_pages_fast(), in the next patch.This is code movement only--all generated code should be identical.
Signed-off-by John Hubbard <[email protected]>Reviewed-by Chris Wilson <[email protected]>Cc Daniel Vetter <[email protected]>Cc David Airlie <[email protected]>Cc Jani Nikula <[email protected]>Cc "Joonas Lahtinen" <[email protected]>Cc Matthew Auld <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Rodrigo Vivi <[email protected]>Cc Souptick Joarder <[email protected]>Cc Tvrtko Ursulin <[email protected]>Signed-off-by Andrew Morton <[email protected]>---
mm/gup.c | 132 ++++++++++++++++++++++++++--------------------------- 1 file changed, 66 insertions(+), 66 deletions(-)mm/gup.c renamed from mm/gup.c~mm-gup-move-__get_user_pages_fast-down-a-few-lines-in-gupc +66
-66@@ -2703,72 +2703,6 @@ static bool gup_fast_permitted(unsigned }#endif -/* -* Like get_user_pages_fast() except it's IRQ-safe in that it won't fall back to -* the regular GUP. -* Note a difference with get_user_pages_fast: this always returns the -* number of pages pinned, 0 if no pages were pinned. -* -* If the architecture does not support this function, simply return with no -* pages pinned. -* -* Careful, careful! COW breaking can go either way, so a non-write -* access can get ambiguous page results. If you call this function without -* 'write' set, you'd better be sure that you're ok with that ambiguity. -*/ -int __get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) -{ -unsigned long len, end; -unsigned long flags; -int nr_pinned =0 ; -/* -* Internally (within mm/gup.c), gup fast variants must set FOLL_GET, -* because gup fast is always a "pin with a +1 page refcount" request. -*/ -unsigned int gup_flags =FOLL_GET ; - -if (write) - gup_flags |=FOLL_WRITE ; - - start =untagged_addr (start) &PAGE_MASK ; - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; - -if (end <= start) -return 0 ; -if (unlikely (!access_ok ((void __user *)start, len))) -return 0 ; - -/* -* Disable interrupts. We use the nested form as we can already have -* interrupts disabled by get_futex_key. -* -* With interrupts disabled, we block page table pages from being -* freed from under us. See struct mmu_table_batch comments in -* include/asm-generic/tlb.h for more details. -* -* We do not adopt an rcu_read_lock(.) here as we also want to -* block IPIs that come from THPs splitting. -* -* NOTE! We allow read-only gup_fast() here, but you'd better be -* careful about possible COW pages. You'll get _a_ COW page, but -* not necessarily the one you intended to get depending on what -* COW event happens after this. COW may break the page copy in a -* random direction. -*/ - -if (IS_ENABLED (CONFIG_HAVE_FAST_GUP ) && -gup_fast_permitted (start, end)) { -local_irq_save (flags); -gup_pgd_range (start, end, gup_flags, pages, &nr_pinned); -local_irq_restore (flags); - } - -return nr_pinned; -} -EXPORT_SYMBOL_GPL (__get_user_pages_fast); -static int __gup_longterm_unlocked (unsigned long start,int nr_pages,unsigned int gup_flags,struct page **pages) {@@ -2848,6 +2782,72 @@ static int internal_get_user_pages_fast(return ret; } +/* +* Like get_user_pages_fast() except it's IRQ-safe in that it won't fall back to +* the regular GUP. +* Note a difference with get_user_pages_fast: this always returns the +* number of pages pinned, 0 if no pages were pinned. +* +* If the architecture does not support this function, simply return with no +* pages pinned. +* +* Careful, careful! COW breaking can go either way, so a non-write +* access can get ambiguous page results. If you call this function without +* 'write' set, you'd better be sure that you're ok with that ambiguity. +*/ +int __get_user_pages_fast (unsigned long start,int nr_pages,int write, +struct page **pages) +{ +unsigned long len, end; +unsigned long flags; +int nr_pinned =0 ; +/* +* Internally (within mm/gup.c), gup fast variants must set FOLL_GET, +* because gup fast is always a "pin with a +1 page refcount" request. +*/ +unsigned int gup_flags =FOLL_GET ; + +if (write) + gup_flags |=FOLL_WRITE ; + + start =untagged_addr (start) &PAGE_MASK ; + len = (unsigned long ) nr_pages <<PAGE_SHIFT ; + end = start + len; + +if (end <= start) +return 0 ; +if (unlikely (!access_ok ((void __user *)start, len))) +return 0 ; + +/* +* Disable interrupts. We use the nested form as we can already have +* interrupts disabled by get_futex_key. +* +* With interrupts disabled, we block page table pages from being +* freed from under us. See struct mmu_table_batch comments in +* include/asm-generic/tlb.h for more details. +* +* We do not adopt an rcu_read_lock(.) here as we also want to +* block IPIs that come from THPs splitting. +* +* NOTE! We allow read-only gup_fast() here, but you'd better be +* careful about possible COW pages. You'll get _a_ COW page, but +* not necessarily the one you intended to get depending on what +* COW event happens after this. COW may break the page copy in a +* random direction. +*/ + +if (IS_ENABLED (CONFIG_HAVE_FAST_GUP ) && +gup_fast_permitted (start, end)) { +local_irq_save (flags); +gup_pgd_range (start, end, gup_flags, pages, &nr_pinned); +local_irq_restore (flags); + } + +return nr_pinned; +} +EXPORT_SYMBOL_GPL (__get_user_pages_fast); +/** * get_user_pages_fast() - pin user pages in memory * @start: starting user address _
Andrew Morton
maintainerPATCH 4/131 mm/gup: refactor and de-duplicate gup_fast() code
In reply to Andrew MortonFrom: John Hubbard <[email protected]>
Subject: mm/gup: refactor and de-duplicate gup_fast() code
There were two nearly identical sets of code for gup_fast() style of walking the page tables with interrupts disabled. This has lead to the usual maintenance problems that arise from having duplicated code.
There is already a core internal routine in gup.c for gup_fast(), so just enhance it very slightly: allow skipping the fall-back to "slow" (regular) get_user_pages(), via the new FOLL_FAST_ONLY flag. Then, just call internal_get_user_pages_fast() from __get_user_pages_fast(), and adjust the API to match pre-existing API behavior.
There is a change in behavior from this refactoring: the nested form of interrupt disabling is used in all gup_fast() variants now. That's because there is only one place that interrupt disabling for page walking is done, and so the safer form is required. This should, if anything, eliminate possible (rare) bugs, because the non-nested form of enabling interrupts was fragile at best.
---
include/linux/mm.h | 1 mm/gup.c | 61 ++++++++++++++++++++----------------------- 2 files changed, 30 insertions(+), 32 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-gup-refactor-and-de-duplicate-gup_fast-code +1 -0
@@ -2816,6 +2816,7 @@ struct page *follow_page(struct vm_area_#define FOLL_LONGTERM 0x10000/* mapping lifetime is indefinite: see below */ #define FOLL_SPLIT_PMD 0x20000/* split huge pmd before returning */ #define FOLL_PIN 0x40000/* pages must be released via unpin_user_page */ +#define FOLL_FAST_ONLY 0x80000/* gup_fast: prevent fall-back to slow gup */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each
mm/gup.c renamed from mm/gup.c~mm-gup-refactor-and-de-duplicate-gup_fast-code +27 -25
93 lines not on this page · raw message
Re: [patch 004/131] mm/gup: refactor and de-duplicate gup_fast() code
Quoted 7 lines · Andrew Morton
From: John Hubbard <[email protected]>
Subject: mm/gup: refactor and de-duplicategup_fast()codeThere were two nearly identical sets of code for
gup_fast()style of walking the page tables with interrupts disabled. This has lead to the usual maintenance problems that arise from having duplicated code.
Andrew, this is actually an example of why you absolutely should *not* rebase your series in the middle of the development tree.
Now you've rebased it on top of my commit 17839856fd58 ("gup: document and work around "COW can break either way" issue") and in the process you broke the result completely for read-only pages.
Now it uses FOLL_WRITE (because that's what internal_get_user_pages_fast() does), which will disallow read-only pages (in order to handle them properly for COW in the slow path), and then the fact that the slow-path is entirely disabled for this case means that it doesn't work at all.
This "rebase onto whatever random base Linus has today" absolutely has *got* to stop.
It's not ok for git trees, but it's not ok for these patch-queues either. It means that all the testing your patch queue got in linux-next is completely worthless, because what you send me is something very different from what was tested. Exactly as with the git trees, where I tell people constantly not to rebase their patches.
Give me a base that it has been tested on, and a series that has actually been tested. Not this "rebased for your convenience" thing.
I'd _much_ rather get a merge conflict when your patch series changes something that somebody else also changed.
Because then I know something clashed, and if I screw up the merge, I only have myself to blame. If it's a very complex merge, I'll ask for help.
That would be much better than getting a patch-bomb with 131 patches that all _look_ sane and build cleanly, but can be randomly broken because they got rebased hours before with no testing.
The "let me fix things up onto a daily snapshot" really is a completely broken model. You are making it _harder_ for me, not easier, because now I have to look for subtle issues in every single commit rather than the big honking clue of "oh, I got a merge error, I'll need to really look at it".
It so happened that with this one, I was very aware of the rebase, because you rebased on a patch that I wrote so when I looked through the patches I went "Hmm.."
What about all the other times when I wouldn't have noticed and been so aware of what changed recently?
Again: merge conflicts are *much* better than silently rebasing and hiding problems.
Linus
Re: [patch 004/131] mm/gup: refactor and de-duplicate gup_fast() code
Quoted 6 lines · Linus Torvalds
Now it uses
FOLL_WRITE(because that's whatinternal_get_user_pages_fast()does), which will disallow read-only pages (in order to handle them properly for COW in the slow path), and then the fact that the slow-path is entirely disabled for this case means that it doesn't work at all.
I have tried to fix it up, partly by editing the patches directly, and partly by then trying to fix up comments after-the-fact.
The end result looks possibly correct after it all. But it would have been easier had I just had a merge conflict to deal with, rather than trying to fix up patches.
Will do more testing etc before really merging and then pushing out.
Linus
Re: [patch 004/131] mm/gup: refactor and de-duplicate gup_fast() code
Quoted 3 lines · Linus Torvalds
I have tried to fix it up, partly by editing the patches directly, and partly by then trying to fix up comments after-the-fact.
The end result passes the smell test, boots for me, and looks like it might work.
But I don't have any good real-world test for this, and I hope and assume that John has something GPU-related that actually uses the code and cares. Presumably there was _something_ that triggered those changes to de-duplicate that code?
So please give it a look. Because of how I edited the patches (and Andrew edited them before me), what is attributed to John Hubbard isn't really the same as the patch he originally wrote.
If I broke something in the process, feel free to let me know in less than polite terms. But it look better than the intermediate situation that definitely looked like it would just fail entirely on any read-only mappings due to not being able to fall back on the slow case.
The drm code probably doesn't even care about the possible ambiguity with GUP picking a COW page that might later break the other way.
Linus
John Hubbard
reviewerRe: [patch 004/131] mm/gup: refactor and de-duplicate gup_fast() code
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I have tried to fix it up, partly by editing the patches directly, and partly by then trying to fix up comments after-the-fact.
Quoted 7 lines
The end result passes the smell test, boots for me, and looks like it might work.
But I don't have any good real-world test for this, and I hope and assume that John has something GPU-related that actually uses the code and cares. Presumably there was _something_ that triggered those changes to de-duplicate that code?
Yes: the Intel i915 driver required a pin_user_pages*() variant of the gup fast-only code. So the next 2 patches put the refactored code into use:
2170ecfa7688 drm/i915: convert get_user_pages() --> pin_user_pages() 104acc327648 mm/gup: introduce pin_user_pages_fast_only()
Quoted 5 lines
So please give it a look. Because of how I edited the patches (and Andrew edited them before me), what is attributed to John Hubbard isn't really the same as the patch he originally wrote.
Looking at it now. I'm pleased to see that the fix is basically identical to a local fix that I was testing an hour ago. The only difference is the name and type of the local fast_flags variable. An unsigned long is larger than the API requires, but that is of course fine for now.
As for testing, the original version of this the was part of a 4-part series [1] that ended up converting Intel i915 to use pin_user_pages*(). And Chris Wilson (+cc) was kind enough to run some drm/i915 CI tests on that and they passed at the time.
Also, I have a set of xfstests and a few other things exercise a fair amount of get_user_pages*() and pin_user_pages*(). Running those now. But my run time testing is not set up for stress testing, and it's a very narrow look at things. But so far it looks promising.
[1] https://lore.kernel.org/r/[email protected]
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Andrew Morton
maintainerPATCH 5/131 mm/gup: introduce pin_user_pages_fast_only()
In reply to Andrew MortonFrom: John Hubbard <[email protected]>
Subject: mm/gup: introduce pin_user_pages_fast_only()
This is the FOLL_PIN equivalent of __get_user_pages_fast(), except with a more descriptive name, and gup_flags instead of a boolean "write" in the argument list.
---
include/linux/mm.h | 2 ++ mm/gup.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+)
include/linux/mm.h renamed from include/linux/mm.h~mm-gup-introduce-pin_user_pages_fast_only +2 -0
@@ -1827,6 +1827,8 @@ extern int mprotect_fixup(struct vm_area*/ int __get_user_pages_fast (unsigned long start,int nr_pages,int write,struct page **pages); +int pin_user_pages_fast_only (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages);/* * per-process(per-mm_struct) statistics. */
mm/gup.c renamed from mm/gup.c~mm-gup-introduce-pin_user_pages_fast_only +36 -0
@@ -2913,6 +2913,42 @@ int pin_user_pages_fast(unsigned long st }EXPORT_SYMBOL_GPL (pin_user_pages_fast); +/* +* This is the FOLL_PIN equivalent of __get_user_pages_fast(). Behavior is the +* same, except that this one sets FOLL_PIN instead of FOLL_GET. +* +* The API rules are the same, too: no negative values may be returned. +*/ +int pin_user_pages_fast_only (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) +{ +int nr_pinned; + +/* +* FOLL_GET and FOLL_PIN are mutually exclusive. Note that the API +* rules require returning 0, rather than -errno: +*/ +if (WARN_ON_ONCE (gup_flags &FOLL_GET )) +return 0 ; +/* +* FOLL_FAST_ONLY is required in order to match the API description of +* this routine: no fall back to regular ("slow") GUP. +*/ + gup_flags |= (FOLL_PIN |FOLL_FAST_ONLY ); + nr_pinned =internal_get_user_pages_fast (start, nr_pages, gup_flags, + pages); +/* +* This routine is not allowed to return negative values. However, +* internal_get_user_pages_fast() *can* return -errno. Therefore, +* correct for that here: +*/ +if (nr_pinned <0 ) + nr_pinned =0 ; + +return nr_pinned; +} +EXPORT_SYMBOL_GPL (pin_user_pages_fast_only); +/** * pin_user_pages_remote() - pin pages of a remote process (task != current) *
_
Andrew Morton
maintainerPATCH 6/131 drm/i915: convert get_user_pages() --> pin_user_pages()
In reply to Andrew MortonFrom: John Hubbard <[email protected]>
Subject: drm/i915: convert get_user_pages() --> pin_user_pages()
This code was using get_user_pages*(), in a "Case 2" scenario (DMA/RDMA), using the categorization from [1]. That means that it's time to convert the get_user_pages*() + put_page() calls to pin_user_pages*() + unpin_user_pages() calls.
There is some helpful background in [2]: basically, this is a small part of fixing a long-standing disconnect between pinning pages, and file systems' use of those pages.
[1] Documentation/core-api/pin_user_pages.rst
[2] "Explicit pinning of user-space pages":
https://lwn.net/Articles/807108/
---
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 22 ++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_userptr.c renamed from drivers/gpu/drm/i915/gem/i915_gem_userptr.c~drm-i915-convert-get_user_pages-pin_user_pages +13 -9
@@ -471,7 +471,7 @@ __i915_gem_userptr_get_pages_worker(strudown_read (&mm->mmap_sem); locked =1 ; } - ret = get_user_pages_remote + ret = pin_user_pages_remote (work->task, mm, obj->userptr.ptr + pinned * PAGE_SIZE, npages - pinned,
@@ -507,7 +507,7 @@ __i915_gem_userptr_get_pages_worker(stru }mutex_unlock (&obj->mm.lock); -release_pages (pvec, pinned); +unpin_user_pages (pvec, pinned);kvfree (pvec);i915_gem_object_put (obj);
@@ -564,6 +564,7 @@ static int i915_gem_userptr_get_pages(ststruct sg_table *pages;bool active;int pinned; +unsigned int gup_flags =0 ;/* If userspace should engineer that these pages are replaced in * the vma between us binding this page into the GTT and completion
@@ -606,11 +607,14 @@ static int i915_gem_userptr_get_pages(st* * We may or may not care. */ -if (pvec)/* defer to worker if malloc fails */ - pinned =__get_user_pages_fast (obj->userptr.ptr, - num_pages, - !i915_gem_object_is_readonly (obj), - pvec); +if (pvec) { +/* defer to worker if malloc fails */ +if (!i915_gem_object_is_readonly (obj)) + gup_flags |=FOLL_WRITE ; + pinned =pin_user_pages_fast_only (obj->userptr.ptr, + num_pages, gup_flags, + pvec); + } } active = false;
@@ -628,7 +632,7 @@ static int i915_gem_userptr_get_pages(st__i915_gem_userptr_set_active (obj, true);if (IS_ERR (pages)) -release_pages (pvec, pinned); +unpin_user_pages (pvec, pinned);kvfree (pvec);return PTR_ERR_OR_ZERO (pages);
@@ -683,7 +687,7 @@ i915_gem_userptr_put_pages(struct drm_i9 }mark_page_accessed (page); -put_page (page); +unpin_user_page (page); } obj->mm.dirty = false;
_
Andrew Morton
maintainerPATCH 7/131 mm/gup: might_lock_read(mmap_sem) in get_user_pages_fast()
In reply to Andrew MortonFrom: John Hubbard <[email protected]>
Subject: mm/gup: might_lock_read(mmap_sem) in get_user_pages_fast()
Instead of scattering these assertions across the drivers, do this assertion inside the core of get_user_pages_fast*() functions. That also includes pin_user_pages_fast*() routines.
Add a might_lock_read(mmap_sem) call to internal_get_user_pages_fast().
---
mm/gup.c | 3 +++ 1 file changed, 3 insertions(+)
mm/gup.c renamed from mm/gup.c~mm-gup-might_lock_readmmap_sem-in-get_user_pages_fast +3 -0
@@ -2739,6 +2739,9 @@ static int internal_get_user_pages_fast( FOLL_FAST_ONLY)))return -EINVAL ; +if (!(gup_flags &FOLL_FAST_ONLY )) +might_lock_read (¤t->mm->mmap_sem); + start =untagged_addr (start) &PAGE_MASK ; addr = start; len = (unsigned long ) nr_pages <<PAGE_SHIFT ;
_
Andrew Morton
maintainerPATCH 8/131 kasan: stop tests being eliminated as dead code with FORTIFY_SOURCE
In reply to Andrew MortonFrom: Daniel Axtens <[email protected]>
Subject: kasan: stop tests being eliminated as dead code with FORTIFY_SOURCE
Patch series "Fix some incompatibilites between KASAN and FORTIFY_SOURCE", v4.
3 KASAN self-tests fail on a kernel with both KASAN and FORTIFY_SOURCE:
memchr, memcmp and strlen.
When FORTIFY_SOURCE is on, a number of functions are replaced with fortified versions, which attempt to check the sizes of the operands. However, these functions often directly invoke __builtin_foo() once they have performed the fortify check. The compiler can detect that the results of these functions are not used, and knows that they have no other side effects, and so can eliminate them as dead code.
Why are only memchr, memcmp and strlen affected?
================================================
Of string and string-like functions, kasan_test tests:
* strchr -> not affected, no fortified version * strrchr -> likewise * strcmp -> likewise * strncmp -> likewise
* strnlen -> not affected, the fortify source implementation calls the underlying strnlen implementation which is instrumented, not a builtin
* strlen -> affected, the fortify souce implementation calls a __builtin version which the compiler can determine is dead.
* memchr -> likewise * memcmp -> likewise
* memset -> not affected, the compiler knows that memset writes to its
first argument and therefore is not dead.
Why does this not affect the functions normally?
================================================
In string.h, these functions are not marked as __pure, so the compiler cannot know that they do not have side effects. If relevant functions are marked as __pure in string.h, we see the following warnings and the functions are elided:
lib/test_kasan.c: In function `kasan_memchr':lib/test_kasan.c:606:2: warning: statement with no effect [-Wunused-value]
memchr(ptr, '1', size + 1);
^~~~~~~~~~~~~~~~~~~~~~~~~~lib/test_kasan.c: In function `kasan_memcmp':lib/test_kasan.c:622:2: warning: statement with no effect [-Wunused-value]
memcmp(ptr, arr, size+1);
^~~~~~~~~~~~~~~~~~~~~~~~lib/test_kasan.c: In function `kasan_strings':lib/test_kasan.c:645:2: warning: statement with no effect [-Wunused-value]
strchr(ptr, '1');
^~~~~~~~~~~~~~~~
...
This annotation would make sense to add and could be added at any point, so the behaviour of test_kasan.c should change.
The fix
=======
Make all the functions that are pure write their results to a global, which makes them live. The strlen and memchr tests now pass.
The memcmp test still fails to trigger, which is addressed in the next patch.
---
lib/test_kasan.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
lib/test_kasan.c renamed from lib/test_kasan.c~kasan-stop-tests-being-eliminated-as-dead-code-with-fortify_source +19 -10
@@ -24,6 +24,14 @@#include <asm/page.h> /* +* We assign some test results to these globals to make sure the tests +* are not eliminated as dead code. +*/ + +int kasan_int_result; +void *kasan_ptr_result; + +/* * Note: test functions are marked noinline so that their names appear in * reports. */
@@ -622,7 +630,7 @@ static noinline void __init kasan_memchrif (!ptr)return ; -memchr (ptr,'1' , size +1 ); + kasan_ptr_result =memchr (ptr,'1' , size +1 );kfree (ptr); }
@@ -638,7 +646,7 @@ static noinline void __init kasan_memcmpreturn ;memset (arr,0 ,sizeof (arr)); -memcmp (ptr, arr, size+1 ); + kasan_int_result =memcmp (ptr, arr, size +1 );kfree (ptr); }
@@ -661,22 +669,22 @@ static noinline void __init kasan_string* will likely point to zeroed byte. */ ptr +=16 ; -strchr (ptr,'1' ); + kasan_ptr_result =strchr (ptr,'1' );pr_info ("use-after-free in strrchr\n" ); -strrchr (ptr,'1' ); + kasan_ptr_result =strrchr (ptr,'1' );pr_info ("use-after-free in strcmp\n" ); -strcmp (ptr,"2" ); + kasan_int_result =strcmp (ptr,"2" );pr_info ("use-after-free in strncmp\n" ); -strncmp (ptr,"2" ,1 ); + kasan_int_result =strncmp (ptr,"2" ,1 );pr_info ("use-after-free in strlen\n" ); -strlen (ptr); + kasan_int_result =strlen (ptr);pr_info ("use-after-free in strnlen\n" ); -strnlen (ptr,1 ); + kasan_int_result =strnlen (ptr,1 ); } static noinline void __init kasan_bitops(void)
@@ -743,11 +751,12 @@ static noinline void __init kasan_bitops__test_and_change_bit (BITS_PER_LONG +BITS_PER_BYTE , bits);pr_info ("out-of-bounds in test_bit\n" ); - (void )test_bit (BITS_PER_LONG +BITS_PER_BYTE , bits); + kasan_int_result =test_bit (BITS_PER_LONG +BITS_PER_BYTE , bits);#if defined(clear_bit_unlock_is_negative_byte)pr_info ("out-of-bounds in clear_bit_unlock_is_negative_byte\n" ); -clear_bit_unlock_is_negative_byte (BITS_PER_LONG +BITS_PER_BYTE , bits); + kasan_int_result =clear_bit_unlock_is_negative_byte (BITS_PER_LONG + +BITS_PER_BYTE , bits);#endif kfree (bits); }
_
Andrew Morton
maintainerPATCH 9/131 string.h: fix incompatibility between FORTIFY_SOURCE and KASAN
In reply to Andrew MortonFrom: Daniel Axtens <[email protected]>
Subject: string.h: fix incompatibility between FORTIFY_SOURCE and KASAN
The memcmp KASAN self-test fails on a kernel with both KASAN and FORTIFY_SOURCE.
When FORTIFY_SOURCE is on, a number of functions are replaced with fortified versions, which attempt to check the sizes of the operands. However, these functions often directly invoke __builtin_foo() once they have performed the fortify check. Using __builtins may bypass KASAN checks if the compiler decides to inline it's own implementation as sequence of instructions, rather than emit a function call that goes out to a KASAN-instrumented implementation.
Why is only memcmp affected?
============================
Of the string and string-like functions that kasan_test tests, only memcmp is replaced by an inline sequence of instructions in my testing on x86 with gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2).
I believe this is due to compiler heuristics. For example, if I annotate kmalloc calls with the alloc_size annotation (and disable some fortify compile-time checking!), the compiler will replace every memset except the one in kmalloc_uaf_memset with inline instructions. (I have some WIP patches to add this annotation.)
Does this affect other functions in string.h?
=============================================
Yes. Anything that uses __builtin_* rather than __real_* could be affected. This looks like:
- strncpy
- strcat
- strlen
- strlcpy maybe, under some circumstances?
- strncat under some circumstances
- memset
- memcpy
- memmove
- memcmp (as noted)
- memchr
- strcpy
Whether a function call is emitted always depends on the compiler. Most bugs should get caught by FORTIFY_SOURCE, but the missed memcmp test shows that this is not always the case.
Isn't FORTIFY_SOURCE disabled with KASAN?
========================================-
The string headers on all arches supporting KASAN disable fortify with kasan, but only when address sanitisation is _also_ disabled. For example from x86:
#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__) /*
* For files that are not instrumented (e.g. mm/slub.c) we
* should use not instrumented version of mem* functions. */
#define memcpy(dst, src, len) __memcpy(dst, src, len) #define memmove(dst, src, len) __memmove(dst, src, len) #define memset(s, c, n) __memset(s, c, n)
#ifndef __NO_FORTIFY
#define __NO_FORTIFY /* FORTIFY_SOURCE uses __builtin_memcpy, etc. */ #endif
#endif
This comes from commit 6974f0c4555e ("include/linux/string.h: add the option of fortified string.h functions"), and doesn't work when KASAN is enabled and the file is supposed to be sanitised - as with test_kasan.c
I'm pretty sure this is not wrong, but not as expansive it should be:
* we shouldn't use __builtin_memcpy etc in files where we don't have instrumentation - it could devolve into a function call to memcpy, which will be instrumented. Rather, we should use __memcpy which by convention is not instrumented.
* we also shouldn't be using __builtin_memcpy when we have a KASAN instrumented file, because it could be replaced with inline asm that will not be instrumented.
What is correct behaviour?
==========================
Firstly, there is some overlap between fortification and KASAN: both provide some level of _runtime_ checking. Only fortify provides compile-time checking.
KASAN and fortify can pick up different things at runtime:
- Some fortify functions, notably the string functions, could easily be modified to consider sub-object sizes (e.g. members within a struct), and I have some WIP patches to do this. KASAN cannot detect these because it cannot insert poision between members of a struct.
- KASAN can detect many over-reads/over-writes when the sizes of both operands are unknown, which fortify cannot.
So there are a couple of options:
1) Flip the test: disable fortify in santised files and enable it in unsanitised files. This at least stops us missing KASAN checking, but we lose the fortify checking.
2) Make the fortify code always call out to real versions. Do this only for KASAN, for fear of losing the inlining opportunities we get from __builtin_*.
(We can't use kasan_check_{read,write}: because the fortify functions are _extern inline_, you can't include _static_ inline functions without a compiler warning. kasan_check_{read,write} are static inline so we can't use them even when they would otherwise be suitable.)
Take approach 2 and call out to real versions when KASAN is enabled.
Use __underlying_foo to distinguish from __real_foo: __real_foo always refers to the kernel's implementation of foo, __underlying_foo could be either the kernel implementation or the __builtin_foo implementation.
This is sometimes enough to make the memcmp test succeed with FORTIFY_SOURCE enabled. It is at least enough to get the function call into the module. One more fix is needed to make it reliable: see the next patch.
---
include/linux/string.h | 60 +++++++++++++++++++++++++++++++-------- 1 file changed, 48 insertions(+), 12 deletions(-)
include/linux/string.h renamed from include/linux/string.h~stringh-fix-incompatibility-between-fortify_source-and-kasan +48 -12
@@ -272,6 +272,31 @@ void __read_overflow3(void) __compiletim void __write_overflow(void) __compiletime_error("detected write beyond size of object passed as 1st parameter");#if !defined(__NO_FORTIFY) && defined(__OPTIMIZE__) && defined(CONFIG_FORTIFY_SOURCE ) + +#ifdef CONFIG_KASAN +extern void *__underlying_memchr (const void *p,int c,__kernel_size_t size)__RENAME (memchr); +extern int __underlying_memcmp(const void *p, const void *q, __kernel_size_t size) __RENAME(memcmp); +extern void *__underlying_memcpy (void *p,const void *q,__kernel_size_t size)__RENAME (memcpy); +extern void *__underlying_memmove (void *p,const void *q,__kernel_size_t size)__RENAME (memmove); +extern void *__underlying_memset (void *p,int c,__kernel_size_t size)__RENAME (memset); +extern char *__underlying_strcat (char *p,const char *q)__RENAME (strcat); +extern char *__underlying_strcpy (char *p,const char *q)__RENAME (strcpy); +extern __kernel_size_t __underlying_strlen(const char *p) __RENAME(strlen); +extern char *__underlying_strncat (char *p,const char *q,__kernel_size_t count)__RENAME (strncat); +extern char *__underlying_strncpy (char *p,const char *q,__kernel_size_t size)__RENAME (strncpy); +#else +#define __underlying_memchr __builtin_memchr +#define __underlying_memcmp __builtin_memcmp +#define __underlying_memcpy __builtin_memcpy +#define __underlying_memmove __builtin_memmove +#define __underlying_memset __builtin_memset +#define __underlying_strcat __builtin_strcat +#define __underlying_strcpy __builtin_strcpy +#define __underlying_strlen __builtin_strlen +#define __underlying_strncat __builtin_strncat +#define __underlying_strncpy __builtin_strncpy +#endif + __FORTIFY_INLINE char *strncpy(char *p, const char *q, __kernel_size_t size) {size_t p_size =__builtin_object_size (p,0 );
@@ -279,14 +304,14 @@ __FORTIFY_INLINE char *strncpy(char *p,__write_overflow ();if (p_size < size)fortify_panic (__func__); -return __builtin_strncpy (p, q, size); +return __underlying_strncpy (p, q, size); } __FORTIFY_INLINE char *strcat(char *p, const char *q) {size_t p_size =__builtin_object_size (p,0 );if (p_size == (size_t )-1 ) -return __builtin_strcat (p, q); +return __underlying_strcat (p, q);if (strlcat (p, q, p_size) >= p_size)fortify_panic (__func__);return p;
@@ -300,7 +325,7 @@ __FORTIFY_INLINE __kernel_size_t strlen(/* Work around gcc excess stack consumption issue */ if (p_size == (size_t )-1 || (__builtin_constant_p (p[p_size -1 ]) && p[p_size -1 ] =='\0' )) -return __builtin_strlen (p); +return __underlying_strlen (p); ret =strnlen (p, p_size);if (p_size <= ret)fortify_panic (__func__);
@@ -333,7 +358,7 @@ __FORTIFY_INLINE size_t strlcpy(char *p,__write_overflow ();if (len >= p_size)fortify_panic (__func__); -__builtin_memcpy (p, q, len); +__underlying_memcpy (p, q, len); p[len] ='\0' ; }return ret;
@@ -346,12 +371,12 @@ __FORTIFY_INLINE char *strncat(char *p,size_t p_size =__builtin_object_size (p,0 );size_t q_size =__builtin_object_size (q,0 );if (p_size == (size_t )-1 && q_size == (size_t )-1 ) -return __builtin_strncat (p, q, count); +return __underlying_strncat (p, q, count); p_len =strlen (p); copy_len =strnlen (q, count);if (p_size < p_len + copy_len +1 )fortify_panic (__func__); -__builtin_memcpy (p + p_len, q, copy_len); +__underlying_memcpy (p + p_len, q, copy_len); p[p_len + copy_len] ='\0' ;return p; }
@@ -363,7 +388,7 @@ __FORTIFY_INLINE void *memset(void *p, i__write_overflow ();if (p_size < size)fortify_panic (__func__); -return __builtin_memset (p, c, size); +return __underlying_memset (p, c, size); } __FORTIFY_INLINE void *memcpy(void *p, const void *q, __kernel_size_t size)
@@ -378,7 +403,7 @@ __FORTIFY_INLINE void *memcpy(void *p, c }if (p_size < size || q_size < size)fortify_panic (__func__); -return __builtin_memcpy (p, q, size); +return __underlying_memcpy (p, q, size); } __FORTIFY_INLINE void *memmove(void *p, const void *q, __kernel_size_t size)
@@ -393,7 +418,7 @@ __FORTIFY_INLINE void *memmove(void *p, }if (p_size < size || q_size < size)fortify_panic (__func__); -return __builtin_memmove (p, q, size); +return __underlying_memmove (p, q, size); }extern void *__real_memscan (void *,int ,__kernel_size_t )__RENAME (memscan);
@@ -419,7 +444,7 @@ __FORTIFY_INLINE int memcmp(const void * }if (p_size < size || q_size < size)fortify_panic (__func__); -return __builtin_memcmp (p, q, size); +return __underlying_memcmp (p, q, size); } __FORTIFY_INLINE void *memchr(const void *p, int c, __kernel_size_t size)
@@ -429,7 +454,7 @@ __FORTIFY_INLINE void *memchr(const void__read_overflow ();if (p_size < size)fortify_panic (__func__); -return __builtin_memchr (p, c, size); +return __underlying_memchr (p, c, size); }void *__real_memchr_inv (const void *s,int c,size_t n)__RENAME (memchr_inv);
@@ -460,11 +485,22 @@ __FORTIFY_INLINE char *strcpy(char *p, csize_t p_size =__builtin_object_size (p,0 );size_t q_size =__builtin_object_size (q,0 );if (p_size == (size_t )-1 && q_size == (size_t )-1 ) -return __builtin_strcpy (p, q); +return __underlying_strcpy (p, q);memcpy (p, q,strlen (q) +1 ); return p; } +/* Don't use these outside the FORITFY_SOURCE implementation */ +#undef __underlying_memchr +#undef __underlying_memcmp +#undef __underlying_memcpy +#undef __underlying_memmove +#undef __underlying_memset +#undef __underlying_strcat +#undef __underlying_strcpy +#undef __underlying_strlen +#undef __underlying_strncat +#undef __underlying_strncpy#endif /**
_
Andrew Morton
maintainerPATCH 10/131 mm: clarify __GFP_MEMALLOC usage
In reply to Andrew MortonFrom: Michal Hocko <[email protected]>
Subject: mm: clarify __GFP_MEMALLOC usage
It seems that the existing documentation is not explicit about the expected usage and potential risks enough. While it is calls out that users have to free memory when using this flag it is not really apparent that users have to careful to not deplete memory reserves and that they should implement some sort of throttling wrt. freeing process.
This is partly based on Neil's explanation [1].
Let's also call out that a pre allocated pool allocator should be considered.
[1] http://lkml.kernel.org/r/[email protected]
---
include/linux/gfp.h | 5 +++++ 1 file changed, 5 insertions(+)
include/linux/gfp.h renamed from include/linux/gfp.h~mm-clarify-__gfp_memalloc-usage +5 -0
@@ -110,6 +110,11 @@ struct vm_area_struct;* the caller guarantees the allocation will allow more memory to be freed * very shortly e.g. process exiting or swapping. Users either should * be the MM or co-ordinating closely with the VM (e.g. swap over NFS). +* Users of this flag have to be extremely careful to not deplete the reserve +* completely and implement a throttling mechanism which controls the +* consumption of the reserve based on the amount of freed memory. +* Usage of a pre-allocated pool (e.g. mempool) should be always considered +* before using this flag. * * %__GFP_NOMEMALLOC is used to explicitly forbid access to emergency reserves. * This takes precedence over the %__GFP_MEMALLOC flag if both are set.
_
Andrew Morton
maintainerPATCH 11/131 mm: memblock: replace dereferences of memblock_region.nid with API calls
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: memblock: replace dereferences of memblock_region.nid with API calls
Patch series "mm: rework free_area_init*() funcitons".
After the discussion [1] about removal of CONFIG_NODES_SPAN_OTHER_NODES and CONFIG_HAVE_MEMBLOCK_NODE_MAP options, I took it a bit further and updated the node/zone initialization.
Since all architectures have memblock, it is possible to use only the newer version of free_area_init_node() that calculates the zone and node boundaries based on memblock node mapping and architectural limits on possible zone PFNs.
The architectures that still determined zone and hole sizes can be switched to the generic code and the old code that took those zone and hole sizes can be simply removed.
And, since it all started from the removal of CONFIG_NODES_SPAN_OTHER_NODES, the memmap_init() is now updated to iterate over memblocks and so it does not need to perform early_pfn_to_nid() query for every PFN.
[1] https://lore.kernel.org/lkml/[email protected]
This patch (of 21):
There are several places in the code that directly dereference memblock_region.nid despite this field being defined only when CONFIG_HAVE_MEMBLOCK_NODE_MAP=y.
Replace these with calls to memblock_get_region_nid() to improve code robustness and to avoid possible breakage when CONFIG_HAVE_MEMBLOCK_NODE_MAP will be removed.
---
arch/arm64/mm/numa.c | 9 ++++++--- arch/x86/mm/numa.c | 6 ++++-- mm/memblock.c | 8 +++++--- mm/page_alloc.c | 4 ++-- 4 files changed, 17 insertions(+), 10 deletions(-)
arch/arm64/mm/numa.c renamed from arch/arm64/mm/numa.c~mm-memblock-replace-dereferences-of-memblock_regionnid-with-api-calls +6 -3
@@ -350,13 +350,16 @@ static int __init numa_register_nodes(vostruct memblock_region *mblk;/* Check that valid nid is set to memblks */ -for_each_memblock (memory, mblk) -if (mblk->nid ==NUMA_NO_NODE || mblk->nid >=MAX_NUMNODES ) { +for_each_memblock (memory, mblk) { +int mblk_nid =memblock_get_region_node (mblk); + +if (mblk_nid ==NUMA_NO_NODE || mblk_nid >=MAX_NUMNODES ) {pr_warn ("Warning: invalid memblk node %d [mem %#010Lx-%#010Lx]\n" , - mblk->nid, mblk->base, + mblk_nid, mblk->base, mblk->base + mblk->size -1 );return -EINVAL ; } + }/* Finally register nodes. */ for_each_node_mask (nid, numa_nodes_parsed) {
arch/x86/mm/numa.c renamed from arch/x86/mm/numa.c~mm-memblock-replace-dereferences-of-memblock_regionnid-with-api-calls +4 -2
@@ -517,8 +517,10 @@ static void __init numa_clear_kernel_nod* reserve specific pages for Sandy Bridge graphics. ] */ for_each_memblock (reserved, mb_region) { -if (mb_region->nid !=MAX_NUMNODES ) -node_set (mb_region->nid, reserved_nodemask); +int nid =memblock_get_region_node (mb_region); + +if (nid !=MAX_NUMNODES ) +node_set (nid, reserved_nodemask); }/*
mm/memblock.c renamed from mm/memblock.c~mm-memblock-replace-dereferences-of-memblock_regionnid-with-api-calls +5 -3
@@ -1207,13 +1207,15 @@ void __init_memblock __next_mem_pfn_rang {struct memblock_type *type = &memblock.memory;struct memblock_region *r; +int r_nid;while (++*idx < type->cnt) { r = &type->regions[*idx]; + r_nid =memblock_get_region_node (r);if (PFN_UP (r->base) >=PFN_DOWN (r->base + r->size))continue ; -if (nid ==MAX_NUMNODES || nid == r->nid) +if (nid ==MAX_NUMNODES || nid == r_nid)break ; }if (*idx >= type->cnt) {
@@ -1226,7 +1228,7 @@ void __init_memblock __next_mem_pfn_rangif (out_end_pfn) *out_end_pfn =PFN_DOWN (r->base + r->size);if (out_nid) - *out_nid = r->nid; + *out_nid = r_nid; }/**
@@ -1810,7 +1812,7 @@ int __init_memblock memblock_search_pfn_ *start_pfn =PFN_DOWN (type->regions[mid].base); *end_pfn =PFN_DOWN (type->regions[mid].base + type->regions[mid].size); - return type->regions[mid].nid; + return memblock_get_region_node(&type->regions[mid]); }#endif
mm/page_alloc.c renamed from mm/page_alloc.c~mm-memblock-replace-dereferences-of-memblock_regionnid-with-api-calls +2 -2
@@ -7220,7 +7220,7 @@ static void __init find_zone_movable_pfnif (!memblock_is_hotpluggable (r))continue ; - nid = r->nid; + nid =memblock_get_region_node (r); usable_startpfn =PFN_DOWN (r->base); zone_movable_pfn[nid] = zone_movable_pfn[nid] ?
@@ -7241,7 +7241,7 @@ static void __init find_zone_movable_pfnif (memblock_is_mirror (r))continue ; - nid = r->nid; + nid =memblock_get_region_node (r); usable_startpfn =memblock_region_memory_base_pfn (r);
_
Andrew Morton
maintainerPATCH 12/131 mm: make early_pfn_to_nid() and related defintions close to each other
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: make early_pfn_to_nid() and related defintions close to each other
early_pfn_to_nid() and its helper __early_pfn_to_nid() are spread around include/linux/mm.h, include/linux/mmzone.h and mm/page_alloc.c.
Drop unused stub for __early_pfn_to_nid() and move its actual generic implementation close to its users.
---
include/linux/mm.h | 4 +-- include/linux/mmzone.h | 9 ------ mm/page_alloc.c | 51 +++++++++++++++++++-------------------- 3 files changed, 27 insertions(+), 37 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-make-early_pfn_to_nid-and-related-defintions-close-to-each-other +2 -2
@@ -2445,9 +2445,9 @@ extern void sparse_memory_present_with_a#if !defined(CONFIG_HAVE_MEMBLOCK_NODE_MAP ) && \ !defined(CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID ) -static inline int __early_pfn_to_nid (unsigned long pfn, -struct mminit_pfnnid_cache *state) +static inline int early_pfn_to_nid (unsigned long pfn) { +BUILD_BUG_ON (IS_ENABLED (CONFIG_NUMA ));return 0 ; }#else
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-make-early_pfn_to_nid-and-related-defintions-close-to-each-other +0 -9
@@ -1080,15 +1080,6 @@ static inline struct zoneref *first_zone#include <asm/sparsemem.h> #endif -#if !defined(CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID ) && \ - !defined(CONFIG_HAVE_MEMBLOCK_NODE_MAP ) -static inline unsigned long early_pfn_to_nid (unsigned long pfn) -{ -BUILD_BUG_ON (IS_ENABLED (CONFIG_NUMA )); -return 0 ; -} -#endif -#ifdef CONFIG_FLATMEM #define pfn_to_nid (pfn) (0)#endif
mm/page_alloc.c renamed from mm/page_alloc.c~mm-make-early_pfn_to_nid-and-related-defintions-close-to-each-other +25 -25
59 lines not on this page · raw message
Andrew Morton
maintainerPATCH 13/131 mm: remove CONFIG_HAVE_MEMBLOCK_NODE_MAP option
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: remove CONFIG_HAVE_MEMBLOCK_NODE_MAP option
CONFIG_HAVE_MEMBLOCK_NODE_MAP is used to differentiate initialization of nodes and zones structures between the systems that have region to node mapping in memblock and those that don't.
Currently all the NUMA architectures enable this option and for the non-NUMA systems we can presume that all the memory belongs to node 0 and therefore the compile time configuration option is not required.
The remaining few architectures that use DISCONTIGMEM without NUMA are easily updated to use memblock_add_node() instead of memblock_add() and thus have proper correspondence of memblock regions to NUMA nodes.
Still, free_area_init_node() must have a backward compatible version because its semantics with and without CONFIG_HAVE_MEMBLOCK_NODE_MAP is different. Once all the architectures will use the new semantics, the entire compatibility layer can be dropped.
To avoid addition of extra run time memory to store node id for architectures that keep memblock but have only a single node, the node id field of the memblock_region is guarded by CONFIG_NEED_MULTIPLE_NODES and the corresponding accessors presume that in those cases it is always 0.
---
Documentation/features/vm/numa-memblock/arch-support.txt | 34 --- arch/alpha/mm/numa.c | 4 arch/arm64/Kconfig | 1 arch/ia64/Kconfig | 1 arch/m68k/mm/motorola.c | 4 arch/microblaze/Kconfig | 1 arch/mips/Kconfig | 1 arch/powerpc/Kconfig | 1 arch/riscv/Kconfig | 1 arch/s390/Kconfig | 1 arch/sh/Kconfig | 1 arch/sparc/Kconfig | 1 arch/x86/Kconfig | 1 include/linux/memblock.h | 8 include/linux/mm.h | 12 - include/linux/mmzone.h | 2 mm/Kconfig | 3 mm/memblock.c | 11 - mm/memory_hotplug.c | 4 mm/page_alloc.c | 101 +++++----- 20 files changed, 74 insertions(+), 119 deletions(-)
arch/alpha/mm/numa.c renamed from arch/alpha/mm/numa.c~mm-remove-config_have_memblock_node_map-option +2 -2
@@ -144,8 +144,8 @@ setup_memory_node(int nid, void *kernel_if (!nid && (node_max_pfn < end_kernel_pfn || node_min_pfn > start_kernel_pfn))panic ("kernel loaded out of ram" ); -memblock_add (PFN_PHYS (node_min_pfn), - (node_max_pfn - node_min_pfn) <<PAGE_SHIFT ); +memblock_add_node (PFN_PHYS (node_min_pfn), + (node_max_pfn - node_min_pfn) <<PAGE_SHIFT , nid);/* Zone start phys-addr must be 2^(MAX_ORDER-1) aligned. Note that we round this down, not up - node memory
arch/arm64/Kconfig renamed from arch/arm64/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -162,7 +162,6 @@ config ARM64 select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTING - select HAVE_MEMBLOCK_NODE_MAP if NUMA select HAVE_NMI select HAVE_PATA_PLATFORM select HAVE_PERF_EVENTS
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -31,7 +31,6 @@ config IA64 select HAVE_FUNCTION_TRACER select TTY select HAVE_ARCH_TRACEHOOK - select HAVE_MEMBLOCK_NODE_MAP select HAVE_VIRT_CPU_ACCOUNTING select DMA_NONCOHERENT_MMAP select ARCH_HAS_SYNC_DMA_FOR_CPU
arch/m68k/mm/motorola.c renamed from arch/m68k/mm/motorola.c~mm-remove-config_have_memblock_node_map-option +2 -2
@@ -386,7 +386,7 @@ void __init paging_init(void) min_addr = m68k_memory[0 ].addr; max_addr = min_addr + m68k_memory[0 ].size; -memblock_add (m68k_memory[0 ].addr, m68k_memory[0 ].size); +memblock_add_node (m68k_memory[0 ].addr, m68k_memory[0 ].size,0 );for (i =1 ; i < m68k_num_memory;) {if (m68k_memory[i].addr < min_addr) { printk("Ignoring memory chunk at 0x%lx:0x%lx before the first chunk\n",
@@ -397,7 +397,7 @@ void __init paging_init(void) (m68k_num_memory - i) * sizeof(struct m68k_mem_info));continue ; } -memblock_add (m68k_memory[i].addr, m68k_memory[i].size); +memblock_add_node (m68k_memory[i].addr, m68k_memory[i].size, i); addr = m68k_memory[i].addr + m68k_memory[i].size;if (addr > max_addr) max_addr = addr;
arch/microblaze/Kconfig renamed from arch/microblaze/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -32,7 +32,6 @@ config MICROBLAZE select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER - select HAVE_MEMBLOCK_NODE_MAP select HAVE_OPROFILE select HAVE_PCI select IRQ_DOMAIN
arch/mips/Kconfig renamed from arch/mips/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -72,7 +72,6 @@ config MIPS select HAVE_KPROBES select HAVE_KRETPROBES select HAVE_LD_DEAD_CODE_DATA_ELIMINATION - select HAVE_MEMBLOCK_NODE_MAP select HAVE_MOD_ARCH_SPECIFIC select HAVE_NMI select HAVE_OPROFILE
arch/powerpc/Kconfig renamed from arch/powerpc/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -211,7 +211,6 @@ config PPC select HAVE_KRETPROBES select HAVE_LD_DEAD_CODE_DATA_ELIMINATION select HAVE_LIVEPATCH if HAVE_DYNAMIC_FTRACE_WITH_REGS - select HAVE_MEMBLOCK_NODE_MAP select HAVE_MOD_ARCH_SPECIFIC select HAVE_NMI if PERF_EVENTS || (PPC64 && PPC_BOOK3S) select HAVE_HARDLOCKUP_DETECTOR_ARCH if (PPC64 && PPC_BOOK3S)
arch/riscv/Kconfig renamed from arch/riscv/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -32,7 +32,6 @@ config RISCV select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_SECCOMP_FILTER select HAVE_ASM_MODVERSIONS - select HAVE_MEMBLOCK_NODE_MAP select HAVE_DMA_CONTIGUOUS if MMU select HAVE_FUTEX_CMPXCHG if FUTEX select HAVE_PERF_EVENTS
arch/s390/Kconfig renamed from arch/s390/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -162,7 +162,6 @@ config S390 select HAVE_LIVEPATCH select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select HAVE_MEMBLOCK_NODE_MAP select HAVE_MEMBLOCK_PHYS_MAP select MMU_GATHER_NO_GATHER select HAVE_MOD_ARCH_SPECIFIC
arch/sh/Kconfig renamed from arch/sh/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -9,7 +9,6 @@ config SUPERH select CLKDEV_LOOKUP select DMA_DECLARE_COHERENT select HAVE_IDE if HAS_IOPORT_MAP - select HAVE_MEMBLOCK_NODE_MAP select HAVE_OPROFILE select HAVE_ARCH_TRACEHOOK select HAVE_PERF_EVENTS
arch/sparc/Kconfig renamed from arch/sparc/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -65,7 +65,6 @@ config SPARC64 select HAVE_KRETPROBES select HAVE_KPROBES select MMU_GATHER_RCU_TABLE_FREE if SMP - select HAVE_MEMBLOCK_NODE_MAP select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD
arch/x86/Kconfig renamed from arch/x86/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -1
@@ -192,7 +192,6 @@ config X86 select HAVE_KRETPROBES select HAVE_KVM select HAVE_LIVEPATCH if X86_64 - select HAVE_MEMBLOCK_NODE_MAP select HAVE_MIXED_BREAKPOINTS_REGS select HAVE_MOD_ARCH_SPECIFIC select HAVE_MOVE_PMD
Documentation/features/vm/numa-memblock/arch-support.txt deleted +0 -34
@@ -1,34 +0,0 @@ -# -# Feature name: numa-memblock -# Kconfig: HAVE_MEMBLOCK_NODE_MAP -# description: arch supports NUMA aware memblocks -# - ----------------------- - | arch |status| - ----------------------- - | alpha: | TODO | - | arc: | .. | - | arm: | .. | - | arm64: | ok | - | c6x: | .. | - | csky: | .. | - | h8300: | .. | - | hexagon: | .. | - | ia64: | ok | - | m68k: | .. | - | microblaze: | ok | - | mips: | ok | - | nds32: | TODO | - | nios2: | .. | - | openrisc: | .. | - | parisc: | .. | - | powerpc: | ok | - | riscv: | ok | - | s390: | ok | - | sh: | ok | - | sparc: | ok | - | um: | .. | - | unicore32: | .. | - | x86: | ok | - | xtensa: | .. | - -----------------------
include/linux/memblock.h renamed from include/linux/memblock.h~mm-remove-config_have_memblock_node_map-option +3 -5
@@ -50,7 +50,7 @@ struct memblock_region {phys_addr_t base;phys_addr_t size;enum memblock_flags flags; -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP +#ifdef CONFIG_NEED_MULTIPLE_NODES int nid;#endif };
@@ -215,7 +215,6 @@ static inline bool memblock_is_nomap(strreturn m->flags &MEMBLOCK_NOMAP ; } -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP int memblock_search_pfn_nid (unsigned long pfn,unsigned long *start_pfn,unsigned long *end_pfn); void __next_mem_pfn_range(int *idx, int nid, unsigned long *out_start_pfn,
@@ -234,7 +233,6 @@ void __next_mem_pfn_range(int *idx, int#define for_each_mem_pfn_range (i, nid, p_start, p_end, p_nid) \ for (i = -1, __next_mem_pfn_range(&i, nid, p_start, p_end, p_nid); \ i >= 0; __next_mem_pfn_range(&i, nid, p_start, p_end, p_nid)) -#endif /* CONFIG_HAVE_MEMBLOCK_NODE_MAP */ #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT void __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone,
@@ -310,10 +308,10 @@ void __next_mem_pfn_range_in_zone(u64 *ifor_each_mem_range_rev (i, &memblock.memory, &memblock.reserved, \ nid, flags, p_start, p_end, p_nid) -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP int memblock_set_node (phys_addr_t base,phys_addr_t size,struct memblock_type *type,int nid); +#ifdef CONFIG_NEED_MULTIPLE_NODES static inline void memblock_set_region_node (struct memblock_region *r,int nid) { r->nid = nid;
@@ -332,7 +330,7 @@ static inline int memblock_get_region_no {return 0 ; } -#endif /* CONFIG_HAVE_MEMBLOCK_NODE_MAP */ +#endif /* CONFIG_NEED_MULTIPLE_NODES */ /* Flags for memblock allocation APIs */ #define MEMBLOCK_ALLOC_ANYWHERE (~(phys_addr_t)0)
include/linux/mm.h renamed from include/linux/mm.h~mm-remove-config_have_memblock_node_map-option +2 -10
@@ -2401,9 +2401,8 @@ static inline unsigned long get_num_physreturn phys_pages; } -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP /* -* With CONFIG_HAVE_MEMBLOCK_NODE_MAP set, an architecture may initialise its +* Using memblock node mappings, an architecture may initialise its * zones, allocate the backing mem_map and account for memory holes in a more * architecture independent manner. This is a substitute for creating the * zone_sizes[] and zholes_size[] arrays and passing them to
@@ -2424,9 +2423,6 @@ static inline unsigned long get_num_phys* registered physical page range. Similarly * sparse_memory_present_with_active_regions() calls memory_present() for * each range when SPARSEMEM is enabled. -* -* See mm/page_alloc.c for more information on each function exposed by -* CONFIG_HAVE_MEMBLOCK_NODE_MAP. */ extern void free_area_init_nodes (unsigned long *max_zone_pfn);unsigned long node_map_pfn_alignment (void );
@@ -2441,13 +2437,9 @@ extern void free_bootmem_with_active_reg unsigned long max_low_pfn);extern void sparse_memory_present_with_active_regions (int nid); -#endif /* CONFIG_HAVE_MEMBLOCK_NODE_MAP */ - -#if !defined(CONFIG_HAVE_MEMBLOCK_NODE_MAP ) && \ - !defined(CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID ) +#ifndef CONFIG_NEED_MULTIPLE_NODES static inline int early_pfn_to_nid (unsigned long pfn) { -BUILD_BUG_ON (IS_ENABLED (CONFIG_NUMA ));return 0 ; }#else
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-remove-config_have_memblock_node_map-option +1 -1
@@ -876,7 +876,7 @@ extern int movable_zone;#ifdef CONFIG_HIGHMEM static inline int zone_movable_is_highmem (void ) { -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP +#ifdef CONFIG_NEED_MULTIPLE_NODES return movable_zone ==ZONE_HIGHMEM ;#else return (ZONE_MOVABLE -1 ) ==ZONE_HIGHMEM ;
mm/Kconfig renamed from mm/Kconfig~mm-remove-config_have_memblock_node_map-option +0 -3
@@ -126,9 +126,6 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. -config HAVE_MEMBLOCK_NODE_MAP - bool - config HAVE_MEMBLOCK_PHYS_MAP bool
mm/memblock.c renamed from mm/memblock.c~mm-remove-config_have_memblock_node_map-option +5 -6
@@ -620,7 +620,7 @@ repeat:* area, insert that portion. */ if (rbase > base) { -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP +#ifdef CONFIG_NEED_MULTIPLE_NODES WARN_ON (nid !=memblock_get_region_node (rgn));#endif WARN_ON (flags != rgn->flags);
@@ -1197,7 +1197,6 @@ void __init_memblock __next_mem_range_re *idx =ULLONG_MAX ; } -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP /* * Common iterator interface used to define for_each_mem_pfn_range(). */
@@ -1247,6 +1246,7 @@ void __init_memblock __next_mem_pfn_rang int __init_memblock memblock_set_node(phys_addr_t base, phys_addr_t size,struct memblock_type *type,int nid) { +#ifdef CONFIG_NEED_MULTIPLE_NODES int start_rgn, end_rgn;int i, ret;
@@ -1258,9 +1258,10 @@ int __init_memblock memblock_set_node(phmemblock_set_region_node (&type->regions[i], nid);memblock_merge_regions (type); +#endif return 0 ; } -#endif /* CONFIG_HAVE_MEMBLOCK_NODE_MAP */ +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /** * __next_mem_pfn_range_in_zone - iterator for for_each_*_range_in_zone()
@@ -1799,7 +1800,6 @@ bool __init_memblock memblock_is_map_memreturn !memblock_is_nomap (&memblock.memory.regions[i]); } -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP int __init_memblock memblock_search_pfn_nid(unsigned long pfn,unsigned long *start_pfn,unsigned long *end_pfn) {
@@ -1814,7 +1814,6 @@ int __init_memblock memblock_search_pfn_return memblock_get_region_node (&type->regions[mid]); } -#endif /** * memblock_is_region_memory - check if a region is a subset of memory
@@ -1905,7 +1904,7 @@ static void __init_memblock memblock_dum size = rgn->size; end = base + size -1 ; flags = rgn->flags; -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP +#ifdef CONFIG_NEED_MULTIPLE_NODES if (memblock_get_region_node (rgn) !=MAX_NUMNODES )snprintf (nid_buf,sizeof (nid_buf)," on node %d" ,memblock_get_region_node (rgn));
mm/memory_hotplug.c renamed from mm/memory_hotplug.c~mm-remove-config_have_memblock_node_map-option +0 -4
@@ -1372,11 +1372,7 @@ check_pages_isolated_cb(unsigned long ststatic int __init cmdline_parse_movable_node (char *p) { -#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP movable_node_enabled = true; -#else -pr_warn ("movable_node parameter depends on CONFIG_HAVE_MEMBLOCK_NODE_MAP to work properly\n" ); -#endif return 0 ; }early_param ("movable_node" , cmdline_parse_movable_node);
mm/page_alloc.c renamed from mm/page_alloc.c~mm-remove-config_have_memblock_node_map-option +59 -41
208 lines not on this page · raw message
Andrew Morton
maintainerPATCH 14/131 mm: free_area_init: use maximal zone PFNs rather than zone sizes
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: free_area_init: use maximal zone PFNs rather than zone sizes
Currently, architectures that use free_area_init() to initialize memory map and node and zone structures need to calculate zone and hole sizes. We can use free_area_init_nodes() instead and let it detect the zone boundaries while the architectures will only have to supply the possible limits for the zones.
---
arch/alpha/mm/init.c | 16 ++++++---------- arch/c6x/mm/init.c | 8 +++----- arch/h8300/mm/init.c | 6 +++--- arch/hexagon/mm/init.c | 6 +++--- arch/m68k/mm/init.c | 6 +++--- arch/m68k/mm/mcfmmu.c | 9 +++------ arch/nds32/mm/init.c | 11 ++++------- arch/nios2/mm/init.c | 8 +++----- arch/openrisc/mm/init.c | 9 +++------ arch/um/kernel/mem.c | 12 ++++-------- include/linux/mm.h | 2 +- mm/page_alloc.c | 5 ++--- 12 files changed, 38 insertions(+), 60 deletions(-)
arch/alpha/mm/init.c renamed from arch/alpha/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +6 -10
@@ -243,21 +243,17 @@ callback_init(void * kernel_end)*/ void __init paging_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; -unsigned long dma_pfn, high_pfn; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , }; +unsigned long dma_pfn; dma_pfn =virt_to_phys ((char *)MAX_DMA_ADDRESS ) >>PAGE_SHIFT ; - high_pfn = max_pfn = max_low_pfn; + max_pfn = max_low_pfn; -if (dma_pfn >= high_pfn) - zones_size[ZONE_DMA ] = high_pfn; -else { - zones_size[ZONE_DMA ] = dma_pfn; - zones_size[ZONE_NORMAL ] = high_pfn - dma_pfn; - } + max_zone_pfn[ZONE_DMA ] = dma_pfn; + max_zone_pfn[ZONE_NORMAL ] = max_pfn;/* Initialize mem_map[]. */ -free_area_init (zones_size); +free_area_init (max_zone_pfn);/* Initialize the kernel's ZERO_PGE. */ memset ((void *)ZERO_PGE ,0 ,PAGE_SIZE );
arch/c6x/mm/init.c renamed from arch/c6x/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +3 -5
@@ -33,7 +33,7 @@ EXPORT_SYMBOL(empty_zero_page);void __init paging_init (void ) {struct pglist_data *pgdat =NODE_DATA (0 ); -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , }; empty_zero_page = (unsigned long )memblock_alloc (PAGE_SIZE ,PAGE_SIZE );
@@ -49,11 +49,9 @@ void __init paging_init(void)/* * Define zones */ - zones_size[ZONE_NORMAL ] = (memory_end -PAGE_OFFSET ) >>PAGE_SHIFT ; - pgdat->node_zones[ZONE_NORMAL ].zone_start_pfn = -__pa (PAGE_OFFSET ) >>PAGE_SHIFT ; + max_zone_pfn[ZONE_NORMAL ] = memory_end >>PAGE_SHIFT ; -free_area_init (zones_size); +free_area_init (max_zone_pfn); }void __init mem_init (void )
arch/h8300/mm/init.c renamed from arch/h8300/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +3 -3
@@ -83,10 +83,10 @@ void __init paging_init(void) start_mem, end_mem); { -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , }; - zones_size[ZONE_NORMAL ] = (end_mem -PAGE_OFFSET ) >>PAGE_SHIFT ; -free_area_init (zones_size); + max_zone_pfn[ZONE_NORMAL ] = end_mem >>PAGE_SHIFT ; +free_area_init (max_zone_pfn); } }
arch/hexagon/mm/init.c renamed from arch/hexagon/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +3 -3
@@ -91,7 +91,7 @@ void sync_icache_dcache(pte_t pte)*/ void __init paging_init (void ) { -unsigned long zones_sizes[MAX_NR_ZONES ] = {0 , }; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , };/* * This is not particularly well documented anywhere, but
@@ -101,9 +101,9 @@ void __init paging_init(void)* adjust accordingly. */ - zones_sizes[ZONE_NORMAL ] = max_low_pfn; + max_zone_pfn[ZONE_NORMAL ] = max_low_pfn; -free_area_init (zones_sizes);/* sets up the zonelists and mem_map */ +free_area_init (max_zone_pfn);/* sets up the zonelists and mem_map */ /* * Start of high memory area. Will probably need something more
arch/m68k/mm/init.c renamed from arch/m68k/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +3 -3
@@ -84,7 +84,7 @@ void __init paging_init(void)* page_alloc get different views of the world. */ unsigned long end_mem = memory_end &PAGE_MASK ; -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , }; high_memory = (void *) end_mem;
@@ -98,8 +98,8 @@ void __init paging_init(void)*/ set_fs (USER_DS ); - zones_size[ZONE_DMA ] = (end_mem -PAGE_OFFSET ) >>PAGE_SHIFT ; - free_area_init(zones_size); + max_zone_pfn[ZONE_DMA ] = end_mem >>PAGE_SHIFT ; + free_area_init(max_zone_pfn); }#endif /* CONFIG_MMU */
arch/m68k/mm/mcfmmu.c renamed from arch/m68k/mm/mcfmmu.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +3 -6
@@ -39,7 +39,7 @@ void __init paging_init(void)pte_t *pg_table;unsigned long address, size;unsigned long next_pgtable, bootmem_end; -unsigned long zones_size[MAX_NR_ZONES ]; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 };enum zone_type zone;int i;
@@ -80,11 +80,8 @@ void __init paging_init(void) } current->mm =NULL ; - -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) - zones_size[zone] =0x0 ; - zones_size[ZONE_DMA ] = num_pages; -free_area_init (zones_size); + max_zone_pfn[ZONE_DMA ] =PFN_DOWN (_ramend); +free_area_init (max_zone_pfn); }int cf_tlb_miss (struct pt_regs *regs,int write,int dtlb,int extension_word)
arch/nds32/mm/init.c renamed from arch/nds32/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +4 -7
@@ -31,16 +31,13 @@ EXPORT_SYMBOL(empty_zero_page);static void __init zone_sizes_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ]; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 }; -/* Clear the zone sizes */ -memset (zones_size,0 ,sizeof (zones_size)); - - zones_size[ZONE_NORMAL ] = max_low_pfn; + max_zone_pfn[ZONE_NORMAL ] = max_low_pfn;#ifdef CONFIG_HIGHMEM - zones_size[ZONE_HIGHMEM ] = max_pfn; + max_zone_pfn[ZONE_HIGHMEM ] = max_pfn;#endif -free_area_init (zones_size); +free_area_init (max_zone_pfn); }
arch/nios2/mm/init.c renamed from arch/nios2/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +3 -5
@@ -46,17 +46,15 @@ pgd_t *pgd_current;*/ void __init paging_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ]; - -memset (zones_size,0 ,sizeof (zones_size)); +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 };pagetable_init (); pgd_current = swapper_pg_dir; - zones_size[ZONE_NORMAL ] = max_mapnr; + max_zone_pfn[ZONE_NORMAL ] = max_mapnr;/* pass the memory from the bootmem allocator to the main allocator */ -free_area_init (zones_size); +free_area_init (max_zone_pfn);flush_dcache_range ((unsigned long )empty_zero_page, (unsigned long )empty_zero_page +PAGE_SIZE );
arch/openrisc/mm/init.c renamed from arch/openrisc/mm/init.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +3 -6
@@ -45,17 +45,14 @@ DEFINE_PER_CPU(struct mmu_gather, mmu_gastatic void __init zone_sizes_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ]; - -/* Clear the zone sizes */ -memset (zones_size,0 ,sizeof (zones_size)); +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 };/* * We use only ZONE_NORMAL */ - zones_size[ZONE_NORMAL ] = max_low_pfn; + max_zone_pfn[ZONE_NORMAL ] = max_low_pfn; -free_area_init (zones_size); +free_area_init (max_zone_pfn); }extern const char _s_kernel_ro[], _e_kernel_ro[];
arch/um/kernel/mem.c renamed from arch/um/kernel/mem.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +4 -8
@@ -158,8 +158,8 @@ static void __init fixaddr_user_init( vovoid __init paging_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ], vaddr; -int i; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 }; +unsigned long vaddr; empty_zero_page = (unsigned long *)memblock_alloc_low (PAGE_SIZE ,PAGE_SIZE );
@@ -167,12 +167,8 @@ void __init paging_init(void)panic ("%s: Failed to allocate %lu bytes align=%lx\n" , __func__,PAGE_SIZE ,PAGE_SIZE ); -for (i =0 ; i <ARRAY_SIZE (zones_size); i++) - zones_size[i] =0 ; - - zones_size[ZONE_NORMAL ] = (end_iomem >>PAGE_SHIFT ) - - (uml_physmem >>PAGE_SHIFT ); -free_area_init (zones_size); + max_zone_pfn[ZONE_NORMAL ] = end_iomem >>PAGE_SHIFT ; +free_area_init (max_zone_pfn);/* * Fixed mappings, only the page table structure has to be
include/linux/mm.h renamed from include/linux/mm.h~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +1 -1
@@ -2329,7 +2329,7 @@ static inline spinlock_t *pud_lock(struc }extern void __init pagecache_init (void ); -extern void free_area_init (unsigned long * zones_size); +extern void free_area_init (unsigned long * max_zone_pfn);extern void __init free_area_init_node (int nid,unsigned long * zones_size,unsigned long zone_start_pfn,unsigned long *zholes_size);extern void free_initmem (void );
mm/page_alloc.c renamed from mm/page_alloc.c~mm-free_area_init-use-maximal-zone-pfns-rather-than-zone-sizes +2 -3
@@ -7712,11 +7712,10 @@ void __init set_dma_reserve(unsigned lon dma_reserve = new_dma_reserve; } -void __init free_area_init (unsigned long *zones_size) +void __init free_area_init (unsigned long *max_zone_pfn) {init_unavailable_mem (); -free_area_init_node (0 , zones_size, -__pa (PAGE_OFFSET ) >>PAGE_SHIFT ,NULL ); +free_area_init_nodes (max_zone_pfn); }static int page_alloc_cpu_dead (unsigned int cpu)
_
Andrew Morton
maintainerPATCH 15/131 mm: use free_area_init() instead of free_area_init_nodes()
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: use free_area_init() instead of free_area_init_nodes()
free_area_init() has effectively became a wrapper for free_area_init_nodes() and there is no point of keeping it. Still free_area_init() name is shorter and more general as it does not imply necessity to initialize multiple nodes.
Rename free_area_init_nodes() to free_area_init(), update the callers and drop old version of free_area_init().
---
arch/arm64/mm/init.c | 2 +- arch/ia64/mm/contig.c | 2 +- arch/ia64/mm/discontig.c | 2 +- arch/microblaze/mm/init.c | 2 +- arch/mips/loongson64/numa.c | 2 +- arch/mips/mm/init.c | 2 +- arch/mips/sgi-ip27/ip27-memory.c | 2 +- arch/powerpc/mm/mem.c | 2 +- arch/riscv/mm/init.c | 2 +- arch/s390/mm/init.c | 2 +- arch/sh/mm/init.c | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/x86/mm/init.c | 2 +- include/linux/mm.h | 7 +++---- mm/page_alloc.c | 10 ++-------- 15 files changed, 18 insertions(+), 25 deletions(-)
arch/arm64/mm/init.c renamed from arch/arm64/mm/init.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -206,7 +206,7 @@ static void __init zone_sizes_init(unsig#endif max_zone_pfns[ZONE_NORMAL ] = max; -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); }#else
arch/ia64/mm/contig.c renamed from arch/ia64/mm/contig.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -210,6 +210,6 @@ paging_init (void) printk("Virtual mem_map starts at 0x%p\n", mem_map); }#endif /* !CONFIG_VIRTUAL_MEM_MAP */ -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); zero_page_memmap_ptr =virt_to_page (ia64_imva (empty_zero_page)); }
arch/ia64/mm/discontig.c renamed from arch/ia64/mm/discontig.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -627,7 +627,7 @@ void __init paging_init(void) max_zone_pfns[ZONE_DMA32 ] = max_dma;#endif max_zone_pfns[ZONE_NORMAL ] = max_pfn; -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); zero_page_memmap_ptr =virt_to_page (ia64_imva (empty_zero_page)); }
arch/microblaze/mm/init.c renamed from arch/microblaze/mm/init.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -112,7 +112,7 @@ static void __init paging_init(void)#endif /* We don't have holes in memory map */ -free_area_init_nodes (zones_size); +free_area_init (zones_size); }void __init setup_memory (void )
arch/mips/loongson64/numa.c renamed from arch/mips/loongson64/numa.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -247,7 +247,7 @@ void __init paging_init(void) zones_size[ZONE_DMA32 ] =MAX_DMA32_PFN ;#endif zones_size[ZONE_NORMAL ] = max_low_pfn; -free_area_init_nodes (zones_size); +free_area_init (zones_size); }void __init mem_init (void )
arch/mips/mm/init.c renamed from arch/mips/mm/init.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -418,7 +418,7 @@ void __init paging_init(void) }#endif -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); }#ifdef CONFIG_64BIT
arch/mips/sgi-ip27/ip27-memory.c renamed from arch/mips/sgi-ip27/ip27-memory.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -419,7 +419,7 @@ void __init paging_init(void)pagetable_init (); zones_size[ZONE_NORMAL ] = max_low_pfn; -free_area_init_nodes (zones_size); +free_area_init (zones_size); }void __init mem_init (void )
arch/powerpc/mm/mem.c renamed from arch/powerpc/mm/mem.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -271,7 +271,7 @@ void __init paging_init(void) max_zone_pfns[ZONE_HIGHMEM ] = max_pfn;#endif -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns);mark_nonram_nosave (); }
arch/riscv/mm/init.c renamed from arch/riscv/mm/init.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -39,7 +39,7 @@ static void __init zone_sizes_init(void)#endif max_zone_pfns[ZONE_NORMAL ] = max_low_pfn; -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); }static void setup_zero_page (void )
arch/s390/mm/init.c renamed from arch/s390/mm/init.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -122,7 +122,7 @@ void __init paging_init(void)memset (max_zone_pfns,0 ,sizeof (max_zone_pfns)); max_zone_pfns[ZONE_DMA ] =PFN_DOWN (MAX_DMA_ADDRESS ); max_zone_pfns[ZONE_NORMAL ] = max_low_pfn; -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); }void mark_rodata_ro (void )
arch/sh/mm/init.c renamed from arch/sh/mm/init.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -334,7 +334,7 @@ void __init paging_init(void)memset (max_zone_pfns,0 ,sizeof (max_zone_pfns)); max_zone_pfns[ZONE_NORMAL ] = max_low_pfn; -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); }unsigned int mem_init_done =0 ;
arch/sparc/mm/init_64.c renamed from arch/sparc/mm/init_64.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -2488,7 +2488,7 @@ void __init paging_init(void) max_zone_pfns[ZONE_NORMAL ] = end_pfn; -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); }printk ("Booting Linux...\n" );
arch/x86/mm/init.c renamed from arch/x86/mm/init.c~mm-use-free_area_init-instead-of-free_area_init_nodes +1 -1
@@ -947,7 +947,7 @@ void __init zone_sizes_init(void) max_zone_pfns[ZONE_HIGHMEM ] = max_pfn;#endif -free_area_init_nodes (max_zone_pfns); +free_area_init (max_zone_pfns); }__visible DEFINE_PER_CPU_SHARED_ALIGNED (struct tlb_state ,cpu_tlbstate ) = {
include/linux/mm.h renamed from include/linux/mm.h~mm-use-free_area_init-instead-of-free_area_init_nodes +3 -4
@@ -2329,7 +2329,6 @@ static inline spinlock_t *pud_lock(struc }extern void __init pagecache_init (void ); -extern void free_area_init (unsigned long * max_zone_pfn);extern void __init free_area_init_node (int nid,unsigned long * zones_size,unsigned long zone_start_pfn,unsigned long *zholes_size);extern void free_initmem (void );
@@ -2410,21 +2409,21 @@ static inline unsigned long get_num_phys* * An architecture is expected to register range of page frames backed by * physical memory with memblock_add[_node]() before calling -* free_area_init_nodes() passing in the PFN each zone ends at. At a basic +* free_area_init() passing in the PFN each zone ends at. At a basic * usage, an architecture is expected to do something like * * unsigned long max_zone_pfns[MAX_NR_ZONES] = {max_dma, max_normal_pfn, * max_highmem_pfn}; * for_each_valid_physical_page_range() * memblock_add_node(base, size, nid) -* free_area_init_nodes(max_zone_pfns); +* free_area_init(max_zone_pfns); * * free_bootmem_with_active_regions() calls free_bootmem_node() for each * registered physical page range. Similarly * sparse_memory_present_with_active_regions() calls memory_present() for * each range when SPARSEMEM is enabled. */ -extern void free_area_init_nodes (unsigned long *max_zone_pfn); +void free_area_init (unsigned long *max_zone_pfn);unsigned long node_map_pfn_alignment (void );unsigned long __absent_pages_in_range (int nid,unsigned long start_pfn,unsigned long end_pfn);
mm/page_alloc.c renamed from mm/page_alloc.c~mm-use-free_area_init-instead-of-free_area_init_nodes +2 -7
@@ -7440,7 +7440,7 @@ static void check_for_memory(pg_data_t * }/** -* free_area_init_nodes - Initialise all pg_data_t and zone data +* free_area_init - Initialise all pg_data_t and zone data * @max_zone_pfn: an array of max PFNs for each zone * * This will call free_area_init_node() for each active node in the system.
@@ -7452,7 +7452,7 @@ static void check_for_memory(pg_data_t ** starts where the previous one ended. For example, ZONE_DMA32 starts * at arch_max_dma_pfn. */ -void __init free_area_init_nodes (unsigned long *max_zone_pfn) +void __init free_area_init (unsigned long *max_zone_pfn) {unsigned long start_pfn, end_pfn;int i, nid;
@@ -7712,12 +7712,6 @@ void __init set_dma_reserve(unsigned lon dma_reserve = new_dma_reserve; } -void __init free_area_init (unsigned long *max_zone_pfn) -{ -init_unavailable_mem (); -free_area_init_nodes (max_zone_pfn); -}
Andrew Morton
maintainerPATCH 16/131 alpha: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: alpha: simplify detection of memory zone boundaries
free_area_init() only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/alpha/mm/numa.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)
arch/alpha/mm/numa.c renamed from arch/alpha/mm/numa.c~alpha-simplify-detection-of-memory-zone-boundaries +4 -14
@@ -202,8 +202,7 @@ setup_memory(void *kernel_end)void __init paging_init (void ) { -unsigned int nid; -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , };unsigned long dma_local_pfn;/*
@@ -215,19 +214,10 @@ void __init paging_init(void)*/ dma_local_pfn =virt_to_phys ((char *)MAX_DMA_ADDRESS ) >>PAGE_SHIFT ; -for_each_online_node (nid) { -unsigned long start_pfn =NODE_DATA (nid)->node_start_pfn; -unsigned long end_pfn = start_pfn +NODE_DATA (nid)->node_present_pages; + max_zone_pfn[ZONE_DMA ] = dma_local_pfn; + max_zone_pfn[ZONE_NORMAL ] = max_pfn; -if (dma_local_pfn >= end_pfn - start_pfn) - zones_size[ZONE_DMA ] = end_pfn - start_pfn; -else { - zones_size[ZONE_DMA ] = dma_local_pfn; - zones_size[ZONE_NORMAL ] = (end_pfn - start_pfn) - dma_local_pfn; - } -node_set_state (nid,N_NORMAL_MEMORY ); -free_area_init_node (nid, zones_size, start_pfn,NULL ); - } +free_area_init (max_zone_pfn);/* Initialize the kernel's ZERO_PGE. */ memset ((void *)ZERO_PGE ,0 ,PAGE_SIZE );
_
Andrew Morton
maintainerPATCH 17/131 arm: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: arm: simplify detection of memory zone boundaries
free_area_init() only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/arm/mm/init.c | 66 ++++--------------------------------------- 1 file changed, 7 insertions(+), 59 deletions(-)
arch/arm/mm/init.c renamed from arch/arm/mm/init.c~arm-simplify-detection-of-memory-zone-boundaries +5 -56
@@ -92,18 +92,6 @@ EXPORT_SYMBOL(arm_dma_zone_size);*/ phys_addr_t arm_dma_limit;unsigned long arm_dma_pfn_limit; - -static void __init arm_adjust_dma_zone (unsigned long *size,unsigned long *hole, -unsigned long dma_size) -{ -if (size[0 ] <= dma_size) -return ; - - size[ZONE_NORMAL ] = size[0 ] - dma_size; - size[ZONE_DMA ] = dma_size; - hole[ZONE_NORMAL ] = hole[0 ]; - hole[ZONE_DMA ] =0 ; -}#endif void __init setup_dma_zone (const struct machine_desc *mdesc)
@@ -121,56 +109,16 @@ void __init setup_dma_zone(const structstatic void __init zone_sizes_init (unsigned long min,unsigned long max_low,unsigned long max_high) { -unsigned long zone_size[MAX_NR_ZONES ], zhole_size[MAX_NR_ZONES ]; -struct memblock_region *reg; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 }; -/* -* initialise the zones. -*/ -memset (zone_size,0 ,sizeof (zone_size)); - -/* -* The memory size has already been determined. If we need -* to do anything fancy with the allocation of this memory -* to the zones, now is the time to do it. -*/ - zone_size[0 ] = max_low - min; -#ifdef CONFIG_HIGHMEM - zone_size[ZONE_HIGHMEM ] = max_high - max_low; +#ifdef CONFIG_ZONE_DMA + max_zone_pfn[ZONE_DMA ] =min (arm_dma_pfn_limit, max_low);#endif - -/* -* Calculate the size of the holes. -* holes = node_size - sum(bank_sizes) -*/ -memcpy (zhole_size, zone_size,sizeof (zhole_size)); -for_each_memblock (memory, reg) { -unsigned long start =memblock_region_memory_base_pfn (reg); -unsigned long end =memblock_region_memory_end_pfn (reg); - -if (start < max_low) { -unsigned long low_end =min (end, max_low); - zhole_size[0 ] -= low_end - start; - } + max_zone_pfn[ZONE_NORMAL ] = max_low;#ifdef CONFIG_HIGHMEM -if (end > max_low) { -unsigned long high_start =max (start, max_low); - zhole_size[ZONE_HIGHMEM ] -= end - high_start; - } + max_zone_pfn[ZONE_HIGHMEM ] = max_high;#endif - } - -#ifdef CONFIG_ZONE_DMA -/* -* Adjust the sizes according to any special requirements for -* this machine type. -*/ -if (arm_dma_zone_size) -arm_adjust_dma_zone (zone_size, zhole_size, - arm_dma_zone_size >>PAGE_SHIFT ); -#endif
Andrew Morton
maintainerPATCH 18/131 arm64: simplify detection of memory zone boundaries for UMA configs
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: arm64: simplify detection of memory zone boundaries for UMA configs
The free_area_init() function only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/arm64/mm/init.c | 54 ----------------------------------------- 1 file changed, 54 deletions(-)
arch/arm64/mm/init.c renamed from arch/arm64/mm/init.c~arm64-simplify-detection-of-memory-zone-boundaries-for-uma-configs +0 -53
@@ -192,8 +192,6 @@ static phys_addr_t __init max_zone_phys(return min (offset + (1ULL << zone_bits),memblock_end_of_DRAM ()); } -#ifdef CONFIG_NUMA -static void __init zone_sizes_init (unsigned long min,unsigned long max) {unsigned long max_zone_pfns[MAX_NR_ZONES ] = {0 };
@@ -209,58 +207,6 @@ static void __init zone_sizes_init(unsigfree_area_init (max_zone_pfns); } -#else - -static void __init zone_sizes_init (unsigned long min,unsigned long max) -{ -struct memblock_region *reg; -unsigned long zone_size[MAX_NR_ZONES ], zhole_size[MAX_NR_ZONES ]; -unsigned long __maybe_unused max_dma, max_dma32; - -memset (zone_size,0 ,sizeof (zone_size)); - - max_dma = max_dma32 = min; -#ifdef CONFIG_ZONE_DMA - max_dma = max_dma32 =PFN_DOWN (arm64_dma_phys_limit); - zone_size[ZONE_DMA ] = max_dma - min; -#endif -#ifdef CONFIG_ZONE_DMA32 - max_dma32 =PFN_DOWN (arm64_dma32_phys_limit); - zone_size[ZONE_DMA32 ] = max_dma32 - max_dma; -#endif - zone_size[ZONE_NORMAL ] = max - max_dma32; - -memcpy (zhole_size, zone_size,sizeof (zhole_size)); - -for_each_memblock (memory, reg) { -unsigned long start =memblock_region_memory_base_pfn (reg); -unsigned long end =memblock_region_memory_end_pfn (reg); - -#ifdef CONFIG_ZONE_DMA -if (start >= min && start < max_dma) { -unsigned long dma_end =min (end, max_dma); - zhole_size[ZONE_DMA ] -= dma_end - start; - start = dma_end; - } -#endif -#ifdef CONFIG_ZONE_DMA32 -if (start >= max_dma && start < max_dma32) { -unsigned long dma32_end =min (end, max_dma32); - zhole_size[ZONE_DMA32 ] -= dma32_end - start; - start = dma32_end; - } -#endif -if (start >= max_dma32 && start < max) { -unsigned long normal_end =min (end, max); - zhole_size[ZONE_NORMAL ] -= normal_end - start; - } - } - -free_area_init_node (0 , zone_size, min, zhole_size); -} - -#endif /* CONFIG_NUMA */
Andrew Morton
maintainerPATCH 19/131 csky: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: csky: simplify detection of memory zone boundaries
The free_area_init() function only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/csky/kernel/setup.c | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-)
arch/csky/kernel/setup.c renamed from arch/csky/kernel/setup.c~csky-simplify-detection-of-memory-zone-boundaries +11 -15
@@ -26,7 +26,9 @@ struct screen_info screen_info = {static void __init csky_memblock_init (void ) { -unsigned long zone_size[MAX_NR_ZONES ]; +unsigned long lowmem_size =PFN_DOWN (LOWMEM_LIMIT -PHYS_OFFSET_OFFSET ); +unsigned long sseg_size =PFN_DOWN (SSEG_SIZE -PHYS_OFFSET_OFFSET ); +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 };signed long size;memblock_reserve (__pa (_stext), _end - _stext);
@@ -36,28 +38,22 @@ static void __init csky_memblock_init(vomemblock_dump_all (); -memset (zone_size,0 ,sizeof (zone_size)); - min_low_pfn =PFN_UP (memblock_start_of_DRAM ()); max_low_pfn = max_pfn =PFN_DOWN (memblock_end_of_DRAM ()); size = max_pfn - min_low_pfn; -if (size <=PFN_DOWN (SSEG_SIZE -PHYS_OFFSET_OFFSET )) - zone_size[ZONE_NORMAL ] = size; -else if (size <PFN_DOWN (LOWMEM_LIMIT -PHYS_OFFSET_OFFSET )) { - zone_size[ZONE_NORMAL ] = -PFN_DOWN (SSEG_SIZE -PHYS_OFFSET_OFFSET ); - max_low_pfn = min_low_pfn + zone_size[ZONE_NORMAL ]; - }else { - zone_size[ZONE_NORMAL ] = -PFN_DOWN (LOWMEM_LIMIT -PHYS_OFFSET_OFFSET ); - max_low_pfn = min_low_pfn + zone_size[ZONE_NORMAL ]; +if (size >= lowmem_size) { + max_low_pfn = min_low_pfn + lowmem_size;write_mmu_msa1 (read_mmu_msa0 () +SSEG_SIZE ); + }else if (size > sseg_size) { + max_low_pfn = min_low_pfn + sseg_size; } + max_zone_pfn[ZONE_NORMAL ] = max_low_pfn; +#ifdef CONFIG_HIGHMEM - zone_size[ZONE_HIGHMEM ] = max_pfn - max_low_pfn; + max_zone_pfn[ZONE_HIGHMEM ] = max_pfn; highstart_pfn = max_low_pfn; highend_pfn = max_pfn;
@@ -66,7 +62,7 @@ static void __init csky_memblock_init(vodma_contiguous_reserve (0 ); -free_area_init_node (0 , zone_size, min_low_pfn,NULL ); +free_area_init (max_zone_pfn); }void __init setup_arch (char **cmdline_p)
_
Andrew Morton
maintainerPATCH 20/131 m68k: mm: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: m68k: mm: simplify detection of memory zone boundaries
free_area_init() only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/m68k/mm/motorola.c | 11 +++++------ arch/m68k/mm/sun3mmu.c | 10 +++------- 2 files changed, 8 insertions(+), 13 deletions(-)
arch/m68k/mm/motorola.c renamed from arch/m68k/mm/motorola.c~m68k-mm-simplify-detection-of-memory-zone-boundaries +5 -6
@@ -365,7 +365,7 @@ static void __init map_node(int node)*/ void __init paging_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , };unsigned long min_addr, max_addr;unsigned long addr;int i;
@@ -448,11 +448,10 @@ void __init paging_init(void)#ifdef DEBUG printk ("before free_area_init\n" );#endif -for (i =0 ; i < m68k_num_memory; i++) { - zones_size[ZONE_DMA ] = m68k_memory[i].size >>PAGE_SHIFT ; -free_area_init_node (i, zones_size, - m68k_memory[i].addr >>PAGE_SHIFT ,NULL ); +for (i =0 ; i < m68k_num_memory; i++)if (node_present_pages (i))node_set_state (i,N_NORMAL_MEMORY ); - } + + max_zone_pfn[ZONE_DMA ] =memblock_end_of_DRAM (); +free_area_init (max_zone_pfn); }
arch/m68k/mm/sun3mmu.c renamed from arch/m68k/mm/sun3mmu.c~m68k-mm-simplify-detection-of-memory-zone-boundaries +3 -6
@@ -42,7 +42,7 @@ void __init paging_init(void)unsigned long address;unsigned long next_pgtable;unsigned long bootmem_end; -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , };unsigned long size; empty_zero_page =memblock_alloc (PAGE_SIZE ,PAGE_SIZE );
@@ -89,14 +89,10 @@ void __init paging_init(void) current->mm =NULL ;/* memory sizing is a hack stolen from motorola.c.. hope it works for us */ - zones_size[ZONE_DMA ] = ((unsigned long )high_memory -PAGE_OFFSET ) >>PAGE_SHIFT ; + max_zone_pfn[ZONE_DMA ] = ((unsigned long )high_memory) >>PAGE_SHIFT ;/* I really wish I knew why the following change made things better... -- Sam */ -/* free_area_init(zones_size); */ -free_area_init_node (0 , zones_size, - (__pa (PAGE_OFFSET ) >>PAGE_SHIFT ) +1 ,NULL ); +free_area_init (max_zone_pfn); } -
Andrew Morton
maintainerPATCH 21/131 parisc: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: parisc: simplify detection of memory zone boundaries
free_area_init() only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/parisc/mm/init.c | 22 +++------------------- 1 file changed, 3 insertions(+), 19 deletions(-)
arch/parisc/mm/init.c renamed from arch/parisc/mm/init.c~parisc-simplify-detection-of-memory-zone-boundaries +2 -17
@@ -675,27 +675,11 @@ static void __init gateway_init(void)static void __init parisc_bootmem_free (void ) { -unsigned long zones_size[MAX_NR_ZONES ] = {0 , }; -unsigned long holes_size[MAX_NR_ZONES ] = {0 , }; -unsigned long mem_start_pfn = ~0UL , mem_end_pfn =0 , mem_size_pfn =0 ; -int i; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 , }; -for (i =0 ; i < npmem_ranges; i++) { -unsigned long start = pmem_ranges[i].start_pfn; -unsigned long size = pmem_ranges[i].pages; -unsigned long end = start + size; + max_zone_pfn[0 ] =memblock_end_of_DRAM (); -if (mem_start_pfn > start) - mem_start_pfn = start; -if (mem_end_pfn < end) - mem_end_pfn = end; - mem_size_pfn += size; - } - - zones_size[0 ] = mem_end_pfn - mem_start_pfn; - holes_size[0 ] = zones_size[0 ] - mem_size_pfn;
Andrew Morton
maintainerPATCH 22/131 sparc32: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: sparc32: simplify detection of memory zone boundaries
free_area_init() only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/sparc/mm/srmmu.c | 21 +++++---------------- 1 file changed, 5 insertions(+), 16 deletions(-)
arch/sparc/mm/srmmu.c renamed from arch/sparc/mm/srmmu.c~sparc32-simplify-detection-of-memory-zone-boundaries +4 -14
@@ -1008,24 +1008,13 @@ void __init srmmu_paging_init(void)kmap_init (); { -unsigned long zones_size[MAX_NR_ZONES ]; -unsigned long zholes_size[MAX_NR_ZONES ]; -unsigned long npages; -int znum; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 }; -for (znum =0 ; znum <MAX_NR_ZONES ; znum++) - zones_size[znum] = zholes_size[znum] =0 ; + max_zone_pfn[ZONE_DMA ] = max_low_pfn; + max_zone_pfn[ZONE_NORMAL ] = max_low_pfn; + max_zone_pfn[ZONE_HIGHMEM ] = highend_pfn; - npages = max_low_pfn - pfn_base; - - zones_size[ZONE_DMA ] = npages; - zholes_size[ZONE_DMA ] = npages - pages_avail; - - npages = highend_pfn - max_low_pfn; - zones_size[ZONE_HIGHMEM ] = npages; - zholes_size[ZONE_HIGHMEM ] = npages -calc_highpages ();
Andrew Morton
maintainerPATCH 23/131 unicore32: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: unicore32: simplify detection of memory zone boundaries
free_area_init() only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/unicore32/include/asm/memory.h | 2 - arch/unicore32/include/mach/memory.h | 6 +-- arch/unicore32/kernel/pci.c | 14 +------- arch/unicore32/mm/init.c | 43 +++++-------------------- 4 files changed, 15 insertions(+), 50 deletions(-)
arch/unicore32/include/asm/memory.h renamed from arch/unicore32/include/asm/memory.h~unicore32-simplify-detection-of-memory-zone-boundaries +1 -1
@@ -60,7 +60,7 @@#ifndef __ASSEMBLY__#ifndef arch_adjust_zones -#define arch_adjust_zones (size, holes) do { } while (0) +#define arch_adjust_zones (max_zone_pfn) do { } while (0)#endif /*
arch/unicore32/include/mach/memory.h renamed from arch/unicore32/include/mach/memory.h~unicore32-simplify-detection-of-memory-zone-boundaries +3 -3
@@ -25,10 +25,10 @@#if !defined(__ASSEMBLY__) && defined(CONFIG_PCI ) -void puv3_pci_adjust_zones (unsigned long *size,unsigned long *holes); +void puv3_pci_adjust_zones (unsigned long *max_zone_pfn); -#define arch_adjust_zones (size, holes) \ - puv3_pci_adjust_zones(size, holes) +#define arch_adjust_zones (max_zone_pfn) \ + puv3_pci_adjust_zones(max_zone_pfn)#endif
arch/unicore32/kernel/pci.c renamed from arch/unicore32/kernel/pci.c~unicore32-simplify-detection-of-memory-zone-boundaries +2 -12
@@ -133,21 +133,11 @@ static int pci_puv3_map_irq(const struct* This is really ugly and we need a better way of specifying * DMA-capable regions of memory. */ -void __init puv3_pci_adjust_zones (unsigned long *zone_size, -unsigned long *zhole_size) +void __init puv3_pci_adjust_zones (unsigned long max_zone_pfn) {unsigned int sz =SZ_128M >>PAGE_SHIFT ; -/* -* Only adjust if > 128M on current system -*/ -if (zone_size[0 ] <= sz) -return ; - - zone_size[1 ] = zone_size[0 ] - sz; - zone_size[0 ] = sz; - zhole_size[1 ] = zhole_size[0 ]; - zhole_size[0 ] =0 ; + max_zone_pfn[ZONE_DMA ] = sz; }/*
arch/unicore32/mm/init.c renamed from arch/unicore32/mm/init.c~unicore32-simplify-detection-of-memory-zone-boundaries +9 -34
66 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 24/131 xtensa: simplify detection of memory zone boundaries
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: xtensa: simplify detection of memory zone boundaries
free_area_init() only requires the definition of maximal PFN for each of the supported zone rater than calculation of actual zone sizes and the sizes of the holes between the zones.
After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP the free_area_init() is available to all architectures.
Using this function instead of free_area_init_node() simplifies the zone detection.
---
arch/xtensa/mm/init.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/xtensa/mm/init.c renamed from arch/xtensa/mm/init.c~xtensa-simplify-detection-of-memory-zone-boundaries +4 -4
@@ -70,13 +70,13 @@ void __init bootmem_init(void)void __init zones_init (void ) {/* All pages are DMA-able, so we put them all in the DMA zone. */ -unsigned long zones_size[MAX_NR_ZONES ] = { - [ZONE_NORMAL ] = max_low_pfn -ARCH_PFN_OFFSET , +unsigned long max_zone_pfn[MAX_NR_ZONES ] = { + [ZONE_NORMAL ] = max_low_pfn, #ifdef CONFIG_HIGHMEM - [ZONE_HIGHMEM ] = max_pfn - max_low_pfn, + [ZONE_HIGHMEM ] = max_pfn, #endif }; -free_area_init_node (0 , zones_size,ARCH_PFN_OFFSET ,NULL ); +free_area_init (max_zone_pfn); }#ifdef CONFIG_HIGHMEM
_
Andrew Morton
maintainerPATCH 25/131 mm: memmap_init: iterate over memblock regions rather that check each PFN
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: mm: memmap_init: iterate over memblock regions rather that check each PFN
When called during boot the memmap_init_zone() function checks if each PFN is valid and actually belongs to the node being initialized using early_pfn_valid() and early_pfn_in_nid().
Each such check may cost up to O(log(n)) where n is the number of memory banks, so for large amount of memory overall time spent in early_pfn*() becomes substantial.
Since the information is anyway present in memblock, we can iterate over memblock memory regions in memmap_init() and only call memmap_init_zone() for PFN ranges that are know to be valid and in the appropriate node.
---
mm/compaction.c | 4 +++- mm/page_alloc.c | 43 ++++++++++++++++--------------------------- 2 files changed, 19 insertions(+), 28 deletions(-)
mm/compaction.c renamed from mm/compaction.c~mm-memmap_init-iterate-over-memblock-regions-rather-that-check-each-pfn +3 -1
@@ -1409,7 +1409,9 @@ fast_isolate_freepages(struct compact_co cc->free_pfn = highest; }else {if (cc->direct_compaction &&pfn_valid (min_pfn)) { - page =pfn_to_page (min_pfn); + page =pageblock_pfn_to_page (min_pfn, +pageblock_end_pfn (min_pfn), + cc->zone); cc->free_pfn = min_pfn; } }
mm/page_alloc.c renamed from mm/page_alloc.c~mm-memmap_init-iterate-over-memblock-regions-rather-that-check-each-pfn +16 -27
62 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 26/131 mm: remove early_pfn_in_nid() and CONFIG_NODES_SPAN_OTHER_NODES
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: remove early_pfn_in_nid() and CONFIG_NODES_SPAN_OTHER_NODES
The memmap_init() function was made to iterate over memblock regions and as the result the early_pfn_in_nid() function became obsolete. Since CONFIG_NODES_SPAN_OTHER_NODES is only used to pick a stub or a real implementation of early_pfn_in_nid(), it is also not needed anymore.
Remove both early_pfn_in_nid() and the CONFIG_NODES_SPAN_OTHER_NODES.
---
arch/powerpc/Kconfig | 9 --------- arch/sparc/Kconfig | 9 --------- arch/x86/Kconfig | 9 --------- mm/page_alloc.c | 20 -------------------- 4 files changed, 47 deletions(-)
arch/powerpc/Kconfig renamed from arch/powerpc/Kconfig~mm-remove-early_pfn_in_nid-and-config_nodes_span_other_nodes +0 -9
@@ -686,15 +686,6 @@ config ARCH_MEMORY_PROBE def_bool y depends on MEMORY_HOTPLUG -# Some NUMA nodes have memory ranges that span -# other nodes. Even though a pfn is valid and -# between a node's start and end pfns, it may not -# reside on that node. See memmap_init_zone() -# for details. -config NODES_SPAN_OTHER_NODES - def_bool y - depends on NEED_MULTIPLE_NODES - config STDBINUTILS bool "Using standard binutils settings" depends on 44x
arch/sparc/Kconfig renamed from arch/sparc/Kconfig~mm-remove-early_pfn_in_nid-and-config_nodes_span_other_nodes +0 -9
@@ -286,15 +286,6 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. -# Some NUMA nodes have memory ranges that span -# other nodes. Even though a pfn is valid and -# between a node's start and end pfns, it may not -# reside on that node. See memmap_init_zone() -# for details. -config NODES_SPAN_OTHER_NODES - def_bool y - depends on NEED_MULTIPLE_NODES - config ARCH_SPARSEMEM_ENABLE def_bool y if SPARC64 select SPARSEMEM_VMEMMAP_ENABLE
arch/x86/Kconfig renamed from arch/x86/Kconfig~mm-remove-early_pfn_in_nid-and-config_nodes_span_other_nodes +0 -9
@@ -1583,15 +1583,6 @@ config X86_64_ACPI_NUMA ---help--- Enable ACPI SRAT based node topology detection. -# Some NUMA nodes have memory ranges that span -# other nodes. Even though a pfn is valid and -# between a node's start and end pfns, it may not -# reside on that node. See memmap_init_zone() -# for details. -config NODES_SPAN_OTHER_NODES - def_bool y - depends on X86_64_ACPI_NUMA - config NUMA_EMU bool "NUMA emulation" depends on NUMA
mm/page_alloc.c renamed from mm/page_alloc.c~mm-remove-early_pfn_in_nid-and-config_nodes_span_other_nodes +0 -20
@@ -1541,26 +1541,6 @@ int __meminit early_pfn_to_nid(unsigned }#endif /* CONFIG_NEED_MULTIPLE_NODES */ -#ifdef CONFIG_NODES_SPAN_OTHER_NODES -/* Only safe to use early in boot when initialisation is single-threaded */ -static inline bool __meminit early_pfn_in_nid (unsigned long pfn,int node) -{ -int nid; - - nid =__early_pfn_to_nid (pfn, &early_pfnnid_cache); -if (nid >=0 && nid != node) -return false; -return true; -} - -#else -static inline bool __meminit early_pfn_in_nid (unsigned long pfn,int node) -{ -return true; -} -#endif - -void __init memblock_free_pages (struct page *page,unsigned long pfn,unsigned int order) {
_
Andrew Morton
maintainerPATCH 27/131 mm: free_area_init: allow defining max_zone_pfn in descending order
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: free_area_init: allow defining max_zone_pfn in descending order
Some architectures (e.g. ARC) have the ZONE_HIGHMEM zone below the ZONE_NORMAL. Allowing free_area_init() parse max_zone_pfn array even it is sorted in descending order allows using free_area_init() on such architectures.
Add top -> down traversal of max_zone_pfn array in free_area_init() and use the latter in ARC node/zone initialization.
---
arch/arc/mm/init.c | 41 ++++++++++++----------------------------- include/linux/mm.h | 1 + mm/page_alloc.c | 26 +++++++++++++++++++++----- 3 files changed, 34 insertions(+), 34 deletions(-)
arch/arc/mm/init.c renamed from arch/arc/mm/init.c~mm-free_area_init-allow-defining-max_zone_pfn-in-descending-order +12 -29
@@ -63,11 +63,13 @@ void __init early_init_dt_add_memory_arc low_mem_sz = size; in_use =1 ; +memblock_add_node (base, size,0 ); }else {#ifdef CONFIG_HIGHMEM high_mem_start = base; high_mem_sz = size; in_use =1 ; +memblock_add_node (base, size,1 );#endif }
@@ -75,6 +77,11 @@ void __init early_init_dt_add_memory_arc base, TO_MB(size), !in_use ? "Not used":""); } +bool arch_has_descending_max_zone_pfns (void ) +{ +return !IS_ENABLED (CONFIG_ARC_HAS_PAE40 ); +} +/* * First memory setup routine called from setup_arch() * 1. setup swapper's mm @init_mm
@@ -83,8 +90,7 @@ void __init early_init_dt_add_memory_arc*/ void __init setup_arch_memory (void ) { -unsigned long zones_size[MAX_NR_ZONES ]; -unsigned long zones_holes[MAX_NR_ZONES ]; +unsigned long max_zone_pfn[MAX_NR_ZONES ] = {0 }; init_mm.start_code = (unsigned long )_text; init_mm.end_code = (unsigned long )_etext;
@@ -115,7 +121,6 @@ void __init setup_arch_memory(void)* the crash */ -memblock_add_node (low_mem_start, low_mem_sz,0 );memblock_reserve (CONFIG_LINUX_LINK_BASE ,__pa (_end) -CONFIG_LINUX_LINK_BASE );
@@ -133,22 +138,7 @@ void __init setup_arch_memory(void)memblock_dump_all ();/*----------------- node/zones setup --------------------------*/ -memset (zones_size,0 ,sizeof (zones_size)); -memset (zones_holes,0 ,sizeof (zones_holes)); - - zones_size[ZONE_NORMAL ] = max_low_pfn - min_low_pfn; - zones_holes[ZONE_NORMAL ] =0 ; - -/* -* We can't use the helper free_area_init(zones[]) because it uses -* PAGE_OFFSET to compute the @min_low_pfn which would be wrong -* when our kernel doesn't start at PAGE_OFFSET, i.e. -* PAGE_OFFSET != CONFIG_LINUX_RAM_BASE -*/ -free_area_init_node (0 ,/* node-id */ - zones_size,/* num pages per zone */ - min_low_pfn,/* first pfn of node */ - zones_holes);/* holes */ + max_zone_pfn[ZONE_NORMAL ] = max_low_pfn;#ifdef CONFIG_HIGHMEM /*
@@ -168,20 +158,13 @@ void __init setup_arch_memory(void) min_high_pfn =PFN_DOWN (high_mem_start); max_high_pfn =PFN_DOWN (high_mem_start + high_mem_sz); - zones_size[ZONE_NORMAL ] =0 ; - zones_holes[ZONE_NORMAL ] =0 ; - - zones_size[ZONE_HIGHMEM ] = max_high_pfn - min_high_pfn; - zones_holes[ZONE_HIGHMEM ] =0 ; - -free_area_init_node (1 ,/* node-id */ - zones_size,/* num pages per zone */ - min_high_pfn,/* first pfn of node */ - zones_holes);/* holes */ + max_zone_pfn[ZONE_HIGHMEM ] = max_high_pfn; high_memory = (void *)(min_high_pfn <<PAGE_SHIFT );kmap_init ();#endif + +free_area_init (max_zone_pfn); }/*
include/linux/mm.h renamed from include/linux/mm.h~mm-free_area_init-allow-defining-max_zone_pfn-in-descending-order +1 -0
@@ -2473,6 +2473,7 @@ extern void setup_per_cpu_pageset(void);extern int min_free_kbytes;extern int watermark_boost_factor;extern int watermark_scale_factor; +extern bool arch_has_descending_max_zone_pfns (void );/* nommu.c */ extern atomic_long_t mmap_pages_allocated;
mm/page_alloc.c renamed from mm/page_alloc.c~mm-free_area_init-allow-defining-max_zone_pfn-in-descending-order +21 -5
@@ -7408,6 +7408,15 @@ static void check_for_memory(pg_data_t * } } +/* +* Some architecturs, e.g. ARC may have ZONE_HIGHMEM below ZONE_NORMAL. For +* such cases we allow max_zone_pfn sorted in the descending order +*/ +bool __weak arch_has_descending_max_zone_pfns (void ) +{ +return false; +} +/** * free_area_init - Initialise all pg_data_t and zone data * @max_zone_pfn: an array of max PFNs for each zone
@@ -7424,7 +7433,8 @@ static void check_for_memory(pg_data_t *void __init free_area_init (unsigned long *max_zone_pfn) {unsigned long start_pfn, end_pfn; -int i, nid; +int i, nid, zone; +bool descending;/* Record where the zone boundaries are */ memset(arch_zone_lowest_possible_pfn, 0,
@@ -7433,14 +7443,20 @@ void __init free_area_init(unsigned long sizeof(arch_zone_highest_possible_pfn)); start_pfn =find_min_pfn_with_active_regions (); + descending =arch_has_descending_max_zone_pfns ();for (i =0 ; i <MAX_NR_ZONES ; i++) { -if (i ==ZONE_MOVABLE ) +if (descending) + zone =MAX_NR_ZONES - i -1 ; +else + zone = i; + +if (zone ==ZONE_MOVABLE )continue ; - end_pfn =max (max_zone_pfn[i], start_pfn); - arch_zone_lowest_possible_pfn[i] = start_pfn; - arch_zone_highest_possible_pfn[i] = end_pfn; + end_pfn =max (max_zone_pfn[zone], start_pfn); + arch_zone_lowest_possible_pfn[zone] = start_pfn; + arch_zone_highest_possible_pfn[zone] = end_pfn; start_pfn = end_pfn; }
_
Andrew Morton
maintainerPATCH 28/131 mm: rename free_area_init_node() to free_area_init_memoryless_node()
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: rename free_area_init_node() to free_area_init_memoryless_node()
free_area_init_node() is only used by x86 to initialize a memory-less nodes. Make its name reflect this and drop all the function parameters except node ID as they are anyway zero.
---
arch/x86/mm/numa.c | 5 +---- include/linux/mm.h | 9 +++------ mm/page_alloc.c | 7 ++----- 3 files changed, 6 insertions(+), 15 deletions(-)
arch/x86/mm/numa.c renamed from arch/x86/mm/numa.c~mm-rename-free_area_init_node-to-free_area_init_memoryless_node +1 -4
@@ -737,12 +737,9 @@ void __init x86_numa_init(void)static void __init init_memory_less_node (int nid) { -unsigned long zones_size[MAX_NR_ZONES ] = {0 }; -unsigned long zholes_size[MAX_NR_ZONES ] = {0 }; -/* Allocate and initialize node data. Memory-less node is now online.*/ alloc_node_data (nid); -free_area_init_node (nid, zones_size,0 , zholes_size); +free_area_init_memoryless_node (nid);/* * All zonelists will be built later in start_kernel() after per cpu
include/linux/mm.h renamed from include/linux/mm.h~mm-rename-free_area_init_node-to-free_area_init_memoryless_node +3 -6
@@ -2329,8 +2329,7 @@ static inline spinlock_t *pud_lock(struc }extern void __init pagecache_init (void ); -extern void __init free_area_init_node (int nid,unsigned long * zones_size, -unsigned long zone_start_pfn,unsigned long *zholes_size); +extern void __init free_area_init_memoryless_node (int nid);extern void free_initmem (void );/*
@@ -2402,10 +2401,8 @@ static inline unsigned long get_num_phys/* * Using memblock node mappings, an architecture may initialise its -* zones, allocate the backing mem_map and account for memory holes in a more -* architecture independent manner. This is a substitute for creating the -* zone_sizes[] and zholes_size[] arrays and passing them to -* free_area_init_node() +* zones, allocate the backing mem_map and account for memory holes in an +* architecture independent manner. * * An architecture is expected to register range of page frames backed by * physical memory with memblock_add[_node]() before calling
mm/page_alloc.c renamed from mm/page_alloc.c~mm-rename-free_area_init_node-to-free_area_init_memoryless_node +2 -5
@@ -6974,12 +6974,9 @@ static void __init __free_area_init_nodefree_area_init_core (pgdat); } -void __init free_area_init_node (int nid,unsigned long *zones_size, -unsigned long node_start_pfn, -unsigned long *zholes_size) +void __init free_area_init_memoryless_node (int nid) { -__free_area_init_node (nid, zones_size, node_start_pfn, zholes_size, - true); +__free_area_init_node (nid,NULL ,0 ,NULL , false); }#if !defined(CONFIG_FLAT_NODE_MEM_MAP )
_
Andrew Morton
maintainerPATCH 29/131 mm: clean up free_area_init_node() and its helpers
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: clean up free_area_init_node() and its helpers
free_area_init_node() now always uses memblock info and the zone PFN limits so it does not need the backwards compatibility functions to calculate the zone spanned and absent pages. The removal of the compat_ versions of zone_{abscent,spanned}_pages_in_node() in turn, makes zone_size and zhole_size parameters unused.
The node_start_pfn is determined by get_pfn_range_for_nid(), so there is no need to pass it to free_area_init_node().
As a result, the only required parameter to free_area_init_node() is the node ID, all the rest are removed along with no longer used compat_zone_{abscent,spanned}_pages_in_node() helpers.
---
mm/page_alloc.c | 104 +++++++++------------------------------------- 1 file changed, 22 insertions(+), 82 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-clean-up-free_area_init_node-and-its-helpers +22 -82
@@ -6436,8 +6436,7 @@ static unsigned long __init zone_spannedunsigned long node_start_pfn, unsigned long node_end_pfn, unsigned long *zone_start_pfn, - unsigned long *zone_end_pfn, - unsigned long *ignored) + unsigned long *zone_end_pfn) { unsigned long zone_low = arch_zone_lowest_possible_pfn[zone_type];unsigned long zone_high = arch_zone_highest_possible_pfn[zone_type];
@@ -6501,8 +6500,7 @@ unsigned long __init absent_pages_in_ranstatic unsigned long __init zone_absent_pages_in_node (int nid,unsigned long zone_type,unsigned long node_start_pfn, -unsigned long node_end_pfn, -unsigned long *ignored) +unsigned long node_end_pfn) {unsigned long zone_low = arch_zone_lowest_possible_pfn[zone_type];unsigned long zone_high = arch_zone_highest_possible_pfn[zone_type];
@@ -6549,43 +6547,9 @@ static unsigned long __init zone_absent_return nr_absent; } -static inline unsigned long __init compat_zone_spanned_pages_in_node (int nid, -unsigned long zone_type, -unsigned long node_start_pfn, -unsigned long node_end_pfn, -unsigned long *zone_start_pfn, -unsigned long *zone_end_pfn, -unsigned long *zones_size) -{ -unsigned int zone; - - *zone_start_pfn = node_start_pfn; -for (zone =0 ; zone < zone_type; zone++) - *zone_start_pfn += zones_size[zone]; - - *zone_end_pfn = *zone_start_pfn + zones_size[zone_type]; - -return zones_size[zone_type]; -} - -static inline unsigned long __init compat_zone_absent_pages_in_node (int nid, -unsigned long zone_type, -unsigned long node_start_pfn, -unsigned long node_end_pfn, -unsigned long *zholes_size) -{ -if (!zholes_size) -return 0 ; - -return zholes_size[zone_type]; -} -static void __init calculate_node_totalpages (struct pglist_data *pgdat,unsigned long node_start_pfn, -unsigned long node_end_pfn, -unsigned long *zones_size, -unsigned long *zholes_size, -bool compat) +unsigned long node_end_pfn) {unsigned long realtotalpages =0 , totalpages =0 ;enum zone_type i;
@@ -6596,31 +6560,14 @@ static void __init calculate_node_totalpunsigned long spanned, absent;unsigned long size, real_size; -if (compat) { - spanned =compat_zone_spanned_pages_in_node ( - pgdat->node_id, i, - node_start_pfn, - node_end_pfn, - &zone_start_pfn, - &zone_end_pfn, - zones_size); - absent =compat_zone_absent_pages_in_node ( - pgdat->node_id, i, - node_start_pfn, - node_end_pfn, - zholes_size); - }else { - spanned =zone_spanned_pages_in_node (pgdat->node_id, i, - node_start_pfn, - node_end_pfn, - &zone_start_pfn, - &zone_end_pfn, - zones_size); - absent =zone_absent_pages_in_node (pgdat->node_id, i, - node_start_pfn, - node_end_pfn, - zholes_size); - } + spanned =zone_spanned_pages_in_node (pgdat->node_id, i, + node_start_pfn, + node_end_pfn, + &zone_start_pfn, + &zone_end_pfn); + absent =zone_absent_pages_in_node (pgdat->node_id, i, + node_start_pfn, + node_end_pfn); size = spanned; real_size = size - absent;
@@ -6942,10 +6889,7 @@ static inline void pgdat_set_deferred_rastatic inline void pgdat_set_deferred_range (pg_data_t *pgdat) {}#endif -static void __init __free_area_init_node (int nid,unsigned long *zones_size, -unsigned long node_start_pfn, -unsigned long *zholes_size, -bool compat) +static void __init free_area_init_node (int nid) {pg_data_t *pgdat =NODE_DATA (nid);unsigned long start_pfn =0 ;
@@ -6954,19 +6898,16 @@ static void __init __free_area_init_node/* pg_data_t should be reset to zero when it's allocated */ WARN_ON (pgdat->nr_zones || pgdat->kswapd_classzone_idx); +get_pfn_range_for_nid (nid, &start_pfn, &end_pfn); + pgdat->node_id = nid; - pgdat->node_start_pfn = node_start_pfn; + pgdat->node_start_pfn = start_pfn; pgdat->per_cpu_nodestats =NULL ; -if (!compat) { -get_pfn_range_for_nid (nid, &start_pfn, &end_pfn); -pr_info ("Initmem setup node %d [mem %#018Lx-%#018Lx]\n" , nid, - (u64 )start_pfn <<PAGE_SHIFT , - end_pfn ? ((u64 )end_pfn <<PAGE_SHIFT ) -1 :0 ); - }else { - start_pfn = node_start_pfn; - } -calculate_node_totalpages (pgdat, start_pfn, end_pfn, - zones_size, zholes_size, compat); + +pr_info ("Initmem setup node %d [mem %#018Lx-%#018Lx]\n" , nid, + (u64 )start_pfn <<PAGE_SHIFT , + end_pfn ? ((u64 )end_pfn <<PAGE_SHIFT ) -1 :0 ); +calculate_node_totalpages (pgdat, start_pfn, end_pfn);alloc_node_mem_map (pgdat);pgdat_set_deferred_range (pgdat);
@@ -6976,7 +6917,7 @@ static void __init __free_area_init_nodevoid __init free_area_init_memoryless_node (int nid) { -__free_area_init_node (nid,NULL ,0 ,NULL , false); +free_area_init_node (nid); }#if !defined(CONFIG_FLAT_NODE_MEM_MAP )
@@ -7506,8 +7447,7 @@ void __init free_area_init(unsigned longinit_unavailable_mem ();for_each_online_node (nid) {pg_data_t *pgdat =NODE_DATA (nid); -__free_area_init_node (nid,NULL , -find_min_pfn_for_node (nid),NULL , false); +free_area_init_node (nid);/* Any memory on that node */ if (pgdat->node_present_pages)
_
Andrew Morton
maintainerPATCH 30/131 mm: simplify find_min_pfn_with_active_regions()
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: mm: simplify find_min_pfn_with_active_regions()
find_min_pfn_with_active_regions() calls find_min_pfn_for_node() with nid parameter set to MAX_NUMNODES. This makes the find_min_pfn_for_node() traverse all memblock memory regions although the first PFN in the system can be easily found with memblock_start_of_DRAM().
Use memblock_start_of_DRAM() in find_min_pfn_with_active_regions() and drop now unused find_min_pfn_for_node().
---
mm/page_alloc.c | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-simplify-find_min_pfn_with_active_regions +0 -17
@@ -7066,24 +7066,6 @@ unsigned long __init node_map_pfn_alignmreturn ~accl_mask +1 ; } -/* Find the lowest pfn for a node */ -static unsigned long __init find_min_pfn_for_node (int nid) -{ -unsigned long min_pfn =ULONG_MAX ; -unsigned long start_pfn; -int i; - -for_each_mem_pfn_range (i, nid, &start_pfn,NULL ,NULL ) - min_pfn =min (min_pfn, start_pfn); - -if (min_pfn ==ULONG_MAX ) { -pr_warn ("Could not find start_pfn for node %d\n" , nid); -return 0 ; - } - -return min_pfn; -}
Andrew Morton
maintainerPATCH 31/131 docs/vm: update memory-models documentation
In reply to Andrew MortonFrom: Mike Rapoport <[email protected]>
Subject: docs/vm: update memory-models documentation
To reflect the updates to free_area_init() family of functions.
---
Documentation/vm/memory-model.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
Documentation/vm/memory-model.rst renamed from Documentation/vm/memory-model.rst~docs-vm-update-memory-models-documentation +4 -5
@@ -46,11 +46,10 @@ maps the entire physical memory. For mos have entries in the `mem_map` array. The `struct page` objects corresponding to the holes are never fully initialized. -To allocate the `mem_map` array, architecture specific setup code -should call :c:func:`free_area_init_node` function or its convenience -wrapper :c:func:`free_area_init`. Yet, the mappings array is not -usable until the call to :c:func:`memblock_free_all` that hands all -the memory to the page allocator. +To allocate the `mem_map` array, architecture specific setup code should +call :c:func:`free_area_init` function. Yet, the mappings array is not +usable until the call to :c:func:`memblock_free_all` that hands all the +memory to the page allocator. If an architecture enables `CONFIG_ARCH_HAS_HOLES_MEMORYMODEL` option, it may free parts of the `mem_map` array that do not cover the
_
Andrew Morton
maintainerPATCH 32/131 mm/page_alloc.c: bad_[reason|flags] is not necessary when PageHWPoison
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm/page_alloc.c: bad_[reason|flags] is not necessary when PageHWPoison
Patch series "mm/page_alloc.c: cleanup on check page", v3.
This patchset does some cleanup related to check page.
1. Remove unnecessary bad_reason assignment
2. Remove bad_flags to bad_page()
3. Rename function for naming convention
4. Extract common part to check page
Thanks for suggestions from David Rientjes and Anshuman Khandual.
This patch (of 5):
Since function returns directly, bad_[reason|flags] is not used any where. And move this to the first.
This is a following cleanup for commit e570f56cccd21 ("mm:check_new_page_bad() directly returns in __PG_HWPOISON case")
---
mm/page_alloc.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-bad_-is-not-necessary-when-pagehwpoison +5 -7
@@ -2097,19 +2097,17 @@ static void check_new_page_bad(struct paconst char *bad_reason =NULL ;unsigned long bad_flags =0 ; +if (unlikely (page->flags & __PG_HWPOISON)) { +/* Don't complain about hwpoisoned pages */ +page_mapcount_reset (page);/* remove PageBuddy */ +return ; + }if (unlikely (atomic_read (&page->_mapcount) !=-1 )) bad_reason ="nonzero mapcount" ;if (unlikely (page->mapping !=NULL )) bad_reason ="non-NULL mapping" ;if (unlikely (page_ref_count (page) !=0 )) bad_reason ="nonzero _refcount" ; -if (unlikely (page->flags & __PG_HWPOISON)) { - bad_reason ="HWPoisoned (hardware-corrupted)" ; - bad_flags = __PG_HWPOISON; -/* Don't complain about hwpoisoned pages */ -page_mapcount_reset (page);/* remove PageBuddy */ -return ; - }if (unlikely (page->flags &PAGE_FLAGS_CHECK_AT_PREP )) { bad_reason ="PAGE_FLAGS_CHECK_AT_PREP flag set" ; bad_flags =PAGE_FLAGS_CHECK_AT_PREP ;
_
Andrew Morton
maintainerPATCH 33/131 mm/page_alloc.c: bad_flags is not necessary for bad_page()
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm/page_alloc.c: bad_flags is not necessary for bad_page()
After commit 5b57b8f22709 ("mm/debug.c: always print flags in dump_page()"), page->flags is always printed for a bad page. It is not necessary to have bad_flags any more.
---
mm/page_alloc.c | 34 ++++++++++------------------------ 1 file changed, 10 insertions(+), 24 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-bad_flags-is-not-necessary-for-bad_page +10 -24
@@ -607,8 +607,7 @@ static inline int __maybe_unused bad_ran }#endif -static void bad_page (struct page *page,const char *reason, -unsigned long bad_flags) +static void bad_page (struct page *page,const char *reason) {static unsigned long resume;static unsigned long nr_shown;
@@ -637,10 +636,6 @@ static void bad_page(struct page *page,pr_alert ("BUG: Bad page state in process %s pfn:%05lx\n" , current->comm,page_to_pfn (page));__dump_page (page, reason); - bad_flags &= page->flags; -if (bad_flags) -pr_alert ("bad because of flags: %#lx(%pGp)\n" , - bad_flags, &bad_flags);dump_page_owner (page);print_modules ();
@@ -1077,11 +1072,7 @@ static inline bool page_expected_state(sstatic void free_pages_check_bad (struct page *page) { -const char *bad_reason; -unsigned long bad_flags; - - bad_reason =NULL ; - bad_flags =0 ; +const char *bad_reason =NULL ;if (unlikely (atomic_read (&page->_mapcount) !=-1 )) bad_reason ="nonzero mapcount" ;
@@ -1089,15 +1080,13 @@ static void free_pages_check_bad(struct bad_reason ="non-NULL mapping" ;if (unlikely (page_ref_count (page) !=0 )) bad_reason ="nonzero _refcount" ; -if (unlikely (page->flags &PAGE_FLAGS_CHECK_AT_FREE )) { +if (unlikely (page->flags &PAGE_FLAGS_CHECK_AT_FREE )) bad_reason ="PAGE_FLAGS_CHECK_AT_FREE flag(s) set" ; - bad_flags =PAGE_FLAGS_CHECK_AT_FREE ; - }#ifdef CONFIG_MEMCG if (unlikely (page->mem_cgroup)) bad_reason ="page still charged to cgroup" ;#endif -bad_page (page, bad_reason, bad_flags); +bad_page (page, bad_reason); }static inline int free_pages_check (struct page *page)
@@ -1128,7 +1117,7 @@ static int free_tail_pages_check(structcase 1 :/* the first tail page: ->mapping may be compound_mapcount() */ if (unlikely (compound_mapcount (page))) { -bad_page (page,"nonzero compound_mapcount" ,0 ); +bad_page (page,"nonzero compound_mapcount" ); goto out; }break ;
@@ -1140,17 +1129,17 @@ static int free_tail_pages_check(structbreak ;default :if (page->mapping !=TAIL_MAPPING ) { -bad_page (page,"corrupted mapping in tail page" ,0 ); +bad_page (page,"corrupted mapping in tail page" ); goto out; }break ; }if (unlikely (!PageTail (page))) { -bad_page (page,"PageTail not set" ,0 ); +bad_page (page,"PageTail not set" ); goto out; }if (unlikely (compound_head (page) != head_page)) { -bad_page (page,"compound_head not consistent" ,0 ); +bad_page (page,"compound_head not consistent" ); goto out; } ret =0 ;
@@ -2095,7 +2084,6 @@ static inline void expand(struct zone *zstatic void check_new_page_bad (struct page *page) {const char *bad_reason =NULL ; -unsigned long bad_flags =0 ;if (unlikely (page->flags & __PG_HWPOISON)) {/* Don't complain about hwpoisoned pages */
@@ -2108,15 +2096,13 @@ static void check_new_page_bad(struct pa bad_reason ="non-NULL mapping" ;if (unlikely (page_ref_count (page) !=0 )) bad_reason ="nonzero _refcount" ; -if (unlikely (page->flags &PAGE_FLAGS_CHECK_AT_PREP )) { +if (unlikely (page->flags &PAGE_FLAGS_CHECK_AT_PREP )) bad_reason ="PAGE_FLAGS_CHECK_AT_PREP flag set" ; - bad_flags =PAGE_FLAGS_CHECK_AT_PREP ; - }#ifdef CONFIG_MEMCG if (unlikely (page->mem_cgroup)) bad_reason ="page still charged to cgroup" ;#endif -bad_page (page, bad_reason, bad_flags); +bad_page (page, bad_reason); }/*
_
Andrew Morton
maintainerPATCH 34/131 mm/page_alloc.c: rename free_pages_check_bad() to check_free_page_bad()
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm/page_alloc.c: rename free_pages_check_bad() to check_free_page_bad()
free_pages_check_bad() is the counterpart of check_new_page_bad(). Rename it to use the same naming convention.
---
mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-rename-free_pages_check_bad-to-check_free_page_bad +2 -2
@@ -1070,7 +1070,7 @@ static inline bool page_expected_state(sreturn true; } -static void free_pages_check_bad (struct page *page) +static void check_free_page_bad (struct page *page) {const char *bad_reason =NULL ;
@@ -1095,7 +1095,7 @@ static inline int free_pages_check(strucreturn 0 ;/* Something has gone sideways, find it */ -free_pages_check_bad (page); +check_free_page_bad (page);return 1 ; }
_
Andrew Morton
maintainerPATCH 35/131 mm/page_alloc.c: rename free_pages_check() to check_free_page()
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm/page_alloc.c: rename free_pages_check() to check_free_page()
free_pages_check() is the counterpart of check_new_page(). Rename it to use the same naming convention.
---
mm/page_alloc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-rename-free_pages_check-to-check_free_page +5 -5
@@ -1089,7 +1089,7 @@ static void check_free_page_bad(struct pbad_page (page, bad_reason); } -static inline int free_pages_check (struct page *page) +static inline int check_free_page (struct page *page) {if (likely (page_expected_state (page,PAGE_FLAGS_CHECK_AT_FREE )))return 0 ;
@@ -1181,7 +1181,7 @@ static __always_inline bool free_pages_pfor (i =1 ; i < (1 << order); i++) {if (compound) bad +=free_tail_pages_check (page, page + i); -if (unlikely (free_pages_check (page + i))) { +if (unlikely (check_free_page (page + i))) { bad++;continue ; }
@@ -1193,7 +1193,7 @@ static __always_inline bool free_pages_pif (memcg_kmem_enabled () &&PageKmemcg (page))__memcg_kmem_uncharge_page (page, order);if (check_free) - bad +=free_pages_check (page); + bad +=check_free_page (page);if (bad)return false;
@@ -1240,7 +1240,7 @@ static bool free_pcp_prepare(struct pagestatic bool bulkfree_pcp_prepare (struct page *page) {if (debug_pagealloc_enabled_static ()) -return free_pages_check (page); +return check_free_page (page);else return false; }
@@ -1261,7 +1261,7 @@ static bool free_pcp_prepare(struct pagestatic bool bulkfree_pcp_prepare (struct page *page) { -return free_pages_check (page); +return check_free_page (page); }#endif /* CONFIG_DEBUG_VM */
_
Andrew Morton
maintainerPATCH 36/131 mm/page_alloc.c: extract check_[new|free]_page_bad() common part to page_bad_reason()
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm/page_alloc.c: extract check_[new|free]_page_bad() common part to page_bad_reason()
We share similar code in check_[new|free]_page_bad() to get the page's bad reason.
Let's extract it and reduce code duplication.
---
mm/page_alloc.c | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-extract-check__page_bad-common-part-to-page_bad_reason +14 -5
@@ -1070,7 +1070,7 @@ static inline bool page_expected_state(sreturn true; } -static void check_free_page_bad (struct page *page) +static const char *page_bad_reason (struct page *page,unsigned long flags) {const char *bad_reason =NULL ;
@@ -1080,13 +1080,23 @@ static void check_free_page_bad(struct p bad_reason ="non-NULL mapping" ;if (unlikely (page_ref_count (page) !=0 )) bad_reason ="nonzero _refcount" ; -if (unlikely (page->flags &PAGE_FLAGS_CHECK_AT_FREE )) - bad_reason ="PAGE_FLAGS_CHECK_AT_FREE flag(s) set" ; +if (unlikely (page->flags & flags)) { +if (flags ==PAGE_FLAGS_CHECK_AT_PREP ) + bad_reason ="PAGE_FLAGS_CHECK_AT_PREP flag(s) set" ; +else + bad_reason ="PAGE_FLAGS_CHECK_AT_FREE flag(s) set" ; + }#ifdef CONFIG_MEMCG if (unlikely (page->mem_cgroup)) bad_reason ="page still charged to cgroup" ;#endif -bad_page (page, bad_reason); +return bad_reason; +} + +static void check_free_page_bad (struct page *page) +{ +bad_page (page, +page_bad_reason (page,PAGE_FLAGS_CHECK_AT_FREE )); }static inline int check_free_page (struct page *page)
@@ -2083,26 +2093,14 @@ static inline void expand(struct zone *zstatic void check_new_page_bad (struct page *page) { -const char *bad_reason =NULL ;
Andrew Morton
maintainerPATCH 37/131 mm,page_alloc,cma: conditionally prefer cma pageblocks for movable allocations
In reply to Andrew MortonFrom: Roman Gushchin <[email protected]>
Subject: mm,page_alloc,cma: conditionally prefer cma pageblocks for movable allocations
Currently a cma area is barely used by the page allocator because it's used only as a fallback from movable, however kswapd tries hard to make sure that the fallback path isn't used.
This results in a system evicting memory and pushing data into swap, while lots of CMA memory is still available. This happens despite the fact that alloc_contig_range is perfectly capable of moving any movable allocations out of the way of an allocation.
To effectively use the cma area let's alter the rules: if the zone has more free cma pages than the half of total free pages in the zone, use cma pageblocks first and fallback to movable blocks in the case of failure.
---
mm/page_alloc.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
mm/page_alloc.c renamed from mm/page_alloc.c~mmpage_alloccma-conditionally-prefer-cma-pageblocks-for-movable-allocations +14 -0
@@ -2752,6 +2752,20 @@ __rmqueue(struct zone *zone, unsigned in {struct page *page; +#ifdef CONFIG_CMA +/* +* Balance movable allocations between regular and CMA areas by +* allocating from CMA when over half of the zone's free memory +* is in the CMA area. +*/ +if (migratetype ==MIGRATE_MOVABLE && +zone_page_state (zone,NR_FREE_CMA_PAGES ) > +zone_page_state (zone,NR_FREE_PAGES ) /2 ) { + page =__rmqueue_cma_fallback (zone, order); +if (page) +return page; + } +#endif retry: page =__rmqueue_smallest (zone, order, migratetype);if (unlikely (!page)) {
_
Andrew Morton
maintainerPATCH 38/131 mm/page_alloc.c: remove unused free_bootmem_with_active_regions
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: mm/page_alloc.c: remove unused free_bootmem_with_active_regions
Since commit 397dc00e249ec64e10 ("mips: sgi-ip27: switch from DISCONTIGMEM to SPARSEMEM"), the last caller of free_bootmem_with_active_regions() was gone. Now no user calls it any more.
Let's remove it.
---
include/linux/mm.h | 4 ---- mm/page_alloc.c | 25 ------------------------- 2 files changed, 29 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-remove-unused-free_bootmem_with_active_regions +0 -4
@@ -2415,8 +2415,6 @@ static inline unsigned long get_num_phys* memblock_add_node(base, size, nid) * free_area_init(max_zone_pfns); * -* free_bootmem_with_active_regions() calls free_bootmem_node() for each -* registered physical page range. Similarly * sparse_memory_present_with_active_regions() calls memory_present() for * each range when SPARSEMEM is enabled. */
@@ -2429,8 +2427,6 @@ extern unsigned long absent_pages_in_ranextern void get_pfn_range_for_nid (unsigned int nid,unsigned long *start_pfn,unsigned long *end_pfn);extern unsigned long find_min_pfn_with_active_regions (void ); -extern void free_bootmem_with_active_regions (int nid, -unsigned long max_low_pfn);extern void sparse_memory_present_with_active_regions (int nid);#ifndef CONFIG_NEED_MULTIPLE_NODES
mm/page_alloc.c renamed from mm/page_alloc.c~mm-remove-unused-free_bootmem_with_active_regions +0 -23
@@ -6296,31 +6296,6 @@ void __meminit init_currently_empty_zone }/** -* free_bootmem_with_active_regions - Call memblock_free_early_nid for each active range -* @nid: The node to free memory on. If MAX_NUMNODES, all nodes are freed. -* @max_low_pfn: The highest PFN that will be passed to memblock_free_early_nid -* -* If an architecture guarantees that all ranges registered contain no holes -* and may be freed, this this function may be used instead of calling -* memblock_free_early_nid() manually. -*/ -void __init free_bootmem_with_active_regions (int nid,unsigned long max_low_pfn) -{ -unsigned long start_pfn, end_pfn; -int i, this_nid; - -for_each_mem_pfn_range (i, nid, &start_pfn, &end_pfn, &this_nid) { - start_pfn =min (start_pfn, max_low_pfn); - end_pfn =min (end_pfn, max_low_pfn); - -if (start_pfn < end_pfn) -memblock_free_early_nid (PFN_PHYS (start_pfn), - (end_pfn - start_pfn) <<PAGE_SHIFT , - this_nid); - } -}
Andrew Morton
maintainerPATCH 39/131 mm/page_alloc.c: only tune sysctl_lowmem_reserve_ratio value once when changing it
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: mm/page_alloc.c: only tune sysctl_lowmem_reserve_ratio value once when changing it
Patch series "improvements about lowmem_reserve and /proc/zoneinfo", v2.
This patch (of 3):
When people write to /proc/sys/vm/lowmem_reserve_ratio to change sysctl_lowmem_reserve_ratio[], setup_per_zone_lowmem_reserve() is called to recalculate all ->lowmem_reserve[] for each zone of all nodes as below:
static void setup_per_zone_lowmem_reserve(void)
{
...
for_each_online_pgdat(pgdat) {
for (j = 0; j < MAX_NR_ZONES; j++) {
...
while (idx) {
...
if (sysctl_lowmem_reserve_ratio[idx] < 1) {
sysctl_lowmem_reserve_ratio[idx] = 0;
lower_zone->lowmem_reserve[j] = 0;
} else {
...
}
}
}
}Meanwhile, here, sysctl_lowmem_reserve_ratio[idx] will be tuned if its value is smaller than '1'. As we know, sysctl_lowmem_reserve_ratio[] is set for zone without regarding to which node it belongs to. That means the tuning will be done on all nodes, even though it has been done in the first node.
And the tuning will be done too even when init_per_zone_wmark_min() calls setup_per_zone_lowmem_reserve(), where actually nobody tries to change sysctl_lowmem_reserve_ratio[].
So now move the tuning into lowmem_reserve_ratio_sysctl_handler(), to make code logic more reasonable.
---
mm/page_alloc.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-only-tune-sysctl_lowmem_reserve_ratio-value-once-when-changing-it +9 -2
@@ -7704,8 +7704,7 @@ static void setup_per_zone_lowmem_reserv idx--; lower_zone = pgdat->node_zones + idx; -if (sysctl_lowmem_reserve_ratio[idx] <1 ) { - sysctl_lowmem_reserve_ratio[idx] =0 ; +if (!sysctl_lowmem_reserve_ratio[idx]) { lower_zone->lowmem_reserve[j] =0 ; }else { lower_zone->lowmem_reserve[j] =
@@ -7970,7 +7969,15 @@ int sysctl_min_slab_ratio_sysctl_handlerint lowmem_reserve_ratio_sysctl_handler (struct ctl_table *table,int write,void __user *buffer,size_t *length,loff_t *ppos) { +int i; +proc_dointvec_minmax (table, write, buffer, length, ppos); + +for (i =0 ; i <MAX_NR_ZONES ; i++) { +if (sysctl_lowmem_reserve_ratio[i] <1 ) + sysctl_lowmem_reserve_ratio[i] =0 ; + } +setup_per_zone_lowmem_reserve ();return 0 ; }
_
Andrew Morton
maintainerPATCH 40/131 mm/page_alloc.c: clear out zone->lowmem_reserve[] if the zone is empty
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: mm/page_alloc.c: clear out zone->lowmem_reserve[] if the zone is empty
When requesting memory allocation from a specific zone is not satisfied, it will fall to lower zone to try allocating memory. In this case, lower zone's ->lowmem_reserve[] will help protect its own memory resource. The higher the relevant ->lowmem_reserve[] is, the harder the upper zone can get memory from this lower zone.
However, this protection mechanism should be applied to populated zone, but not an empty zone. So filling ->lowmem_reserve[] for empty zone is not necessary, and may mislead people that it's valid data in that zone.
Node 2, zone DMA
pages free 0
min 0
low 0
high 0
spanned 0
present 0
managed 0
protection: (0, 0, 1024, 1024)
Node 2, zone DMA32
pages free 0
min 0
low 0
high 0
spanned 0
present 0
managed 0
protection: (0, 0, 1024, 1024)
Node 2, zone Normal
per-node stats
nr_inactive_anon 0
nr_active_anon 143
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_slab_reclaimable 45
nr_slab_unreclaimable 254Here clear out zone->lowmem_reserve[] if zone is empty.
---
mm/page_alloc.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-clear-out-zone-lowmem_reserve-if-the-zone-is-empty +3 -1
@@ -7704,8 +7704,10 @@ static void setup_per_zone_lowmem_reserv idx--; lower_zone = pgdat->node_zones + idx; -if (!sysctl_lowmem_reserve_ratio[idx]) { +if (!sysctl_lowmem_reserve_ratio[idx] || + !zone_managed_pages (lower_zone)) { lower_zone->lowmem_reserve[j] =0 ; +continue ; }else { lower_zone->lowmem_reserve[j] = managed_pages / sysctl_lowmem_reserve_ratio[idx];
_
Andrew Morton
maintainerPATCH 41/131 mm/vmstat.c: do not show lowmem reserve protection information of empty zone
In reply to Andrew MortonFrom: Baoquan He <[email protected]>
Subject: mm/vmstat.c: do not show lowmem reserve protection information of empty zone
Because the lowmem reserve protection of a zone can't tell anything if the zone is empty, except of adding one more line in /proc/zoneinfo.
Let's remove it from that zone's showing.
---
mm/vmstat.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
mm/vmstat.c renamed from mm/vmstat.c~mm-vmstatc-do-not-show-lowmem-reserve-protection-information-of-empty-zone +6 -6
@@ -1592,6 +1592,12 @@ static void zoneinfo_show_print(struct s zone->present_pages, zone_managed_pages(zone)); +/* If unpopulated, no other information is useful */ +if (!populated_zone (zone)) { +seq_putc (m,'\n' ); +return ; + } +seq_printf (m,"\n protection: (%ld" , zone->lowmem_reserve[0 ]);
@@ -1599,12 +1605,6 @@ static void zoneinfo_show_print(struct sseq_printf (m,", %ld" , zone->lowmem_reserve[i]);seq_putc (m,')' ); -/* If unpopulated, no other information is useful */ -if (!populated_zone (zone)) { -seq_putc (m,'\n' ); -return ; - } -for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++)seq_printf (m,"\n %-12s %lu" ,zone_stat_name (i),zone_page_state (zone, i));
_
Andrew Morton
maintainerPATCH 42/131 mm/page_alloc: use ac->high_zoneidx for classzone_idx
In reply to Andrew MortonFrom: Joonsoo Kim <[email protected]>
Subject: mm/page_alloc: use ac->high_zoneidx for classzone_idx
Patch series "integrate classzone_idx and high_zoneidx", v5.
This patchset is followup of the problem reported and discussed two years ago [1, 2]. The problem this patchset solves is related to the classzone_idx on the NUMA system. It causes a problem when the lowmem reserve protection exists for some zones on a node that do not exist on other nodes.
This problem was reported two years ago, and, at that time, the solution got general agreements [2]. But it was not upstreamed.
[1]: http://lkml.kernel.org/r/20180102063528.GG30397@yexl-desktop
[2]: http://lkml.kernel.org/r/[email protected]
This patch (of 2):
Currently, we use classzone_idx to calculate lowmem reserve proetection for an allocation request. This classzone_idx causes a problem on NUMA systems when the lowmem reserve protection exists for some zones on a node that do not exist on other nodes.
Before further explanation, I should first clarify how to compute the classzone_idx and the high_zoneidx.
- ac->high_zoneidx is computed via the arcane gfp_zone(gfp_mask) and represents the index of the highest zone the allocation can use
- classzone_idx was supposed to be the index of the highest zone on the local node that the allocation can use, that is actually available in the system
Think about following example. Node 0 has 4 populated zone, DMA/DMA32/NORMAL/MOVABLE. Node 1 has 1 populated zone, NORMAL. Some zones, such as MOVABLE, doesn't exist on node 1 and this makes following difference.
Assume that there is an allocation request whose gfp_zone(gfp_mask) is the zone, MOVABLE. Then, it's high_zoneidx is 3. If this allocation is initiated on node 0, it's classzone_idx is 3 since actually available/usable zone on local (node 0) is MOVABLE. If this allocation is initiated on node 1, it's classzone_idx is 2 since actually available/usable zone on local (node 1) is NORMAL.
You can see that classzone_idx of the allocation request are different according to their starting node, even if their high_zoneidx is the same.
Think more about these two allocation requests. If they are processed on local, there is no problem. However, if allocation is initiated on node 1 are processed on remote, in this example, at the NORMAL zone on node 0, due to memory shortage, problem occurs. Their different classzone_idx leads to different lowmem reserve and then different min watermark. See the following example.
root@ubuntu:/sys/devices/system/memory# cat /proc/zoneinfo
Node 0, zone DMA
per-node stats
...
pages free 3965
min 5
low 8
high 11
spanned 4095
present 3998
managed 3977
protection: (0, 2961, 4928, 5440)
...
Node 0, zone DMA32
pages free 757955
min 1129
low 1887
high 2645
spanned 1044480
present 782303
managed 758116
protection: (0, 0, 1967, 2479)
...
Node 0, zone Normal
pages free 459806
min 750
low 1253
high 1756
spanned 524288
present 524288
managed 503620
protection: (0, 0, 0, 4096)
...
Node 0, zone Movable
pages free 130759
min 195
low 326
high 457
spanned 1966079
present 131072
managed 131072
protection: (0, 0, 0, 0)
...
Node 1, zone DMA
pages free 0
min 0
low 0
high 0
spanned 0
present 0
managed 0
protection: (0, 0, 1006, 1006)
Node 1, zone DMA32
pages free 0
min 0
low 0
high 0
spanned 0
present 0
managed 0
protection: (0, 0, 1006, 1006)
Node 1, zone Normal
per-node stats
...
pages free 233277
min 383
low 640
high 897
spanned 262144
present 262144
managed 257744
protection: (0, 0, 0, 0)
...
Node 1, zone Movable
pages free 0
min 0
low 0
high 0
spanned 262144
present 0
managed 0
protection: (0, 0, 0, 0)- static min watermark for the NORMAL zone on node 0 is 750.
- lowmem reserve for the request with classzone idx 3 at the NORMAL on node 0 is 4096.
- lowmem reserve for the request with classzone idx 2 at the NORMAL on node 0 is 0.
So, overall min watermark is:
allocation initiated on node 0 (classzone_idx 3): 750 + 4096 = 4846 allocation initiated on node 1 (classzone_idx 2): 750 + 0 = 750
Allocation initiated on node 1 will have some precedence than allocation initiated on node 0 because min watermark of the former allocation is lower than the other. So, allocation initiated on node 1 could succeed on node 0 when allocation initiated on node 0 could not, and, this could cause too many numa_miss allocation. Then, performance could be downgraded.
Recently, there was a regression report about this problem on CMA patches since CMA memory are placed in ZONE_MOVABLE by those patches. I checked that problem is disappeared with this fix that uses high_zoneidx for classzone_idx.
http://lkml.kernel.org/r/20180102063528.GG30397@yexl-desktop
Using high_zoneidx for classzone_idx is more consistent way than previous approach because system's memory layout doesn't affect anything to it. With this patch, both classzone_idx on above example will be 3 so will have the same min watermark.
allocation initiated on node 0: 750 + 4096 = 4846
allocation initiated on node 1: 750 + 4096 = 4846
One could wonder if there is a side effect that allocation initiated on node 1 will use higher bar when allocation is handled on local since classzone_idx could be higher than before. It will not happen because the zone without managed page doesn't contributes lowmem_reserve at all.
---
mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h renamed from mm/internal.h~mm-page_alloc-use-ac-high_zoneidx-for-classzone_idx +1 -1
@@ -144,7 +144,7 @@ struct alloc_context {bool spread_dirty_pages; }; -#define ac_classzone_idx (ac) zonelist_zone_idx(ac->preferred_zoneref) +#define ac_classzone_idx (ac) (ac->high_zoneidx)/* * Locate the struct page for both the matching buddy in our
_
Andrew Morton
maintainerPATCH 43/131 mm/page_alloc: integrate classzone_idx and high_zoneidx
In reply to Andrew MortonFrom: Joonsoo Kim <[email protected]>
Subject: mm/page_alloc: integrate classzone_idx and high_zoneidx
classzone_idx is just different name for high_zoneidx now. So, integrate them and add some comment to struct alloc_context in order to reduce future confusion about the meaning of this variable.
The accessor, ac_classzone_idx() is also removed since it isn't needed after integration.
In addition to integration, this patch also renames high_zoneidx to highest_zoneidx since it represents more precise meaning.
---
include/linux/compaction.h | 9 +- include/linux/mmzone.h | 12 +-- include/trace/events/compaction.h | 22 +++-- include/trace/events/vmscan.h | 14 ++- mm/compaction.c | 64 ++++++++--------- mm/internal.h | 21 ++++- mm/memory_hotplug.c | 6 - mm/oom_kill.c | 4 - mm/page_alloc.c | 60 ++++++++-------- mm/slab.c | 4 - mm/slub.c | 4 - mm/vmscan.c | 105 ++++++++++++++-------------- 12 files changed, 175 insertions(+), 150 deletions(-)
include/linux/compaction.h renamed from include/linux/compaction.h~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +5 -4
@@ -97,7 +97,7 @@ extern enum compact_result try_to_compac struct page **page);extern void reset_isolation_suitable (pg_data_t *pgdat);extern enum compact_result compaction_suitable (struct zone *zone,int order, -unsigned int alloc_flags,int classzone_idx); +unsigned int alloc_flags,int highest_zoneidx);extern void defer_compaction (struct zone *zone,int order);extern bool compaction_deferred (struct zone *zone,int order);
@@ -182,7 +182,7 @@ bool compaction_zonelist_suitable(structextern int kcompactd_run (int nid);extern void kcompactd_stop (int nid); -extern void wakeup_kcompactd (pg_data_t *pgdat,int order,int classzone_idx); +extern void wakeup_kcompactd (pg_data_t *pgdat,int order,int highest_zoneidx);#else static inline void reset_isolation_suitable (pg_data_t *pgdat)
@@ -190,7 +190,7 @@ static inline void reset_isolation_suita }static inline enum compact_result compaction_suitable (struct zone *zone,int order, -int alloc_flags,int classzone_idx) +int alloc_flags,int highest_zoneidx) {return COMPACT_SKIPPED ; }
@@ -232,7 +232,8 @@ static inline void kcompactd_stop(int ni { } -static inline void wakeup_kcompactd (pg_data_t *pgdat,int order,int classzone_idx) +static inline void wakeup_kcompactd (pg_data_t *pgdat, +int order,int highest_zoneidx) { }
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +6 -6
@@ -699,13 +699,13 @@ typedef struct pglist_data {struct task_struct *kswapd;/* Protected by mem_hotplug_begin/end() */ int kswapd_order; -enum zone_type kswapd_classzone_idx; +enum zone_type kswapd_highest_zoneidx;int kswapd_failures;/* Number of 'reclaimed == 0' runs */ #ifdef CONFIG_COMPACTION int kcompactd_max_order; -enum zone_type kcompactd_classzone_idx; +enum zone_type kcompactd_highest_zoneidx;wait_queue_head_t kcompactd_wait;struct task_struct *kcompactd;#endif
@@ -783,15 +783,15 @@ static inline bool pgdat_is_empty(pg_datvoid build_all_zonelists (pg_data_t *pgdat);void wakeup_kswapd (struct zone *zone,gfp_t gfp_mask,int order, -enum zone_type classzone_idx); +enum zone_type highest_zoneidx);bool __zone_watermark_ok (struct zone *z,unsigned int order,unsigned long mark, -int classzone_idx,unsigned int alloc_flags, +int highest_zoneidx,unsigned int alloc_flags,long free_pages);bool zone_watermark_ok (struct zone *z,unsigned int order, -unsigned long mark,int classzone_idx, +unsigned long mark,int highest_zoneidx,unsigned int alloc_flags);bool zone_watermark_ok_safe (struct zone *z,unsigned int order, -unsigned long mark,int classzone_idx); +unsigned long mark,int highest_zoneidx);enum memmap_context {MEMMAP_EARLY ,MEMMAP_HOTPLUG ,
include/trace/events/compaction.h renamed from include/trace/events/compaction.h~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +13 -9
@@ -314,40 +314,44 @@ TRACE_EVENT(mm_compaction_kcompactd_sleeDECLARE_EVENT_CLASS (kcompactd_wake_template, - TP_PROTO(int nid, int order, enum zone_type classzone_idx), + TP_PROTO(int nid, int order, enum zone_type highest_zoneidx), -TP_ARGS (nid, order, classzone_idx), +TP_ARGS (nid, order, highest_zoneidx),TP_STRUCT__entry ( __field(int, nid) __field(int, order) - __field(enum zone_type, classzone_idx) + __field(enum zone_type, highest_zoneidx) ),TP_fast_assign ( __entry->nid = nid; __entry->order = order; - __entry->classzone_idx = classzone_idx; + __entry->highest_zoneidx = highest_zoneidx; ), +/* +* classzone_idx is previous name of the highest_zoneidx. +* Reason not to change it is the ABI requirement of the tracepoint. +*/ TP_printk ("nid=%d order=%d classzone_idx=%-8s" , __entry->nid, __entry->order, -__print_symbolic (__entry->classzone_idx,ZONE_TYPE )) +__print_symbolic (__entry->highest_zoneidx,ZONE_TYPE )) );DEFINE_EVENT (kcompactd_wake_template, mm_compaction_wakeup_kcompactd, - TP_PROTO(int nid, int order, enum zone_type classzone_idx), + TP_PROTO(int nid, int order, enum zone_type highest_zoneidx), -TP_ARGS (nid, order, classzone_idx) +TP_ARGS (nid, order, highest_zoneidx) );DEFINE_EVENT (kcompactd_wake_template, mm_compaction_kcompactd_wake, - TP_PROTO(int nid, int order, enum zone_type classzone_idx), + TP_PROTO(int nid, int order, enum zone_type highest_zoneidx), -TP_ARGS (nid, order, classzone_idx) +TP_ARGS (nid, order, highest_zoneidx) );#endif
include/trace/events/vmscan.h renamed from include/trace/events/vmscan.h~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +9 -5
@@ -265,7 +265,7 @@ TRACE_EVENT(mm_shrink_slab_end, ); TRACE_EVENT(mm_vmscan_lru_isolate, - TP_PROTO(int classzone_idx, + TP_PROTO(int highest_zoneidx, int order, unsigned long nr_requested, unsigned long nr_scanned,
@@ -274,10 +274,10 @@ TRACE_EVENT(mm_vmscan_lru_isolate,isolate_mode_t isolate_mode, int lru), - TP_ARGS(classzone_idx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, isolate_mode, lru), + TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, isolate_mode, lru), TP_STRUCT__entry( - __field(int, classzone_idx) + __field(int, highest_zoneidx) __field(int, order) __field(unsigned long, nr_requested) __field(unsigned long, nr_scanned)
@@ -288,7 +288,7 @@ TRACE_EVENT(mm_vmscan_lru_isolate, ), TP_fast_assign( - __entry->classzone_idx = classzone_idx; + __entry->highest_zoneidx = highest_zoneidx; __entry->order = order; __entry->nr_requested = nr_requested; __entry->nr_scanned = nr_scanned;
@@ -298,9 +298,13 @@ TRACE_EVENT(mm_vmscan_lru_isolate, __entry->lru = lru; ), +/* +* classzone is previous name of the highest_zoneidx. +* Reason not to change it is the ABI requirement of the tracepoint. +*/ TP_printk("isolate_mode=%d classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_skipped=%lu nr_taken=%lu lru=%s", __entry->isolate_mode, - __entry->classzone_idx, + __entry->highest_zoneidx, __entry->order, __entry->nr_requested, __entry->nr_scanned,
mm/compaction.c renamed from mm/compaction.c~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +32 -32
195 lines not on this page · raw message
mm/internal.h renamed from mm/internal.h~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +15 -6
@@ -127,10 +127,10 @@ extern pmd_t *mm_find_pmd(struct mm_stru* between functions involved in allocations, including the alloc_pages* * family of functions. * -* nodemask, migratetype and high_zoneidx are initialized only once in +* nodemask, migratetype and highest_zoneidx are initialized only once in * __alloc_pages_nodemask() and then never change. * -* zonelist, preferred_zone and classzone_idx are set first in +* zonelist, preferred_zone and highest_zoneidx are set first in * __alloc_pages_nodemask() for the fast path, and might be later changed * in __alloc_pages_slowpath(). All other functions pass the whole strucure * by a const pointer.
@@ -140,12 +140,21 @@ struct alloc_context {nodemask_t *nodemask;struct zoneref *preferred_zoneref;int migratetype; -enum zone_type high_zoneidx; + +/* +* highest_zoneidx represents highest usable zone index of +* the allocation request. Due to the nature of the zone, +* memory on lower zone than the highest_zoneidx will be +* protected by lowmem_reserve[highest_zoneidx]. +* +* highest_zoneidx is also used by reclaim/compaction to limit +* the target zone since higher zone than this index cannot be +* usable for this allocation request. +*/ +enum zone_type highest_zoneidx;bool spread_dirty_pages; }; -#define ac_classzone_idx (ac) (ac->high_zoneidx) -/* * Locate the struct page for both the matching buddy in our * pair (buddy1) and the combined O(n+1) page they form (page).
@@ -224,7 +233,7 @@ struct compact_control {int order;/* order a direct compactor needs */ int migratetype;/* migratetype of direct compactor */ const unsigned int alloc_flags;/* alloc flags of a direct compactor */ -const int classzone_idx;/* zone index of a direct compactor */ +const int highest_zoneidx;/* zone index of a direct compactor */ enum migrate_mode mode;/* Async or sync migration mode */ bool ignore_skip_hint;/* Scan blocks even if marked skip */ bool no_set_skip_hint;/* Don't mark blocks for skipping */
mm/memory_hotplug.c renamed from mm/memory_hotplug.c~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +3 -3
@@ -879,13 +879,13 @@ static pg_data_t __ref *hotadd_new_pgdat }else {int cpu;/* -* Reset the nr_zones, order and classzone_idx before reuse. -* Note that kswapd will init kswapd_classzone_idx properly +* Reset the nr_zones, order and highest_zoneidx before reuse. +* Note that kswapd will init kswapd_highest_zoneidx properly * when it starts in the near future. */ pgdat->nr_zones =0 ; pgdat->kswapd_order =0 ; - pgdat->kswapd_classzone_idx =0 ; + pgdat->kswapd_highest_zoneidx =0 ;for_each_online_cpu (cpu) {struct per_cpu_nodestat *p;
mm/oom_kill.c renamed from mm/oom_kill.c~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +2 -2
@@ -254,7 +254,7 @@ static enum oom_constraint constrained_a {struct zone *zone;struct zoneref *z; -enum zone_type high_zoneidx =gfp_zone (oc->gfp_mask); +enum zone_type highest_zoneidx =gfp_zone (oc->gfp_mask);bool cpuset_limited = false;int nid;
@@ -294,7 +294,7 @@ static enum oom_constraint constrained_a/* Check this allocation failure is caused by cpuset's wall function */ for_each_zone_zonelist_nodemask (zone, z, oc->zonelist, - high_zoneidx, oc->nodemask) + highest_zoneidx, oc->nodemask)if (!cpuset_zone_allowed (zone, oc->gfp_mask)) cpuset_limited = true;
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +31 -29
189 lines not on this page · raw message
mm/slab.c renamed from mm/slab.c~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +2 -2
@@ -3106,7 +3106,7 @@ static void *fallback_alloc(struct kmem_struct zonelist *zonelist;struct zoneref *z;struct zone *zone; -enum zone_type high_zoneidx =gfp_zone (flags); +enum zone_type highest_zoneidx =gfp_zone (flags);void *obj =NULL ;struct page *page;int nid;
@@ -3124,7 +3124,7 @@ retry:* Look through allowed nodes for objects available * from existing per node queues. */ -for_each_zone_zonelist (zone, z, zonelist, high_zoneidx) { +for_each_zone_zonelist (zone, z, zonelist, highest_zoneidx) { nid =zone_to_nid (zone); if (cpuset_zone_allowed(zone, flags) &&
mm/slub.c renamed from mm/slub.c~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +2 -2
@@ -1938,7 +1938,7 @@ static void *get_any_partial(struct kmemstruct zonelist *zonelist;struct zoneref *z;struct zone *zone; -enum zone_type high_zoneidx =gfp_zone (flags); +enum zone_type highest_zoneidx =gfp_zone (flags);void *object;unsigned int cpuset_mems_cookie;
@@ -1967,7 +1967,7 @@ static void *get_any_partial(struct kmem do { cpuset_mems_cookie = read_mems_allowed_begin(); zonelist = node_zonelist(mempolicy_slab_node(), flags); - for_each_zone_zonelist(zone, z, zonelist, high_zoneidx) { + for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) { struct kmem_cache_node *n; n = get_node(s, zone_to_nid(zone));
mm/vmscan.c renamed from mm/vmscan.c~mm-page_alloc-integrate-classzone_idx-and-high_zoneidx +55 -50
289 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 44/131 mm/page_alloc.c: use NODE_MASK_NONE in build_zonelists()
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm/page_alloc.c: use NODE_MASK_NONE in build_zonelists()
Slightly simplify the code by initializing user_mask with NODE_MASK_NONE, instead of later calling nodes_clear(). This saves a line of code.
---
mm/page_alloc.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-use-node_mask_none-in-build_zonelists +1 -2
@@ -5692,14 +5692,13 @@ static void build_zonelists(pg_data_t *p {static int node_order[MAX_NUMNODES ];int node, load, nr_nodes =0 ; -nodemask_t used_mask; +nodemask_t used_mask =NODE_MASK_NONE ;int local_node, prev_node;/* NUMA-aware ordering of nodes */ local_node = pgdat->node_id; load = nr_online_nodes; prev_node = local_node; -nodes_clear (used_mask);memset (node_order,0 ,sizeof (node_order));while ((node =find_next_best_node (local_node, &used_mask)) >=0 ) {
_
Andrew Morton
maintainerPATCH 45/131 mm: rename gfpflags_to_migratetype to gfp_migratetype for same convention
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm: rename gfpflags_to_migratetype to gfp_migratetype for same convention
Pageblock migrate type is encoded in GFP flags, just as zone_type and zonelist.
Currently we use gfp_zone() and gfp_zonelist() to extract related information, it would be proper to use the same naming convention for migrate type.
---
include/linux/gfp.h | 2 +- mm/compaction.c | 2 +- mm/page_alloc.c | 4 ++-- mm/page_owner.c | 7 +++---- 4 files changed, 7 insertions(+), 8 deletions(-)
include/linux/gfp.h renamed from include/linux/gfp.h~mm-rename-gfpflags_to_migratetype-to-gfp_migratetype-for-same-convention +1 -1
@@ -312,7 +312,7 @@ struct vm_area_struct;#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)#define GFP_MOVABLE_SHIFT 3 -static inline int gfpflags_to_migratetype (const gfp_t gfp_flags) +static inline int gfp_migratetype (const gfp_t gfp_flags) {VM_WARN_ON ((gfp_flags &GFP_MOVABLE_MASK ) ==GFP_MOVABLE_MASK );BUILD_BUG_ON ((1UL <<GFP_MOVABLE_SHIFT ) != ___GFP_MOVABLE);
mm/compaction.c renamed from mm/compaction.c~mm-rename-gfpflags_to_migratetype-to-gfp_migratetype-for-same-convention +1 -1
@@ -2100,7 +2100,7 @@ compact_zone(struct compact_control *cc,INIT_LIST_HEAD (&cc->freepages);INIT_LIST_HEAD (&cc->migratepages); - cc->migratetype =gfpflags_to_migratetype (cc->gfp_mask); + cc->migratetype =gfp_migratetype (cc->gfp_mask); ret =compaction_suitable (cc->zone, cc->order, cc->alloc_flags, cc->highest_zoneidx);/* Compaction is likely to fail */
mm/page_alloc.c renamed from mm/page_alloc.c~mm-rename-gfpflags_to_migratetype-to-gfp_migratetype-for-same-convention +2 -2
@@ -4285,7 +4285,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) alloc_flags |=ALLOC_HARDER ;#ifdef CONFIG_CMA -if (gfpflags_to_migratetype (gfp_mask) ==MIGRATE_MOVABLE ) +if (gfp_migratetype (gfp_mask) ==MIGRATE_MOVABLE ) alloc_flags |=ALLOC_CMA ;#endif return alloc_flags;
@@ -4735,7 +4735,7 @@ static inline bool prepare_alloc_pages(g ac->highest_zoneidx =gfp_zone (gfp_mask); ac->zonelist =node_zonelist (preferred_nid, gfp_mask); ac->nodemask = nodemask; - ac->migratetype =gfpflags_to_migratetype (gfp_mask); + ac->migratetype =gfp_migratetype (gfp_mask);if (cpusets_enabled ()) { *alloc_mask |= __GFP_HARDWALL;
mm/page_owner.c renamed from mm/page_owner.c~mm-rename-gfpflags_to_migratetype-to-gfp_migratetype-for-same-convention +3 -4
@@ -312,8 +312,7 @@ void pagetypeinfo_showmixedcount_print(scontinue ; page_owner =get_page_owner (page_ext); - page_mt =gfpflags_to_migratetype ( - page_owner->gfp_mask); + page_mt =gfp_migratetype (page_owner->gfp_mask);if (pageblock_mt != page_mt) {if (is_migrate_cma (pageblock_mt)) count[MIGRATE_MOVABLE ]++;
@@ -359,7 +358,7 @@ print_page_owner(char __user *buf, size_/* Print information relevant to grouping pages by mobility */ pageblock_mt =get_pageblock_migratetype (page); - page_mt =gfpflags_to_migratetype (page_owner->gfp_mask); + page_mt =gfp_migratetype (page_owner->gfp_mask); ret += snprintf(kbuf + ret, count - ret, "PFN %lu type %s Block %lu type %s Flags %#lx(%pGp)\n", pfn,
@@ -416,7 +415,7 @@ void __dump_page_owner(struct page *page page_owner =get_page_owner (page_ext); gfp_mask = page_owner->gfp_mask; - mt =gfpflags_to_migratetype (gfp_mask); + mt =gfp_migratetype (gfp_mask);if (!test_bit (PAGE_EXT_OWNER , &page_ext->flags)) {pr_alert ("page_owner info is not present (never set?)\n" );
_
Andrew Morton
maintainerPATCH 46/131 mm/page_alloc.c: reset numa stats for boot pagesets
In reply to Andrew MortonFrom: Sandipan Das <[email protected]>
Subject: mm/page_alloc.c: reset numa stats for boot pagesets
Initially, the per-cpu pagesets of each zone are set to the boot pagesets. The real pagesets are allocated later but before that happens, page allocations do occur and the numa stats for the boot pagesets get incremented since they are common to all zones at that point.
The real pagesets, however, are allocated for the populated zones only. Unpopulated zones, like those associated with memory-less nodes, continue using the boot pageset and end up skewing the numa stats of the corresponding node.
E.g.
$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 0 size: 0 MB
node 0 free: 0 MB
node 1 cpus: 4 5 6 7
node 1 size: 8131 MB
node 1 free: 6980 MB
node distances:
node 0 1
0: 10 40
1: 40 10 $ numastat
node0 node1
numa_hit 108 56495
numa_miss 0 0
numa_foreign 0 0
interleave_hit 0 4537
local_node 108 31547
other_node 0 24948Hence, the boot pageset stats need to be cleared after the real pagesets are allocated.
After this point, the stats of the boot pagesets do not change as page allocations requested for a memory-less node will either fail (if __GFP_THISNODE is used) or get fulfilled by a preferred zone of a different node based on the fallback zonelist.
---
mm/page_alloc.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-reset-numa-stats-for-boot-pagesets +15 -0
@@ -6250,10 +6250,25 @@ void __init setup_per_cpu_pageset(void) {struct pglist_data *pgdat;struct zone *zone; +int __maybe_unused cpu;for_each_populated_zone (zone)setup_zone_pageset (zone); +#ifdef CONFIG_NUMA +/* +* Unpopulated zones continue using the boot pagesets. +* The numa stats for these pagesets need to be reset. +* Otherwise, they will end up skewing the stats of +* the nodes these zones are associated with. +*/ +for_each_possible_cpu (cpu) { +struct per_cpu_pageset *pcp = &per_cpu (boot_pageset, cpu); +memset (pcp->vm_numa_stat_diff,0 , +sizeof (pcp->vm_numa_stat_diff)); + } +#endif +for_each_online_pgdat (pgdat) pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat);
_
Andrew Morton
maintainerPATCH 47/131 mm, page_alloc: reset the zone->watermark_boost early
In reply to Andrew MortonFrom: Charan Teja Reddy <[email protected]>
Subject: mm, page_alloc: reset the zone->watermark_boost early
Updating the zone watermarks by any means, like min_free_kbytes, water_mark_scale_factor etc, when ->watermark_boost is set will result in higher low and high watermarks than the user asked.
Below are the steps to reproduce the problem on system setup of Android kernel running on Snapdragon hardware.
1) Default settings of the system are as below:
#cat /proc/sys/vm/min_free_kbytes = 5162
#cat /proc/zoneinfo | grep -e boost -e low -e "high " -e min -e Node
Node 0, zone Normal min 797 low 8340 high 8539
2) Monitor the zone->watermark_boost(by adding a debug print in the kernel) and whenever it is greater than zero value, write the same value of min_free_kbytes obtained from step 1.
#echo 5162 > /proc/sys/vm/min_free_kbytes
3) Then read the zone watermarks in the system while the ->watermark_boost is zero. This should show the same values of watermarks as step 1 but shown a higher values than asked.
#cat /proc/zoneinfo | grep -e boost -e low -e "high " -e min -e Node
Node 0, zone Normal min 797 low 21148 high 21347
These higher values are because of updating the zone watermarks using the macro min_wmark_pages(zone) which also adds the zone->watermark_boost.
#define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost)
So the steps that lead to the issue are:
1) On the extfrag event, watermarks are boosted by storing the required value in ->watermark_boost.
2) User tries to update the zone watermarks level in the system through min_free_kbytes or watermark_scale_factor.
3) Later, when kswapd woke up, it resets the zone->watermark_boost to zero.
In step 2), we use the min_wmark_pages() macro to store the watermarks in the zone structure thus the values are always offsetted by ->watermark_boost value. This can be avoided by resetting the ->watermark_boost to zero before it is used.
---
mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-reset-the-zone-watermark_boost-early +1 -1
@@ -7788,9 +7788,9 @@ static void __setup_per_zone_wmarks(voidmult_frac (zone_managed_pages (zone), watermark_scale_factor, 10000)); + zone->watermark_boost =0 ; zone->_watermark[WMARK_LOW ] =min_wmark_pages (zone) + tmp; zone->_watermark[WMARK_HIGH ] =min_wmark_pages (zone) + tmp *2 ; - zone->watermark_boost =0 ;spin_unlock_irqrestore (&zone->lock, flags); }
_
Andrew Morton
maintainerPATCH 48/131 mm/page_alloc: restrict and formalize compound_page_dtors[]
In reply to Andrew MortonFrom: Anshuman Khandual <[email protected]>
Subject: mm/page_alloc: restrict and formalize compound_page_dtors[]
Restrict elements in compound_page_dtors[] array per NR_COMPOUND_DTORS and explicitly position them according to enum compound_dtor_id. This improves protection against possible misalignment between compound_page_dtors[] and enum compound_dtor_id later on.
---
include/linux/mm.h | 2 +- mm/page_alloc.c | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-page_alloc-restrict-and-formalize-compound_page_dtors +1 -1
@@ -867,7 +867,7 @@ enum compound_dtor_id {#endif NR_COMPOUND_DTORS, }; -extern compound_page_dtor *const compound_page_dtors[]; +extern compound_page_dtor *const compound_page_dtors[NR_COMPOUND_DTORS ];static inline void set_compound_page_dtor (struct page *page,enum compound_dtor_id compound_dtor)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-restrict-and-formalize-compound_page_dtors +5 -5
@@ -302,14 +302,14 @@ const char * const migratetype_names[MIG#endif }; -compound_page_dtor *const compound_page_dtors[] = { -NULL , - free_compound_page, +compound_page_dtor *const compound_page_dtors[NR_COMPOUND_DTORS ] = { + [NULL_COMPOUND_DTOR ] =NULL , + [COMPOUND_PAGE_DTOR ] = free_compound_page, #ifdef CONFIG_HUGETLB_PAGE - free_huge_page, + [HUGETLB_PAGE_DTOR ] = free_huge_page, #endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE - free_transhuge_page, + [TRANSHUGE_PAGE_DTOR ] = free_transhuge_page, #endif };
_
Andrew Morton
maintainerPATCH 49/131 mm/pagealloc.c: call touch_nmi_watchdog() on max order boundaries in deferred init
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: mm/pagealloc.c: call touch_nmi_watchdog() on max order boundaries in deferred init
Patch series "initialize deferred pages with interrupts enabled", v4.
Keep interrupts enabled during deferred page initialization in order to make code more modular and allow jiffies to update.
Original approach, and discussion can be found here:
http://lkml.kernel.org/r/[email protected]
This patch (of 3):
deferred_init_memmap() disables interrupts the entire time, so it calls touch_nmi_watchdog() periodically to avoid soft lockup splats. Soon it will run with interrupts enabled, at which point cond_resched() should be used instead.
deferred_grow_zone() makes the same watchdog calls through code shared with deferred init but will continue to run with interrupts disabled, so it can't call cond_resched().
Pull the watchdog calls up to these two places to allow the first to be changed later, independently of the second. The frequency reduces from twice per pageblock (init and free) to once per max order block.
---
mm/page_alloc.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-call-touch_nmi_watchdog-on-max-order-boundaries-in-deferred-init +4 -3
@@ -1693,7 +1693,6 @@ static void __init deferred_free_pages(u }else if (!(pfn & nr_pgmask)) {deferred_free_range (pfn - nr_free, nr_free); nr_free =1 ; -touch_nmi_watchdog (); }else { nr_free++; }
@@ -1723,7 +1722,6 @@ static unsigned long __init deferred_incontinue ; }else if (!page || !(pfn & nr_pgmask)) { page =pfn_to_page (pfn); -touch_nmi_watchdog (); }else { page++; }
@@ -1863,8 +1861,10 @@ static int __init deferred_init_memmap(v* that we can avoid introducing any issues with the buddy * allocator. */ -while (spfn < epfn) +while (spfn < epfn) { nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); +touch_nmi_watchdog (); + } zone_empty:pgdat_resize_unlock (pgdat, &flags);
@@ -1948,6 +1948,7 @@ deferred_grow_zone(struct zone *zone, un first_deferred_pfn = spfn; nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); +touch_nmi_watchdog ();/* We should only stop along section boundaries */ if ((first_deferred_pfn ^ spfn) <PAGES_PER_SECTION )
_
Andrew Morton
maintainerPATCH 50/131 mm: initialize deferred pages with interrupts enabled
In reply to Andrew MortonFrom: Pavel Tatashin <[email protected]>
Subject: mm: initialize deferred pages with interrupts enabled
Initializing struct pages is a long task and keeping interrupts disabled for the duration of this operation introduces a number of problems.
1. jiffies are not updated for long period of time, and thus incorrect time is reported. See proposed solution and discussion here:
lkml/[email protected]
2. It prevents farther improving deferred page initialization by allowing intra-node multi-threading.
We are keeping interrupts disabled to solve a rather theoretical problem that was never observed in real world (See 3a2d7fa8a3d5).
Let's keep interrupts enabled. In case we ever encounter a scenario where an interrupt thread wants to allocate large amount of memory this early in boot we can deal with that by growing zone (see deferred_grow_zone()) by the needed amount before starting deferred_init_memmap() threads.
Before:
[ 1.232459] node 0 initialised, 12058412 pages in 1ms
After:
[ 1.632580] node 0 initialised, 12051227 pages in 436ms
---
include/linux/mmzone.h | 2 ++ mm/page_alloc.c | 20 +++++++------------- 2 files changed, 9 insertions(+), 13 deletions(-)
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-initialize-deferred-pages-with-interrupts-enabled +2 -0
@@ -680,6 +680,8 @@ typedef struct pglist_data {/* * Must be held any time you expect node_start_pfn, * node_present_pages, node_spanned_pages or nr_zones to stay constant. +* Also synchronizes pgdat->first_deferred_pfn during deferred page +* init. * * pgdat_resize_lock() and pgdat_resize_unlock() are provided to * manipulate node_size_lock without checking for CONFIG_MEMORY_HOTPLUG
mm/page_alloc.c renamed from mm/page_alloc.c~mm-initialize-deferred-pages-with-interrupts-enabled +7 -11
@@ -1844,6 +1844,13 @@ static int __init deferred_init_memmap(vBUG_ON (pgdat->first_deferred_pfn >pgdat_end_pfn (pgdat)); pgdat->first_deferred_pfn =ULONG_MAX ; +/* +* Once we unlock here, the zone cannot be grown anymore, thus if an +* interrupt thread must allocate this early in boot, zone must be +* pre-grown prior to start of deferred page initialization. +*/ +pgdat_resize_unlock (pgdat, &flags); +/* Only the highest zone is deferred so find it */ for (zid =0 ; zid <MAX_NR_ZONES ; zid++) { zone = pgdat->node_zones + zid;
@@ -1866,8 +1873,6 @@ static int __init deferred_init_memmap(vtouch_nmi_watchdog (); } zone_empty: -pgdat_resize_unlock (pgdat, &flags); -/* Sanity check that the next zone really is unpopulated */ WARN_ON (++zid <MAX_NR_ZONES &&populated_zone (++zone));
@@ -1910,17 +1915,6 @@ deferred_grow_zone(struct zone *zone, unpgdat_resize_lock (pgdat, &flags);/* -* If deferred pages have been initialized while we were waiting for -* the lock, return true, as the zone was grown. The caller will retry -* this zone. We won't return to this function since the caller also -* has this static branch. -*/ -if (!static_branch_unlikely (&deferred_pages)) { -pgdat_resize_unlock (pgdat, &flags); -return true; - }
Andrew Morton
maintainerPATCH 51/131 mm: call cond_resched() from deferred_init_memmap()
In reply to Andrew MortonFrom: Pavel Tatashin <[email protected]>
Subject: mm: call cond_resched() from deferred_init_memmap()
Now that deferred pages are initialized with interrupts enabled we can replace touch_nmi_watchdog() with cond_resched(), as it was before 3a2d7fa8a3d5.
For now, we cannot do the same in deferred_grow_zone() as it is still initializes pages with interrupts disabled.
This change fixes RCU problem described in
https://lkml.kernel.org/r/[email protected]
[ 60.474005] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 60.475000] rcu: 1-...0: (0 ticks this GP) idle=02a/1/0x4000000000000000 softirq=1/1 fqs=15000 [ 60.475000] rcu: (detected by 0, t=60002 jiffies, g=-1199, q=1) [ 60.475000] Sending NMI from CPU 0 to CPUs 1: [ 1.760091] NMI backtrace for cpu 1 [ 1.760091] CPU: 1 PID: 20 Comm: pgdatinit0 Not tainted 4.18.0-147.9.1.el8_1.x86_64 #1 [ 1.760091] Hardware name: Red Hat KVM, BIOS 1.13.0-1.module+el8.2.0+5520+4e5817f3 04/01/2014 [ 1.760091] RIP: 0010:__init_single_page.isra.65+0x10/0x4f [ 1.760091] Code: 48 83 cf 63 48 89 f8 0f 1f 40 00 48 89 c6 48 89 d7 e8 6b 18 80 ff 66 90 5b c3 31 c0 b9 10 00 00 00 49 89 f8 48 c1 e6 33 f3 ab <b8> 07 00 00 00 48 c1 e2 36 41 c7 40 34 01 00 00 00 48 c1 e0 33 41 [ 1.760091] RSP: 0000:ffffba783123be40 EFLAGS: 00000006 [ 1.760091] RAX: 0000000000000000 RBX: fffffad34405e300 RCX: 0000000000000000 [ 1.760091] RDX: 0000000000000000 RSI: 0010000000000000 RDI: fffffad34405e340 [ 1.760091] RBP: 0000000033f3177e R08: fffffad34405e300 R09: 0000000000000002 [ 1.760091] R10: 000000000000002b R11: ffff98afb691a500 R12: 0000000000000002 [ 1.760091] R13: 0000000000000000 R14: 000000003f03ea00 R15: 000000003e10178c [ 1.760091] FS: 0000000000000000(0000) GS:ffff9c9ebeb00000(0000) knlGS:0000000000000000 [ 1.760091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.760091] CR2: 00000000ffffffff CR3: 000000a1cf20a001 CR4: 00000000003606e0 [ 1.760091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1.760091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1.760091] Call Trace: [ 1.760091] deferred_init_pages+0x8f/0xbf [ 1.760091] deferred_init_memmap+0x184/0x29d [ 1.760091] ? deferred_free_pages.isra.97+0xba/0xba [ 1.760091] kthread+0x112/0x130 [ 1.760091] ? kthread_flush_work_fn+0x10/0x10 [ 1.760091] ret_from_fork+0x35/0x40 [ 89.123011] node 0 initialised, 1055935372 pages in 88650ms
---
mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-call-cond_resched-from-deferred_init_memmap +1 -1
@@ -1870,7 +1870,7 @@ static int __init deferred_init_memmap(v*/ while (spfn < epfn) { nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); -touch_nmi_watchdog (); +cond_resched (); } zone_empty:/* Sanity check that the next zone really is unpopulated */
_
Andrew Morton
maintainerPATCH 52/131 padata: remove exit routine
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: padata: remove exit routine
Patch series "padata: parallelize deferred page init", v3.
Deferred struct page init is a bottleneck in kernel boot--the biggest for us and probably others. Optimizing it maximizes availability for large-memory systems and allows spinning up short-lived VMs as needed without having to leave them running. It also benefits bare metal machines hosting VMs that are sensitive to downtime. In projects such as VMM Fast Restart[1], where guest state is preserved across kexec reboot, it helps prevent application and network timeouts in the guests.
So, multithread deferred init to take full advantage of system memory bandwidth.
Extend padata, a framework that handles many parallel singlethreaded jobs, to handle multithreaded jobs as well by adding support for splitting up the work evenly, specifying a minimum amount of work that's appropriate for one helper thread to do, load balancing between helpers, and coordinating them. More documentation in patches 4 and 8.
This series is the first step in a project to address other memory proportional bottlenecks in the kernel such as pmem struct page init, vfio page pinning, hugetlb fallocate, and munmap. Deferred page init doesn't require concurrency limits, resource control, or priority adjustments like these other users will because it happens during boot when the system is otherwise idle and waiting for page init to finish.
This has been run on a variety of x86 systems and speeds up kernel boot by 4% to 49%, saving up to 1.6 out of 4 seconds. Patch 6 has more numbers.
This patch (of 8):
padata_driver_exit() is unnecessary because padata isn't built as a module and doesn't exit.
padata's init routine will soon allocate memory, so getting rid of the exit function now avoids pointless code to free it.
---
kernel/padata.c | 6 ------ 1 file changed, 6 deletions(-)
kernel/padata.c renamed from kernel/padata.c~padata-remove-exit-routine +0 -6
@@ -1074,10 +1074,4 @@ static __init int padata_driver_init(voi }module_init (padata_driver_init); -static __exit void padata_driver_exit(void) -{ -cpuhp_remove_multi_state (CPUHP_PADATA_DEAD ); -cpuhp_remove_multi_state (hp_online); -} -module_exit (padata_driver_exit);#endif
_
Andrew Morton
maintainerPATCH 53/131 padata: initialize earlier
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: padata: initialize earlier
padata will soon initialize the system's struct pages in parallel, so it needs to be ready by page_alloc_init_late().
The error return from padata_driver_init() triggers an initcall warning, so add a warning to padata_init() to avoid silent failure.
---
include/linux/padata.h | 6 ++++++ init/main.c | 2 ++ kernel/padata.c | 17 ++++++++--------- 3 files changed, 16 insertions(+), 9 deletions(-)
include/linux/padata.h renamed from include/linux/padata.h~padata-initialize-earlier +6 -0
@@ -166,6 +166,12 @@ struct padata_instance {#define PADATA_INVALID 4 }; +#ifdef CONFIG_PADATA +extern void __init padata_init (void ); +#else +static inline void __init padata_init (void ) {} +#endif +extern struct padata_instance *padata_alloc_possible (const char *name);extern void padata_free (struct padata_instance *pinst);extern struct padata_shell *padata_alloc_shell (struct padata_instance *pinst);
init/main.c renamed from init/main.c~padata-initialize-earlier +2 -0
@@ -63,6 +63,7 @@#include <linux/debugobjects.h> #include <linux/lockdep.h> #include <linux/kmemleak.h> +#include <linux/padata.h> #include <linux/pid_namespace.h> #include <linux/device/driver.h> #include <linux/kthread.h>
@@ -1482,6 +1483,7 @@ static noinline void __init kernel_init_smp_init ();sched_init_smp (); +padata_init ();page_alloc_init_late ();/* Initialize page ext after all struct pages are initialized. */ page_ext_init ();
kernel/padata.c renamed from kernel/padata.c~padata-initialize-earlier +8 -9
@@ -31,7 +31,6 @@#include <linux/slab.h> #include <linux/sysfs.h> #include <linux/rcupdate.h> -#include <linux/module.h> #define MAX_OBJ_NUM 1000
@@ -1052,26 +1051,26 @@ void padata_free_shell(struct padata_she }EXPORT_SYMBOL (padata_free_shell); -#ifdef CONFIG_HOTPLUG_CPU - -static __init int padata_driver_init (void ) +void __init padata_init (void ) { +#ifdef CONFIG_HOTPLUG_CPU int ret; ret =cpuhp_setup_state_multi (CPUHP_AP_ONLINE_DYN ,"padata:online" , padata_cpu_online,NULL );if (ret <0 ) -return ret; + goto err; hp_online = ret; ret =cpuhp_setup_state_multi (CPUHP_PADATA_DEAD ,"padata:dead" ,NULL , padata_cpu_dead);if (ret <0 ) {cpuhp_remove_multi_state (hp_online); -return ret; + goto err; } -return 0 ; -} -module_init (padata_driver_init); +return ; +err: +pr_warn ("padata: initialization failed\n" );#endif +}
_
Andrew Morton
maintainerPATCH 54/131 padata: allocate work structures for parallel jobs from a pool
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: padata: allocate work structures for parallel jobs from a pool
padata allocates per-CPU, per-instance work structs for parallel jobs. A do_parallel call assigns a job to a sequence number and hashes the number to a CPU, where the job will eventually run using the corresponding work.
This approach fit with how padata used to bind a job to each CPU round-robin, makes less sense after commit bfde23ce200e6 ("padata: unbind parallel jobs from specific CPUs") because a work isn't bound to a particular CPU anymore, and isn't needed at all for multithreaded jobs because they don't have sequence numbers.
Replace the per-CPU works with a preallocated pool, which allows sharing them between existing padata users and the upcoming multithreaded user. The pool will also facilitate setting NUMA-aware concurrency limits with later users.
The pool is sized according to the number of possible CPUs. With this limit, MAX_OBJ_NUM no longer makes sense, so remove it.
If the global pool is exhausted, a parallel job is run in the current task instead to throttle a system trying to do too much in parallel.
---
include/linux/padata.h | 8 -- kernel/padata.c | 118 +++++++++++++++++++++++++-------------- 2 files changed, 78 insertions(+), 48 deletions(-)
include/linux/padata.h renamed from include/linux/padata.h~padata-allocate-work-structures-for-parallel-jobs-from-a-pool +1 -7
@@ -24,7 +24,6 @@* @list: List entry, to attach to the padata lists. * @pd: Pointer to the internal control structure. * @cb_cpu: Callback cpu for serializatioon. -* @cpu: Cpu for parallelization. * @seq_nr: Sequence number of the parallelized data object. * @info: Used to pass information from the parallel to the serial function. * @parallel: Parallel execution function.
@@ -34,7 +33,6 @@ struct padata_priv {struct list_head list;struct parallel_data *pd;int cb_cpu; -int cpu;unsigned int seq_nr;int info;void (*parallel)(struct padata_priv *padata);
@@ -68,15 +66,11 @@ struct padata_serial_queue {/** * struct padata_parallel_queue - The percpu padata parallel queue * -* @parallel: List to wait for parallelization. * @reorder: List to wait for reordering after parallel processing. -* @work: work struct for parallelization. * @num_obj: Number of objects that are processed by this cpu. */ struct padata_parallel_queue { -struct padata_list parallel;struct padata_list reorder; -struct work_struct work;atomic_t num_obj; };
@@ -111,7 +105,7 @@ struct parallel_data {struct padata_parallel_queue __percpu *pqueue;struct padata_serial_queue __percpu *squeue;atomic_t refcnt; -atomic_t seq_nr; +unsigned int seq_nr;unsigned int processed;int cpu;struct padata_cpumask cpumask;
kernel/padata.c renamed from kernel/padata.c~padata-allocate-work-structures-for-parallel-jobs-from-a-pool +77 -41
190 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 55/131 padata: add basic support for multithreaded jobs
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: padata: add basic support for multithreaded jobs
Sometimes the kernel doesn't take full advantage of system memory bandwidth, leading to a single CPU spending excessive time in initialization paths where the data scales with memory size.
Multithreading naturally addresses this problem.
Extend padata, a framework that handles many parallel yet singlethreaded jobs, to also handle multithreaded jobs by adding support for splitting up the work evenly, specifying a minimum amount of work that's appropriate for one helper thread to do, load balancing between helpers, and coordinating them.
This is inspired by work from Pavel Tatashin and Steve Sistare.
---
include/linux/padata.h | 29 +++++++ kernel/padata.c | 152 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 178 insertions(+), 3 deletions(-)
include/linux/padata.h renamed from include/linux/padata.h~padata-add-basic-support-for-multithreaded-jobs +29 -0
@@ -4,6 +4,9 @@* * Copyright (C) 2008, 2009 secunet Security Networks AG * Copyright (C) 2008, 2009 Steffen Klassert <[email protected]> +* +* Copyright (c) 2020 Oracle and/or its affiliates. +* Author: Daniel Jordan <[email protected]> */ #ifndef PADATA_H
@@ -131,6 +134,31 @@ struct padata_shell { };/** +* struct padata_mt_job - represents one multithreaded job +* +* @thread_fn: Called for each chunk of work that a padata thread does. +* @fn_arg: The thread function argument. +* @start: The start of the job (units are job-specific). +* @size: size of this node's work (units are job-specific). +* @align: Ranges passed to the thread function fall on this boundary, with the +* possible exceptions of the beginning and end of the job. +* @min_chunk: The minimum chunk size in job-specific units. This allows +* the client to communicate the minimum amount of work that's +* appropriate for one worker thread to do at once. +* @max_threads: Max threads to use for the job, actual number may be less +* depending on task size and minimum chunk size. +*/ +struct padata_mt_job { +void (*thread_fn)(unsigned long start,unsigned long end,void *arg); +void *fn_arg; +unsigned long start; +unsigned long size; +unsigned long align; +unsigned long min_chunk; +int max_threads; +}; + +/** * struct padata_instance - The overall control structure. * * @cpu_online_node: Linkage for CPU online callback.
@@ -173,6 +201,7 @@ extern void padata_free_shell(struct padextern int padata_do_parallel (struct padata_shell *ps,struct padata_priv *padata,int *cb_cpu);extern void padata_do_serial (struct padata_priv *padata); +extern void __init padata_do_multithreaded (struct padata_mt_job *job);extern int padata_set_cpumask (struct padata_instance *pinst,int cpumask_type,cpumask_var_t cpumask);extern int padata_start (struct padata_instance *pinst);
kernel/padata.c renamed from kernel/padata.c~padata-add-basic-support-for-multithreaded-jobs +149 -3
205 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 56/131 mm: don't track number of pages during deferred initialization
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: mm: don't track number of pages during deferred initialization
Deferred page init used to report the number of pages initialized:
node 0 initialised, 32439114 pages in 97ms
Tracking this makes the code more complicated when using multiple threads. Given that the statistic probably has limited value, especially since a zone grows on demand so that the page count can vary, just remove it.
The boot message now looks like
node 0 deferred pages initialised in 97ms
---
mm/page_alloc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-dont-track-number-of-pages-during-deferred-initialization +4 -4
@@ -1820,7 +1820,7 @@ static int __init deferred_init_memmap(v {pg_data_t *pgdat = data;const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id); -unsigned long spfn =0 , epfn =0 , nr_pages =0 ; +unsigned long spfn =0 , epfn =0 ;unsigned long first_init_pfn, flags;unsigned long start = jiffies;struct zone *zone;
@@ -1869,15 +1869,15 @@ static int __init deferred_init_memmap(v* allocator. */ while (spfn < epfn) { - nr_pages +=deferred_init_maxorder (&i, zone, &spfn, &epfn); +deferred_init_maxorder (&i, zone, &spfn, &epfn);cond_resched (); } zone_empty:/* Sanity check that the next zone really is unpopulated */ WARN_ON (++zid <MAX_NR_ZONES &&populated_zone (++zone)); -pr_info ("node %d initialised, %lu pages in %ums\n" , - pgdat->node_id, nr_pages,jiffies_to_msecs (jiffies - start)); +pr_info ("node %d deferred pages initialised in %ums\n" , + pgdat->node_id,jiffies_to_msecs (jiffies - start));pgdat_init_report_one_done ();return 0 ;
_
Andrew Morton
maintainerPATCH 57/131 mm: parallelize deferred_init_memmap()
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: mm: parallelize deferred_init_memmap()
Deferred struct page init is a significant bottleneck in kernel boot. Optimizing it maximizes availability for large-memory systems and allows spinning up short-lived VMs as needed without having to leave them running. It also benefits bare metal machines hosting VMs that are sensitive to downtime. In projects such as VMM Fast Restart[1], where guest state is preserved across kexec reboot, it helps prevent application and network timeouts in the guests.
Multithread to take full advantage of system memory bandwidth.
The maximum number of threads is capped at the number of CPUs on the node because speedups always improve with additional threads on every system tested, and at this phase of boot, the system is otherwise idle and waiting on page init to finish.
Helper threads operate on section-aligned ranges to both avoid false sharing when setting the pageblock's migrate type and to avoid accessing uninitialized buddy pages, though max order alignment is enough for the latter.
The minimum chunk size is also a section. There was benefit to using multiple threads even on relatively small memory (1G) systems, and this is the smallest size that the alignment allows.
The time (milliseconds) is the slowest node to initialize since boot blocks until all nodes finish. intel_pstate is loaded in active mode without hwp and with turbo enabled, and intel_idle is active as well.
Intel(R) Xeon(R) Platinum 8167M CPU @ 2.00GHz (Skylake, bare metal)
2 nodes * 26 cores * 2 threads = 104 CPUs
384G/node = 768G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 4089.7 ( 8.1) -- 1785.7 ( 7.6)
2% ( 1) 1.7% 4019.3 ( 1.5) 3.8% 1717.7 ( 11.8)
12% ( 6) 34.9% 2662.7 ( 2.9) 79.9% 359.3 ( 0.6)
25% ( 13) 39.9% 2459.0 ( 3.6) 91.2% 157.0 ( 0.0)
37% ( 19) 39.2% 2485.0 ( 29.7) 90.4% 172.0 ( 28.6)
50% ( 26) 39.3% 2482.7 ( 25.7) 90.3% 173.7 ( 30.0)
75% ( 39) 39.0% 2495.7 ( 5.5) 89.4% 190.0 ( 1.0)
100% ( 52) 40.2% 2443.7 ( 3.8) 92.3% 138.0 ( 1.0)
Intel(R) Xeon(R) CPU E5-2699C v4 @ 2.20GHz (Broadwell, kvm guest)
1 node * 16 cores * 2 threads = 32 CPUs
192G/node = 192G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 1988.7 ( 9.6) -- 1096.0 ( 11.5)
3% ( 1) 1.1% 1967.0 ( 17.6) 0.3% 1092.7 ( 11.0)
12% ( 4) 41.1% 1170.3 ( 14.2) 73.8% 287.0 ( 3.6)
25% ( 8) 47.1% 1052.7 ( 21.9) 83.9% 177.0 ( 13.5)
38% ( 12) 48.9% 1016.3 ( 12.1) 86.8% 144.7 ( 1.5)
50% ( 16) 48.9% 1015.7 ( 8.1) 87.8% 134.0 ( 4.4)
75% ( 24) 49.1% 1012.3 ( 3.1) 88.1% 130.3 ( 2.3)
100% ( 32) 49.5% 1004.0 ( 5.3) 88.5% 125.7 ( 2.1)
Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, bare metal)
2 nodes * 18 cores * 2 threads = 72 CPUs
128G/node = 256G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 1680.0 ( 4.6) -- 627.0 ( 4.0)
3% ( 1) 0.3% 1675.7 ( 4.5) -0.2% 628.0 ( 3.6)
11% ( 4) 25.6% 1250.7 ( 2.1) 67.9% 201.0 ( 0.0)
25% ( 9) 30.7% 1164.0 ( 17.3) 81.8% 114.3 ( 17.7)
36% ( 13) 31.4% 1152.7 ( 10.8) 84.0% 100.3 ( 17.9)
50% ( 18) 31.5% 1150.7 ( 9.3) 83.9% 101.0 ( 14.1)
75% ( 27) 31.7% 1148.0 ( 5.6) 84.5% 97.3 ( 6.4)
100% ( 36) 32.0% 1142.3 ( 4.0) 85.6% 90.0 ( 1.0)
AMD EPYC 7551 32-Core Processor (Zen, kvm guest)
1 node * 8 cores * 2 threads = 16 CPUs
64G/node = 64G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 1029.3 ( 25.1) -- 240.7 ( 1.5)
6% ( 1) -0.6% 1036.0 ( 7.8) -2.2% 246.0 ( 0.0)
12% ( 2) 11.8% 907.7 ( 8.6) 44.7% 133.0 ( 1.0)
25% ( 4) 13.9% 886.0 ( 10.6) 62.6% 90.0 ( 6.0)
38% ( 6) 17.8% 845.7 ( 14.2) 69.1% 74.3 ( 3.8)
50% ( 8) 16.8% 856.0 ( 22.1) 72.9% 65.3 ( 5.7)
75% ( 12) 15.4% 871.0 ( 29.2) 79.8% 48.7 ( 7.4)
100% ( 16) 21.0% 813.7 ( 21.0) 80.5% 47.0 ( 5.2)Server-oriented distros that enable deferred page init sometimes run in small VMs, and they still benefit even though the fraction of boot time saved is smaller:
AMD EPYC 7551 32-Core Processor (Zen, kvm guest)
1 node * 2 cores * 2 threads = 4 CPUs
16G/node = 16G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 716.0 ( 14.0) -- 49.7 ( 0.6)
25% ( 1) 1.8% 703.0 ( 5.3) -4.0% 51.7 ( 0.6)
50% ( 2) 1.6% 704.7 ( 1.2) 43.0% 28.3 ( 0.6)
75% ( 3) 2.7% 696.7 ( 13.1) 49.7% 25.0 ( 0.0)
100% ( 4) 4.1% 687.0 ( 10.4) 55.7% 22.0 ( 0.0)
Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz (Haswell, kvm guest)
1 node * 2 cores * 2 threads = 4 CPUs
14G/node = 14G memory
kernel boot deferred init
------------------------ ------------------------
node% (thr) speedup time_ms (stdev) speedup time_ms (stdev)
( 0) -- 787.7 ( 6.4) -- 122.3 ( 0.6)
25% ( 1) 0.2% 786.3 ( 10.8) -2.5% 125.3 ( 2.1)
50% ( 2) 5.9% 741.0 ( 13.9) 37.6% 76.3 ( 19.7)
75% ( 3) 8.3% 722.0 ( 19.0) 49.9% 61.3 ( 3.2)
100% ( 4) 9.3% 714.7 ( 9.5) 56.4% 53.3 ( 1.5)On Josh's 96-CPU and 192G memory system:
Without this patch series:
[ 0.487132] node 0 initialised, 23398907 pages in 292ms
[ 0.499132] node 1 initialised, 24189223 pages in 304ms
...
[ 0.629376] Run /sbin/init as init process
With this patch series:
[ 0.231435] node 1 initialised, 24189223 pages in 32ms
[ 0.236718] node 0 initialised, 23398907 pages in 36ms[1] https://static.sched.com/hosted_files/kvmforum2019/66/VMM-fast-restart_kvmforum2019.pdf
---
mm/Kconfig | 6 +++--- mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 43 insertions(+), 9 deletions(-)
mm/Kconfig renamed from mm/Kconfig~mm-parallelize-deferred_init_memmap +3 -3
@@ -747,13 +747,13 @@ config DEFERRED_STRUCT_PAGE_INIT depends on SPARSEMEM depends on !NEED_PER_CPU_KM depends on 64BIT + select PADATA help Ordinarily all struct pages are initialised during early boot in a single thread. On very large machines this can take a considerable amount of time. If this option is set, large machines will bring up - a subset of memmap at boot and then initialise the rest in parallel - by starting one-off "pgdatinitX" kernel thread for each node X. This - has a potential performance impact on processes running early in the + a subset of memmap at boot and then initialise the rest in parallel. + This has a potential performance impact on tasks running early in the lifetime of the system until these kthreads finish the initialisation.
mm/page_alloc.c renamed from mm/page_alloc.c~mm-parallelize-deferred_init_memmap +40 -6
72 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 58/131 mm: make deferred init's max threads arch-specific
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: mm: make deferred init's max threads arch-specific
Using padata during deferred init has only been tested on x86, so for now limit it to this architecture.
If another arch wants this, it can find the max thread limit that's best for it and override deferred_page_init_max_threads().
---
arch/x86/mm/init_64.c | 12 ++++++++++++ include/linux/memblock.h | 3 +++ mm/page_alloc.c | 13 ++++++++----- 3 files changed, 23 insertions(+), 5 deletions(-)
arch/x86/mm/init_64.c renamed from arch/x86/mm/init_64.c~mm-make-deferred-inits-max-threads-arch-specific +12 -0
@@ -1265,6 +1265,18 @@ void __init mem_init(void)mem_init_print_info (NULL ); } +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +int __init deferred_page_init_max_threads (const struct cpumask *node_cpumask) +{ +/* +* More CPUs always led to greater speedups on tested systems, up to +* all the nodes' CPUs. Use all since the system is otherwise idle +* now. +*/ +return max_t (int,cpumask_weight (node_cpumask),1 ); +} +#endif +int kernel_set_to_readonly;void mark_rodata_ro (void )
include/linux/memblock.h renamed from include/linux/memblock.h~mm-make-deferred-inits-max-threads-arch-specific +3 -0
@@ -273,6 +273,9 @@ void __next_mem_pfn_range_in_zone(u64 *i#define for_each_free_mem_pfn_range_in_zone_from (i, zone, p_start, p_end) \ for (; i != U64_MAX; \ __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end)) + +int __init deferred_page_init_max_threads (const struct cpumask *node_cpumask); +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ /**
mm/page_alloc.c renamed from mm/page_alloc.c~mm-make-deferred-inits-max-threads-arch-specific +8 -5
@@ -1836,6 +1836,13 @@ deferred_init_memmap_chunk(unsigned long } } +/* An arch may override for more concurrency. */ +__weak int __init +deferred_page_init_max_threads (const struct cpumask *node_cpumask) +{ +return 1 ; +} +/* Initialise remaining memory on a node */ static int __init deferred_init_memmap (void *data) {
@@ -1884,11 +1891,7 @@ static int __init deferred_init_memmap(v first_init_pfn)) goto zone_empty; -/* -* More CPUs always led to greater speedups on tested systems, up to -* all the nodes' CPUs. Use all since the system is otherwise idle now. -*/ - max_threads =max (cpumask_weight (cpumask),1u ); + max_threads =deferred_page_init_max_threads (cpumask);while (spfn < epfn) {unsigned long epfn_align =ALIGN (epfn,PAGES_PER_SECTION );
_
Andrew Morton
maintainerPATCH 59/131 padata: document multithreaded jobs
In reply to Andrew MortonFrom: Daniel Jordan <[email protected]>
Subject: padata: document multithreaded jobs
Add Documentation for multithreaded jobs.
---
Documentation/core-api/padata.rst | 41 +++++++++++++++++++++------- 1 file changed, 31 insertions(+), 10 deletions(-)
Documentation/core-api/padata.rst renamed from Documentation/core-api/padata.rst~padata-document-multithreaded-jobs +31 -10
@@ -4,23 +4,26 @@ The padata parallel execution mechanism ======================================= -:Date: December 2019 +:Date: May 2020 Padata is a mechanism by which the kernel can farm jobs out to be done in -parallel on multiple CPUs while retaining their ordering. It was developed for -use with the IPsec code, which needs to be able to perform encryption and -decryption on large numbers of packets without reordering those packets. The -crypto developers made a point of writing padata in a sufficiently general -fashion that it could be put to other uses as well. +parallel on multiple CPUs while optionally retaining their ordering. -Usage -===== +It was originally developed for IPsec, which needs to perform encryption and +decryption on large numbers of packets without reordering those packets. This +is currently the sole consumer of padata's serialized job support. + +Padata also supports multithreaded jobs, splitting up the job evenly while load +balancing and coordinating between threads. + +Running Serialized Jobs +======================= Initializing ------------ -The first step in using padata is to set up a padata_instance structure for -overall control of how jobs are to be run:: +The first step in using padata to run serialized jobs is to set up a +padata_instance structure for overall control of how jobs are to be run:: #include <linux/padata.h>
@@ -162,6 +165,24 @@ functions that correspond to the allocat It is the user's responsibility to ensure all outstanding jobs are complete before any of the above are called. +Running Multithreaded Jobs +========================== + +A multithreaded job has a main thread and zero or more helper threads, with the +main thread participating in the job and then waiting until all helpers have +finished. padata splits the job into units called chunks, where a chunk is a +piece of the job that one thread completes in one call to the thread function. + +A user has to do three things to run a multithreaded job. First, describe the +job by defining a padata_mt_job structure, which is explained in the Interface +section. This includes a pointer to the thread function, which padata will +call each time it assigns a job chunk to a thread. Then, define the thread +function, which accepts three arguments, ``start``, ``end``, and ``arg``, where +the first two delimit the range that the thread operates on and the last is a +pointer to the job's shared state, if any. Prepare the shared state, which is +typically allocated on the main thread's stack. Last, call +padata_do_multithreaded(), which will return once the job is finished. + Interface =========
_
Andrew Morton
maintainerPATCH 60/131 mm/page_alloc.c: add missing newline
In reply to Andrew MortonFrom: Chen Tao <[email protected]>
Subject: mm/page_alloc.c: add missing newline
Add missing line breaks on pr_warn().
---
mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_allocc-add-missing-line-breaks +1 -1
@@ -7182,7 +7182,7 @@ static void __init find_zone_movable_pfn }if (mem_below_4gb_not_mirrored) -pr_warn ("This configuration results in unmirrored kernel memory." ); +pr_warn ("This configuration results in unmirrored kernel memory.\n" ); goto out2; }
_
Andrew Morton
maintainerPATCH 61/131 khugepaged: add self test
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: khugepaged: add self test
Patch series "thp/khugepaged improvements and CoW semantics", v4.
The patchset adds khugepaged selftest (anon-THP only for now), expands cases khugepaged can handle and switches anon-THP copy-on-write handling to 4k.
This patch (of 8):
The test checks if khugepaged is able to recover huge page where we expect to do so. It only covers anon-THP for now.
Currently the test shows few failures. They are going to be addressed by the following patches.
---
tools/testing/selftests/vm/.gitignore | 1 tools/testing/selftests/vm/Makefile | 1 tools/testing/selftests/vm/khugepaged.c | 952 ++++++++++++++++++++++ 3 files changed, 954 insertions(+)
tools/testing/selftests/vm/.gitignore renamed from tools/testing/selftests/vm/.gitignore~khugepaged-add-self-test +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only hugepage-mmap hugepage-shm +khugepaged map_hugetlb map_populate thuge-gen
tools/testing/selftests/vm/khugepaged.c new file +952 -0
952 lines not on this page · raw message
tools/testing/selftests/vm/Makefile renamed from tools/testing/selftests/vm/Makefile~khugepaged-add-self-test +1 -0
@@ -20,6 +20,7 @@ TEST_GEN_FILES += on-fault-limit TEST_GEN_FILES += thuge-gen TEST_GEN_FILES += transhuge-stress TEST_GEN_FILES += userfaultfd +TEST_GEN_FILES += khugepaged ifneq (,$(filter $(MACHINE),arm64 ia64 mips64 parisc64 ppc64 ppc64le riscv64 s390x sh64 sparc64 x86_64)) TEST_GEN_FILES += va_128TBswitch
_
Andrew Morton
maintainerPATCH 62/131 khugepaged: do not stop collapse if less than half PTEs are referenced
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: khugepaged: do not stop collapse if less than half PTEs are referenced
__collapse_huge_page_swapin() checks the number of referenced PTE to decide if the memory range is hot enough to justify swapin.
We have few problems with the approach:
- It is way too late: we can do the check much earlier and safe time. khugepaged_scan_pmd() already knows if we have any pages to swap in and number of referenced page.
- It stops collapse altogether if there's not enough referenced pages, not only swappingin.
Fix it by making the right check early. We also can avoid additional page table scanning if khugepaged_scan_pmd() haven't found any swap entries.
---
mm/khugepaged.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-)
mm/khugepaged.c renamed from mm/khugepaged.c~khugepaged-do-not-stop-collapse-if-less-than-half-ptes-are-referenced +11 -16
@@ -899,11 +899,6 @@ static bool __collapse_huge_page_swapin( .pgoff = linear_page_index(vma, address), }; -/* we only decide to swapin, if there is enough young ptes */ -if (referenced <HPAGE_PMD_NR /2 ) { -trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,0 ); -return false; - } vmf.pte =pte_offset_map (pmd, address);for (; vmf.address < address +HPAGE_PMD_NR *PAGE_SIZE ; vmf.pte++, vmf.address +=PAGE_SIZE ) {
@@ -943,7 +938,7 @@ static bool __collapse_huge_page_swapin(static void collapse_huge_page (struct mm_struct *mm,unsigned long address,struct page **hpage, -int node,int referenced) +int node,int referenced,int unmapped) {pmd_t *pmd, _pmd;pte_t *pte;
@@ -1000,7 +995,8 @@ static void collapse_huge_page(struct mm* If it fails, we release mmap_sem and jump out_nolock. * Continuing to collapse causes inconsistency. */ -if (!__collapse_huge_page_swapin (mm, vma, address, pmd, referenced)) { +if (unmapped && !__collapse_huge_page_swapin (mm, vma, address, + pmd, referenced)) {mem_cgroup_cancel_charge (new_page, memcg, true);up_read (&mm->mmap_sem); goto out_nolock;
@@ -1233,22 +1229,21 @@ static int khugepaged_scan_pmd(struct mm mmu_notifier_test_young(vma->vm_mm, address)) referenced++; } -if (writable) { -if (referenced) { - result =SCAN_SUCCEED ; - ret =1 ; - }else { - result =SCAN_LACK_REFERENCED_PAGE ; - } - }else { +if (!writable) { result =SCAN_PAGE_RO ; + }else if (!referenced || (unmapped && referenced <HPAGE_PMD_NR /2 )) { + result =SCAN_LACK_REFERENCED_PAGE ; + }else { + result =SCAN_SUCCEED ; + ret =1 ; } out_unmap:pte_unmap_unlock (pte, ptl);if (ret) { node =khugepaged_find_target_node ();/* collapse_huge_page will return with the mmap_sem released */ -collapse_huge_page (mm, address, hpage, node, referenced); +collapse_huge_page (mm, address, hpage, node, + referenced, unmapped); } out: trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced,
_
Andrew Morton
maintainerPATCH 63/131 khugepaged: drain all LRU caches before scanning pages
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: khugepaged: drain all LRU caches before scanning pages
Having a page in LRU add cache offsets page refcount and gives false-negative on PageLRU(). It reduces collapse success rate.
Drain all LRU add caches before scanning. It happens relatively rare and should not disturb the system too much.
---
mm/khugepaged.c | 2 ++ 1 file changed, 2 insertions(+)
mm/khugepaged.c renamed from mm/khugepaged.c~khugepaged-drain-all-lru-caches-before-scanning-pages +2 -0
@@ -2079,6 +2079,8 @@ static void khugepaged_do_scan(void)barrier ();/* write khugepaged_pages_to_scan to local stack */ +lru_add_drain_all (); +while (progress < pages) {if (!khugepaged_prealloc_page (&hpage, &wait))break ;
_
Andrew Morton
maintainerPATCH 64/131 khugepaged: drain LRU add pagevec after swapin
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: khugepaged: drain LRU add pagevec after swapin
collapse_huge_page() tries to swap in pages that are part of the PMD range. Just swapped in page goes though LRU add cache. The cache gets extra reference on the page.
The extra reference can lead to the collapse fail: the following __collapse_huge_page_isolate() would check refcount and abort collapse seeing unexpected refcount.
The fix is to drain local LRU add cache in __collapse_huge_page_swapin() if we successfully swapped in any pages.
---
mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/khugepaged.c renamed from mm/khugepaged.c~khugepaged-drain-lru-add-pagevec-after-swapin +5 -0
@@ -931,6 +931,11 @@ static bool __collapse_huge_page_swapin( } vmf.pte--;pte_unmap (vmf.pte); + +/* Drain LRU add pagevec to remove extra pin on the swapped in pages */ +if (swapped_in) +lru_add_drain (); +trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,1 );return true; }
_
Andrew Morton
maintainerPATCH 65/131 khugepaged: allow to collapse a page shared across fork
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: khugepaged: allow to collapse a page shared across fork
The page can be included into collapse as long as it doesn't have extra pins (from GUP or otherwise).
Logic to check the refcount is moved to a separate function. For pages in swap cache, add compound_nr(page) to the expected refcount, in order to handle the compound page case. This is in preparation for the following patch.
VM_BUG_ON_PAGE() was removed from __collapse_huge_page_copy() as the invariant it checks is no longer valid: the source can be mapped multiple times now.
---
mm/khugepaged.c | 46 +++++++++++++++++++++++++++++++++++++--------- 1 file changed, 37 insertions(+), 9 deletions(-)
mm/khugepaged.c renamed from mm/khugepaged.c~khugepaged-allow-to-collapse-a-page-shared-across-fork +37 -9
@@ -526,6 +526,17 @@ static void release_pte_pages(pte_t *pte } } +static bool is_refcount_suitable (struct page *page) +{ +int expected_refcount; + + expected_refcount =total_mapcount (page); +if (PageSwapCache (page)) + expected_refcount +=compound_nr (page); + +return page_count (page) == expected_refcount; +} +static int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address,pte_t *pte)
@@ -578,11 +589,17 @@ static int __collapse_huge_page_isolate( }/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. +* Check if the page has any GUP (or other external) pins. +* +* The page table that maps the page has been already unlinked +* from the page table tree and this process cannot get +* an additinal pin on the page. +* +* New pins can come later if the page is shared across fork, +* but not from this process. The other process cannot write to +* the page, only trigger CoW. */ -if (page_count (page) !=1 +PageSwapCache (page)) { +if (!is_refcount_suitable (page)) {unlock_page (page); result =SCAN_PAGE_COUNT ; goto out;
@@ -669,7 +686,6 @@ static void __collapse_huge_page_copy(pt }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -VM_BUG_ON_PAGE (page_mapcount (src_page) !=1 , src_page);release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to
@@ -1221,11 +1237,23 @@ static int khugepaged_scan_pmd(struct mm }/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. +* Check if the page has any GUP (or other external) pins. +* +* Here the check is racy it may see totmal_mapcount > refcount +* in some cases. +* For example, one process with one forked child process. +* The parent has the PMD split due to MADV_DONTNEED, then +* the child is trying unmap the whole PMD, but khugepaged +* may be scanning the parent between the child has +* PageDoubleMap flag cleared and dec the mapcount. So +* khugepaged may see total_mapcount > refcount. +* +* But such case is ephemeral we could always retry collapse +* later. However it may report false positive if the page +* has excessive GUP pins (i.e. 512). Anyway the same check +* will be done again later the risk seems low. */ -if (page_count (page) !=1 +PageSwapCache (page)) { +if (!is_refcount_suitable (page)) { result =SCAN_PAGE_COUNT ; goto out_unmap; }
_
Andrew Morton
maintainerPATCH 66/131 khugepaged: allow to collapse PTE-mapped compound pages
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: khugepaged: allow to collapse PTE-mapped compound pages
We can collapse PTE-mapped compound pages. We only need to avoid handling them more than once: lock/unlock page only once if it's present in the PMD range multiple times as it handled on compound level. The same goes for LRU isolation and putback.
---
mm/khugepaged.c | 99 ++++++++++++++++++++++++++++++---------------- 1 file changed, 65 insertions(+), 34 deletions(-)
mm/khugepaged.c renamed from mm/khugepaged.c~khugepaged-allow-to-collapse-pte-mapped-compound-pages +65 -34
@@ -512,17 +512,30 @@ void __khugepaged_exit(struct mm_structstatic void release_pte_page (struct page *page) { -dec_node_page_state (page,NR_ISOLATED_ANON +page_is_file_lru (page)); +mod_node_page_state (page_pgdat (page), +NR_ISOLATED_ANON +page_is_file_lru (page), + -compound_nr (page));unlock_page (page);putback_lru_page (page); } -static void release_pte_pages (pte_t *pte,pte_t *_pte) +static void release_pte_pages (pte_t *pte,pte_t *_pte, +struct list_head *compound_pagelist) { +struct page *page, *tmp; +while (--_pte >= pte) {pte_t pteval = *_pte; -if (!pte_none (pteval) && !is_zero_pfn (pte_pfn (pteval))) -release_pte_page (pte_page (pteval)); + + page =pte_page (pteval); +if (!pte_none (pteval) && !is_zero_pfn (pte_pfn (pteval)) && + !PageCompound (page)) +release_pte_page (page); + } + +list_for_each_entry_safe (page, tmp, compound_pagelist, lru) { +list_del (&page->lru); +release_pte_page (page); } }
@@ -539,7 +552,8 @@ static bool is_refcount_suitable(structstatic int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address, -pte_t *pte) +pte_t *pte, +struct list_head *compound_pagelist) {struct page *page =NULL ;pte_t *_pte;
@@ -569,13 +583,21 @@ static int __collapse_huge_page_isolate( goto out; } -/* TODO: teach khugepaged to collapse THP mapped with pte */ +VM_BUG_ON_PAGE (!PageAnon (page), page); +if (PageCompound (page)) { - result =SCAN_PAGE_COMPOUND ; - goto out; - } +struct page *p; + page =compound_head (page); -VM_BUG_ON_PAGE (!PageAnon (page), page); +/* +* Check if we have dealt with the compound page +* already +*/ +list_for_each_entry (p, compound_pagelist, lru) { +if (page == p) + goto next; + } + }/* * We can do it before isolate_lru_page because the
@@ -604,19 +626,15 @@ static int __collapse_huge_page_isolate( result =SCAN_PAGE_COUNT ; goto out; } -if (pte_write (pteval)) { - writable = true; - }else { -if (PageSwapCache (page) && - !reuse_swap_page (page,NULL )) { -unlock_page (page); - result =SCAN_SWAP_CACHE_PAGE ; - goto out; - } +if (!pte_write (pteval) &&PageSwapCache (page) && + !reuse_swap_page (page,NULL )) {/* -* Page is not in the swap cache. It can be collapsed -* into a THP. +* Page is in the swap cache and cannot be re-used. +* It cannot be collapsed into a THP. */ +unlock_page (page); + result =SCAN_SWAP_CACHE_PAGE ; + goto out; }/*
@@ -628,16 +646,23 @@ static int __collapse_huge_page_isolate( result =SCAN_DEL_PAGE_LRU ; goto out; } -inc_node_page_state (page, -NR_ISOLATED_ANON +page_is_file_lru (page)); +mod_node_page_state (page_pgdat (page), +NR_ISOLATED_ANON +page_is_file_lru (page), +compound_nr (page));VM_BUG_ON_PAGE (!PageLocked (page), page);VM_BUG_ON_PAGE (PageLRU (page), page); +if (PageCompound (page)) +list_add_tail (&page->lru, compound_pagelist); +next:/* There should be enough young pte to collapse the page */ if (pte_young (pteval) ||page_is_young (page) ||PageReferenced (page) ||mmu_notifier_test_young (vma->vm_mm, address)) referenced++; + +if (pte_write (pteval)) + writable = true; }if (likely (writable)) {if (likely (referenced)) {
@@ -651,7 +676,7 @@ static int __collapse_huge_page_isolate( } out: -release_pte_pages (pte, _pte); +release_pte_pages (pte, _pte, compound_pagelist);trace_mm_collapse_huge_page_isolate (page, none_or_zero, referenced, writable, result);return 0 ;
@@ -660,13 +685,14 @@ out:static void __collapse_huge_page_copy (pte_t *pte,struct page *page,struct vm_area_struct *vma,unsigned long address, -spinlock_t *ptl) +spinlock_t *ptl, +struct list_head *compound_pagelist) { +struct page *src_page, *tmp;pte_t *_pte;for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, page++, address +=PAGE_SIZE ) {pte_t pteval = *_pte; -struct page *src_page;if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {clear_user_highpage (page, address);
@@ -686,7 +712,8 @@ static void __collapse_huge_page_copy(pt }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -release_pte_page (src_page); +if (!PageCompound (src_page)) +release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats
@@ -703,6 +730,11 @@ static void __collapse_huge_page_copy(ptfree_page_and_swap_cache (src_page); } } + +list_for_each_entry_safe (src_page, tmp, compound_pagelist, lru) { +list_del (&src_page->lru); +release_pte_page (src_page); + } }static void khugepaged_alloc_sleep (void )
@@ -961,6 +993,7 @@ static void collapse_huge_page(struct mmstruct page **hpage, int node, int referenced, int unmapped) { +LIST_HEAD (compound_pagelist );pmd_t *pmd, _pmd;pte_t *pte;pgtable_t pgtable;
@@ -1061,7 +1094,8 @@ static void collapse_huge_page(struct mmmmu_notifier_invalidate_range_end (&range);spin_lock (pte_ptl); - isolated =__collapse_huge_page_isolate (vma, address, pte); + isolated =__collapse_huge_page_isolate (vma, address, pte, + &compound_pagelist);spin_unlock (pte_ptl);if (unlikely (!isolated)) {
@@ -1086,7 +1120,8 @@ static void collapse_huge_page(struct mm*/ anon_vma_unlock_write (vma->anon_vma); -__collapse_huge_page_copy (pte, new_page, vma, address, pte_ptl); +__collapse_huge_page_copy (pte, new_page, vma, address, pte_ptl, + &compound_pagelist);pte_unmap (pte);__SetPageUptodate (new_page); pgtable =pmd_pgtable (_pmd);
@@ -1205,11 +1240,7 @@ static int khugepaged_scan_pmd(struct mm goto out_unmap; } -/* TODO: teach khugepaged to collapse THP mapped with pte */ -if (PageCompound (page)) { - result =SCAN_PAGE_COMPOUND ; - goto out_unmap; - } + page =compound_head (page);/* * Record which node the original page is from and save this
_
Andrew Morton
maintainerPATCH 67/131 thp: change CoW semantics for anon-THP
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: thp: change CoW semantics for anon-THP
Currently we have different copy-on-write semantics for anon- and file-THP. For anon-THP we try to allocate huge page on the write fault, but on file-THP we split PMD and allocate 4k page.
Arguably, file-THP semantics is more desirable: we don't necessary want to unshare full PMD range from the parent on the first access. This is the primary reason THP is unusable for some workloads, like Redis.
The original THP refcounting didn't allow to have PTE-mapped compound pages, so we had no options, but to allocate huge page on CoW (with fallback to 512 4k pages).
The current refcounting doesn't have such limitations and we can cut a lot of complex code out of fault path.
khugepaged is now able to recover THP from such ranges if the configuration allows.
---
mm/huge_memory.c | 250 ++++----------------------------------------- 1 file changed, 25 insertions(+), 225 deletions(-)
mm/huge_memory.c renamed from mm/huge_memory.c~thp-change-cow-semantics-for-anon-thp +25 -225
@@ -1255,263 +1255,63 @@ unlock:spin_unlock (vmf->ptl); } -static vm_fault_t do_huge_pmd_wp_page_fallback (struct vm_fault *vmf, -pmd_t orig_pmd,struct page *page) -{ -struct vm_area_struct *vma = vmf->vma; -unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -struct mem_cgroup *memcg; -pgtable_t pgtable; -pmd_t _pmd; -int i; -vm_fault_t ret =0 ; -struct page **pages; -struct mmu_notifier_range range; - - pages =kmalloc_array (HPAGE_PMD_NR ,sizeof (struct page *), -GFP_KERNEL ); -if (unlikely (!pages)) { - ret |=VM_FAULT_OOM ; - goto out; - } - -for (i =0 ; i <HPAGE_PMD_NR ; i++) { - pages[i] =alloc_page_vma_node (GFP_HIGHUSER_MOVABLE , vma, - vmf->address,page_to_nid (page)); -if (unlikely (!pages[i] || -mem_cgroup_try_charge_delay (pages[i], vma->vm_mm, -GFP_KERNEL , &memcg, false))) { -if (pages[i]) -put_page (pages[i]); -while (--i >=0 ) { - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, - false); -put_page (pages[i]); - } -kfree (pages); - ret |=VM_FAULT_OOM ; - goto out; - } -set_page_private (pages[i], (unsigned long )memcg); - } - -for (i =0 ; i <HPAGE_PMD_NR ; i++) { -copy_user_highpage (pages[i], page + i, - haddr +PAGE_SIZE * i, vma); -__SetPageUptodate (pages[i]); -cond_resched (); - } - -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, vma->vm_mm, - haddr, haddr +HPAGE_PMD_SIZE ); -mmu_notifier_invalidate_range_start (&range); - - vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) - goto out_free_pages; -VM_BUG_ON_PAGE (!PageHead (page), page); - -/* -* Leave pmd empty until pte is filled note we must notify here as -* concurrent CPU thread might write to new page before the call to -* mmu_notifier_invalidate_range_end() happens which can lead to a -* device seeing memory write in different order than CPU. -* -* See Documentation/vm/mmu_notifier.rst -*/ -pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd); - - pgtable =pgtable_trans_huge_withdraw (vma->vm_mm, vmf->pmd); -pmd_populate (vma->vm_mm, &_pmd, pgtable); - -for (i =0 ; i <HPAGE_PMD_NR ; i++, haddr +=PAGE_SIZE ) { -pte_t entry; - entry =mk_pte (pages[i], vma->vm_page_prot); - entry =maybe_mkwrite (pte_mkdirty (entry), vma); - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -page_add_new_anon_rmap (pages[i], vmf->vma, haddr, false); -mem_cgroup_commit_charge (pages[i], memcg, false, false); -lru_cache_add_active_or_unevictable (pages[i], vma); - vmf->pte =pte_offset_map (&_pmd, haddr); -VM_BUG_ON (!pte_none (*vmf->pte)); -set_pte_at (vma->vm_mm, haddr, vmf->pte, entry); -pte_unmap (vmf->pte); - } -kfree (pages); - -smp_wmb ();/* make pte visible before pmd */ -pmd_populate (vma->vm_mm, vmf->pmd, pgtable); -page_remove_rmap (page, true); -spin_unlock (vmf->ptl); - -/* -* No need to double call mmu_notifier->invalidate_range() callback as -* the above pmdp_huge_clear_flush_notify() did already call it. -*/ -mmu_notifier_invalidate_range_only_end (&range); - - ret |=VM_FAULT_WRITE ; -put_page (page); - -out: -return ret; - -out_free_pages: -spin_unlock (vmf->ptl); -mmu_notifier_invalidate_range_end (&range); -for (i =0 ; i <HPAGE_PMD_NR ; i++) { - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, false); -put_page (pages[i]); - } -kfree (pages); - goto out; -} -vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd) {struct vm_area_struct *vma = vmf->vma; -struct page *page =NULL , *new_page; -struct mem_cgroup *memcg; +struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -struct mmu_notifier_range range; -gfp_t huge_gfp;/* for allocation and charge */ -vm_fault_t ret =0 ; vmf->ptl =pmd_lockptr (vma->vm_mm, vmf->pmd);VM_BUG_ON_VMA (!vma->anon_vma, vma); +if (is_huge_zero_pmd (orig_pmd)) - goto alloc; + goto fallback; +spin_lock (vmf->ptl); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) - goto out_unlock; + +if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) { +spin_unlock (vmf->ptl); +return 0 ; + } page =pmd_page (orig_pmd);VM_BUG_ON_PAGE (!PageCompound (page) || !PageHead (page), page); -/* -* We can only reuse the page if nobody else maps the huge page or it's -* part. -*/ + +/* Lock page for reuse_swap_page() */ if (!trylock_page (page)) {get_page (page);spin_unlock (vmf->ptl);lock_page (page);spin_lock (vmf->ptl);if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) { +spin_unlock (vmf->ptl);unlock_page (page);put_page (page); - goto out_unlock; +return 0 ; }put_page (page); } + +/* +* We can only reuse the page if nobody else maps the huge page or it's +* part. +*/ if (reuse_swap_page (page,NULL )) {pmd_t entry; entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); -if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 )) +if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); - ret |=VM_FAULT_WRITE ;unlock_page (page); - goto out_unlock; - } -unlock_page (page); -get_page (page); -spin_unlock (vmf->ptl); -alloc: -if (__transparent_hugepage_enabled (vma) && - !transparent_hugepage_debug_cow ()) { - huge_gfp =alloc_hugepage_direct_gfpmask (vma); - new_page =alloc_hugepage_vma (huge_gfp, vma, haddr,HPAGE_PMD_ORDER ); - }else - new_page =NULL ; - -if (likely (new_page)) { -prep_transhuge_page (new_page); - }else { -if (!page) { -split_huge_pmd (vma, vmf->pmd, vmf->address); - ret |=VM_FAULT_FALLBACK ; - }else { - ret =do_huge_pmd_wp_page_fallback (vmf, orig_pmd, page); -if (ret &VM_FAULT_OOM ) { -split_huge_pmd (vma, vmf->pmd, vmf->address); - ret |=VM_FAULT_FALLBACK ; - } -put_page (page); - } -count_vm_event (THP_FAULT_FALLBACK ); - goto out; - } - -if (unlikely (mem_cgroup_try_charge_delay (new_page, vma->vm_mm, - huge_gfp, &memcg, true))) { -put_page (new_page); -split_huge_pmd (vma, vmf->pmd, vmf->address); -if (page) -put_page (page); - ret |=VM_FAULT_FALLBACK ; -count_vm_event (THP_FAULT_FALLBACK ); -count_vm_event (THP_FAULT_FALLBACK_CHARGE ); - goto out; - } - -count_vm_event (THP_FAULT_ALLOC ); -count_memcg_events (memcg,THP_FAULT_ALLOC ,1 ); - -if (!page) -clear_huge_page (new_page, vmf->address,HPAGE_PMD_NR ); -else -copy_user_huge_page (new_page, page, vmf->address, - vma,HPAGE_PMD_NR ); -__SetPageUptodate (new_page); - -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, vma->vm_mm, - haddr, haddr +HPAGE_PMD_SIZE ); -mmu_notifier_invalidate_range_start (&range); - -spin_lock (vmf->ptl); -if (page) -put_page (page); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) {spin_unlock (vmf->ptl); -mem_cgroup_cancel_charge (new_page, memcg, true); -put_page (new_page); - goto out_mn; - }else { -pmd_t entry; - entry =mk_huge_pmd (new_page, vma->vm_page_prot); - entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); -pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd); -page_add_new_anon_rmap (new_page, vma, haddr, true); -mem_cgroup_commit_charge (new_page, memcg, false, true); -lru_cache_add_active_or_unevictable (new_page, vma); -set_pmd_at (vma->vm_mm, haddr, vmf->pmd, entry); -update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); -if (!page) { -add_mm_counter (vma->vm_mm,MM_ANONPAGES ,HPAGE_PMD_NR ); - }else { -VM_BUG_ON_PAGE (!PageHead (page), page); -page_remove_rmap (page, true); -put_page (page); - } - ret |=VM_FAULT_WRITE ; +return VM_FAULT_WRITE ; } + +unlock_page (page);spin_unlock (vmf->ptl); -out_mn: -/* -* No need to double call mmu_notifier->invalidate_range() callback as -* the above pmdp_huge_clear_flush_notify() did already call it. -*/ -mmu_notifier_invalidate_range_only_end (&range); -out: -return ret; -out_unlock: -spin_unlock (vmf->ptl); -return ret; +fallback: +__split_huge_pmd (vma, vmf->pmd, vmf->address, false,NULL ); +return VM_FAULT_FALLBACK ; }/*
_
Andrew Morton
maintainerPATCH 68/131 khugepaged: introduce 'max_ptes_shared' tunable
In reply to Andrew MortonFrom: "Kirill A. Shutemov" <[email protected]>
Subject: khugepaged: introduce 'max_ptes_shared' tunable
'max_ptes_shared' specifies how many pages can be shared across multiple processes. Exceeding the number would block the collapse::
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared
A higher value may increase memory footprint for some workloads.
By default, at least half of pages has to be not shared.
---
Documentation/admin-guide/mm/transhuge.rst | 7 + include/trace/events/huge_memory.h | 3 mm/khugepaged.c | 52 ++++++++++- tools/testing/selftests/vm/khugepaged.c | 83 +++++++++++++++++++ 4 files changed, 140 insertions(+), 5 deletions(-)
Documentation/admin-guide/mm/transhuge.rst renamed from Documentation/admin-guide/mm/transhuge.rst~khugepaged-introduce-max_ptes_shared-tunable +7 -0
@@ -220,6 +220,13 @@ memory. A lower value can prevent THPs f collapsed, resulting fewer pages being collapsed into THPs, and lower memory access performance. +``max_ptes_shared`` specifies how many pages can be shared across multiple +processes. Exceeding the number would block the collapse:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared + +A higher value may increase memory footprint for some workloads. + Boot parameter ==============
include/trace/events/huge_memory.h renamed from include/trace/events/huge_memory.h~khugepaged-introduce-max_ptes_shared-tunable +2 -1
@@ -12,6 +12,8 @@ EM( SCAN_SUCCEED, "succeeded") \ EM( SCAN_PMD_NULL, "pmd_null") \ EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \ + EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ + EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \ EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \ EM( SCAN_PTE_UFFD_WP, "pte_uffd_wp") \EM (SCAN_PAGE_RO ,"no_writable_page" ) \
@@ -31,7 +33,6 @@ EM( SCAN_DEL_PAGE_LRU, "could_not_delete_page_from_lru")\ EM( SCAN_ALLOC_HUGE_PAGE_FAIL, "alloc_huge_page_failed") \ EM( SCAN_CGROUP_CHARGE_FAIL, "ccgroup_charge_failed") \ - EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ EM( SCAN_TRUNCATED, "truncated") \EMe (SCAN_PAGE_HAS_PRIVATE ,"page_has_private" ) \
mm/khugepaged.c renamed from mm/khugepaged.c~khugepaged-introduce-max_ptes_shared-tunable +48 -4
114 lines not on this page · raw message
tools/testing/selftests/vm/khugepaged.c renamed from tools/testing/selftests/vm/khugepaged.c~khugepaged-introduce-max_ptes_shared-tunable +83 -0
125 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 69/131 hugetlbfs: add arch_hugetlb_valid_size
In reply to Andrew MortonFrom: Mike Kravetz <[email protected]>
Subject: hugetlbfs: add arch_hugetlb_valid_size
Patch series "Clean up hugetlb boot command line processing", v4.
Longpeng(Mike) reported a weird message from hugetlb command line processing and proposed a solution [1]. While the proposed patch does address the specific issue, there are other related issues in command line processing. As hugetlbfs evolved, updates to command line processing have been made to meet immediate needs and not necessarily in a coordinated manner. The result is that some processing is done in arch specific code, some is done in arch independent code and coordination is problematic. Semantics can vary between architectures.
The patch series does the following:
- Define arch specific arch_hugetlb_valid_size routine used to validate passed huge page sizes.
- Move hugepagesz= command line parsing out of arch specific code and into an arch independent routine.
- Clean up command line processing to follow desired semantics and document those semantics.
[1] https://lore.kernel.org/linux-mm/[email protected]
This patch (of 3):
The architecture independent routine hugetlb_default_setup sets up the default huge pages size. It has no way to verify if the passed value is valid, so it accepts it and attempts to validate at a later time. This requires undocumented cooperation between the arch specific and arch independent code.
For architectures that support more than one huge page size, provide a routine arch_hugetlb_valid_size to validate a huge page size. hugetlb_default_setup can use this to validate passed values.
arch_hugetlb_valid_size will also be used in a subsequent patch to move processing of the "hugepagesz=" in arch specific code to a common routine in arch independent code.
---
arch/arm64/mm/hugetlbpage.c | 17 +++++++++++++---- arch/powerpc/mm/hugetlbpage.c | 20 +++++++++++++------- arch/riscv/mm/hugetlbpage.c | 26 +++++++++++++++++--------- arch/s390/mm/hugetlbpage.c | 16 ++++++++++++---- arch/sparc/mm/init_64.c | 24 ++++++++++++++++-------- arch/x86/mm/hugetlbpage.c | 17 +++++++++++++---- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 21 ++++++++++++++++++--- 8 files changed, 103 insertions(+), 39 deletions(-)
arch/arm64/mm/hugetlbpage.c renamed from arch/arm64/mm/hugetlbpage.c~hugetlbfs-add-arch_hugetlb_valid_size +13 -4
@@ -464,17 +464,26 @@ static int __init hugetlbpage_init(void) }arch_initcall (hugetlbpage_init); -static __init int setup_hugepagesz (char *opt) +bool __init arch_hugetlb_valid_size (unsigned long size) { -unsigned long ps =memparse (opt, &opt); - -switch (ps) { +switch (size) {#ifdef CONFIG_ARM64_4K_PAGES case PUD_SIZE :#endif case CONT_PMD_SIZE :case PMD_SIZE :case CONT_PTE_SIZE : +return true; + } + +return false; +} + +static __init int setup_hugepagesz (char *opt) +{ +unsigned long ps =memparse (opt, &opt); + +if (arch_hugetlb_valid_size (ps)) {add_huge_page_size (ps);return 1 ; }
arch/powerpc/mm/hugetlbpage.c renamed from arch/powerpc/mm/hugetlbpage.c~hugetlbfs-add-arch_hugetlb_valid_size +13 -7
@@ -558,7 +558,7 @@ unsigned long vma_mmu_pagesize(struct vmreturn vma_kernel_pagesize (vma); } -static int __init add_huge_page_size (unsigned long long size) +bool __init arch_hugetlb_valid_size (unsigned long size) {int shift =__ffs (size);int mmu_psize;
@@ -566,20 +566,26 @@ static int __init add_huge_page_size(uns/* Check that it is a page size supported by the hardware and * that it fits within pagetable and slice limits. */ if (size <=PAGE_SIZE || !is_power_of_2 (size)) -return -EINVAL ; +return false; mmu_psize =check_and_get_huge_psize (shift);if (mmu_psize <0 ) -return -EINVAL ; +return false;BUG_ON (mmu_psize_defs[mmu_psize].shift != shift); -/* Return if huge page size has already been setup */ -if (size_to_hstate (size)) -return 0 ; +return true; +} -hugetlb_add_hstate (shift -PAGE_SHIFT ); +static int __init add_huge_page_size (unsigned long long size) +{ +int shift =__ffs (size); + +if (!arch_hugetlb_valid_size ((unsigned long )size)) +return -EINVAL ; +if (!size_to_hstate (size)) +hugetlb_add_hstate (shift -PAGE_SHIFT );return 0 ; }
arch/riscv/mm/hugetlbpage.c renamed from arch/riscv/mm/hugetlbpage.c~hugetlbfs-add-arch_hugetlb_valid_size +17 -9
@@ -12,21 +12,29 @@ int pmd_huge(pmd_t pmd)return pmd_leaf (pmd); } +bool __init arch_hugetlb_valid_size (unsigned long size) +{ +if (size ==HPAGE_SIZE ) +return true; +else if (IS_ENABLED (CONFIG_64BIT ) && size ==PUD_SIZE ) +return true; +else +return false; +} +static __init int setup_hugepagesz (char *opt) {unsigned long ps =memparse (opt, &opt); -if (ps ==HPAGE_SIZE ) { -hugetlb_add_hstate (HPAGE_SHIFT -PAGE_SHIFT ); - }else if (IS_ENABLED (CONFIG_64BIT ) && ps ==PUD_SIZE ) { -hugetlb_add_hstate (PUD_SHIFT -PAGE_SHIFT ); - }else { -hugetlb_bad_size (); -pr_err ("hugepagesz: Unsupported page size %lu M\n" , ps >>20 ); -return 0 ; +if (arch_hugetlb_valid_size (ps)) { +hugetlb_add_hstate (ilog2 (ps) -PAGE_SHIFT ); +return 1 ; } -return 1 ; +hugetlb_bad_size (); +pr_err ("hugepagesz: Unsupported page size %lu M\n" , ps >>20 ); +return 0 ; + }__setup ("hugepagesz=" , setup_hugepagesz);
arch/s390/mm/hugetlbpage.c renamed from arch/s390/mm/hugetlbpage.c~hugetlbfs-add-arch_hugetlb_valid_size +12 -4
@@ -254,16 +254,24 @@ follow_huge_pud(struct mm_struct *mm, unreturn pud_page (*pud) + ((address & ~PUD_MASK ) >>PAGE_SHIFT ); } +bool __init arch_hugetlb_valid_size (unsigned long size) +{ +if (MACHINE_HAS_EDAT1 && size ==PMD_SIZE ) +return true; +else if (MACHINE_HAS_EDAT2 && size ==PUD_SIZE ) +return true; +else +return false; +} +static __init int setup_hugepagesz (char *opt) {unsigned long size;char *string = opt; size =memparse (opt, &opt); -if (MACHINE_HAS_EDAT1 && size ==PMD_SIZE ) { -hugetlb_add_hstate (PMD_SHIFT -PAGE_SHIFT ); - }else if (MACHINE_HAS_EDAT2 && size ==PUD_SIZE ) { -hugetlb_add_hstate (PUD_SHIFT -PAGE_SHIFT ); +if (arch_hugetlb_valid_size (size)) { +hugetlb_add_hstate (ilog2 (size) -PAGE_SHIFT ); }else {hugetlb_bad_size (); pr_err("hugepagesz= specifies an unsupported page size %s\n",
arch/sparc/mm/init_64.c renamed from arch/sparc/mm/init_64.c~hugetlbfs-add-arch_hugetlb_valid_size +16 -8
@@ -360,16 +360,11 @@ static void __init pud_huge_patch(void) __asm__ __volatile__("flush %0" : :"r" (addr)); } -static int __init setup_hugepagesz (char *string) +bool __init arch_hugetlb_valid_size (unsigned long size) { -unsigned long long hugepage_size; -unsigned int hugepage_shift; +unsigned int hugepage_shift =ilog2 (size);unsigned short hv_pgsz_idx;unsigned int hv_pgsz_mask; -int rc =0 ; - - hugepage_size =memparse (string, &string); - hugepage_shift =ilog2 (hugepage_size);switch (hugepage_shift) {case HPAGE_16GB_SHIFT :
@@ -397,7 +392,20 @@ static int __init setup_hugepagesz(char hv_pgsz_mask =0 ; } -if ((hv_pgsz_mask & cpu_pgsz_mask) ==0U ) { +if ((hv_pgsz_mask & cpu_pgsz_mask) ==0U ) +return false; + +return true; +} + +static int __init setup_hugepagesz (char *string) +{ +unsigned long long hugepage_size; +int rc =0 ; + + hugepage_size =memparse (string, &string); + +if (!arch_hugetlb_valid_size ((unsigned long )hugepage_size)) {hugetlb_bad_size ();pr_err ("hugepagesz=%llu not supported by MMU.\n" , hugepage_size);
arch/x86/mm/hugetlbpage.c renamed from arch/x86/mm/hugetlbpage.c~hugetlbfs-add-arch_hugetlb_valid_size +13 -4
@@ -181,13 +181,22 @@ get_unmapped_area:#endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_X86_64 +bool __init arch_hugetlb_valid_size (unsigned long size) +{ +if (size ==PMD_SIZE ) +return true; +else if (size ==PUD_SIZE &&boot_cpu_has (X86_FEATURE_GBPAGES )) +return true; +else +return false; +} +static __init int setup_hugepagesz (char *opt) {unsigned long ps =memparse (opt, &opt); -if (ps ==PMD_SIZE ) { -hugetlb_add_hstate (PMD_SHIFT -PAGE_SHIFT ); - }else if (ps ==PUD_SIZE &&boot_cpu_has (X86_FEATURE_GBPAGES )) { -hugetlb_add_hstate (PUD_SHIFT -PAGE_SHIFT ); + +if (arch_hugetlb_valid_size (ps)) { +hugetlb_add_hstate (ilog2 (ps) -PAGE_SHIFT ); }else {hugetlb_bad_size (); printk(KERN_ERR "hugepagesz: Unsupported page size %lu M\n",
include/linux/hugetlb.h renamed from include/linux/hugetlb.h~hugetlbfs-add-arch_hugetlb_valid_size +1 -0
@@ -521,6 +521,7 @@ int __init alloc_bootmem_huge_page(strucvoid __init hugetlb_bad_size (void );void __init hugetlb_add_hstate (unsigned order); +bool __init arch_hugetlb_valid_size (unsigned long size);struct hstate *size_to_hstate (unsigned long size);#ifndef HUGE_MAX_HSTATE
mm/hugetlb.c renamed from mm/hugetlb.c~hugetlbfs-add-arch_hugetlb_valid_size +18 -3
@@ -3256,6 +3256,12 @@ static int __init hugetlb_init(void) }subsys_initcall (hugetlb_init); +/* Overwritten by architectures with more huge page sizes */ +bool __init __attribute((weak))arch_hugetlb_valid_size (unsigned long size) +{ +return size ==HPAGE_SIZE ; +} +/* Should be called on processing a hugepagesz=... option */ void __init hugetlb_bad_size (void ) {
@@ -3331,12 +3337,21 @@ static int __init hugetlb_nrpages_setup( }__setup ("hugepages=" , hugetlb_nrpages_setup); -static int __init hugetlb_default_setup (char *s) +static int __init default_hugepagesz_setup (char *s) { - default_hstate_size =memparse (s, &s); +unsigned long size; + + size = (unsigned long )memparse (s,NULL ); + +if (!arch_hugetlb_valid_size (size)) { +pr_err ("HugeTLB: unsupported default_hugepagesz %s\n" , s); +return 0 ; + } + + default_hstate_size = size;return 1 ; } -__setup ("default_hugepagesz=" , hugetlb_default_setup); +__setup ("default_hugepagesz=" , default_hugepagesz_setup);static unsigned int cpuset_mems_nr (unsigned int *array) {
_
Andrew Morton
maintainerPATCH 70/131 hugetlbfs: move hugepagesz= parsing to arch independent code
In reply to Andrew MortonFrom: Mike Kravetz <[email protected]>
Subject: hugetlbfs: move hugepagesz= parsing to arch independent code
Now that architectures provide arch_hugetlb_valid_size(), parsing of "hugepagesz=" can be done in architecture independent code. Create a single routine to handle hugepagesz= parsing and remove all arch specific routines. We can also remove the interface hugetlb_bad_size() as this is no longer used outside arch independent code.
This also provides consistent behavior of hugetlbfs command line options. The hugepagesz= option should only be specified once for a specific size, but some architectures allow multiple instances. This appears to be more of an oversight when code was added by some architectures to set up ALL huge pages sizes.
---
arch/arm64/mm/hugetlbpage.c | 15 --------------- arch/powerpc/mm/hugetlbpage.c | 15 --------------- arch/riscv/mm/hugetlbpage.c | 16 ---------------- arch/s390/mm/hugetlbpage.c | 18 ------------------ arch/sparc/mm/init_64.c | 22 ---------------------- arch/x86/mm/hugetlbpage.c | 16 ---------------- include/linux/hugetlb.h | 1 - mm/hugetlb.c | 23 +++++++++++++++++------ 8 files changed, 17 insertions(+), 109 deletions(-)
arch/arm64/mm/hugetlbpage.c renamed from arch/arm64/mm/hugetlbpage.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -15
@@ -478,18 +478,3 @@ bool __init arch_hugetlb_valid_size(unsireturn false; } - -static __init int setup_hugepagesz (char *opt) -{ -unsigned long ps =memparse (opt, &opt); - -if (arch_hugetlb_valid_size (ps)) { -add_huge_page_size (ps); -return 1 ; - } - -hugetlb_bad_size (); -pr_err ("hugepagesz: Unsupported page size %lu K\n" , ps >>10 ); -return 0 ; -} -__setup ("hugepagesz=" , setup_hugepagesz);
arch/powerpc/mm/hugetlbpage.c renamed from arch/powerpc/mm/hugetlbpage.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -15
@@ -589,21 +589,6 @@ static int __init add_huge_page_size(unsreturn 0 ; } -static int __init hugepage_setup_sz (char *str) -{ -unsigned long long size; - - size =memparse (str, &str); - -if (add_huge_page_size (size) !=0 ) { -hugetlb_bad_size (); -pr_err ("Invalid huge page size specified(%llu)\n" , size); - } - -return 1 ; -} -__setup ("hugepagesz=" , hugepage_setup_sz); -static int __init hugetlbpage_init (void ) {bool configured = false;
arch/riscv/mm/hugetlbpage.c renamed from arch/riscv/mm/hugetlbpage.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -16
@@ -22,22 +22,6 @@ bool __init arch_hugetlb_valid_size(unsireturn false; } -static __init int setup_hugepagesz (char *opt) -{ -unsigned long ps =memparse (opt, &opt); - -if (arch_hugetlb_valid_size (ps)) { -hugetlb_add_hstate (ilog2 (ps) -PAGE_SHIFT ); -return 1 ; - } - -hugetlb_bad_size (); -pr_err ("hugepagesz: Unsupported page size %lu M\n" , ps >>20 ); -return 0 ; - -} -__setup ("hugepagesz=" , setup_hugepagesz); -#ifdef CONFIG_CONTIG_ALLOC static __init int gigantic_pages_init (void ) {
arch/s390/mm/hugetlbpage.c renamed from arch/s390/mm/hugetlbpage.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -18
@@ -264,24 +264,6 @@ bool __init arch_hugetlb_valid_size(unsireturn false; } -static __init int setup_hugepagesz (char *opt) -{ -unsigned long size; -char *string = opt; - - size =memparse (opt, &opt); -if (arch_hugetlb_valid_size (size)) { -hugetlb_add_hstate (ilog2 (size) -PAGE_SHIFT ); - }else { -hugetlb_bad_size (); -pr_err ("hugepagesz= specifies an unsupported page size %s\n" , - string); -return 0 ; - } -return 1 ; -} -__setup ("hugepagesz=" , setup_hugepagesz); -static unsigned long hugetlb_get_unmapped_area_bottomup (struct file *file,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags)
arch/sparc/mm/init_64.c renamed from arch/sparc/mm/init_64.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -22
@@ -397,28 +397,6 @@ bool __init arch_hugetlb_valid_size(unsi return true; } - -static int __init setup_hugepagesz (char *string) -{ -unsigned long long hugepage_size; -int rc =0 ; - - hugepage_size =memparse (string, &string); - -if (!arch_hugetlb_valid_size ((unsigned long )hugepage_size)) { -hugetlb_bad_size (); -pr_err ("hugepagesz=%llu not supported by MMU.\n" , - hugepage_size); - goto out; - } - -add_huge_page_size (hugepage_size); - rc =1 ; - -out: -return rc; -} -__setup ("hugepagesz=" , setup_hugepagesz);#endif /* CONFIG_HUGETLB_PAGE */ void update_mmu_cache (struct vm_area_struct *vma,unsigned long address,pte_t *ptep)
arch/x86/mm/hugetlbpage.c renamed from arch/x86/mm/hugetlbpage.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -16
@@ -191,22 +191,6 @@ bool __init arch_hugetlb_valid_size(unsireturn false; } -static __init int setup_hugepagesz (char *opt) -{ -unsigned long ps =memparse (opt, &opt); - -if (arch_hugetlb_valid_size (ps)) { -hugetlb_add_hstate (ilog2 (ps) -PAGE_SHIFT ); - }else { -hugetlb_bad_size (); -printk (KERN_ERR "hugepagesz: Unsupported page size %lu M\n" , - ps >>20 ); -return 0 ; - } -return 1 ; -} -__setup ("hugepagesz=" , setup_hugepagesz); -#ifdef CONFIG_CONTIG_ALLOC static __init int gigantic_pages_init (void ) {
include/linux/hugetlb.h renamed from include/linux/hugetlb.h~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -1
@@ -519,7 +519,6 @@ int huge_add_to_page_cache(struct page *int __init __alloc_bootmem_huge_page (struct hstate *h);int __init alloc_bootmem_huge_page (struct hstate *h); -void __init hugetlb_bad_size (void );void __init hugetlb_add_hstate (unsigned order);bool __init arch_hugetlb_valid_size (unsigned long size);struct hstate *size_to_hstate (unsigned long size);
mm/hugetlb.c renamed from mm/hugetlb.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code +0 -5
@@ -3262,12 +3262,6 @@ bool __init __attribute((weak)) arch_hugreturn size ==HPAGE_SIZE ; } -/* Should be called on processing a hugepagesz=... option */ -void __init hugetlb_bad_size (void ) -{ - parsed_valid_hugepagesz = false; -}
Andrew Morton
maintainerPATCH 71/131 hugetlbfs: remove hugetlb_add_hstate() warning for existing hstate
In reply to Andrew MortonFrom: Mike Kravetz <[email protected]>
Subject: hugetlbfs: remove hugetlb_add_hstate() warning for existing hstate
hugetlb_add_hstate() prints a warning if the hstate already exists. This was originally done as part of kernel command line parsing. If 'hugepagesz=' was specified more than once, the warning
pr_warn("hugepagesz= specified twice, ignoring\n");would be printed.
Some architectures want to enable all huge page sizes. They would call hugetlb_add_hstate for all supported sizes. However, this was done after command line processing and as a result hstates could have already been created for some sizes. To make sure no warning were printed, there would often be code like:
if (!size_to_hstate(size)
hugetlb_add_hstate(ilog2(size) - PAGE_SHIFT)
The only time we want to print the warning is as the result of command line processing. So, remove the warning from hugetlb_add_hstate and add it to the single arch independent routine processing "hugepagesz=". After this, calls to size_to_hstate() in arch specific code can be removed and hugetlb_add_hstate can be called without worrying about warning messages.
---
arch/arm64/mm/hugetlbpage.c | 16 ++++------------ arch/powerpc/mm/hugetlbpage.c | 3 +-- arch/riscv/mm/hugetlbpage.c | 2 +- arch/sparc/mm/init_64.c | 19 ++++--------------- arch/x86/mm/hugetlbpage.c | 2 +- mm/hugetlb.c | 9 ++++++--- 6 files changed, 17 insertions(+), 34 deletions(-)
arch/arm64/mm/hugetlbpage.c renamed from arch/arm64/mm/hugetlbpage.c~hugetlbfs-remove-hugetlb_add_hstate-warning-for-existing-hstate +4 -12
@@ -443,22 +443,14 @@ void huge_ptep_clear_flush(struct vm_areclear_flush (vma->vm_mm, addr, ptep, pgsize, ncontig); } -static void __init add_huge_page_size (unsigned long size) -{ -if (size_to_hstate (size)) -return ; - -hugetlb_add_hstate (ilog2 (size) -PAGE_SHIFT ); -} -static int __init hugetlbpage_init (void ) {#ifdef CONFIG_ARM64_4K_PAGES -add_huge_page_size (PUD_SIZE ); +hugetlb_add_hstate (PUD_SHIFT -PAGE_SHIFT );#endif -add_huge_page_size (CONT_PMD_SIZE ); -add_huge_page_size (PMD_SIZE ); -add_huge_page_size (CONT_PTE_SIZE ); +hugetlb_add_hstate ((CONT_PMD_SHIFT +PMD_SHIFT ) -PAGE_SHIFT ); +hugetlb_add_hstate (PMD_SHIFT -PAGE_SHIFT ); +hugetlb_add_hstate ((CONT_PTE_SHIFT +PAGE_SHIFT ) -PAGE_SHIFT );return 0 ; }
arch/powerpc/mm/hugetlbpage.c renamed from arch/powerpc/mm/hugetlbpage.c~hugetlbfs-remove-hugetlb_add_hstate-warning-for-existing-hstate +1 -2
@@ -584,8 +584,7 @@ static int __init add_huge_page_size(unsif (!arch_hugetlb_valid_size ((unsigned long )size))return -EINVAL ; -if (!size_to_hstate (size)) -hugetlb_add_hstate (shift -PAGE_SHIFT ); +hugetlb_add_hstate (shift -PAGE_SHIFT );return 0 ; }
arch/riscv/mm/hugetlbpage.c renamed from arch/riscv/mm/hugetlbpage.c~hugetlbfs-remove-hugetlb_add_hstate-warning-for-existing-hstate +1 -1
@@ -26,7 +26,7 @@ bool __init arch_hugetlb_valid_size(unsistatic __init int gigantic_pages_init (void ) {/* With CONTIG_ALLOC, we can allocate gigantic pages at runtime */ -if (IS_ENABLED (CONFIG_64BIT ) && !size_to_hstate (1UL <<PUD_SHIFT )) +if (IS_ENABLED (CONFIG_64BIT ))hugetlb_add_hstate (PUD_SHIFT -PAGE_SHIFT );return 0 ; }
arch/sparc/mm/init_64.c renamed from arch/sparc/mm/init_64.c~hugetlbfs-remove-hugetlb_add_hstate-warning-for-existing-hstate +4 -15
@@ -325,23 +325,12 @@ static void __update_mmu_tsb_insert(stru }#ifdef CONFIG_HUGETLB_PAGE -static void __init add_huge_page_size (unsigned long size) -{ -unsigned int order; - -if (size_to_hstate (size)) -return ; - - order =ilog2 (size) -PAGE_SHIFT ; -hugetlb_add_hstate (order); -} -static int __init hugetlbpage_init (void ) { -add_huge_page_size (1UL <<HPAGE_64K_SHIFT ); -add_huge_page_size (1UL <<HPAGE_SHIFT ); -add_huge_page_size (1UL <<HPAGE_256MB_SHIFT ); -add_huge_page_size (1UL <<HPAGE_2GB_SHIFT ); +hugetlb_add_hstate (HPAGE_64K_SHIFT -PAGE_SHIFT ); +hugetlb_add_hstate (HPAGE_SHIFT -PAGE_SHIFT ); +hugetlb_add_hstate (HPAGE_256MB_SHIFT -PAGE_SHIFT ); +hugetlb_add_hstate (HPAGE_2GB_SHIFT -PAGE_SHIFT );return 0 ; }
arch/x86/mm/hugetlbpage.c renamed from arch/x86/mm/hugetlbpage.c~hugetlbfs-remove-hugetlb_add_hstate-warning-for-existing-hstate +1 -1
@@ -195,7 +195,7 @@ bool __init arch_hugetlb_valid_size(unsistatic __init int gigantic_pages_init (void ) {/* With compaction or CMA we can allocate gigantic pages at runtime */ -if (boot_cpu_has (X86_FEATURE_GBPAGES ) && !size_to_hstate (1UL <<PUD_SHIFT )) +if (boot_cpu_has (X86_FEATURE_GBPAGES ))hugetlb_add_hstate (PUD_SHIFT -PAGE_SHIFT );return 0 ; }
mm/hugetlb.c renamed from mm/hugetlb.c~hugetlbfs-remove-hugetlb_add_hstate-warning-for-existing-hstate +6 -3
@@ -3222,8 +3222,7 @@ static int __init hugetlb_init(void) } default_hstate_size =HPAGE_SIZE ; -if (!size_to_hstate (default_hstate_size)) -hugetlb_add_hstate (HUGETLB_PAGE_ORDER ); +hugetlb_add_hstate (HUGETLB_PAGE_ORDER ); } default_hstate_idx =hstate_index (size_to_hstate (default_hstate_size));if (default_hstate_max_huge_pages) {
@@ -3268,7 +3267,6 @@ void __init hugetlb_add_hstate(unsignedunsigned long i;if (size_to_hstate (PAGE_SIZE << order)) { -pr_warn ("hugepagesz= specified twice, ignoring\n" );return ; }BUG_ON (hugetlb_max_hstate >=HUGE_MAX_HSTATE );
@@ -3343,6 +3341,11 @@ static int __init hugepagesz_setup(charreturn 0 ; } +if (size_to_hstate (size)) { +pr_warn ("HugeTLB: hugepagesz %s specified twice, ignoring\n" , s); +return 0 ; + } +hugetlb_add_hstate (ilog2 (size) -PAGE_SHIFT );return 1 ; }
_
Andrew Morton
maintainerPATCH 72/131 hugetlbfs: clean up command line processing
In reply to Andrew MortonFrom: Mike Kravetz <[email protected]>
Subject: hugetlbfs: clean up command line processing
With all hugetlb page processing done in a single file clean up code.
- Make code match desired semantics
- Update documentation with semantics
- Make all warnings and errors messages start with 'HugeTLB:'.
- Consistently name command line parsing routines.
- Warn if !hugepages_supported() and command line parameters have been specified.
- Add comments to code
- Describe some of the subtle interactions
- Describe semantics of command line arguments
This patch also fixes issues with implicitly setting the number of gigantic huge pages to preallocate. Previously on X86 command line,
hugepages=2 default_hugepagesz=1G
would result in zero 1G pages being preallocated and,
# grep HugePages_Total /proc/meminfo
HugePages_Total: 0
# sysctl -a | grep nr_hugepages
vm.nr_hugepages = 2
vm.nr_hugepages_mempolicy = 2
# cat /proc/sys/vm/nr_hugepages
2After this patch 2 gigantic pages will be preallocated and all the proc, sysfs, sysctl and meminfo files will accurately reflect this.
To address the issue with gigantic pages, a small change in behavior was made to command line processing. Previously the command line,
hugepages=128 default_hugepagesz=2M hugepagesz=2M hugepages=256
would result in the allocation of 256 2M huge pages. The value 128 would be ignored without any warning. After this patch, 128 2M pages will be allocated and a warning message will be displayed indicating the value of 256 is ignored. This change in behavior is required because allocation of implicitly specified gigantic pages must be done when the default_hugepagesz= is encountered for gigantic pages. Previously the code waited until later in the boot process (hugetlb_init), to allocate pages of default size. However the bootmem allocator required for gigantic allocations is not available at this time.
---
Documentation/admin-guide/kernel-parameters.txt | 40 ++- Documentation/admin-guide/mm/hugetlbpage.rst | 35 +++ mm/hugetlb.c | 159 +++++++++++--- 3 files changed, 190 insertions(+), 44 deletions(-)
Documentation/admin-guide/kernel-parameters.txt renamed from Documentation/admin-guide/kernel-parameters.txt~hugetlbfs-clean-up-command-line-processing +27 -13
@@ -834,12 +834,15 @@ See also Documentation/networking/decnet.txt. default_hugepagesz= - [same as hugepagesz=] The size of the default - HugeTLB page size. This is the size represented by - the legacy /proc/ hugepages APIs, used for SHM, and - default size when mounting hugetlbfs filesystems. - Defaults to the default architecture's huge page size - if not specified. + [HW] The size of the default HugeTLB page. This is + the size represented by the legacy /proc/ hugepages + APIs. In addition, this is the default hugetlb size + used for shmget(), mmap() and mounting hugetlbfs + filesystems. If not specified, defaults to the + architecture's default huge page size. Huge page + sizes are architecture dependent. See also + Documentation/admin-guide/mm/hugetlbpage.rst. + Format: size[KMG] deferred_probe_timeout= [KNL] Debugging option to set a timeout in seconds for
@@ -1479,13 +1482,24 @@ hugepages using the cma allocator. If enabled, the boot-time allocation of gigantic hugepages is skipped. - hugepages= [HW,X86-32,IA-64] HugeTLB pages to allocate at boot. - hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages. - On x86-64 and powerpc, this option can be specified - multiple times interleaved with hugepages= to reserve - huge pages of different sizes. Valid pages sizes on - x86-64 are 2M (when the CPU supports "pse") and 1G - (when the CPU supports the "pdpe1gb" cpuinfo flag). + hugepages= [HW] Number of HugeTLB pages to allocate at boot. + If this follows hugepagesz (below), it specifies + the number of pages of hugepagesz to be allocated. + If this is the first HugeTLB parameter on the command + line, it specifies the number of pages to allocate for + the default huge page size. See also + Documentation/admin-guide/mm/hugetlbpage.rst. + Format: <integer> + + hugepagesz= + [HW] The size of the HugeTLB pages. This is used in + conjunction with hugepages (above) to allocate huge + pages of a specific size at boot. The pair + hugepagesz=X hugepages=Y can be specified once for + each supported huge page size. Huge page sizes are + architecture dependent. See also + Documentation/admin-guide/mm/hugetlbpage.rst. + Format: size[KMG] hung_task_panic= [KNL] Should the hung task detector generate panics.
Documentation/admin-guide/mm/hugetlbpage.rst renamed from Documentation/admin-guide/mm/hugetlbpage.rst~hugetlbfs-clean-up-command-line-processing +35 -0
@@ -100,6 +100,41 @@ with a huge page size selection paramete be specified in bytes with optional scale suffix [kKmMgG]. The default huge page size may be selected with the "default_hugepagesz=<size>" boot parameter. +Hugetlb boot command line parameter semantics +hugepagesz - Specify a huge page size. Used in conjunction with hugepages + parameter to preallocate a number of huge pages of the specified + size. Hence, hugepagesz and hugepages are typically specified in + pairs such as: + hugepagesz=2M hugepages=512 + hugepagesz can only be specified once on the command line for a + specific huge page size. Valid huge page sizes are architecture + dependent. +hugepages - Specify the number of huge pages to preallocate. This typically + follows a valid hugepagesz or default_hugepagesz parameter. However, + if hugepages is the first or only hugetlb command line parameter it + implicitly specifies the number of huge pages of default size to + allocate. If the number of huge pages of default size is implicitly + specified, it can not be overwritten by a hugepagesz,hugepages + parameter pair for the default size. + For example, on an architecture with 2M default huge page size: + hugepages=256 hugepagesz=2M hugepages=512 + will result in 256 2M huge pages being allocated and a warning message + indicating that the hugepages=512 parameter is ignored. If a hugepages + parameter is preceded by an invalid hugepagesz parameter, it will + be ignored. +default_hugepagesz - Specify the default huge page size. This parameter can + only be specified once on the command line. default_hugepagesz can + optionally be followed by the hugepages parameter to preallocate a + specific number of huge pages of default size. The number of default + sized huge pages to preallocate can also be implicitly specified as + mentioned in the hugepages section above. Therefore, on an + architecture with 2M default huge page size: + hugepages=256 + default_hugepagesz=2M hugepages=256 + hugepages=256 default_hugepagesz=2M + will all result in 256 2M huge pages being allocated. Valid default + huge page size is architecture dependent. + When multiple huge page sizes are supported, ``/proc/sys/vm/nr_hugepages`` indicates the current number of pre-allocated huge pages of the default size. Thus, one can use the following command to dynamically allocate/deallocate
mm/hugetlb.c renamed from mm/hugetlb.c~hugetlbfs-clean-up-command-line-processing +128 -31
239 lines not on this page · raw message
_
Andrew Morton
maintainerPATCH 73/131 hugetlbfs: fix changes to command line processing
In reply to Andrew MortonFrom: Mike Kravetz <[email protected]>
Subject: hugetlbfs: fix changes to command line processing
Previously, a check for hugepages_supported was added before processing hugetlb command line parameters. On some architectures such as powerpc, hugepages_supported() is not set to true until after command line processing. Therefore, no hugetlb command line parameters would be accepted.
Remove the additional checks for hugepages_supported. In hugetlb_init, print a warning if !hugepages_supported and command line parameters were specified.
---
mm/hugetlb.c | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)
mm/hugetlb.c renamed from mm/hugetlb.c~hugetlbfs-move-hugepagesz=-parsing-to-arch-independent-code-fix +4 -15
@@ -3212,8 +3212,11 @@ static int __init hugetlb_init(void) {int i; -if (!hugepages_supported ()) +if (!hugepages_supported ()) { +if (hugetlb_max_hstate || default_hstate_max_huge_pages) +pr_warn ("HugeTLB: huge pages not supported, ignoring associated command-line parameters\n" );return 0 ; + }/* * Make sure HPAGE_SIZE (HUGETLB_PAGE_ORDER) hstate exists. Some
@@ -3315,11 +3318,6 @@ static int __init hugepages_setup(char *unsigned long *mhp;static unsigned long *last_mhp; -if (!hugepages_supported ()) { -pr_warn ("HugeTLB: huge pages not supported, ignoring hugepages = %s\n" , s); -return 0 ; - } -if (!parsed_valid_hugepagesz) {pr_warn ("HugeTLB: hugepages=%s does not follow a valid hugepagesz, ignoring\n" , s); parsed_valid_hugepagesz = true;
@@ -3372,11 +3370,6 @@ static int __init hugepagesz_setup(charstruct hstate *h; parsed_valid_hugepagesz = false; -if (!hugepages_supported ()) { -pr_warn ("HugeTLB: huge pages not supported, ignoring hugepagesz = %s\n" , s); -return 0 ; - } - size = (unsigned long )memparse (s,NULL );if (!arch_hugetlb_valid_size (size)) {
@@ -3424,11 +3417,6 @@ static int __init default_hugepagesz_setunsigned long size; parsed_valid_hugepagesz = false; -if (!hugepages_supported ()) { -pr_warn ("HugeTLB: huge pages not supported, ignoring default_hugepagesz = %s\n" , s); -return 0 ; - }
Andrew Morton
maintainerPATCH 74/131 mm/hugetlb: avoid unnecessary check on pud and pmd entry in huge_pte_offset
In reply to Andrew MortonFrom: Li Xinhai <[email protected]>
Subject: mm/hugetlb: avoid unnecessary check on pud and pmd entry in huge_pte_offset
When huge_pte_offset() is called, the parameter sz can only be PUD_SIZE or PMD_SIZE. If sz is PUD_SIZE and code can reach pud, then *pud must be none, or normal hugetlb entry, or non-present (migration or hwpoisoned) hugetlb entry, and we can directly return pud. When sz is PMD_SIZE, pud must be none or present, and if code can reach pmd, we can directly return pmd.
So after this patch the code is simplified by first check on the parameter sz, and avoid unnecessary checks in current code. Same semantics of existing code is maintained.
More details about relevant commits:
commit 9b19df292c66 ("mm/hugetlb.c: make huge_pte_offset() consistent and document behaviour") changed the code path for pud and pmd handling, see comments about why this patch intends to change it.
...
pud = pud_offset(p4d, addr); if (sz != PUD_SIZE && pud_none(*pud)) // [1] return NULL; /* hugepage or swap? */ if (pud_huge(*pud) || !pud_present(*pud)) // [2] return (pte_t *)pud; pmd = pmd_offset(pud, addr); if (sz != PMD_SIZE && pmd_none(*pmd)) // [3] return NULL; /* hugepage or swap? */ if (pmd_huge(*pmd) || !pmd_present(*pmd)) // [4] return (pte_t *)pmd; return NULL; // [5]
...
[1]: this is necessary, return NULL for sz == PMD_SIZE;
[2]: if sz == PUD_SIZE, all valid values of pud entry will cause return; [3]: dead code, sz != PMD_SIZE never true;
[4]: all valid values of pmd entry will cause return;
[5]: dead code, because of check in [4].
Now, this patch combines [1] and [2] for pud, and combines [3], [4] and
[5] for pmd, so avoid unnecessary checks.
I don't try to catch any invalid values in page table entry, as that will be checked by caller and avoid extra branch in this function. Also no assert on sz must equal PUD_SIZE or PMD_SIZE, since this function only call for hugetlb mapping.
For commit 3c1d7e6ccb64 ("mm/hugetlb: fix a addressing exception caused by huge_pte_offset"), since we don't read the entry more than once now, variable pud_entry and pmd_entry are not needed.
---
mm/hugetlb.c | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-)
mm/hugetlb.c renamed from mm/hugetlb.c~mm-hugetlb-avoid-unnecessary-check-on-pud-and-pmd-entry-in-huge_pte_offset +6 -9
@@ -5469,8 +5469,8 @@ pte_t *huge_pte_alloc(struct mm_struct ** huge_pte_offset() - Walk the page table to resolve the hugepage * entry at address @addr * -* Return: Pointer to page table or swap entry (PUD or PMD) for -* address @addr, or NULL if a p*d_none() entry is encountered and the +* Return: Pointer to page table entry (PUD or PMD) for +* address @addr, or NULL if a !p*d_present() entry is encountered and the * size @sz doesn't match the hugepage size at this level of the page * table. */
@@ -5479,8 +5479,8 @@ pte_t *huge_pte_offset(struct mm_struct {pgd_t *pgd;p4d_t *p4d; -pud_t *pud, pud_entry; -pmd_t *pmd, pmd_entry; +pud_t *pud; +pmd_t *pmd; pgd =pgd_offset (mm, addr);if (!pgd_present (*pgd))
@@ -5490,22 +5490,16 @@ pte_t *huge_pte_offset(struct mm_structreturn NULL ; pud =pud_offset (p4d, addr); - pud_entry =READ_ONCE (*pud); -if (sz !=PUD_SIZE &&pud_none (pud_entry)) -return NULL ; -/* hugepage or swap? */ -if (pud_huge (pud_entry) || !pud_present (pud_entry)) +if (sz ==PUD_SIZE ) +/* must be pud huge, non-present or none */ return (pte_t *)pud;
Andrew Morton
maintainerPATCH 75/131 arm64/mm: drop __HAVE_ARCH_HUGE_PTEP_GET
In reply to Andrew MortonFrom: Anshuman Khandual <[email protected]>
Subject: arm64/mm: drop __HAVE_ARCH_HUGE_PTEP_GET
Patch series "mm/hugetlb: Add some new generic fallbacks", v3.
This series adds the following new generic fallbacks. Before that it drops __HAVE_ARCH_HUGE_PTEP_GET from arm64 platform.
1. is_hugepage_only_range()
2. arch_clear_hugepage_flags()
After this arm (32 bit) remains the sole platform defining it's own huge_ptep_get() via __HAVE_ARCH_HUGE_PTEP_GET.
This patch (of 3):
Platform specific huge_ptep_get() is required only when fetching the huge PTE involves more than just dereferencing the page table pointer. This is not the case on arm64 platform. Hence huge_ptep_pte() can be dropped along with it's __HAVE_ARCH_HUGE_PTEP_GET subscription. Before that, it updates the generic huge_ptep_get() with READ_ONCE() which will prevent known page table issues with THP on arm64.
---
arch/arm64/include/asm/hugetlb.h | 6 ------ include/asm-generic/hugetlb.h | 2 +- 2 files changed, 1 insertion(+), 7 deletions(-)
arch/arm64/include/asm/hugetlb.h renamed from arch/arm64/include/asm/hugetlb.h~arm64-mm-drop-__have_arch_huge_ptep_get +0 -5
@@ -17,12 +17,6 @@extern bool arch_hugetlb_migration_supported (struct hstate *h);#endif -#define __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get (pte_t *ptep) -{ -return READ_ONCE (*ptep); -}
Andrew Morton
maintainerPATCH 76/131 mm/hugetlb: define a generic fallback for is_hugepage_only_range()
In reply to Andrew MortonFrom: Anshuman Khandual <[email protected]>
Subject: mm/hugetlb: define a generic fallback for is_hugepage_only_range()
There are multiple similar definitions for is_hugepage_only_range() on various platforms. Lets just add it's generic fallback definition for platforms that do not override. This help reduce code duplication.
---
arch/arm/include/asm/hugetlb.h | 6 ------ arch/arm64/include/asm/hugetlb.h | 6 ------ arch/ia64/include/asm/hugetlb.h | 1 + arch/mips/include/asm/hugetlb.h | 7 ------- arch/parisc/include/asm/hugetlb.h | 6 ------ arch/powerpc/include/asm/hugetlb.h | 1 + arch/riscv/include/asm/hugetlb.h | 6 ------ arch/s390/include/asm/hugetlb.h | 7 ------- arch/sh/include/asm/hugetlb.h | 6 ------ arch/sparc/include/asm/hugetlb.h | 6 ------ arch/x86/include/asm/hugetlb.h | 6 ------ include/linux/hugetlb.h | 9 +++++++++ 12 files changed, 11 insertions(+), 56 deletions(-)
arch/arm64/include/asm/hugetlb.h renamed from arch/arm64/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -6
@@ -17,12 +17,6 @@extern bool arch_hugetlb_migration_supported (struct hstate *h);#endif -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr,unsigned long len) -{ -return 0 ; -} -static inline void arch_clear_hugepage_flags (struct page *page) {clear_bit (PG_dcache_clean, &page->flags);
arch/arm/include/asm/hugetlb.h renamed from arch/arm/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -6
@@ -14,12 +14,6 @@#include <asm/hugetlb-3level.h> #include <asm-generic/hugetlb.h> -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr,unsigned long len) -{ -return 0 ; -} -static inline void arch_clear_hugepage_flags (struct page *page) {clear_bit (PG_dcache_clean, &page->flags);
arch/ia64/include/asm/hugetlb.h renamed from arch/ia64/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +1 -0
@@ -20,6 +20,7 @@ static inline int is_hugepage_only_rangereturn (REGION_NUMBER (addr) ==RGN_HPAGE ||REGION_NUMBER ((addr)+(len)-1 ) ==RGN_HPAGE ); } +#define is_hugepage_only_range is_hugepage_only_range#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH static inline void huge_ptep_clear_flush(struct vm_area_struct *vma,
arch/mips/include/asm/hugetlb.h renamed from arch/mips/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -7
@@ -11,13 +11,6 @@#include <asm/page.h> -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr, -unsigned long len) -{ -return 0 ; -} -#define __HAVE_ARCH_PREPARE_HUGEPAGE_RANGE static inline int prepare_hugepage_range(struct file *file, unsigned long addr,
arch/parisc/include/asm/hugetlb.h renamed from arch/parisc/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -6
@@ -12,12 +12,6 @@ void set_huge_pte_at(struct mm_struct *mpte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep); -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr, -unsigned long len) { -return 0 ; -} -/* * If the arch doesn't supply something else, assume that hugepage * size aligned regions are ok without further preparation.
arch/powerpc/include/asm/hugetlb.h renamed from arch/powerpc/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +1 -0
@@ -30,6 +30,7 @@ static inline int is_hugepage_only_rangereturn slice_is_hugepage_only_range (mm, addr, len);return 0 ; } +#define is_hugepage_only_range is_hugepage_only_range#define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr,
arch/riscv/include/asm/hugetlb.h renamed from arch/riscv/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -6
@@ -5,12 +5,6 @@#include <asm-generic/hugetlb.h> #include <asm/page.h> -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr, -unsigned long len) { -return 0 ; -} -static inline void arch_clear_hugepage_flags (struct page *page) { }
arch/s390/include/asm/hugetlb.h renamed from arch/s390/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -7
@@ -21,13 +21,6 @@ pte_t huge_ptep_get(pte_t *ptep);pte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep); -static inline bool is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr, -unsigned long len) -{ -return false; -} -/* * If the arch doesn't supply something else, assume that hugepage * size aligned regions are ok without further preparation.
arch/sh/include/asm/hugetlb.h renamed from arch/sh/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -6
@@ -5,12 +5,6 @@#include <asm/cacheflush.h> #include <asm/page.h> -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr, -unsigned long len) { -return 0 ; -} -/* * If the arch doesn't supply something else, assume that hugepage * size aligned regions are ok without further preparation.
arch/sparc/include/asm/hugetlb.h renamed from arch/sparc/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -6
@@ -20,12 +20,6 @@ void set_huge_pte_at(struct mm_struct *mpte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep); -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr, -unsigned long len) { -return 0 ; -} -#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSHstatic inline void huge_ptep_clear_flush (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep)
arch/x86/include/asm/hugetlb.h renamed from arch/x86/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-is_hugepage_only_range +0 -5
@@ -7,12 +7,6 @@#define hugepages_supported () boot_cpu_has(X86_FEATURE_PSE) -static inline int is_hugepage_only_range (struct mm_struct *mm, -unsigned long addr, -unsigned long len) { -return 0 ; -}
Andrew Morton
maintainerPATCH 77/131 mm/hugetlb: define a generic fallback for arch_clear_hugepage_flags()
In reply to Andrew MortonFrom: Anshuman Khandual <[email protected]>
Subject: mm/hugetlb: define a generic fallback for arch_clear_hugepage_flags()
There are multiple similar definitions for arch_clear_hugepage_flags() on various platforms. Lets just add it's generic fallback definition for platforms that do not override. This help reduce code duplication.
---
arch/arm/include/asm/hugetlb.h | 1 + arch/arm64/include/asm/hugetlb.h | 1 + arch/ia64/include/asm/hugetlb.h | 4 ---- arch/mips/include/asm/hugetlb.h | 4 ---- arch/parisc/include/asm/hugetlb.h | 4 ---- arch/powerpc/include/asm/hugetlb.h | 4 ---- arch/riscv/include/asm/hugetlb.h | 4 ---- arch/s390/include/asm/hugetlb.h | 1 + arch/sh/include/asm/hugetlb.h | 1 + arch/sparc/include/asm/hugetlb.h | 4 ---- arch/x86/include/asm/hugetlb.h | 4 ---- include/linux/hugetlb.h | 5 +++++ 12 files changed, 9 insertions(+), 28 deletions(-)
arch/arm64/include/asm/hugetlb.h renamed from arch/arm64/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +1 -0
@@ -21,6 +21,7 @@ static inline void arch_clear_hugepage_f {clear_bit (PG_dcache_clean, &page->flags); } +#define arch_clear_hugepage_flags arch_clear_hugepage_flagsextern pte_t arch_make_huge_pte (pte_t entry,struct vm_area_struct *vma,struct page *page,int writable);
arch/arm/include/asm/hugetlb.h renamed from arch/arm/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +1 -0
@@ -18,5 +18,6 @@ static inline void arch_clear_hugepage_f {clear_bit (PG_dcache_clean, &page->flags); } +#define arch_clear_hugepage_flags arch_clear_hugepage_flags#endif /* _ASM_ARM_HUGETLB_H */
arch/ia64/include/asm/hugetlb.h renamed from arch/ia64/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +0 -4
@@ -28,10 +28,6 @@ static inline void huge_ptep_clear_flush { } -static inline void arch_clear_hugepage_flags (struct page *page) -{ -} -#include <asm-generic/hugetlb.h> #endif /* _ASM_IA64_HUGETLB_H */
arch/mips/include/asm/hugetlb.h renamed from arch/mips/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +0 -4
@@ -75,10 +75,6 @@ static inline int huge_ptep_set_access_f return changed; } -static inline void arch_clear_hugepage_flags (struct page *page) -{ -} -#include <asm-generic/hugetlb.h> #endif /* __ASM_HUGETLB_H */
arch/parisc/include/asm/hugetlb.h renamed from arch/parisc/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +0 -4
@@ -42,10 +42,6 @@ int huge_ptep_set_access_flags(struct vm unsigned long addr, pte_t *ptep, pte_t pte, int dirty); -static inline void arch_clear_hugepage_flags (struct page *page) -{ -} -#include <asm-generic/hugetlb.h> #endif /* _ASM_PARISC64_HUGETLB_H */
arch/powerpc/include/asm/hugetlb.h renamed from arch/powerpc/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +0 -4
@@ -61,10 +61,6 @@ int huge_ptep_set_access_flags(struct vm unsigned long addr, pte_t *ptep, pte_t pte, int dirty); -static inline void arch_clear_hugepage_flags (struct page *page) -{ -} -#include <asm-generic/hugetlb.h> #else /* ! CONFIG_HUGETLB_PAGE */
arch/riscv/include/asm/hugetlb.h renamed from arch/riscv/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +0 -4
@@ -5,8 +5,4 @@#include <asm-generic/hugetlb.h> #include <asm/page.h> -static inline void arch_clear_hugepage_flags (struct page *page) -{ -} -#endif /* _ASM_RISCV_HUGETLB_H */
arch/s390/include/asm/hugetlb.h renamed from arch/s390/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +1 -0
@@ -39,6 +39,7 @@ static inline void arch_clear_hugepage_f {clear_bit (PG_arch_1, &page->flags); } +#define arch_clear_hugepage_flags arch_clear_hugepage_flagsstatic inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz)
arch/sh/include/asm/hugetlb.h renamed from arch/sh/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +1 -0
@@ -30,6 +30,7 @@ static inline void arch_clear_hugepage_f {clear_bit (PG_dcache_clean, &page->flags); } +#define arch_clear_hugepage_flags arch_clear_hugepage_flags#include <asm-generic/hugetlb.h>
arch/sparc/include/asm/hugetlb.h renamed from arch/sparc/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +0 -4
@@ -47,10 +47,6 @@ static inline int huge_ptep_set_access_freturn changed; } -static inline void arch_clear_hugepage_flags (struct page *page) -{ -} -#define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor,
arch/x86/include/asm/hugetlb.h renamed from arch/x86/include/asm/hugetlb.h~mm-hugetlb-define-a-generic-fallback-for-arch_clear_hugepage_flags +0 -3
@@ -7,8 +7,4 @@#define hugepages_supported () boot_cpu_has(X86_FEATURE_PSE) -static inline void arch_clear_hugepage_flags (struct page *page) -{ -}
Andrew Morton
maintainerPATCH 78/131 mm: simplify calling a compound page destructor
In reply to Andrew MortonFrom: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm: simplify calling a compound page destructor
None of the three callers of get_compound_page_dtor() want to know the value; they just want to call the function. Replace it with destroy_compound_page() which calls the dtor for them.
---
include/linux/mm.h | 4 ++-- mm/swap.c | 5 +---- mm/vmscan.c | 4 ++-- 3 files changed, 5 insertions(+), 8 deletions(-)
include/linux/mm.h renamed from include/linux/mm.h~mm-simplify-calling-a-compound-page-destructor +2 -2
@@ -876,10 +876,10 @@ static inline void set_compound_page_dto page[1 ].compound_dtor = compound_dtor; } -static inline compound_page_dtor *get_compound_page_dtor (struct page *page) +static inline void destroy_compound_page (struct page *page) {VM_BUG_ON_PAGE (page[1 ].compound_dtor >=NR_COMPOUND_DTORS , page); -return compound_page_dtors[page[1 ].compound_dtor]; + compound_page_dtors[page[1 ].compound_dtor](page); }static inline unsigned int compound_order (struct page *page)
mm/swap.c renamed from mm/swap.c~mm-simplify-calling-a-compound-page-destructor +1 -4
@@ -102,8 +102,6 @@ static void __put_single_page(struct pagstatic void __put_compound_page (struct page *page) { -compound_page_dtor *dtor; -/* * __page_cache_release() is supposed to be called for thp, not for * hugetlb. This is because hugetlb page does never have PageLRU set
@@ -112,8 +110,7 @@ static void __put_compound_page(struct p*/ if (!PageHuge (page))__page_cache_release (page); - dtor =get_compound_page_dtor (page); - (*dtor)(page); +destroy_compound_page (page); }void __put_page (struct page *page)
mm/vmscan.c renamed from mm/vmscan.c~mm-simplify-calling-a-compound-page-destructor +2 -2
@@ -1438,7 +1438,7 @@ free_it:* appear not as the counts should be low */ if (unlikely (PageTransHuge (page))) - (*get_compound_page_dtor (page))(page); +destroy_compound_page (page);else list_add (&page->lru, &free_pages);continue ;
@@ -1859,7 +1859,7 @@ static unsigned noinline_for_stack move_if (unlikely (PageCompound (page))) {spin_unlock_irq (&pgdat->lru_lock); - (*get_compound_page_dtor (page))(page); +destroy_compound_page (page);spin_lock_irq (&pgdat->lru_lock); }else list_add (&page->lru, &pages_to_free);
_
Andrew Morton
maintainerPATCH 79/131 mm/vmscan.c: use update_lru_size() in update_lru_sizes()
In reply to Andrew MortonFrom: Wei Yang <[email protected]>
Subject: mm/vmscan.c: use update_lru_size() in update_lru_sizes()
We already defined the helper update_lru_size().
Let's use this to reduce code duplication.
---
mm/vmscan.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscanc-use-update_lru_size-in-update_lru_sizes +1 -4
@@ -1602,10 +1602,7 @@ static __always_inline void update_lru_sif (!nr_zone_taken[zid])continue ; -__update_lru_size (lruvec, lru, zid, -nr_zone_taken[zid]); -#ifdef CONFIG_MEMCG -mem_cgroup_update_lru_size (lruvec, lru, zid, -nr_zone_taken[zid]); -#endif +update_lru_size (lruvec, lru, zid, -nr_zone_taken[zid]); } }
_
Andrew Morton
maintainerPATCH 80/131 mm/vmscan: count layzfree pages and fix nr_isolated_* mismatch
In reply to Andrew MortonFrom: Jaewon Kim <[email protected]>
Subject: mm/vmscan: count layzfree pages and fix nr_isolated_* mismatch
Fix an nr_isolate_* mismatch problem between cma and dirty lazyfree pages.
If try_to_unmap_one is used for reclaim and it detects a dirty lazyfree page, then the lazyfree page is changed to a normal anon page having SwapBacked by commit 802a3a92ad7a ("mm: reclaim MADV_FREE pages"). Even with the change, reclaim context correctly counts isolated files because it uses is_file_lru to distinguish file. And the change to anon is not happened if try_to_unmap_one is used for migration. So migration context like compaction also correctly counts isolated files even though it uses page_is_file_lru insted of is_file_lru. Recently page_is_file_cache was renamed to page_is_file_lru by commit 9de4f22a60f7 ("mm: code cleanup for MADV_FREE").
But the nr_isolate_* mismatch problem happens on cma alloc. There is reclaim_clean_pages_from_list which is being used only by cma. It was introduced by commit 02c6de8d757c ("mm: cma: discard clean pages during contiguous allocation instead of migration") to reclaim clean file pages without migration. The cma alloc uses both reclaim_clean_pages_from_list and migrate_pages, and it uses page_is_file_lru to count isolated files. If there are dirty lazyfree pages allocated from cma memory region, the pages are counted as isolated file at the beginging but are counted as isolated anon after finished.
Mem-Info:
Node 0 active_anon:3045904kB inactive_anon:611448kB active_file:14892kB inactive_file:205636kB unevictable:10416kB isolated(anon):0kB isolated(file):37664kB mapped:630216kB dirty:384kB writeback:0kB shmem:42576kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Like log above, there were too much isolated files, 37664kB, which triggers too_many_isolated in reclaim even when there is no actually isolated file in system wide. It could be reproducible by running two programs, writing on MADV_FREE page and doing cma alloc, respectively. Although isolated anon is 0, I found that the internal value of isolated anon was the negative value of isolated file.
Fix this by compensating the isolated count for both LRU lists. Count non-discarded lazyfree pages in shrink_page_list, then compensate the counted number in reclaim_clean_pages_from_list.
---
include/linux/vmstat.h | 1 + mm/vmscan.c | 26 ++++++++++++++++++++------ 2 files changed, 21 insertions(+), 6 deletions(-)
include/linux/vmstat.h renamed from include/linux/vmstat.h~mm-vmscan-count-layzfree-pages-and-fix-nr_isolated_-mismatch +1 -0
@@ -29,6 +29,7 @@ struct reclaim_stat {unsigned nr_activate[2 ];unsigned nr_ref_keep;unsigned nr_unmap_fail; +unsigned nr_lazyfree_fail; };enum writeback_stat_item {
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-count-layzfree-pages-and-fix-nr_isolated_-mismatch +20 -6
@@ -1295,11 +1295,15 @@ static unsigned long shrink_page_list(st*/ if (page_mapped (page)) {enum ttu_flags flags = ttu_flags |TTU_BATCH_FLUSH ; +bool was_swapbacked =PageSwapBacked (page);if (unlikely (PageTransHuge (page))) flags |=TTU_SPLIT_HUGE_PMD ; +if (!try_to_unmap (page, flags)) { stat->nr_unmap_fail += nr_pages; +if (!was_swapbacked &&PageSwapBacked (page)) + stat->nr_lazyfree_fail += nr_pages; goto activate_locked; } }
@@ -1491,8 +1495,8 @@ unsigned long reclaim_clean_pages_from_l .priority = DEF_PRIORITY, .may_unmap = 1, }; -struct reclaim_stat dummy_stat; -unsigned long ret; +struct reclaim_stat stat; +unsigned long nr_reclaimed;struct page *page, *next;LIST_HEAD (clean_pages);
@@ -1504,11 +1508,21 @@ unsigned long reclaim_clean_pages_from_l } } - ret =shrink_page_list (&clean_pages, zone->zone_pgdat, &sc, -TTU_IGNORE_ACCESS , &dummy_stat, true); + nr_reclaimed =shrink_page_list (&clean_pages, zone->zone_pgdat, &sc, +TTU_IGNORE_ACCESS , &stat, true);list_splice (&clean_pages, page_list); -mod_node_page_state (zone->zone_pgdat,NR_ISOLATED_FILE , -ret); -return ret; +mod_node_page_state (zone->zone_pgdat,NR_ISOLATED_FILE , -nr_reclaimed); +/* +* Since lazyfree pages are isolated from file LRU from the beginning, +* they will rotate back to anonymous LRU in the end if it failed to +* discard so isolated count will be mismatched. +* Compensate the isolated count for both LRU lists. +*/ +mod_node_page_state (zone->zone_pgdat,NR_ISOLATED_ANON , + stat.nr_lazyfree_fail); +mod_node_page_state (zone->zone_pgdat,NR_ISOLATED_FILE , + -stat.nr_lazyfree_fail); +return nr_reclaimed; }/*
_
Andrew Morton
maintainerPATCH 81/131 mm/vmscan.c: change prototype for shrink_page_list
In reply to Andrew MortonFrom: Maninder Singh <[email protected]>
Subject: mm/vmscan.c: change prototype for shrink_page_list
commit 3c710c1ad11b ("mm, vmscan extract shrink_page_list reclaim counters into a struct") changed data type for the function, so changing return type for funciton and its caller.
---
mm/internal.h | 2 +- mm/page_alloc.c | 2 +- mm/vmscan.c | 24 ++++++++++++------------ 3 files changed, 14 insertions(+), 14 deletions(-)
mm/internal.h renamed from mm/internal.h~mm-vmscanc-change-prototype-for-shrink_page_list +1 -1
@@ -538,7 +538,7 @@ extern unsigned long __must_check vm_mm unsigned long, unsigned long);extern void set_pageblock_order (void ); -unsigned long reclaim_clean_pages_from_list (struct zone *zone, +unsigned int reclaim_clean_pages_from_list (struct zone *zone,struct list_head *page_list);/* The ALLOC_WMARK bits are used as an index to zone->watermark */ #define ALLOC_WMARK_MIN WMARK_MIN
mm/page_alloc.c renamed from mm/page_alloc.c~mm-vmscanc-change-prototype-for-shrink_page_list +1 -1
@@ -8355,7 +8355,7 @@ static int __alloc_contig_migrate_range( unsigned long start, unsigned long end) {/* This function is based on compact_zone() from compaction.c. */ -unsigned long nr_reclaimed; +unsigned int nr_reclaimed;unsigned long pfn = start;unsigned int tries =0 ;int ret =0 ;
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscanc-change-prototype-for-shrink_page_list +12 -12
@@ -1066,17 +1066,17 @@ static void page_check_dirty_writeback(s/* * shrink_page_list() returns the number of reclaimed pages */ -static unsigned long shrink_page_list (struct list_head *page_list, -struct pglist_data *pgdat, -struct scan_control *sc, -enum ttu_flags ttu_flags, -struct reclaim_stat *stat, -bool ignore_references) +static unsigned int shrink_page_list (struct list_head *page_list, +struct pglist_data *pgdat, +struct scan_control *sc, +enum ttu_flags ttu_flags, +struct reclaim_stat *stat, +bool ignore_references) {LIST_HEAD (ret_pages);LIST_HEAD (free_pages); -unsigned nr_reclaimed =0 ; -unsigned pgactivate =0 ; +unsigned int nr_reclaimed =0 ; +unsigned int pgactivate =0 ;memset (stat,0 ,sizeof (*stat));cond_resched ();
@@ -1487,7 +1487,7 @@ keep:return nr_reclaimed; } -unsigned long reclaim_clean_pages_from_list (struct zone *zone, +unsigned int reclaim_clean_pages_from_list (struct zone *zone,struct list_head *page_list) {struct scan_control sc = {
@@ -1496,7 +1496,7 @@ unsigned long reclaim_clean_pages_from_l .may_unmap = 1, };struct reclaim_stat stat; -unsigned long nr_reclaimed; +unsigned int nr_reclaimed;struct page *page, *next;LIST_HEAD (clean_pages);
@@ -1910,7 +1910,7 @@ shrink_inactive_list(unsigned long nr_to {LIST_HEAD (page_list);unsigned long nr_scanned; -unsigned long nr_reclaimed =0 ; +unsigned int nr_reclaimed =0 ;unsigned long nr_taken;struct reclaim_stat stat;int file =is_file_lru (lru);
@@ -2106,7 +2106,7 @@ static void shrink_active_list(unsignedunsigned long reclaim_pages (struct list_head *page_list) {int nid =NUMA_NO_NODE ; -unsigned long nr_reclaimed =0 ; +unsigned int nr_reclaimed =0 ;LIST_HEAD (node_page_list);struct reclaim_stat dummy_stat;struct page *page;
_
Andrew Morton
maintainerPATCH 82/131 mm/vmscan: update the comment of should_continue_reclaim()
In reply to Andrew MortonFrom: Qiwu Chen <[email protected]>
Subject: mm/vmscan: update the comment of should_continue_reclaim()
try_to_compact_zone() has been replaced by try_to_compact_pages(), which is necessary to be updated in the comment of should_continue_reclaim().
---
mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-update-the-comment-of-should_continue_reclaim +1 -1
@@ -2577,7 +2577,7 @@ static bool in_reclaim_compaction(struct* Reclaim/compaction is used for high-order allocation requests. It reclaims * order-0 pages before compacting the zone. should_continue_reclaim() returns * true if more pages should be reclaimed such that when the page allocator -* calls try_to_compact_zone() that it will have enough free pages to succeed. +* calls try_to_compact_pages() that it will have enough free pages to succeed. * It will give up earlier than that if there is difficulty reclaiming pages. */ static inline bool should_continue_reclaim(struct pglist_data *pgdat,
_
Andrew Morton
maintainerPATCH 83/131 mm: fix NUMA node file count error in replace_page_cache()
In reply to Andrew MortonFrom: Johannes Weiner <[email protected]>
Subject: mm: fix NUMA node file count error in replace_page_cache()
Patch series "mm: memcontrol: charge swapin pages on instantiation", v2.
This patch series reworks memcg to charge swapin pages directly at swapin time, rather than at fault time, which may be much later, or not happen at all.
Changes in version 2:
- prevent double charges on pre-allocated hugepages in khugepaged
- leave shmem swapcache when charging fails to avoid double IO (Joonsoo)
- fix temporary accounting bug by switching rmap<->commit (Joonsoo)
- fix double swap charge bug in cgroup1/cgroup2 code gating
- simplify swapin error checking (Joonsoo)
- mm: memcontrol: document the new swap control behavior (Alex)
- review tags
The delayed swapin charging scheme we have right now causes problems:
- Alex's per-cgroup lru_lock patches rely on pages that have been isolated from the LRU to have a stable page->mem_cgroup; otherwise the lock may change underneath him. Swapcache pages are charged only after they are added to the LRU, and charging doesn't follow the LRU isolation protocol.
- Joonsoo's anon workingset patches need a suitable LRU at the time the page enters the swap cache and displaces the non-resident info. But the correct LRU is only available after charging.
- It's a containment hole / DoS vector. Users can trigger arbitrarily large swap readahead using MADV_WILLNEED. The memory is never charged unless somebody actually touches it.
- It complicates the page->mem_cgroup stabilization rules
In order to charge pages directly at swapin time, the memcg code base needs to be prepared, and several overdue cleanups become a necessity:
To charge pages at swapin time, we need to always have cgroup ownership tracking of swap records. We also cannot rely on page->mapping to tell apart page types at charge time, because that's only set up during a page fault.
To eliminate the page->mapping dependency, memcg needs to ditch its private page type counters (MEMCG_CACHE, MEMCG_RSS, NR_SHMEM) in favor of the generic vmstat counters and accounting sites, such as NR_FILE_PAGES, NR_ANON_MAPPED etc.
To switch to generic vmstat counters, the charge sequence must be adjusted such that page->mem_cgroup is set up by the time these counters are modified.
The series is structured as follows:
1. Bug fixes
2. Decoupling charging from rmap
3. Swap controller integration into memcg
4. Direct swapin charging
This patch (of 19):
When replacing one page with another one in the cache, we have to decrease the file count of the old page's NUMA node and increase the one of the new NUMA node, otherwise the old node leaks the count and the new node eventually underflows its counter.
---
mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c renamed from mm/filemap.c~mm-fix-numa-node-file-count-error-in-replace_page_cache +2 -2
@@ -808,11 +808,11 @@ int replace_page_cache_page(struct page old->mapping =NULL ;/* hugetlb pages do not participate in page cache accounting. */ if (!PageHuge (old)) -__dec_node_page_state (new,NR_FILE_PAGES ); +__dec_node_page_state (old,NR_FILE_PAGES );if (!PageHuge (new))__inc_node_page_state (new,NR_FILE_PAGES );if (PageSwapBacked (old)) -__dec_node_page_state (new,NR_SHMEM ); +__dec_node_page_state (old,NR_SHMEM );if (PageSwapBacked (new))__inc_node_page_state (new,NR_SHMEM );xas_unlock_irqrestore (&xas, flags);
_
Andrew Morton
maintainerPATCH 84/131 mm: memcontrol: fix stat-corrupting race in charge moving
In reply to Andrew MortonFrom: Johannes Weiner <[email protected]>
Subject: mm: memcontrol: fix stat-corrupting race in charge moving
The move_lock is a per-memcg lock, but the VM accounting code that needs to acquire it comes from the page and follows page->mem_cgroup under RCU protection. That means that the page becomes unlocked not when we drop the move_lock, but when we update page->mem_cgroup. And that assignment doesn't imply any memory ordering. If that pointer write gets reordered against the reads of the page state - page_mapped, PageDirty etc. the state may change while we rely on it being stable and we can end up corrupting the counters.
Place an SMP memory barrier to make sure we're done with all page state by the time the new page->mem_cgroup becomes visible.
Also replace the open-coded move_lock with a lock_page_memcg() to make it more obvious what we're serializing against.
---
mm/memcontrol.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-fix-stat-corrupting-race-in-charge-moving +14 -12
@@ -5432,7 +5432,6 @@ static int mem_cgroup_move_account(struc {struct lruvec *from_vec, *to_vec;struct pglist_data *pgdat; -unsigned long flags;unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ;int ret;bool anon;
@@ -5459,18 +5458,13 @@ static int mem_cgroup_move_account(struc from_vec =mem_cgroup_lruvec (from, pgdat); to_vec =mem_cgroup_lruvec (to, pgdat); -spin_lock_irqsave (&from->move_lock, flags); +lock_page_memcg (page);if (!anon &&page_mapped (page)) {__mod_lruvec_state (from_vec,NR_FILE_MAPPED , -nr_pages);__mod_lruvec_state (to_vec,NR_FILE_MAPPED , nr_pages); } -/* -* move_lock grabbed above and caller set from->moving_account, so -* mod_memcg_page_state will serialize updates to PageDirty. -* So mapping should be stable for dirty pages. -*/ if (!anon &&PageDirty (page)) {struct address_space *mapping =page_mapping (page);
@@ -5486,15 +5480,23 @@ static int mem_cgroup_move_account(struc }/* +* All state has been migrated, let's switch to the new memcg. +* * It is safe to change page->mem_cgroup here because the page -* is referenced, charged, and isolated - we can't race with -* uncharging, charging, migration, or LRU putback. +* is referenced, charged, isolated, and locked: we can't race +* with (un)charging, migration, LRU putback, or anything else +* that would rely on a stable page->mem_cgroup. +* +* Note that lock_page_memcg is a memcg lock, not a page lock, +* to save space. As soon as we switch page->mem_cgroup to a +* new memcg that isn't locked, the above state can change +* concurrently again. Make sure we're truly done with it. */ +smp_mb (); -/* caller should have done css_get */ - page->mem_cgroup = to; + page->mem_cgroup = to;/* caller should have done css_get */ -spin_unlock_irqrestore (&from->move_lock, flags); +__unlock_page_memcg (from); ret =0 ;
_