mm: move core MM initialization to mm/mm_init.c
Subsystems (8)
Mike Rapoport
maintainerv2 cover mm: move core MM initialization to mm/mm_init.c
From: "Mike Rapoport (IBM)" <[email protected]>
Also in git:
https://git.kernel.org/rppt/h/mm-init/v2
v2:
* move init_cma_reserved_pageblock() from cma.c to mm_init.c
* rename init_mem_debugging_and_hardening() to mem_debugging_and_hardening_init()
* inline pgtable_init() into mem_core_init()
* add Acked and Reviewed tags (thanks David, hopefully I've picked them right)
v1: https://lore.kernel.org/all/[email protected]
This set moves most of the core MM initialization to mm/mm_init.c.
This largely includes free_area_init() and its helpers, functions used at boot time, mm_init() from init/main.c and some of the functions it calls.
Aside from gaining some more space before mm/page_alloc.c hits 10k lines, this makes mm/page_alloc.c to be mostly about buddy allocator and moves the init code out of the way, which IMO improves maintainability.
Besides, this allows to move a couple of declarations out of include/linux and make them private to mm/.
And as an added bonus there a slight decrease in vmlinux size. For tinyconfig and defconfig on x86 I've got
tinyconfig:
text data bss dec hex filename
853206 289376 1200128 2342710 23bf36 a/vmlinux
853198 289344 1200128 2342670 23bf0e b/vmlinux
defconfig:
text data bss dec hex filename
26152959 9730634 2170884 38054477 244aa4d a/vmlinux
26152945 9730602 2170884 38054431 244aa1f b/vmlinux
Mike Rapoport (IBM) (14):
mips: fix comment about pgtable_init()
mm/page_alloc: add helper for checking if check_pages_enabled
mm: move most of core MM initialization to mm/mm_init.c
mm: handle hashdist initialization in mm/mm_init.c mm/page_alloc: rename page_alloc_init() to page_alloc_init_cpuhp()
init: fold build_all_zonelists() and page_alloc_init_cpuhp() to mm_init() init,mm: move mm_init() to mm/mm_init.c and rename it to mm_core_init()
mm: call {ptlock,pgtable}_cache_init() directly from mm_core_init()
mm: move init_mem_debugging_and_hardening() to mm/mm_init.c init,mm: fold late call to page_ext_init() to page_alloc_init_late()
mm: move mem_init_print_info() to mm_init.c
mm: move kmem_cache_init() declaration to mm/slab.h
mm: move vmalloc_init() declaration to mm/internal.h
MAINTAINERS: extend memblock entry to include MM initialization
MAINTAINERS | 3 +- arch/mips/include/asm/fixmap.h | 2 +- include/linux/gfp.h | 7 +- include/linux/mm.h | 9 +- include/linux/page_ext.h | 2 - include/linux/slab.h | 1 - include/linux/vmalloc.h | 4 - init/main.c | 74 +- mm/cma.c | 1 + mm/internal.h | 52 +- mm/mm_init.c | 2547 +++++++++++++++++++++++++++ mm/page_alloc.c | 2981 +++----------------------------- mm/slab.h | 1 + 13 files changed, 2856 insertions(+), 2828 deletions(-)
base-commit: 4018ab1f7cec061b8425737328edefebdc0ab832
Signature · 1 line
2.35.1
Mike Rapoport
maintainerPATCH v2 1/14 mips: fix comment about pgtable_init()
From: "Mike Rapoport (IBM)" <[email protected]>
Comment about fixrange_init() says that its called from pgtable_init() while the actual caller is pagetabe_init().
Update comment to match the code.
--- arch/mips/include/asm/fixmap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/mips/include/asm/fixmap.h +1 -1
@@ -70,7 +70,7 @@ enum fixed_addresses {#include <asm-generic/fixmap.h> /* -* Called from pgtable_init() +* Called from pagetable_init() */ extern void fixrange_init (unsigned long start,unsigned long end,pgd_t *pgd_base);
Signature · 1 line
2.35.1
Re: [PATCH v2 01/14] mips: fix comment about pgtable_init()
Quoted 9 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
Comment about
fixrange_init()says that its called frompgtable_init()while the actual caller ispagetabe_init().Update comment to match the code.
Signed-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Quoted 17 lines
--- arch/mips/include/asm/fixmap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/mips/include/asm/fixmap.h +1
-1@@ -70,7 +70,7 @@ enum fixed_addresses {#include <asm-generic/fixmap.h> /* -* Called from pgtable_init() +* Called from pagetable_init() */ extern void fixrange_init (unsigned long start,unsigned long end,pgd_t *pgd_base);
Mike Rapoport
maintainerPATCH v2 2/14 mm/page_alloc: add helper for checking if check_pages_enabled
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
Instead of duplicating long static_branch_enabled(&check_pages_enabled) wrap it in a helper function is_check_pages_enabled()
--- mm/page_alloc.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
mm/page_alloc.c +8 -3
@@ -245,6 +245,11 @@ EXPORT_SYMBOL(init_on_free);/* perform sanity checks on struct pages being allocated or freed */ static DEFINE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); +static inline bool is_check_pages_enabled (void ) +{ +return static_branch_unlikely (&check_pages_enabled); +} +static bool _init_on_alloc_enabled_early__read_mostly =IS_ENABLED (CONFIG_INIT_ON_ALLOC_DEFAULT_ON );static int __init early_init_on_alloc (char *buf)
@@ -1443,7 +1448,7 @@ static __always_inline bool free_pages_prepare(struct page *page,for (i =1 ; i < (1 << order); i++) {if (compound) bad +=free_tail_pages_check (page, page + i); -if (static_branch_unlikely (&check_pages_enabled)) { +if (is_check_pages_enabled ()) {if (unlikely (free_page_is_bad (page + i))) { bad++;continue ;
@@ -1456,7 +1461,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->mapping =NULL ;if (memcg_kmem_online () &&PageMemcgKmem (page))__memcg_kmem_uncharge_page (page, order); -if (static_branch_unlikely (&check_pages_enabled)) { +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;if (bad)
@@ -2366,7 +2371,7 @@ static int check_new_page(struct page *page)static inline bool check_new_pages (struct page *page,unsigned int order) { -if (static_branch_unlikely (&check_pages_enabled)) { +if (is_check_pages_enabled ()) {for (int i =0 ; i < (1 << order); i++) {struct page *p = page + i;
Signature · 1 line
2.35.1
Re: [PATCH v2 02/14] mm/page_alloc: add helper for checking if check_pages_enabled
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
Instead of duplicating long
static_branch_enabled(&check_pages_enabled)wrap it in a helper functionis_check_pages_enabled()Signed-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Quoted 47 lines
--- mm/page_alloc.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)mm/page_alloc.c +8
-3@@ -245,6 +245,11 @@ EXPORT_SYMBOL(init_on_free);/* perform sanity checks on struct pages being allocated or freed */ static DEFINE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); +static inline bool is_check_pages_enabled (void ) +{ +return static_branch_unlikely (&check_pages_enabled); +} +static bool _init_on_alloc_enabled_early__read_mostly =IS_ENABLED (CONFIG_INIT_ON_ALLOC_DEFAULT_ON );static int __init early_init_on_alloc (char *buf)@@ -1443,7 +1448,7 @@ static __always_inline bool free_pages_prepare(struct page *page,for (i =1 ; i < (1 << order); i++) {if (compound) bad +=free_tail_pages_check (page, page + i); -if (static_branch_unlikely (&check_pages_enabled)) { +if (is_check_pages_enabled ()) {if (unlikely (free_page_is_bad (page + i))) { bad++;continue ;@@ -1456,7 +1461,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->mapping =NULL ;if (memcg_kmem_online () &&PageMemcgKmem (page))__memcg_kmem_uncharge_page (page, order); -if (static_branch_unlikely (&check_pages_enabled)) { +if (is_check_pages_enabled ()) {if (free_page_is_bad (page)) bad++;if (bad)@@ -2366,7 +2371,7 @@ static int check_new_page(struct page *page)static inline bool check_new_pages (struct page *page,unsigned int order) { -if (static_branch_unlikely (&check_pages_enabled)) { +if (is_check_pages_enabled ()) {for (int i =0 ; i < (1 << order); i++) {struct page *p = page + i;
Mike Rapoport
maintainerPATCH v2 3/14 mm: move most of core MM initialization to mm/mm_init.c
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
The bulk of memory management initialization code is spread all over mm/page_alloc.c and makes navigating through page allocator functionality difficult.
Move most of the functions marked __init and __meminit to mm/mm_init.c to make it better localized and allow some more spare room before mm/page_alloc.c reaches 10k lines.
No functional changes.
--- include/linux/gfp.h | 5 - mm/cma.c | 1 + mm/internal.h | 38 +- mm/mm_init.c | 2304 ++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 2866 ++++--------------------------------------- 5 files changed, 2614 insertions(+), 2600 deletions(-)
include/linux/gfp.h +0 -5
@@ -361,9 +361,4 @@ extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask,#endif void free_contig_range (unsigned long pfn,unsigned long nr_pages); -#ifdef CONFIG_CMA -/* CMA stuff */ -extern void init_cma_reserved_pageblock (struct page *page); -#endif -#endif /* __LINUX_GFP_H */
mm/cma.c +1 -0
@@ -33,6 +33,7 @@#include <linux/kmemleak.h> #include <trace/events/cma.h> +#include "internal.h" #include "cma.h" struct cma cma_areas[MAX_CMA_AREAS ];
mm/internal.h +37 -1
@@ -202,6 +202,8 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);* in mm/page_alloc.c */ +extern char *const zone_names[MAX_NR_ZONES ]; +/* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages*
@@ -366,7 +368,29 @@ extern void __putback_isolated_page(struct page *page, unsigned int order,extern void memblock_free_pages (struct page *page,unsigned long pfn,unsigned int order);extern void __free_pages_core (struct page *page,unsigned int order); + +static inline void prep_compound_head (struct page *page,unsigned int order) +{ +struct folio *folio = (struct folio *)page; + +set_compound_page_dtor (page,COMPOUND_PAGE_DTOR ); +set_compound_order (page, order); +atomic_set (&folio->_entire_mapcount,-1 ); +atomic_set (&folio->_nr_pages_mapped,0 ); +atomic_set (&folio->_pincount,0 ); +} + +static inline void prep_compound_tail (struct page *head,int tail_idx) +{ +struct page *p = head + tail_idx; + + p->mapping =TAIL_MAPPING ; +set_compound_head (p, head); +set_page_private (p,0 ); +} +extern void prep_compound_page (struct page *page,unsigned int order); +extern void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags);extern int user_min_free_kbytes;
@@ -377,6 +401,7 @@ extern void free_unref_page_list(struct list_head *list);extern void zone_pcp_reset (struct zone *zone);extern void zone_pcp_disable (struct zone *zone);extern void zone_pcp_enable (struct zone *zone); +extern void zone_pcp_init (struct zone *zone); extern void *memmap_alloc(phys_addr_t size, phys_addr_t align, phys_addr_t min_addr,
@@ -474,7 +499,12 @@ isolate_migratepages_range(struct compact_control *cc,int __alloc_contig_migrate_range (struct compact_control *cc,unsigned long start,unsigned long end); -#endif + +/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ +void init_cma_reserved_pageblock (struct page *page); + +#endif /* CONFIG_COMPACTION || CONFIG_CMA */ +int find_suitable_fallback (struct free_area *area,unsigned int order,int migratetype,bool only_stealable,bool *can_steal);
@@ -679,6 +709,12 @@ static inline loff_t fadvise_calc_endbyte(loff_t offset, loff_t len) }/* Memory initialisation debug and verification */ +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +DECLARE_STATIC_KEY_TRUE (deferred_pages); + +bool __init deferred_grow_zone (struct zone *zone,unsigned int order); +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ +enum mminit_level {MMINIT_WARNING ,MMINIT_VERIFY ,
mm/mm_init.c +2304 -0
2,314 lines not on this page · raw message
mm/page_alloc.c +272 -2594
601 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.35.1
Part of the diff is not shown here. View the raw message.
Re: [PATCH v2 03/14] mm: move most of core MM initialization to mm/mm_init.c
Quoted 6 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
The bulk of memory management initialization code is spread all over
mm/page_alloc.cand makes navigating through page allocator functionality difficult.
Quoted 8 lines
Move most of the functions marked
__initand__meminittomm/mm_init.cto make it better localized and allow some more spare room beforemm/page_alloc.creaches 10k lines.No functional changes.
Signed-off-by Mike Rapoport (IBM) <[email protected]>
Mike Rapoport
maintainerPATCH v2 4/14 mm: handle hashdist initialization in mm/mm_init.c
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
The hashdist variable must be initialized before the first call to alloc_large_system_hash() and free_area_init() looks like a better place for it than page_alloc_init().
Move hashdist handling to mm/mm_init.c
--- mm/mm_init.c | 22 ++++++++++++++++++++++ mm/page_alloc.c | 18 ------------------ 2 files changed, 22 insertions(+), 18 deletions(-)
mm/mm_init.c +22 -0
@@ -607,6 +607,25 @@ int __meminit early_pfn_to_nid(unsigned long pfn) return nid; } + +int hashdist =HASHDIST_DEFAULT ; + +static int __init set_hashdist (char *str) +{ +if (!str) +return 0 ; + hashdist =simple_strtoul (str, &str,0 ); +return 1 ; +} +__setup ("hashdist=" , set_hashdist); + +static inline void fixup_hashdist (void ) +{ +if (num_node_state (N_MEMORY ) ==1 ) + hashdist =0 ; +} +#else +static inline void fixup_hashdist (void ) {}#endif /* CONFIG_NUMA */ #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
@@ -1855,6 +1874,9 @@ void __init free_area_init(unsigned long *max_zone_pfn) }memmap_init (); + +/* disable hash distribution for systems with a single node */ +fixup_hashdist (); }/**
mm/page_alloc.c +0 -18
@@ -6383,28 +6383,10 @@ static int page_alloc_cpu_online(unsigned int cpu)return 0 ; } -#ifdef CONFIG_NUMA -int hashdist =HASHDIST_DEFAULT ; - -static int __init set_hashdist (char *str) -{ -if (!str) -return 0 ; - hashdist =simple_strtoul (str, &str,0 ); -return 1 ; -} -__setup ("hashdist=" , set_hashdist); -#endif -void __init page_alloc_init (void ) {int ret; -#ifdef CONFIG_NUMA -if (num_node_state (N_MEMORY ) ==1 ) - hashdist =0 ; -#endif - ret = cpuhp_setup_state_nocalls(CPUHP_PAGE_ALLOC, "mm/page_alloc:pcp", page_alloc_cpu_online,
Signature · 1 line
2.35.1
Re: [PATCH v2 04/14] mm: handle hashdist initialization in mm/mm_init.c
Quoted 10 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
The hashdist variable must be initialized before the first call to
alloc_large_system_hash()andfree_area_init()looks like a better place for it thanpage_alloc_init().Move hashdist handling to
mm/mm_init.cSigned-off-by Mike Rapoport (IBM) <[email protected]>
Looks like this will move the fixup_hashdist() call earlier, but can't result in seeing less N_MEMORY nodes than before, right?
I wonder if the whole thing lacks hotplug support anyway, what if system boots with one node and more are added later? Hmm.
Quoted 78 lines
--- mm/mm_init.c | 22 ++++++++++++++++++++++ mm/page_alloc.c | 18 ------------------ 2 files changed, 22 insertions(+), 18 deletions(-)mm/mm_init.c +22
-0@@ -607,6 +607,25 @@ int __meminit early_pfn_to_nid(unsigned long pfn) return nid; } + +int hashdist =HASHDIST_DEFAULT ; + +static int __init set_hashdist (char *str) +{ +if (!str) +return 0 ; + hashdist =simple_strtoul (str, &str,0 ); +return 1 ; +} +__setup ("hashdist=" , set_hashdist); + +static inline void fixup_hashdist (void ) +{ +if (num_node_state (N_MEMORY ) ==1 ) + hashdist =0 ; +} +#else +static inline void fixup_hashdist (void ) {}#endif /* CONFIG_NUMA */ #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT @@ -1855,6 +1874,9 @@ void __init free_area_init(unsigned long *max_zone_pfn) }memmap_init (); + +/* disable hash distribution for systems with a single node */ +fixup_hashdist (); }/** mm/page_alloc.c +0
-18@@ -6383,28 +6383,10 @@ static int page_alloc_cpu_online(unsigned int cpu)return 0 ; } -#ifdef CONFIG_NUMA -int hashdist =HASHDIST_DEFAULT ; - -static int __init set_hashdist (char *str) -{ -if (!str) -return 0 ; - hashdist =simple_strtoul (str, &str,0 ); -return 1 ; -} -__setup ("hashdist=" , set_hashdist); -#endif -void __init page_alloc_init (void ) {int ret; -#ifdef CONFIG_NUMA -if (num_node_state (N_MEMORY ) ==1 ) - hashdist =0 ; -#endif - ret = cpuhp_setup_state_nocalls(CPUHP_PAGE_ALLOC, "mm/page_alloc:pcp", page_alloc_cpu_online,
Mike Rapoport
maintainerRe: [PATCH v2 04/14] mm: handle hashdist initialization in mm/mm_init.c
Quoted 9 lines · Vlastimil Babka
Quoted 8 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
The hashdist variable must be initialized before the first call to
alloc_large_system_hash()andfree_area_init()looks like a better place for it thanpage_alloc_init().Move hashdist handling to
mm/mm_init.c
Quoted 7 lines
Quoted 2 lines
Signed-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Looks like this will move the
fixup_hashdist()call earlier, but can't result in seeing lessN_MEMORYnodes than before, right?
hashdist must be set before the first call to alloc_large_system_hash() and after the nodes present at boot time are initialized, so setting it in the end of free_area_init() is Ok.
Quoted 2 lines
I wonder if the whole thing lacks hotplug support anyway, what if system boots with one node and more are added later? Hmm.
alloc_large_system_hash() is called really early even for !HASH_EARLY cases. Not sure it's feasible to redistribute the hashes allocated with it when new node is added.
Quoted 79 lines
Quoted 78 lines
--- mm/mm_init.c | 22 ++++++++++++++++++++++ mm/page_alloc.c | 18 ------------------ 2 files changed, 22 insertions(+), 18 deletions(-)mm/mm_init.c +22
-0@@ -607,6 +607,25 @@ int __meminit early_pfn_to_nid(unsigned long pfn) return nid; } + +int hashdist =HASHDIST_DEFAULT ; + +static int __init set_hashdist (char *str) +{ +if (!str) +return 0 ; + hashdist =simple_strtoul (str, &str,0 ); +return 1 ; +} +__setup ("hashdist=" , set_hashdist); + +static inline void fixup_hashdist (void ) +{ +if (num_node_state (N_MEMORY ) ==1 ) + hashdist =0 ; +} +#else +static inline void fixup_hashdist (void ) {}#endif /* CONFIG_NUMA */ #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT @@ -1855,6 +1874,9 @@ void __init free_area_init(unsigned long *max_zone_pfn) }memmap_init (); + +/* disable hash distribution for systems with a single node */ +fixup_hashdist (); }/** mm/page_alloc.c +0
-18@@ -6383,28 +6383,10 @@ static int page_alloc_cpu_online(unsigned int cpu)return 0 ; } -#ifdef CONFIG_NUMA -int hashdist =HASHDIST_DEFAULT ; - -static int __init set_hashdist (char *str) -{ -if (!str) -return 0 ; - hashdist =simple_strtoul (str, &str,0 ); -return 1 ; -} -__setup ("hashdist=" , set_hashdist); -#endif -void __init page_alloc_init (void ) {int ret; -#ifdef CONFIG_NUMA -if (num_node_state (N_MEMORY ) ==1 ) - hashdist =0 ; -#endif - ret = cpuhp_setup_state_nocalls(CPUHP_PAGE_ALLOC, "mm/page_alloc:pcp", page_alloc_cpu_online,
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
maintainerPATCH v2 5/14 mm/page_alloc: rename page_alloc_init() to page_alloc_init_cpuhp()
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
The page_alloc_init() name is really misleading because all this function does is sets up CPU hotplug callbacks for the page allocator.
Rename it to page_alloc_init_cpuhp() so that name will reflect what the function does.
--- include/linux/gfp.h | 2 +- init/main.c | 2 +- mm/page_alloc.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
include/linux/gfp.h +1 -1
@@ -319,7 +319,7 @@ extern void page_frag_free(void *addr);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0) -void page_alloc_init (void ); +void page_alloc_init_cpuhp (void );void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp);void drain_all_pages (struct zone *zone);void drain_local_pages (struct zone *zone);
init/main.c +1 -1
@@ -969,7 +969,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)boot_cpu_hotplug_init ();build_all_zonelists (NULL ); -page_alloc_init (); +page_alloc_init_cpuhp ();pr_notice ("Kernel command line: %s\n" , saved_command_line);/* parameters may set static keys */
mm/page_alloc.c +1 -1
@@ -6383,7 +6383,7 @@ static int page_alloc_cpu_online(unsigned int cpu)return 0 ; } -void __init page_alloc_init (void ) +void __init page_alloc_init_cpuhp (void ) {int ret;
Signature · 1 line
2.35.1
Re: [PATCH v2 05/14] mm/page_alloc: rename page_alloc_init() to page_alloc_init_cpuhp()
Quoted 10 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
The
page_alloc_init()name is really misleading because all this function does is sets up CPU hotplug callbacks for the page allocator.Rename it to
page_alloc_init_cpuhp()so that name will reflect what the function does.Signed-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Quoted 45 lines
--- include/linux/gfp.h | 2 +- init/main.c | 2 +- mm/page_alloc.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)include/linux/gfp.h +1
-1@@ -319,7 +319,7 @@ extern void page_frag_free(void *addr);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0) -void page_alloc_init (void ); +void page_alloc_init_cpuhp (void );void drain_zone_pages (struct zone *zone,struct per_cpu_pages *pcp);void drain_all_pages (struct zone *zone);void drain_local_pages (struct zone *zone);init/main.c +1
-1@@ -969,7 +969,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)boot_cpu_hotplug_init ();build_all_zonelists (NULL ); -page_alloc_init (); +page_alloc_init_cpuhp ();pr_notice ("Kernel command line: %s\n" , saved_command_line);/* parameters may set static keys */ mm/page_alloc.c +1
-1@@ -6383,7 +6383,7 @@ static int page_alloc_cpu_online(unsigned int cpu)return 0 ; } -void __init page_alloc_init (void ) +void __init page_alloc_init_cpuhp (void ) {int ret;
Mike Rapoport
maintainerPATCH v2 6/14 init: fold build_all_zonelists() and page_alloc_init_cpuhp() to mm_init()
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
Both build_all_zonelists() and page_alloc_init_cpuhp() must be called after SMP setup is complete but before the page allocator is set up.
Still, they both are a part of memory management initialization, so move them to mm_init().
--- init/main.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
init/main.c +4 -3
@@ -833,6 +833,10 @@ static void __init report_meminit(void)*/ static void __init mm_init (void ) { +/* Initializations relying on SMP setup */ +build_all_zonelists (NULL ); +page_alloc_init_cpuhp (); +/* * page_ext requires contiguous pages, * bigger than MAX_ORDER unless SPARSEMEM.
@@ -968,9 +972,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)smp_prepare_boot_cpu ();/* arch-specific boot-cpu hooks */ boot_cpu_hotplug_init (); -build_all_zonelists (NULL ); -page_alloc_init_cpuhp (); -pr_notice ("Kernel command line: %s\n" , saved_command_line);/* parameters may set static keys */ jump_label_init ();
Signature · 1 line
2.35.1
Re: [PATCH v2 06/14] init: fold build_all_zonelists() and page_alloc_init_cpuhp() to mm_init()
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
Both
build_all_zonelists()andpage_alloc_init_cpuhp()must be called after SMP setup is complete but before the page allocator is set up.Still, they both are a part of memory management initialization, so move them to
mm_init().
Well, logic grouping is one thing, but not breaking a functional order is more important. So this moves both calls to happen later than theyw ere. I guess it could only matter for page_alloc_init_cpuhp() in case cpu hotplugs would be processed in some of the calls we "skipped" over by moving this later. And one of them is setup_arch()... so are we sure no arch does some cpu hotplug for non-boot cpus there?
Quoted 31 lines
Signed-off-by Mike Rapoport (IBM) <[email protected]>--- init/main.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)init/main.c +4
-3@@ -833,6 +833,10 @@ static void __init report_meminit(void)*/ static void __init mm_init (void ) { +/* Initializations relying on SMP setup */ +build_all_zonelists (NULL ); +page_alloc_init_cpuhp (); +/* * page_ext requires contiguous pages, * bigger than MAX_ORDER unless SPARSEMEM. @@ -968,9 +972,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)smp_prepare_boot_cpu ();/* arch-specific boot-cpu hooks */ boot_cpu_hotplug_init (); -build_all_zonelists (NULL ); -page_alloc_init_cpuhp (); -pr_notice ("Kernel command line: %s\n" , saved_command_line);/* parameters may set static keys */ jump_label_init ();
Mike Rapoport
maintainerRe: [PATCH v2 06/14] init: fold build_all_zonelists() and page_alloc_init_cpuhp() to mm_init()
Quoted 9 lines · Vlastimil Babka
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
Both
build_all_zonelists()andpage_alloc_init_cpuhp()must be called after SMP setup is complete but before the page allocator is set up.Still, they both are a part of memory management initialization, so move them to
mm_init().
Quoted 6 lines
Well, logic grouping is one thing, but not breaking a functional order is more important. So this moves both calls to happen later than theyw ere. I guess it could only matter for
page_alloc_init_cpuhp()in case cpu hotplugs would be processed in some of the calls we "skipped" over by moving this later. And one of them issetup_arch()... so are we sure no arch does some cpu hotplug for non-boot cpus there?
mm_init() happens after the point build_all_zonelists() and page_alloc_init_cpuhp() were originally, so they are essentially moved later in the init sequence and in either case called after setup_arch().
We skip the code below and it does not do neither cpu hotplug nor non-memblock allocations.
jump_label_init();
parse_early_param();
after_dashes = parse_args("Booting kernel",
static_command_line, __start___param,
__stop___param - __start___param,
-1, -1, NULL, &unknown_bootoption);
print_unknown_bootoptions();
if (!IS_ERR_OR_NULL(after_dashes))
parse_args("Setting init args", after_dashes, NULL, 0, -1, -1,
NULL, set_init_arg);
if (extra_init_args)
parse_args("Setting extra init args", extra_init_args,
NULL, 0, -1, -1, NULL, set_init_arg);
/* Architectural and non-timekeeping rng init, before allocator init */
random_init_early(command_line);
/*
* These use large bootmem allocations and must precede
* kmem_cache_init()
*/
setup_log_buf(0);
vfs_caches_init_early();
sort_main_extable();
trap_init();Quoted 32 lines
Quoted 31 lines
Signed-off-by Mike Rapoport (IBM) <[email protected]>--- init/main.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)init/main.c +4
-3@@ -833,6 +833,10 @@ static void __init report_meminit(void)*/ static void __init mm_init (void ) { +/* Initializations relying on SMP setup */ +build_all_zonelists (NULL ); +page_alloc_init_cpuhp (); +/* * page_ext requires contiguous pages, * bigger than MAX_ORDER unless SPARSEMEM. @@ -968,9 +972,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)smp_prepare_boot_cpu ();/* arch-specific boot-cpu hooks */ boot_cpu_hotplug_init (); -build_all_zonelists (NULL ); -page_alloc_init_cpuhp (); -pr_notice ("Kernel command line: %s\n" , saved_command_line);/* parameters may set static keys */ jump_label_init ();
Signature · 2 lines
Sincerely yours, Mike.
Vlastimil Babka
Reviewed-byRe: [PATCH v2 06/14] init: fold build_all_zonelists() and page_alloc_init_cpuhp() to mm_init()
Quoted 17 lines · Mike Rapoport
Quoted 9 lines · Vlastimil Babka
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
Both
build_all_zonelists()andpage_alloc_init_cpuhp()must be called after SMP setup is complete but before the page allocator is set up.Still, they both are a part of memory management initialization, so move them to
mm_init().Quoted 6 lines
Well, logic grouping is one thing, but not breaking a functional order is more important. So this moves both calls to happen later than theyw ere. I guess it could only matter for
page_alloc_init_cpuhp()in case cpu hotplugs would be processed in some of the calls we "skipped" over by moving this later. And one of them issetup_arch()... so are we sure no arch does some cpu hotplug for non-boot cpus there?
Quoted 3 lines
mm_init()happens after the pointbuild_all_zonelists()andpage_alloc_init_cpuhp()were originally, so they are essentially moved later in the init sequence and in either case called aftersetup_arch().
Right, I looked at a wrong place in start_kernel() for the original location of the calls, sorry for the noise.
Quoted 20 lines
We skip the code below and it does not do neither cpu hotplug nor non-memblock allocations.
jump_label_init(); parse_early_param(); after_dashes = parse_args("Booting kernel", static_command_line, __start___param, __stop___param - __start___param, -1, -1, NULL, &unknown_bootoption); print_unknown_bootoptions(); if (!IS_ERR_OR_NULL(after_dashes)) parse_args("Setting init args", after_dashes, NULL, 0, -1, -1, NULL, set_init_arg); if (extra_init_args) parse_args("Setting extra init args", extra_init_args, NULL, 0, -1, -1, NULL, set_init_arg); /* Architectural and non-timekeeping rng init, before allocator init */ random_init_early(command_line);
Quoted 9 lines
/* * These use large bootmem allocations and must precede * kmem_cache_init() */ setup_log_buf(0); vfs_caches_init_early(); sort_main_extable(); trap_init();
Yeah, that looks safe.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signed-off-by Mike Rapoport (IBM) <[email protected]>
Mike Rapoport
maintainerPATCH v2 7/14 init,mm: move mm_init() to mm/mm_init.c and rename it to mm_core_init()
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
Make mm_init() a part of mm/ codebase. mm_core_init() better describes what the function does and does not clash with mm_init() in kernel/fork.c
--- include/linux/mm.h | 1 + init/main.c | 71 ++------------------------------------------ mm/mm_init.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 69 deletions(-)
include/linux/mm.h +1 -0
@@ -39,6 +39,7 @@ struct pt_regs;extern int sysctl_page_lock_unfairness; +void mm_core_init (void );void init_mm_internals (void );#ifndef CONFIG_NUMA /* Don't use mapnrs, do it properly */
init/main.c +2 -69
88 lines not on this page · raw message
mm/mm_init.c +73 -0
85 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCH v2 07/14] init,mm: move mm_init() to mm/mm_init.c and rename it to mm_core_init()
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
Make
mm_init()a part ofmm/codebase.mm_core_init()better describes what the function does and does not clash withmm_init()inkernel/fork.cSigned-off-by Mike Rapoport (IBM) <[email protected]>
Mike Rapoport
maintainerPATCH v2 8/14 mm: call {ptlock,pgtable}_cache_init() directly from mm_core_init()
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
and drop pgtable_init() as it has no real value and it's name is misleading.
--- include/linux/mm.h | 6 ------ mm/mm_init.c | 3 ++- 2 files changed, 2 insertions(+), 7 deletions(-)
include/linux/mm.h +0 -6
@@ -2782,12 +2782,6 @@ static inline bool ptlock_init(struct page *page) { return true; }static inline void ptlock_free (struct page *page) {}#endif /* USE_SPLIT_PTE_PTLOCKS */ -static inline void pgtable_init (void ) -{ -ptlock_cache_init (); -pgtable_cache_init (); -} -static inline bool pgtable_pte_page_ctor (struct page *page) {if (!ptlock_init (page))
mm/mm_init.c +2 -1
@@ -2584,7 +2584,8 @@ void __init mm_core_init(void)*/ page_ext_init_flatmem_late ();kmemleak_init (); -pgtable_init (); +ptlock_cache_init (); +pgtable_cache_init ();debug_objects_mem_init ();vmalloc_init ();/* If no deferred init page_ext now, as vmap is fully initialized */
Signature · 1 line
2.35.1
Re: [PATCH v2 08/14] mm: call {ptlock,pgtable}_cache_init() directly from mm_core_init()
Quoted 3 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
and drop
pgtable_init()as it has no real value and it's name is
Its name.
Quoted 3 lines
misleading.
Signed-off-by Mike Rapoport (IBM) <[email protected]>
[...]
MBR, Sergey
Mike Rapoport
maintainerRe: [PATCH v2 08/14] mm: call {ptlock,pgtable}_cache_init() directly from mm_core_init()
Quoted 7 lines · Sergei Shtylyov
Quoted 3 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
and drop
pgtable_init()as it has no real value and it's name isIts name.
oops :)
Andrew, can you replace this patch with the updated version, please?
From 52420723c9bfa84aa48f666330e96f9e5b2f3248 Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (IBM)" <[email protected]>
Date: Sat, 18 Mar 2023 13:55:28 +0200
Subject: [PATCH v3] mm: call {ptlock,pgtable}_cache_init() directly from
mm_core_init()
and drop pgtable_init() as it has no real value and its name is misleading.
--- include/linux/mm.h | 6 ------ mm/mm_init.c | 3 ++- 2 files changed, 2 insertions(+), 7 deletions(-)
include/linux/mm.h +0 -6
@@ -2782,12 +2782,6 @@ static inline bool ptlock_init(struct page *page) { return true; }static inline void ptlock_free (struct page *page) {}#endif /* USE_SPLIT_PTE_PTLOCKS */ -static inline void pgtable_init (void ) -{ -ptlock_cache_init (); -pgtable_cache_init (); -} -static inline bool pgtable_pte_page_ctor (struct page *page) {if (!ptlock_init (page))
mm/mm_init.c +2 -1
@@ -2584,7 +2584,8 @@ void __init mm_core_init(void)*/ page_ext_init_flatmem_late ();kmemleak_init (); -pgtable_init (); +ptlock_cache_init (); +pgtable_cache_init ();debug_objects_mem_init ();vmalloc_init ();/* If no deferred init page_ext now, as vmap is fully initialized */
Signature · 5 lines
2.35.1 -- Sincerely yours, Mike.
David Hildenbrand
Reviewed-byRe: [PATCH v2 08/14] mm: call {ptlock,pgtable}_cache_init() directly from mm_core_init()
Quoted 50 lines · Mike Rapoport
Quoted 7 lines · Sergei Shtylyov
Quoted 3 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
and drop
pgtable_init()as it has no real value and it's name isIts name.
oops :)
Andrew, can you replace this patch with the updated version, please?
From
52420723c9bfa84aa48f666330e96f9e5b2f3248Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (IBM)" <[email protected]>
Date: Sat, 18 Mar 2023 13:55:28 +0200
Subject: [PATCH v3] mm: call {ptlock,pgtable}_cache_init() directly from
mm_core_init()and drop
pgtable_init()as it has no real value and its name is misleading.Signed-off-by Mike Rapoport (IBM) <[email protected]>--- include/linux/mm.h | 6 ------ mm/mm_init.c | 3 ++- 2 files changed, 2 insertions(+), 7 deletions(-)include/linux/mm.h +0
-6@@ -2782,12 +2782,6 @@ static inline bool ptlock_init(struct page *page) { return true; }static inline void ptlock_free (struct page *page) {}#endif /* USE_SPLIT_PTE_PTLOCKS */ -static inline void pgtable_init (void ) -{ -ptlock_cache_init (); -pgtable_cache_init (); -} -static inline bool pgtable_pte_page_ctor (struct page *page) {if (!ptlock_init (page))mm/mm_init.c +0
-0
mm/mm_init.c void __init mm_core_init(void) · line 2591
Quoted 9 lines
mm/mm_init.c +2
-1@@ -2584,7 +2584,8 @@ void __init mm_core_init(void)*/ page_ext_init_flatmem_late ();kmemleak_init (); -pgtable_init (); +ptlock_cache_init (); +pgtable_cache_init ();debug_objects_mem_init ();vmalloc_init ();/* If no deferred init page_ext now, as vmap is fully initialized */
Signature · 3 lines
Thanks, David / dhildenb
Vlastimil Babka
Reviewed-byRe: [PATCH v2 08/14] mm: call {ptlock,pgtable}_cache_init() directly from mm_core_init()
In reply to Mike RapoportQuoted 19 lines · Mike Rapoport
Quoted 7 lines · Sergei Shtylyov
Quoted 3 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
and drop
pgtable_init()as it has no real value and it's name isIts name.
oops :)
Andrew, can you replace this patch with the updated version, please?
From
52420723c9bfa84aa48f666330e96f9e5b2f3248Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (IBM)" <[email protected]>
Date: Sat, 18 Mar 2023 13:55:28 +0200
Subject: [PATCH v3] mm: call {ptlock,pgtable}_cache_init() directly from
mm_core_init()
Quoted 4 lines
and drop
pgtable_init()as it has no real value and its name is misleading.Signed-off-by Mike Rapoport (IBM) <[email protected]>
Quoted 36 lines
--- include/linux/mm.h | 6 ------ mm/mm_init.c | 3 ++- 2 files changed, 2 insertions(+), 7 deletions(-)include/linux/mm.h +0
-6@@ -2782,12 +2782,6 @@ static inline bool ptlock_init(struct page *page) { return true; }static inline void ptlock_free (struct page *page) {}#endif /* USE_SPLIT_PTE_PTLOCKS */ -static inline void pgtable_init (void ) -{ -ptlock_cache_init (); -pgtable_cache_init (); -} -static inline bool pgtable_pte_page_ctor (struct page *page) {if (!ptlock_init (page))mm/mm_init.c +2
-1@@ -2584,7 +2584,8 @@ void __init mm_core_init(void)*/ page_ext_init_flatmem_late ();kmemleak_init (); -pgtable_init (); +ptlock_cache_init (); +pgtable_cache_init ();debug_objects_mem_init ();vmalloc_init ();/* If no deferred init page_ext now, as vmap is fully initialized */
Mike Rapoport
maintainerPATCH v2 9/14 mm: move init_mem_debugging_and_hardening() to mm/mm_init.c
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
init_mem_debugging_and_hardening() is only called from mm_core_init().
Move it close to the caller, make it static and rename it to mem_debugging_and_hardening_init() for consistency with surrounding convention.
--- include/linux/mm.h | 1 - mm/internal.h | 8 ++++ mm/mm_init.c | 91 +++++++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 95 ---------------------------------------------- 4 files changed, 98 insertions(+), 97 deletions(-)
include/linux/mm.h +0 -1
@@ -3394,7 +3394,6 @@ extern int apply_to_existing_page_range(struct mm_struct *mm, unsigned long address, unsigned long size, pte_fn_t fn, void *data); -extern void __init init_mem_debugging_and_hardening (void );#ifdef CONFIG_PAGE_POISONING extern void __kernel_poison_pages (struct page *page,int numpages);extern void __kernel_unpoison_pages (struct page *page,int numpages);
mm/internal.h +8 -0
@@ -204,6 +204,14 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);extern char *const zone_names[MAX_NR_ZONES ]; +/* perform sanity checks on struct pages being allocated or freed */ +DECLARE_STATIC_KEY_MAYBE (CONFIG_DEBUG_VM , check_pages_enabled); + +static inline bool is_check_pages_enabled (void ) +{ +return static_branch_unlikely (&check_pages_enabled); +} +/* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages*
mm/mm_init.c +90 -1
103 lines not on this page · raw message
mm/page_alloc.c +0 -95
107 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCH v2 09/14] mm: move init_mem_debugging_and_hardening() to mm/mm_init.c
Quoted 10 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
init_mem_debugging_and_hardening()is only called frommm_core_init().Move it close to the caller, make it static and rename it to
mem_debugging_and_hardening_init()for consistency with surrounding convention.Signed-off-by Mike Rapoport (IBM) <[email protected]>
Quoted 260 lines
--- include/linux/mm.h | 1 - mm/internal.h | 8 ++++ mm/mm_init.c | 91 +++++++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 95 ---------------------------------------------- 4 files changed, 98 insertions(+), 97 deletions(-)include/linux/mm.h +0
-1@@ -3394,7 +3394,6 @@ extern int apply_to_existing_page_range(struct mm_struct *mm, unsigned long address, unsigned long size, pte_fn_t fn, void *data); -extern void __init init_mem_debugging_and_hardening (void );#ifdef CONFIG_PAGE_POISONING extern void __kernel_poison_pages (struct page *page,int numpages);extern void __kernel_unpoison_pages (struct page *page,int numpages);mm/internal.h +8
-0@@ -204,6 +204,14 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);extern char *const zone_names[MAX_NR_ZONES ]; +/* perform sanity checks on struct pages being allocated or freed */ +DECLARE_STATIC_KEY_MAYBE (CONFIG_DEBUG_VM , check_pages_enabled); + +static inline bool is_check_pages_enabled (void ) +{ +return static_branch_unlikely (&check_pages_enabled); +} +/* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages* mm/mm_init.c +90
-1@@ -2531,6 +2531,95 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn,__free_pages_core (page, order); } +static bool _init_on_alloc_enabled_early__read_mostly + =IS_ENABLED (CONFIG_INIT_ON_ALLOC_DEFAULT_ON ); +static int __init early_init_on_alloc (char *buf) +{ + +return kstrtobool (buf, &_init_on_alloc_enabled_early); +} +early_param ("init_on_alloc" , early_init_on_alloc); + +static bool _init_on_free_enabled_early__read_mostly + =IS_ENABLED (CONFIG_INIT_ON_FREE_DEFAULT_ON ); +static int __init early_init_on_free (char *buf) +{ +return kstrtobool (buf, &_init_on_free_enabled_early); +} +early_param ("init_on_free" , early_init_on_free); + +DEFINE_STATIC_KEY_MAYBE (CONFIG_DEBUG_VM , check_pages_enabled); + +/* +* Enable static keys related to various memory debugging and hardening options. +* Some override others, and depend on early params that are evaluated in the +* order of appearance. So we need to first gather the full picture of what was +* enabled, and then make decisions. +*/ +static void __init mem_debugging_and_hardening_init (void ) +{ +bool page_poisoning_requested = false; +bool want_check_pages = false; + +#ifdef CONFIG_PAGE_POISONING +/* +* Page poisoning is debug page alloc for some arches. If +* either of those options are enabled, enable poisoning. +*/ +if (page_poisoning_enabled () || + (!IS_ENABLED (CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC ) && +debug_pagealloc_enabled ())) { +static_branch_enable (&_page_poisoning_enabled); + page_poisoning_requested = true; + want_check_pages = true; + } +#endif + +if ((_init_on_alloc_enabled_early || _init_on_free_enabled_early) && + page_poisoning_requested) { +pr_info ("mem auto-init: CONFIG_PAGE_POISONING is on, " +"will take precedence over init_on_alloc and init_on_free\n" ); + _init_on_alloc_enabled_early = false; + _init_on_free_enabled_early = false; + } + +if (_init_on_alloc_enabled_early) { + want_check_pages = true; +static_branch_enable (&init_on_alloc); + }else { +static_branch_disable (&init_on_alloc); + } + +if (_init_on_free_enabled_early) { + want_check_pages = true; +static_branch_enable (&init_on_free); + }else { +static_branch_disable (&init_on_free); + } + +if (IS_ENABLED (CONFIG_KMSAN ) && + (_init_on_alloc_enabled_early || _init_on_free_enabled_early)) +pr_info ("mem auto-init: please make sure init_on_alloc and init_on_free are disabled when running KMSAN\n" ); + +#ifdef CONFIG_DEBUG_PAGEALLOC +if (debug_pagealloc_enabled ()) { + want_check_pages = true; +static_branch_enable (&_debug_pagealloc_enabled); + +if (debug_guardpage_minorder ()) +static_branch_enable (&_debug_guardpage_enabled); + } +#endif + +/* +* Any page debugging or hardening option also enables sanity checking +* of struct pages being allocated or freed. With CONFIG_DEBUG_VM it's +* enabled already. +*/ +if (!IS_ENABLED (CONFIG_DEBUG_VM ) && want_check_pages) +static_branch_enable (&check_pages_enabled); +} +/* Report memory auto-initialization states for this boot. */ static void __init report_meminit (void ) {@@ -2570,7 +2659,7 @@ void __init mm_core_init(void)* bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem (); -init_mem_debugging_and_hardening (); +mem_debugging_and_hardening_init ();kfence_alloc_pool ();report_meminit ();kmsan_init_shadow ();mm/page_alloc.c +0
-95@@ -240,31 +240,6 @@ EXPORT_SYMBOL(init_on_alloc);DEFINE_STATIC_KEY_MAYBE (CONFIG_INIT_ON_FREE_DEFAULT_ON , init_on_free);EXPORT_SYMBOL (init_on_free); -/* perform sanity checks on struct pages being allocated or freed */ -static DEFINE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); - -static inline bool is_check_pages_enabled (void ) -{ -return static_branch_unlikely (&check_pages_enabled); -} - -static bool _init_on_alloc_enabled_early__read_mostly - =IS_ENABLED (CONFIG_INIT_ON_ALLOC_DEFAULT_ON ); -static int __init early_init_on_alloc (char *buf) -{ - -return kstrtobool (buf, &_init_on_alloc_enabled_early); -} -early_param ("init_on_alloc" , early_init_on_alloc); - -static bool _init_on_free_enabled_early__read_mostly - =IS_ENABLED (CONFIG_INIT_ON_FREE_DEFAULT_ON ); -static int __init early_init_on_free (char *buf) -{ -return kstrtobool (buf, &_init_on_free_enabled_early); -} -early_param ("init_on_free" , early_init_on_free); -/* * A cached value of the page's pageblock's migratetype, used when the page is * put on a pcplist. Used to avoid the pageblock migratetype lookup when @@ -798,76 +773,6 @@ static inline void clear_page_guard(struct zone *zone, struct page *page, unsigned int order, int migratetype) {}#endif -/* -* Enable static keys related to various memory debugging and hardening options. -* Some override others, and depend on early params that are evaluated in the -* order of appearance. So we need to first gather the full picture of what was -* enabled, and then make decisions. -*/ -void __init init_mem_debugging_and_hardening (void ) -{ -bool page_poisoning_requested = false; -bool want_check_pages = false; - -#ifdef CONFIG_PAGE_POISONING -/* -* Page poisoning is debug page alloc for some arches. If -* either of those options are enabled, enable poisoning. -*/ -if (page_poisoning_enabled () || - (!IS_ENABLED (CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC ) && -debug_pagealloc_enabled ())) { -static_branch_enable (&_page_poisoning_enabled); - page_poisoning_requested = true; - want_check_pages = true; - } -#endif - -if ((_init_on_alloc_enabled_early || _init_on_free_enabled_early) && - page_poisoning_requested) { -pr_info ("mem auto-init: CONFIG_PAGE_POISONING is on, " -"will take precedence over init_on_alloc and init_on_free\n" ); - _init_on_alloc_enabled_early = false; - _init_on_free_enabled_early = false; - } - -if (_init_on_alloc_enabled_early) { - want_check_pages = true; -static_branch_enable (&init_on_alloc); - }else { -static_branch_disable (&init_on_alloc); - } - -if (_init_on_free_enabled_early) { - want_check_pages = true; -static_branch_enable (&init_on_free); - }else { -static_branch_disable (&init_on_free); - } - -if (IS_ENABLED (CONFIG_KMSAN ) && - (_init_on_alloc_enabled_early || _init_on_free_enabled_early)) -pr_info ("mem auto-init: please make sure init_on_alloc and init_on_free are disabled when running KMSAN\n" ); - -#ifdef CONFIG_DEBUG_PAGEALLOC -if (debug_pagealloc_enabled ()) { - want_check_pages = true; -static_branch_enable (&_debug_pagealloc_enabled); - -if (debug_guardpage_minorder ()) -static_branch_enable (&_debug_guardpage_enabled); - } -#endif - -/* -* Any page debugging or hardening option also enables sanity checking -* of struct pages being allocated or freed. With CONFIG_DEBUG_VM it's -* enabled already. -*/ -if (!IS_ENABLED (CONFIG_DEBUG_VM ) && want_check_pages) -static_branch_enable (&check_pages_enabled); -} -static inline void set_buddy_order (struct page *page,unsigned int order) {set_page_private (page, order);
Mike Rapoport
maintainerPATCH v2 10/14 init,mm: fold late call to page_ext_init() to page_alloc_init_late()
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
When deferred initialization of struct pages is enabled, page_ext_init() must be called after all the deferred initialization is done, but there is no point to keep it a separate call from kernel_init_freeable() right after page_alloc_init_late().
Fold the call to page_ext_init() into page_alloc_init_late() and localize deferred_struct_pages variable.
--- include/linux/page_ext.h | 2 -- init/main.c | 4 ---- mm/mm_init.c | 6 +++++- 3 files changed, 5 insertions(+), 7 deletions(-)
include/linux/page_ext.h +0 -2
@@ -29,8 +29,6 @@ struct page_ext_operations {bool need_shared_flags; }; -extern bool deferred_struct_pages; -#ifdef CONFIG_PAGE_EXTENSION /*
init/main.c +0 -4
@@ -62,7 +62,6 @@#include <linux/rmap.h> #include <linux/mempolicy.h> #include <linux/key.h> -#include <linux/page_ext.h> #include <linux/debug_locks.h> #include <linux/debugobjects.h> #include <linux/lockdep.h>
@@ -1561,9 +1560,6 @@ static noinline void __init kernel_init_freeable(void)padata_init ();page_alloc_init_late (); -/* Initialize page ext after all struct pages are initialized. */ -if (deferred_struct_pages) -page_ext_init ();do_basic_setup ();
mm/mm_init.c +5 -1
@@ -225,7 +225,7 @@ static unsigned long nr_kernel_pages __initdata;static unsigned long nr_all_pages__initdata ;static unsigned long dma_reserve__initdata ; -bool deferred_struct_pages__meminitdata ; +static bool deferred_struct_pages__meminitdata ; static DEFINE_PER_CPU(struct per_cpu_nodestat, boot_nodestats);
@@ -2358,6 +2358,10 @@ void __init page_alloc_init_late(void)for_each_populated_zone (zone)set_zone_contiguous (zone); + +/* Initialize page ext after all struct pages are initialized. */ +if (deferred_struct_pages) +page_ext_init (); }#ifndef __HAVE_ARCH_RESERVED_KERNEL_PAGES
Signature · 1 line
2.35.1
Re: [PATCH v2 10/14] init,mm: fold late call to page_ext_init() to page_alloc_init_late()
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
When deferred initialization of struct pages is enabled,
page_ext_init()must be called after all the deferred initialization is done, but there is no point to keep it a separate call fromkernel_init_freeable()right afterpage_alloc_init_late().
Quoted 5 lines
Fold the call to
page_ext_init()intopage_alloc_init_late()and localizedeferred_struct_pagesvariable.Signed-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Mike Rapoport
maintainerPATCH v2 11/14 mm: move mem_init_print_info() to mm_init.c
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
mem_init_print_info() is only called from mm_core_init().
Move it close to the caller and make it static.
--- include/linux/mm.h | 1 - mm/internal.h | 1 + mm/mm_init.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 53 ---------------------------------------------- 4 files changed, 54 insertions(+), 54 deletions(-)
include/linux/mm.h +0 -1
@@ -2925,7 +2925,6 @@ extern unsigned long free_reserved_area(void *start, void *end, int poison, const char *s);extern void adjust_managed_page_count (struct page *page,long count); -extern void mem_init_print_info (void );extern void reserve_bootmem_region (phys_addr_t start,phys_addr_t end);
mm/internal.h +1 -0
@@ -201,6 +201,7 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);/* * in mm/page_alloc.c */ +#define K (x) ((x) << (PAGE_SHIFT-10))extern char *const zone_names[MAX_NR_ZONES ];
mm/mm_init.c +53 -0
65 lines not on this page · raw message
mm/page_alloc.c +0 -53
65 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCH v2 11/14] mm: move mem_init_print_info() to mm_init.c
Quoted 8 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
mem_init_print_info()is only called frommm_core_init().Move it close to the caller and make it static.
Signed-off-by Mike Rapoport (IBM) <[email protected]>
Mike Rapoport
maintainerPATCH v2 12/14 mm: move kmem_cache_init() declaration to mm/slab.h
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
kmem_cache_init() is called only from mm_core_init(), there is no need to declare it in include/linux/slab.h
Move kmem_cache_init() declaration to mm/slab.h
--- include/linux/slab.h | 1 - mm/mm_init.c | 1 + mm/slab.h | 1 + 3 files changed, 2 insertions(+), 1 deletion(-)
include/linux/slab.h +0 -1
@@ -167,7 +167,6 @@ struct mem_cgroup;/* * struct kmem_cache related prototypes */ -void __init kmem_cache_init (void );bool slab_is_available (void ); struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
mm/mm_init.c +1 -0
@@ -27,6 +27,7 @@#include <linux/swap.h> #include <linux/cma.h> #include "internal.h" +#include "slab.h" #include "shuffle.h" #include <asm/setup.h>
mm/slab.h +1 -0
@@ -4,6 +4,7 @@/* * Internal slab definitions */ +void __init kmem_cache_init (void );/* Reuses the bits in struct page */ struct slab {
Signature · 1 line
2.35.1
Re: [PATCH v2 12/14] mm: move kmem_cache_init() declaration to mm/slab.h
Quoted 9 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
kmem_cache_init()is called only frommm_core_init(), there is no need to declare it ininclude/linux/slab.hMove
kmem_cache_init()declaration tomm/slab.hSigned-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Mike Rapoport
maintainerPATCH v2 13/14 mm: move vmalloc_init() declaration to mm/internal.h
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
vmalloc_init() is called only from mm_core_init(), there is no need to declare it in include/linux/vmalloc.h
Move vmalloc_init() declaration to mm/internal.h
--- include/linux/vmalloc.h | 4 ---- mm/internal.h | 5 +++++ 2 files changed, 5 insertions(+), 4 deletions(-)
include/linux/vmalloc.h +0 -4
@@ -131,12 +131,8 @@ extern void *vm_map_ram(struct page **pages, unsigned int count, int node);extern void vm_unmap_aliases (void );#ifdef CONFIG_MMU -extern void __init vmalloc_init (void );extern unsigned long vmalloc_nr_pages (void );#else -static inline void vmalloc_init (void ) -{ -}static inline unsigned long vmalloc_nr_pages (void ) {return 0 ; }#endif
mm/internal.h +5 -0
@@ -900,9 +900,14 @@ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,* mm/vmalloc.c */ #ifdef CONFIG_MMU +void __init vmalloc_init (void );int vmap_pages_range_noflush (unsigned long addr,unsigned long end,pgprot_t prot,struct page **pages,unsigned int page_shift);#else +static inline void vmalloc_init (void ) +{ +} +static inline int vmap_pages_range_noflush (unsigned long addr,unsigned long end,pgprot_t prot,struct page **pages,unsigned int page_shift)
Signature · 1 line
2.35.1
Re: [PATCH v2 13/14] mm: move vmalloc_init() declaration to mm/internal.h
Quoted 9 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
vmalloc_init()is called only frommm_core_init(), there is no need to declare it ininclude/linux/vmalloc.hMove
vmalloc_init()declaration tomm/internal.hSigned-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Mike Rapoport
maintainerPATCH v2 14/14 MAINTAINERS: extend memblock entry to include MM initialization
In reply to Mike RapoportFrom: "Mike Rapoport (IBM)" <[email protected]>
and add mm/mm_init.c to memblock entry in MAINTAINERS
--- MAINTAINERS | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
MAINTAINERS +2 -1
@@ -13368,13 +13368,14 @@ F: arch/powerpc/include/asm/membarrier.h F: include/uapi/linux/membarrier.h F: kernel/sched/membarrier.c -MEMBLOCK +MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION M: Mike Rapoport <[email protected]> L: [email protected] S: Maintained F: Documentation/core-api/boot-time-mm.rst F: include/linux/memblock.h F: mm/memblock.c +F: mm/mm_init.c F: tools/testing/memblock/ MEMORY CONTROLLER DRIVERS
Signature · 1 line
2.35.1
Re: [PATCH v2 14/14] MAINTAINERS: extend memblock entry to include MM initialization
Quoted 6 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
and add
mm/mm_init.cto memblock entry inMAINTAINERSSigned-off-by Mike Rapoport (IBM) <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Quoted 24 lines
--- MAINTAINERS | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)MAINTAINERS +2
-1@@ -13368,13 +13368,14 @@ F: arch/powerpc/include/asm/membarrier.h F: include/uapi/linux/membarrier.h F: kernel/sched/membarrier.c -MEMBLOCK +MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION M: Mike Rapoport <[email protected]> L: [email protected] S: Maintained F: Documentation/core-api/boot-time-mm.rst F: include/linux/memblock.h F: mm/memblock.c +F: mm/mm_init.c F: tools/testing/memblock/ MEMORY CONTROLLER DRIVERS
Quoted 5 lines · Mike Rapoport
From: "Mike Rapoport (IBM)" <[email protected]>
Also in git:
https://git.kernel.org/rppt/h/mm-init/v2
Quoted 7 lines
v2:
* moveinit_cma_reserved_pageblock()fromcma.ctomm_init.c
* renameinit_mem_debugging_and_hardening()tomem_debugging_and_hardening_init()
* inlinepgtable_init()intomem_core_init()
* add Acked and Reviewed tags (thanks David, hopefully I've picked them right)
Sorry, I get lazy on large patches and only ACK instead of checking each and every line :)
Signature · 3 lines
Thanks, David / dhildenb