Deferred page init improvements
Subsystems (6)
v7 cover Deferred page init improvements
This patchset is essentially a refactor of the page initialization logic that is meant to provide for better code reuse while providing a significant improvement in deferred page initialization performance.
In my testing on an x86_64 system with 384GB of RAM I have seen the following. In the case of regular memory initialization the deferred init time was decreased from 3.75s to 1.38s on average. This amounts to a 172% improvement for the deferred memory initialization performance.
I have called out the improvement observed with each patch.
v1->v2:
Fixed build issue on PowerPC due to page struct size being 56 Added new patch that removed __SetPageReserved call for hotplugv2->v3:
Rebased on latest linux-next
Removed patch that had removed __SetPageReserved call from init Added patch that folded __SetPageReserved into set_page_links Tweaked __init_pageblock to use start_pfn to get section_nr instead of pfnv3->v4:
Updated patch description and comments for mm_zero_struct_page patch
Replaced "default" with "case 64"
Removed #ifndef mm_zero_struct_page
Fixed typo in comment that ommited "_from" in kerneldoc for iterator Added Reviewed-by for patches reviewed by Pavel
Added Acked-by from Michal Hocko
Added deferred init times for patches that affect init performance Swapped patches 5 & 6, pulled some code/comments from 4 into 5v4->v5:
Updated Acks/Reviewed-by
Rebased on latest linux-next
Split core bits of zone iterator patch from MAX_ORDER_NR_PAGES initv5->v6:
Rebased on linux-next with previous v5 reverted
Drop the "This patch" or "This change" from patch descriptions. Cleaned up patch descriptions for patches 3 & 4
Fixed kerneldoc for __next_mem_pfn_range_in_zone
Updated several Reviewed-by, and incorporated suggestions from Pavel Added __init_single_page_nolru to patch 5 to consolidate code Refactored iterator in patch 7 and fixed several issuesv6->v7:
Updated MAX_ORDER_NR_PAGES patch to stop on section aligned boundaries Dropped patches 5-7
Will follow-up later with reserved bit rework before resubmitting
---
Alexander Duyck (4):
mm: Use mm_zero_struct_page from SPARC on all 64b architectures
mm: Drop meminit_pfn_in_nid as it is redundant
mm: Implement new zone specific memblock iterator
mm: Initialize MAX_ORDER_NR_PAGES at a time instead of doing larger sections
arch/sparc/include/asm/pgtable_64.h | 30 ----- include/linux/memblock.h | 41 +++++++ include/linux/mm.h | 41 ++++++- mm/memblock.c | 64 ++++++++++ mm/page_alloc.c | 218 ++++++++++++++++++++++------------- 5 files changed, 277 insertions(+), 117 deletions(-)
--
PATCH v7 1/4 mm: Use mm_zero_struct_page from SPARC on all 64b architectures
From: Alexander Duyck <[email protected]>
Use the same approach that was already in use on Sparc on all the architectures that support a 64b long.
This is mostly motivated by the fact that 7 to 10 store/move instructions are likely always going to be faster than having to call into a function that is not specialized for handling page init.
An added advantage to doing it this way is that the compiler can get away with combining writes in the __init_single_page call. As a result the memset call will be reduced to only about 4 write operations, or at least that is what I am seeing with GCC 6.2 as the flags, LRU pointers, and count/mapcount seem to be cancelling out at least 4 of the 8 assignments on my system.
One change I had to make to the function was to reduce the minimum page size to 56 to support some powerpc64 configurations.
This change should introduce no change on SPARC since it already had this code. In the case of x86_64 I saw a reduction from 3.75s to 2.80s when initializing 384GB of RAM per node. Pavel Tatashin tested on a system with Broadcom's Stingray CPU and 48GB of RAM and found that __init_single_page() takes 19.30ns / 64-byte struct page before this patch and with this patch it takes 17.33ns / 64-byte struct page. Mike Rapoport ran a similar test on a OpenPower (S812LC 8348-21C) with Power8 processor and 128GB or RAM. His results per 64-byte struct page were 4.68ns before, and 4.59ns after this patch.
--- arch/sparc/include/asm/pgtable_64.h | 30 -------------------------- include/linux/mm.h | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 38 insertions(+), 33 deletions(-)
arch/sparc/include/asm/pgtable_64.h +0 -30
@@ -231,36 +231,6 @@extern struct page *mem_map_zero;#define ZERO_PAGE (vaddr) (mem_map_zero) -/* This macro must be updated when the size of struct page grows above 80 -* or reduces below 64. -* The idea that compiler optimizes out switch() statement, and only -* leaves clrx instructions -*/ -#define mm_zero_struct_page (pp) do { \ - unsigned long *_pp = (void *)(pp); \ - \ -/* Check that struct page is either 64, 72, or 80 bytes */ \ -BUILD_BUG_ON (sizeof (struct page ) &7 ); \ -BUILD_BUG_ON (sizeof (struct page ) <64 ); \ -BUILD_BUG_ON (sizeof (struct page ) >80 ); \ - \ -switch (sizeof (struct page )) { \ -case 80 : \ - _pp[9 ] =0 ;/* fallthrough */ \ -case 72 : \ - _pp[8 ] =0 ;/* fallthrough */ \ -default : \ - _pp[7 ] =0 ; \ - _pp[6 ] =0 ; \ - _pp[5 ] =0 ; \ - _pp[4 ] =0 ; \ - _pp[3 ] =0 ; \ - _pp[2 ] =0 ; \ - _pp[1 ] =0 ; \ - _pp[0 ] =0 ; \ - } \ -}while (0 ) -/* PFNs are real physical page numbers. However, mem_map only begins to record * per-page information starting at pfn_base. This is to handle systems where * the first physical page in the machine is at some huge physical address,
include/linux/mm.h +38 -3
48 lines not on this page · raw message
PATCH v7 2/4 mm: Drop meminit_pfn_in_nid as it is redundant
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
As best as I can tell the meminit_pfn_in_nid call is completely redundant. The deferred memory initialization is already making use of for_each_free_mem_range which in turn will call into __next_mem_range which will only return a memory range if it matches the node ID provided assuming it is not NUMA_NO_NODE.
I am operating on the assumption that there are no zones or pgdata_t structures that have a NUMA node of NUMA_NO_NODE associated with them. If that is the case then __next_mem_range will never return a memory range that doesn't match the zone's node ID and as such the check is redundant.
So one piece I would like to verify on this is if this works for ia64. Technically it was using a different approach to get the node ID, but it seems to have the node ID also encoded into the memblock. So I am assuming this is okay, but would like to get confirmation on that.
On my x86_64 test system with 384GB of memory per node I saw a reduction in initialization time from 2.80s to 1.85s as a result of this patch.
--- mm/page_alloc.c | 51 ++++++++++++++------------------------------------- 1 file changed, 14 insertions(+), 37 deletions(-)
mm/page_alloc.c +14 -37
@@ -1398,36 +1398,22 @@ int __meminit early_pfn_to_nid(unsigned long pfn)#endif #ifdef CONFIG_NODES_SPAN_OTHER_NODES -static inline bool __meminit __maybe_unused -meminit_pfn_in_nid (unsigned long pfn,int node, -struct mminit_pfnnid_cache *state) +/* Only safe to use early in boot when initialisation is single-threaded */ +static inline bool __meminit early_pfn_in_nid (unsigned long pfn,int node) {int nid; - nid =__early_pfn_to_nid (pfn, state); + nid =__early_pfn_to_nid (pfn, &early_pfnnid_cache);if (nid >=0 && nid != node)return false;return true; } -/* Only safe to use early in boot when initialisation is single-threaded */ -static inline bool __meminit early_pfn_in_nid (unsigned long pfn,int node) -{ -return meminit_pfn_in_nid (pfn, node, &early_pfnnid_cache); -} -#else -static inline bool __meminit early_pfn_in_nid (unsigned long pfn,int node) {return true; } -static inline bool __meminit __maybe_unused -meminit_pfn_in_nid (unsigned long pfn,int node, -struct mminit_pfnnid_cache *state) -{ -return true; -}#endif
@@ -1556,21 +1542,13 @@ static inline void __init pgdat_init_report_one_done(void)* * Then, we check if a current large page is valid by only checking the validity * of the head pfn. -* -* Finally, meminit_pfn_in_nid is checked on systems where pfns can interleave -* within a node: a pfn is between start and end of a node, but does not belong -* to this memory node. */ -static inline bool __init -deferred_pfn_valid (int nid,unsigned long pfn, -struct mminit_pfnnid_cache *nid_init_state) +static inline bool __init deferred_pfn_valid (unsigned long pfn) {if (!pfn_valid_within (pfn))return false;if (!(pfn & (pageblock_nr_pages -1 )) && !pfn_valid (pfn))return false; -if (!meminit_pfn_in_nid (pfn, nid, nid_init_state)) -return false;return true; }
@@ -1578,15 +1556,14 @@ static inline void __init pgdat_init_report_one_done(void)* Free pages to buddy allocator. Try to free aligned pages in * pageblock_nr_pages sizes. */ -static void __init deferred_free_pages (int nid,int zid,unsigned long pfn, +static void __init deferred_free_pages (unsigned long pfn,unsigned long end_pfn) { -struct mminit_pfnnid_cache nid_init_state = { };unsigned long nr_pgmask = pageblock_nr_pages -1 ;unsigned long nr_free =0 ;for (; pfn < end_pfn; pfn++) { -if (!deferred_pfn_valid (nid, pfn, &nid_init_state)) { +if (!deferred_pfn_valid (pfn)) {deferred_free_range (pfn - nr_free, nr_free); nr_free =0 ; }else if (!(pfn & nr_pgmask)) {
@@ -1606,17 +1583,18 @@ static void __init deferred_free_pages(int nid, int zid, unsigned long pfn,* by performing it only once every pageblock_nr_pages. * Return number of pages initialized. */ -static unsigned long __init deferred_init_pages (int nid,int zid, +static unsigned long __init deferred_init_pages (struct zone *zone,unsigned long pfn,unsigned long end_pfn) { -struct mminit_pfnnid_cache nid_init_state = { };unsigned long nr_pgmask = pageblock_nr_pages -1 ; +int nid =zone_to_nid (zone);unsigned long nr_pages =0 ; +int zid =zone_idx (zone);struct page *page =NULL ;for (; pfn < end_pfn; pfn++) { -if (!deferred_pfn_valid (nid, pfn, &nid_init_state)) { +if (!deferred_pfn_valid (pfn)) { page =NULL ;continue ; }else if (!page || !(pfn & nr_pgmask)) {
@@ -1679,12 +1657,12 @@ static int __init deferred_init_memmap(void *data)for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); - nr_pages +=deferred_init_pages (nid, zid, spfn, epfn); + nr_pages +=deferred_init_pages (zone, spfn, epfn); }for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); -deferred_free_pages (nid, zid, spfn, epfn); +deferred_free_pages (spfn, epfn); }pgdat_resize_unlock (pgdat, &flags);
@@ -1716,7 +1694,6 @@ static int __init deferred_init_memmap(void *data) static noinline bool __initdeferred_grow_zone (struct zone *zone,unsigned int order) { -int zid =zone_idx (zone);int nid =zone_to_nid (zone);pg_data_t *pgdat =NODE_DATA (nid);unsigned long nr_pages_needed =ALIGN (1 << order,PAGES_PER_SECTION );
@@ -1766,7 +1743,7 @@ static int __init deferred_init_memmap(void *data)while (spfn < epfn && nr_pages < nr_pages_needed) { t =ALIGN (spfn +PAGES_PER_SECTION ,PAGES_PER_SECTION ); first_deferred_pfn =min (t, epfn); - nr_pages +=deferred_init_pages (nid, zid, spfn, + nr_pages +=deferred_init_pages (zone, spfn, first_deferred_pfn); spfn = first_deferred_pfn; }
@@ -1778,7 +1755,7 @@ static int __init deferred_init_memmap(void *data)for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); epfn = min_t(unsigned long, first_deferred_pfn, PFN_DOWN(epa)); -deferred_free_pages (nid, zid, spfn, epfn); +deferred_free_pages (spfn, epfn);if (first_deferred_pfn = epfn)break ;
PATCH v7 3/4 mm: Implement new zone specific memblock iterator
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
Introduce a new iterator for_each_free_mem_pfn_range_in_zone.
This iterator will take care of making sure a given memory range provided is in fact contained within a zone. It takes are of all the bounds checking we were doing in deferred_grow_zone, and deferred_init_memmap. In addition it should help to speed up the search a bit by iterating until the end of a range is greater than the start of the zone pfn range, and will exit completely if the start is beyond the end of the zone.
--- include/linux/memblock.h | 25 ++++++++++++++++++ mm/memblock.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 31 +++++++++------------- 3 files changed, 101 insertions(+), 19 deletions(-)
include/linux/memblock.h +25 -0
@@ -240,6 +240,31 @@ void __next_mem_pfn_range(int *idx, int nid, unsigned long *out_start_pfn, i >= 0; __next_mem_pfn_range(&i, nid, p_start, p_end, p_nid))#endif /* CONFIG_HAVE_MEMBLOCK_NODE_MAP */ +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +void __next_mem_pfn_range_in_zone (u64 *idx,struct zone *zone, +unsigned long *out_spfn, +unsigned long *out_epfn); +/** +* for_each_free_mem_range_in_zone - iterate through zone specific free +* memblock areas +* @i: u64 used as loop variable +* @zone: zone in which all of the memory blocks reside +* @p_start: ptr to phys_addr_t for start address of the range, can be %NULL +* @p_end: ptr to phys_addr_t for end address of the range, can be %NULL +* +* Walks over free (memory && !reserved) areas of memblock in a specific +* zone. Available once memblock and an empty zone is initialized. The main +* assumption is that the zone start, end, and pgdat have been associated. +* This way we can use the zone to determine NUMA node, and if a given part +* of the memblock is valid for the zone. +*/ +#define for_each_free_mem_pfn_range_in_zone (i, zone, p_start, p_end) \ + for (i = 0, \ + __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end); \ + i != U64_MAX; \ + __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end)) +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ +/** * for_each_free_mem_range - iterate through free memblock areas * @i: u64 used as loop variable
mm/memblock.c +64 -0
70 lines not on this page · raw message
mm/page_alloc.c +12 -19
@@ -1613,11 +1613,9 @@ static unsigned long __init deferred_init_pages(struct zone *zone,static int __init deferred_init_memmap (void *data) {pg_data_t *pgdat = data; -int nid = pgdat->node_id;unsigned long start = jiffies;unsigned long nr_pages =0 ;unsigned long spfn, epfn, first_init_pfn, flags; -phys_addr_t spa, epa;int zid;struct zone *zone;const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id);
@@ -1654,14 +1652,12 @@ static int __init deferred_init_memmap(void *data)* freeing pages we can access pages that are ahead (computing buddy * page in __free_one_page()). */ -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn); nr_pages +=deferred_init_pages (zone, spfn, epfn); } -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn);deferred_free_pages (spfn, epfn); }pgdat_resize_unlock (pgdat, &flags);
@@ -1669,8 +1665,8 @@ static int __init deferred_init_memmap(void *data)/* Sanity check that the next zone really is unpopulated */ WARN_ON (++zid <MAX_NR_ZONES &&populated_zone (++zone)); -pr_info ("node %d initialised, %lu pages in %ums\n" , nid, nr_pages, -jiffies_to_msecs (jiffies - start)); +pr_info ("node %d initialised, %lu pages in %ums\n" , + pgdat->node_id, nr_pages,jiffies_to_msecs (jiffies - start));pgdat_init_report_one_done ();return 0 ;
@@ -1694,13 +1690,11 @@ static int __init deferred_init_memmap(void *data) static noinline bool __initdeferred_grow_zone (struct zone *zone,unsigned int order) { -int nid =zone_to_nid (zone); -pg_data_t *pgdat =NODE_DATA (nid);unsigned long nr_pages_needed =ALIGN (1 << order,PAGES_PER_SECTION ); +pg_data_t *pgdat = zone->zone_pgdat;unsigned long nr_pages =0 ;unsigned long first_init_pfn, spfn, epfn, t, flags;unsigned long first_deferred_pfn = pgdat->first_deferred_pfn; -phys_addr_t spa, epa;u64 i;/* Only the last zone may have deferred pages */
@@ -1736,9 +1730,8 @@ static int __init deferred_init_memmap(void *data)return false; } -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn);while (spfn < epfn && nr_pages < nr_pages_needed) { t =ALIGN (spfn +PAGES_PER_SECTION ,PAGES_PER_SECTION );
@@ -1752,9 +1745,9 @@ static int __init deferred_init_memmap(void *data)break ; } -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, first_deferred_pfn, PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn); + epfn = min_t(unsigned long, first_deferred_pfn, epfn);deferred_free_pages (spfn, epfn);if (first_deferred_pfn = epfn)
Re: [mm PATCH v7 3/4] mm: Implement new zone specific memblock iterator
Quoted 11 lines · Alexander Duyck
From: Alexander Duyck <alexander.h.duyck-VuQAYsv1563Yd54FQh9/[email protected]>
Introduce a new iterator
for_each_free_mem_pfn_range_in_zone.This iterator will take care of making sure a given memory range provided is in fact contained within a zone. It takes are of all the bounds checking we were doing in
deferred_grow_zone, anddeferred_init_memmap. In addition it should help to speed up the search a bit by iterating until the end of a range is greater than the start of the zone pfn range, and will exit completely if the start is beyond the end of the zone.
Quoted 2 lines
Quoted 204 lines
--- include/linux/memblock.h | 25 ++++++++++++++++++ mm/memblock.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 31 +++++++++------------- 3 files changed, 101 insertions(+), 19 deletions(-)include/linux/memblock.h +25
-0@@ -240,6 +240,31 @@ void __next_mem_pfn_range(int *idx, int nid, unsigned long *out_start_pfn, i >= 0; __next_mem_pfn_range(&i, nid, p_start, p_end, p_nid))#endif /* CONFIG_HAVE_MEMBLOCK_NODE_MAP */ +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +void __next_mem_pfn_range_in_zone (u64 *idx,struct zone *zone, +unsigned long *out_spfn, +unsigned long *out_epfn); +/** +* for_each_free_mem_range_in_zone - iterate through zone specific free +* memblock areas +* @i: u64 used as loop variable +* @zone: zone in which all of the memory blocks reside +* @p_start: ptr to phys_addr_t for start address of the range, can be %NULL +* @p_end: ptr to phys_addr_t for end address of the range, can be %NULL +* +* Walks over free (memory && !reserved) areas of memblock in a specific +* zone. Available once memblock and an empty zone is initialized. The main +* assumption is that the zone start, end, and pgdat have been associated. +* This way we can use the zone to determine NUMA node, and if a given part +* of the memblock is valid for the zone. +*/ +#define for_each_free_mem_pfn_range_in_zone (i, zone, p_start, p_end) \ + for (i = 0, \ + __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end); \ + i != U64_MAX; \ + __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end)) +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ +/** * for_each_free_mem_range - iterate through free memblock areas * @i: u64 used as loop variable mm/memblock.c +64
-0@@ -1255,6 +1255,70 @@ int __init_memblock memblock_set_node(phys_addr_t base, phys_addr_t size,return 0 ; }#endif /* CONFIG_HAVE_MEMBLOCK_NODE_MAP */ +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +/** +* __next_mem_pfn_range_in_zone - iterator for for_each_*_range_in_zone() +* +* @idx: pointer to u64 loop variable +* @zone: zone in which all of the memory blocks reside +* @out_spfn: ptr to ulong for start pfn of the range, can be %NULL +* @out_epfn: ptr to ulong for end pfn of the range, can be %NULL +* +* This function is meant to be a zone/pfn specific wrapper for the +* for_each_mem_range type iterators. Specifically they are used in the +* deferred memory init routines and as such we were duplicating much of +* this logic throughout the code. So instead of having it in multiple +* locations it seemed like it would make more sense to centralize this to +* one new iterator that does everything they need. +*/ +void __init_memblock +__next_mem_pfn_range_in_zone (u64 *idx,struct zone *zone, +unsigned long *out_spfn,unsigned long *out_epfn) +{ +int zone_nid =zone_to_nid (zone); +phys_addr_t spa, epa; +int nid; + +__next_mem_range (idx, zone_nid,MEMBLOCK_NONE , + &memblock.memory, &memblock.reserved, + &spa, &epa, &nid); + +while (*idx !=U64_MAX ) { +unsigned long epfn =PFN_DOWN (epa); +unsigned long spfn =PFN_UP (spa); + +/* +* Verify the end is at least past the start of the zone and +* that we have at least one PFN to initialize. +*/ +if (zone->zone_start_pfn < epfn && spfn < epfn) { +/* if we went too far just stop searching */ +if (zone_end_pfn (zone) <= spfn) { + *idx =U64_MAX ; +break ; + } + +if (out_spfn) + *out_spfn =max (zone->zone_start_pfn, spfn); +if (out_epfn) + *out_epfn =min (zone_end_pfn (zone), epfn); + +return ; + } + +__next_mem_range (idx, zone_nid,MEMBLOCK_NONE , + &memblock.memory, &memblock.reserved, + &spa, &epa, &nid); + } + +/* signal end of iteration */ +if (out_spfn) + *out_spfn =ULONG_MAX ; +if (out_epfn) + *out_epfn =0 ; +} + +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ /** * memblock_alloc_range_nid - allocate boot memory block mm/page_alloc.c +12
-19@@ -1613,11 +1613,9 @@ static unsigned long __init deferred_init_pages(struct zone *zone,static int __init deferred_init_memmap (void *data) {pg_data_t *pgdat = data; -int nid = pgdat->node_id;unsigned long start = jiffies;unsigned long nr_pages =0 ;unsigned long spfn, epfn, first_init_pfn, flags; -phys_addr_t spa, epa;int zid;struct zone *zone;const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id);@@ -1654,14 +1652,12 @@ static int __init deferred_init_memmap(void *data)* freeing pages we can access pages that are ahead (computing buddy * page in __free_one_page()). */ -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn); nr_pages +=deferred_init_pages (zone, spfn, epfn); } -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn);deferred_free_pages (spfn, epfn); }pgdat_resize_unlock (pgdat, &flags);@@ -1669,8 +1665,8 @@ static int __init deferred_init_memmap(void *data)/* Sanity check that the next zone really is unpopulated */ WARN_ON (++zid <MAX_NR_ZONES &&populated_zone (++zone)); -pr_info ("node %d initialised, %lu pages in %ums\n" , nid, nr_pages, -jiffies_to_msecs (jiffies - start)); +pr_info ("node %d initialised, %lu pages in %ums\n" , + pgdat->node_id, nr_pages,jiffies_to_msecs (jiffies - start));pgdat_init_report_one_done ();return 0 ;@@ -1694,13 +1690,11 @@ static int __init deferred_init_memmap(void *data) static noinline bool __initdeferred_grow_zone (struct zone *zone,unsigned int order) { -int nid =zone_to_nid (zone); -pg_data_t *pgdat =NODE_DATA (nid);unsigned long nr_pages_needed =ALIGN (1 << order,PAGES_PER_SECTION ); +pg_data_t *pgdat = zone->zone_pgdat;unsigned long nr_pages =0 ;unsigned long first_init_pfn, spfn, epfn, t, flags;unsigned long first_deferred_pfn = pgdat->first_deferred_pfn; -phys_addr_t spa, epa;u64 i;/* Only the last zone may have deferred pages */ @@ -1736,9 +1730,8 @@ static int __init deferred_init_memmap(void *data)return false; } -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, zone_end_pfn(zone), PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn);while (spfn < epfn && nr_pages < nr_pages_needed) { t =ALIGN (spfn +PAGES_PER_SECTION ,PAGES_PER_SECTION );@@ -1752,9 +1745,9 @@ static int __init deferred_init_memmap(void *data)break ; } -for_each_free_mem_range (i, nid,MEMBLOCK_NONE , &spa, &epa,NULL ) { - spfn = max_t(unsigned long, first_init_pfn, PFN_UP(spa)); - epfn = min_t(unsigned long, first_deferred_pfn, PFN_DOWN(epa)); +for_each_free_mem_pfn_range_in_zone (i, zone, &spfn, &epfn) { + spfn = max_t(unsigned long, first_init_pfn, spfn); + epfn = min_t(unsigned long, first_deferred_pfn, epfn);deferred_free_pages (spfn, epfn);if (first_deferred_pfn == epfn)
Signature · 2 lines
Sincerely yours, Mike.
PATCH v7 4/4 mm: Initialize MAX_ORDER_NR_PAGES at a time instead of doing larger sections
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
Add yet another iterator, for_each_free_mem_range_in_zone_from, and then use it to support initializing and freeing pages in groups no larger than MAX_ORDER_NR_PAGES. By doing this we can greatly improve the cache locality of the pages while we do several loops over them in the init and freeing process.
We are able to tighten the loops further as a result of the "from" iterator as we can perform the initial checks for first_init_pfn in our first call to the iterator, and continue without the need for those checks via the "from" iterator. I have added this functionality in the function called deferred_init_mem_pfn_range_in_zone that primes the iterator and causes us to exit if we encounter any failure.
On my x86_64 test system with 384GB of memory per node I saw a reduction in initialization time from 1.85s to 1.38s as a result of this patch.
--- include/linux/memblock.h | 16 +++++ mm/page_alloc.c | 162 ++++++++++++++++++++++++++++++++++------------ 2 files changed, 137 insertions(+), 41 deletions(-)
include/linux/memblock.h +16 -0
@@ -263,6 +263,22 @@ void __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone,__next_mem_pfn_range_in_zone (&i, zone, p_start, p_end); \ i !=U64_MAX ; \ __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end)) + +/** +* for_each_free_mem_range_in_zone_from - iterate through zone specific +* free memblock areas from a given point +* @i: u64 used as loop variable +* @zone: zone in which all of the memory blocks reside +* @p_start: ptr to phys_addr_t for start address of the range, can be %NULL +* @p_end: ptr to phys_addr_t for end address of the range, can be %NULL +* +* Walks over free (memory && !reserved) areas of memblock in a specific +* zone, continuing from current position. Available as soon as memblock is +* initialized. +*/ +#define for_each_free_mem_pfn_range_in_zone_from (i, zone, p_start, p_end) \ + for (; i != U64_MAX; \ + __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end))#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ /**
mm/page_alloc.c +121 -41
205 lines not on this page · raw message