Transparent on-demand struct page initialization embedded in the buddy allocator

Currently, when free_all_bootmem() calls __free_pages_memory(), the number of contiguous pages that __free_pages_memory() passes to the buddy allocator is limited to BITS_PER_LONG. In order to be able to free only the first page of a 2MiB chunk, we need that to be increased to PTRS_PER_PMD.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>

---
 mm/nobootmem.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

mm/nobootmem.c +4 -4

index bdd3fa2..3b512ca 100644
@@ -83,10 +83,10 @@ void __init free_bootmem_late(unsigned long addr, unsigned long size)
 static void __init __free_pages_memory(unsigned long start, unsigned long end)
 {
 	unsigned long i, start_aligned, end_aligned;
-	int order = ilog2(BITS_PER_LONG);
+	int order = ilog2(max(BITS_PER_LONG, PTRS_PER_PMD));
 
-	start_aligned = (start + (BITS_PER_LONG - 1)) & ~(BITS_PER_LONG - 1);
-	end_aligned = end & ~(BITS_PER_LONG - 1);
+	start_aligned = (start + ((1UL << order) - 1)) & ~((1UL << order) - 1);
+	end_aligned = end & ~((1UL << order) - 1);
 
 	if (end_aligned <= start_aligned) {
 		for (i = start; i < end; i++)

@@ -98,7 +98,7 @@ static void __init __free_pages_memory(unsigned long start, unsigned long end)
 	for (i = start; i < start_aligned; i++)
 		__free_pages_bootmem(pfn_to_page(i), 0);
 
-	for (i = start_aligned; i < end_aligned; i += BITS_PER_LONG)
+	for (i = start_aligned; i < end_aligned; i += 1 << order)
 		__free_pages_bootmem(pfn_to_page(i), order);
 
 	for (i = end_aligned; i < end; i++)

Signature · 1 line

1.8.2.1

At the very least, I think we could change to:
static void __init __free_pages_memory(unsigned long start, unsigned long end)

{
        int order;

        while (start < end) {
                order = ffs(start);

                while (start + (1UL << order) > end)
                        order--;

                __free_pages_bootmem(start, order);

                start += (1UL << order);
        }
}

Currently, when free_all_bootmem() calls __free_pages_memory(), the number of contiguous pages that __free_pages_memory() passes to the buddy allocator is limited to BITS_PER_LONG. In order to be able to

free only the first page of a 2MiB chunk, we need that to be increased to PTRS_PER_PMD.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>

---
  mm/nobootmem.c | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)

mm/nobootmem.c +3 -3

index bdd3fa2..3b512ca 100644
@@ -83,10 +83,10 @@ void __init free_bootmem_late(unsigned long addr, unsigned long size)
  static void __init __free_pages_memory(unsigned long start, unsigned long end)
  {
  	unsigned long i, start_aligned, end_aligned;
-	int order = ilog2(BITS_PER_LONG);
+	int order = ilog2(max(BITS_PER_LONG, PTRS_PER_PMD));
 
-	start_aligned = (start + (BITS_PER_LONG - 1)) & ~(BITS_PER_LONG - 1);
-	end_aligned = end & ~(BITS_PER_LONG - 1);
+	start_aligned = (start + ((1UL << order) - 1)) & ~((1UL << order) - 1);
+	end_aligned = end & ~((1UL << order) - 1);
 
  	if (end_aligned <= start_aligned) {
  		for (i = start; i < end; i++)

Hi Robin,

Quoted 3 lines · Robin Holt

Currently, when free_all_bootmem() calls __free_pages_memory(), the number of contiguous pages that __free_pages_memory() passes to the buddy allocator is limited to BITS_PER_LONG. In order to be able to

I fail to understand this. Why the original page number is BITS_PER_LONG?

Currently, memmap_init_zone() has all the smarts for initializing a single page. When we convert to initializing pages in a 2MiB chunk, we will need to do this equivalent work from two separate places so we are breaking out a helper function.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>

---
 mm/mm_init.c    |  2 +-
 mm/page_alloc.c | 75 +++++++++++++++++++++++++++++++++------------------------
 2 files changed, 45 insertions(+), 32 deletions(-)

mm/mm_init.c +1 -1

index c280a02..be8a539 100644
@@ -128,7 +128,7 @@ void __init mminit_verify_pageflags_layout(void)
        BUG_ON(or_mask != add_mask);
 }
 
-void __meminit mminit_verify_page_links(struct page *page, enum zone_type zone,
+void mminit_verify_page_links(struct page *page, enum zone_type zone,
                        unsigned long nid, unsigned long pfn)
 {
        BUG_ON(page_to_nid(page) != nid);

mm/page_alloc.c +43 -22

index c3edb62..635b131 100644
@@ -697,6 +697,49 @@ static void free_one_page(struct zone *zone, struct page *page, int order,
        spin_unlock(&zone->lock);
 }
 
+static void __init_single_page(struct page *page, unsigned long zone, int nid, int reserved)
+{
+       unsigned long pfn = page_to_pfn(page);
+       struct zone *z = &NODE_DATA(nid)->node_zones[zone];
+
+       set_page_links(page, zone, nid, pfn);
+       mminit_verify_page_links(page, zone, nid, pfn);
+       init_page_count(page);
+       page_mapcount_reset(page);
+       page_nid_reset_last(page);
+       if (reserved) {
+               SetPageReserved(page);
+       } else {
+               ClearPageReserved(page);
+               set_page_count(page, 0);
+       }
+       /*
+        * Mark the block movable so that blocks are reserved for
+        * movable at startup. This will force kernel allocations
+        * to reserve their blocks rather than leaking throughout
+        * the address space during boot when many long-lived
+        * kernel allocations are made. Later some blocks near
+        * the start are marked MIGRATE_RESERVE by
+        * setup_zone_migrate_reserve()
+        *
+        * bitmap is created for zone's valid pfn range. but memmap
+        * can be created for invalid pages (for alignment)
+        * check here not to call set_pageblock_migratetype() against
+        * pfn out of zone.
+        */
+       if ((z->zone_start_pfn <= pfn)
+           && (pfn < zone_end_pfn(z))
+           && !(pfn & (pageblock_nr_pages - 1)))
+               set_pageblock_migratetype(page, MIGRATE_MOVABLE);
+
+       INIT_LIST_HEAD(&page->lru);
+#ifdef WANT_PAGE_VIRTUAL
+       /* The shift won't overflow because ZONE_NORMAL is below 4G. */
+       if (!is_highmem_idx(zone))
+               set_page_address(page, __va(pfn << PAGE_SHIFT));
+#endif
+}
+
 static bool free_pages_prepare(struct page *page, unsigned int order)
 {
        int i;

@@ -3934,37 +3977,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
                                continue;
                }
                page = pfn_to_page(pfn);
-               set_page_links(page, zone, nid, pfn);
-               mminit_verify_page_links(page, zone, nid, pfn);
-               init_page_count(page);
-               page_mapcount_reset(page);
-               page_nid_reset_last(page);
-               SetPageReserved(page);
-               /*
-                * Mark the block movable so that blocks are reserved for
-                * movable at startup. This will force kernel allocations
-                * to reserve their blocks rather than leaking throughout
-                * the address space during boot when many long-lived
-                * kernel allocations are made. Later some blocks near
-                * the start are marked MIGRATE_RESERVE by
-                * setup_zone_migrate_reserve()
-                *
-                * bitmap is created for zone's valid pfn range. but memmap
-                * can be created for invalid pages (for alignment)
-                * check here not to call set_pageblock_migratetype() against
-                * pfn out of zone.
-                */
-               if ((z->zone_start_pfn <= pfn)
-                   && (pfn < zone_end_pfn(z))

Quoted 122 lines · Robin Holt

Currently, memmap_init_zone() has all the smarts for initializing a single page. When we convert to initializing pages in a 2MiB chunk, we will need to do this equivalent work from two separate places so we are breaking out a helper function.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>

---
 mm/mm_init.c    |  2 +-
 mm/page_alloc.c | 75 +++++++++++++++++++++++++++++++++------------------------
 2 files changed, 45 insertions(+), 32 deletions(-)

mm/mm_init.c +1 -1

index c280a02..be8a539 100644
@@ -128,7 +128,7 @@ void __init mminit_verify_pageflags_layout(void)
        BUG_ON(or_mask != add_mask);
 }
 
-void __meminit mminit_verify_page_links(struct page *page, enum zone_type zone,
+void mminit_verify_page_links(struct page *page, enum zone_type zone,
                        unsigned long nid, unsigned long pfn)
 {
        BUG_ON(page_to_nid(page) != nid);

mm/page_alloc.c +43 -28

index c3edb62..635b131 100644
@@ -697,6 +697,49 @@ static void free_one_page(struct zone *zone, struct page *page, int order,
        spin_unlock(&zone->lock);
 }
 
+static void __init_single_page(struct page *page, unsigned long zone, int nid, int reserved)
+{
+       unsigned long pfn = page_to_pfn(page);
+       struct zone *z = &NODE_DATA(nid)->node_zones[zone];
+
+       set_page_links(page, zone, nid, pfn);
+       mminit_verify_page_links(page, zone, nid, pfn);
+       init_page_count(page);
+       page_mapcount_reset(page);
+       page_nid_reset_last(page);
+       if (reserved) {
+               SetPageReserved(page);
+       } else {
+               ClearPageReserved(page);
+               set_page_count(page, 0);
+       }
+       /*
+        * Mark the block movable so that blocks are reserved for
+        * movable at startup. This will force kernel allocations
+        * to reserve their blocks rather than leaking throughout
+        * the address space during boot when many long-lived
+        * kernel allocations are made. Later some blocks near
+        * the start are marked MIGRATE_RESERVE by
+        * setup_zone_migrate_reserve()
+        *
+        * bitmap is created for zone's valid pfn range. but memmap
+        * can be created for invalid pages (for alignment)
+        * check here not to call set_pageblock_migratetype() against
+        * pfn out of zone.
+        */
+       if ((z->zone_start_pfn <= pfn)
+           && (pfn < zone_end_pfn(z))
+           && !(pfn & (pageblock_nr_pages - 1)))
+               set_pageblock_migratetype(page, MIGRATE_MOVABLE);
+
+       INIT_LIST_HEAD(&page->lru);
+#ifdef WANT_PAGE_VIRTUAL
+       /* The shift won't overflow because ZONE_NORMAL is below 4G. */
+       if (!is_highmem_idx(zone))
+               set_page_address(page, __va(pfn << PAGE_SHIFT));
+#endif
+}
+
 static bool free_pages_prepare(struct page *page, unsigned int order)
 {
        int i;

@@ -3934,37 +3977,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
                                continue;
                }
                page = pfn_to_page(pfn);
-               set_page_links(page, zone, nid, pfn);
-               mminit_verify_page_links(page, zone, nid, pfn);
-               init_page_count(page);
-               page_mapcount_reset(page);
-               page_nid_reset_last(page);
-               SetPageReserved(page);
-               /*
-                * Mark the block movable so that blocks are reserved for
-                * movable at startup. This will force kernel allocations
-                * to reserve their blocks rather than leaking throughout
-                * the address space during boot when many long-lived
-                * kernel allocations are made. Later some blocks near
-                * the start are marked MIGRATE_RESERVE by
-                * setup_zone_migrate_reserve()
-                *
-                * bitmap is created for zone's valid pfn range. but memmap
-                * can be created for invalid pages (for alignment)
-                * check here not to call set_pageblock_migratetype() against
-                * pfn out of zone.
-                */
-               if ((z->zone_start_pfn <= pfn)
-                   && (pfn < zone_end_pfn(z))
-                   && !(pfn & (pageblock_nr_pages - 1)))
-                       set_pageblock_migratetype(page, MIGRATE_MOVABLE);
-
-               INIT_LIST_HEAD(&page->lru);
-#ifdef WANT_PAGE_VIRTUAL
-               /* The shift won't overflow because ZONE_NORMAL is below 4G. */

mm/page_alloc.c void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, · line 3980

Quoted 4 lines

mm/page_alloc.c +1 -3

-               if (!is_highmem_idx(zone))
-                       set_page_address(page, __va(pfn << PAGE_SHIFT));
-#endif
+               __init_single_page(page, zone, nid, 1);

Can you
move page = pfn_to_page(pfn) into __init_single_page and pass pfn directly?

During boot of large memory machines, a significant portion of boot is spent initializing the struct page array. The vast majority of those pages are not referenced during boot.

Change this over to only initializing the pages when they are actually allocated.

Besides the advantage of boot speed, this allows us the chance to use normal performance monitoring tools to determine where the bulk of time is spent during page initialization.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>

---
 include/linux/mm.h         |  11 +++++
 include/linux/page-flags.h |   5 +-
 mm/nobootmem.c             |   5 ++
 mm/page_alloc.c            | 117 +++++++++++++++++++++++++++++++++++++++++++--
 4 files changed, 132 insertions(+), 6 deletions(-)

include/linux/mm.h +11 -0

index e0c8528..3de08b5 100644
@@ -1330,8 +1330,19 @@ static inline void __free_reserved_page(struct page *page)
        __free_page(page);
 }
 
+extern void __reserve_bootmem_region(phys_addr_t start, phys_addr_t end);
+
+static inline void __reserve_bootmem_page(struct page *page)
+{
+       phys_addr_t start = page_to_pfn(page) << PAGE_SHIFT;
+       phys_addr_t end = start + PAGE_SIZE;
+
+       __reserve_bootmem_region(start, end);
+}
+
 static inline void free_reserved_page(struct page *page)
 {
+       __reserve_bootmem_page(page);
        __free_reserved_page(page);
        adjust_managed_page_count(page, 1);
 }

include/linux/page-flags.h +4 -1

index 6d53675..79e8eb7 100644
@@ -83,6 +83,7 @@ enum pageflags {
        PG_owner_priv_1,        /* Owner use. If pagecache, fs may use*/
        PG_arch_1,
        PG_reserved,
+       PG_uninitialized2mib,   /* Is this the right spot? ntz - Yes - rmh */
        PG_private,             /* If pagecache, has fs-private data */
        PG_private_2,           /* If pagecache, has fs aux data */
        PG_writeback,           /* Page is under writeback */

@@ -211,6 +212,8 @@ PAGEFLAG(SwapBacked, swapbacked) __CLEARPAGEFLAG(SwapBacked, swapbacked)
 
 __PAGEFLAG(SlobFree, slob_free)
 
+PAGEFLAG(Uninitialized2Mib, uninitialized2mib)
+
 /*
  * Private page markings that may be used by the filesystem that owns the page
  * for its own purposes.

@@ -499,7 +502,7 @@ static inline void ClearPageSlabPfmemalloc(struct page *page)
 #define PAGE_FLAGS_CHECK_AT_FREE \
        (1 << PG_lru     | 1 << PG_locked    | \
         1 << PG_private | 1 << PG_private_2 | \
-        1 << PG_writeback | 1 << PG_reserved | \
+        1 << PG_writeback | 1 << PG_reserved | 1 << PG_uninitialized2mib | \
         1 << PG_slab    | 1 << PG_swapcache | 1 << PG_active | \
         1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON | \
         __PG_COMPOUND_LOCK)

mm/nobootmem.c +0 -0

index 3b512ca..e3a386d 100644

mm/nobootmem.c +0 -0

        for_each_online_pgdat(pgdat)
                reset_node_lowmem_managed_pages(pgdat);
 

mm/page_alloc.c +32 -0

index 635b131..fe51eb9 100644
@@ -740,6 +740,54 @@ static void __init_single_page(struct page *page, unsigned long zone, int nid, i
 #endif
 }
 
+static void expand_page_initialization(struct page *basepage)
+{
+       unsigned long pfn = page_to_pfn(basepage);
+       unsigned long end_pfn = pfn + PTRS_PER_PMD;
+       unsigned long zone = page_zonenum(basepage);
+       int reserved = PageReserved(basepage);
+       int nid = page_to_nid(basepage);
+
+       ClearPageUninitialized2Mib(basepage);
+
+       for( pfn++; pfn < end_pfn; pfn++ )
+               __init_single_page(pfn_to_page(pfn), zone, nid, reserved);
+}
+
+void ensure_pages_are_initialized(unsigned long start_pfn,
+                                 unsigned long end_pfn)
+{
+       unsigned long aligned_start_pfn = start_pfn & ~(PTRS_PER_PMD - 1);
+       unsigned long aligned_end_pfn;
+       struct page *page;
+
+       aligned_end_pfn = end_pfn & ~(PTRS_PER_PMD - 1);
+       aligned_end_pfn += PTRS_PER_PMD;
+       while (aligned_start_pfn < aligned_end_pfn) {
+               if (pfn_valid(aligned_start_pfn)) {
+                       page = pfn_to_page(aligned_start_pfn);
+
+                       if(PageUninitialized2Mib(page))
+                               expand_page_initialization(page);
+               }
+
+               aligned_start_pfn += PTRS_PER_PMD;

mm/page_alloc.c +24 -2

+
 static bool free_pages_prepare(struct page *page, unsigned int order)
 {
        int i;

@@ -751,7 +799,10 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
        if (PageAnon(page))
                page->mapping = NULL;
        for (i = 0; i < (1 << order); i++)
-               bad += free_pages_check(page + i);
+               if (PageUninitialized2Mib(page + i))
+                       i += PTRS_PER_PMD - 1;
+               else
+                       bad += free_pages_check(page + i);
        if (bad)
                return false;
 

@@ -795,13 +846,22 @@ void __meminit __free_pages_bootmem(struct page *page, unsigned int order)
        unsigned int loop;
 
        prefetchw(page);
-       for (loop = 0; loop < nr_pages; loop++) {
+       for (loop = 0; loop < nr_pages; ) {
                struct page *p = &page[loop];
 
                if (loop + 1 < nr_pages)
                        prefetchw(p + 1);
+
+               if ((PageUninitialized2Mib(p)) &&
+                   ((loop + PTRS_PER_PMD) > nr_pages))
+                       ensure_page_is_initialized(p);
+
                __ClearPageReserved(p);
                set_page_count(p, 0);
+               if (PageUninitialized2Mib(p))
+                       loop += PTRS_PER_PMD;
+               else
+                       loop += 1;
        }
 
        page_zone(page)->managed_pages += 1 << order;

@@ -856,6 +916,7 @@ static inline void expand(struct zone *zone, struct page *page,
                area--;
                high--;
                size >>= 1;
+               ensure_page_is_initialized(page);
                VM_BUG_ON(bad_range(zone, &page[size]));
 
 #ifdef CONFIG_DEBUG_PAGEALLOC

@@ -903,6 +964,10 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
+
+               if (PageUninitialized2Mib(p))
+                       expand_page_initialization(page);
+
                if (unlikely(check_new_page(p)))
                        return 1;
        }

@@ -985,6 +1050,7 @@ int move_freepages(struct zone *zone,
        unsigned long order;
        int pages_moved = 0;
 
+       ensure_pages_are_initialized(page_to_pfn(start_page), page_to_pfn(end_page));
 #ifndef CONFIG_HOLES_IN_ZONE
        /*
         * page_zone is not safe to call in this context when

@@ -3859,6 +3925,9 @@ static int pageblock_is_reserved(unsigned long start_pfn, unsigned long end_pfn)
        for (pfn = start_pfn; pfn < end_pfn; pfn++) {
                if (!pfn_valid_within(pfn) || PageReserved(pfn_to_page(pfn)))
                        return 1;
+
+               if (PageUninitialized2Mib(pfn_to_page(pfn)))
+                       pfn += PTRS_PER_PMD;
        }
        return 0;
 }

mm/page_alloc.c +25 -2

+{
+       unsigned long validate_end_pfn = pfn + size;
+
+       if (pfn & (size - 1))
+               return 1;
+
+       if (pfn + size >= end_pfn)
+               return 1;
+
+       while (pfn < validate_end_pfn)
+       {
+               if (!early_pfn_valid(pfn))
+                       return 1;
+               if (!early_pfn_in_nid(pfn, nid))
+                       return 1;
+               pfn++;
+       }
+
+       return size;
+}
+
 /*
  * Initially all pages are reserved - free ones are freed
  * up by free_all_bootmem() once the early boot process is

@@ -3964,20 +4056,34 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
                highest_memmap_pfn = end_pfn - 1;
 
        z = &NODE_DATA(nid)->node_zones[zone];
-       for (pfn = start_pfn; pfn < end_pfn; pfn++) {
+       for (pfn = start_pfn; pfn < end_pfn; ) {
                /*
                 * There can be holes in boot-time mem_map[]s
                 * handed to this function.  They do not
                 * exist on hotplugged memory.
                 */
+               int pfns = 1;
                if (context == MEMMAP_EARLY) {
-                       if (!early_pfn_valid(pfn))
+                       if (!early_pfn_valid(pfn)) {
+                               pfn++;
                                continue;

mm/page_alloc.c +6 -0

+
                page = pfn_to_page(pfn);
                __init_single_page(page, zone, nid, 1);
+
+               if (pfns > 1)
+                       SetPageUninitialized2Mib(page);
+
+               pfn += pfns;
        }
 }
 

Quoted 3 lines

PG_reserved,
+ PG_uninitialized2mib, /* Is this the right spot? ntz - Yes - rmh */
PG_private, /* If pagecache, has fs-private data */

Quoted 3 lines · Yinghai Lu

Quoted 3 lines

PG_reserved,
+ PG_uninitialized2mib, /* Is this the right spot? ntz - Yes - rmh */
PG_private, /* If pagecache, has fs-private data */

The comment here is WTF...

Quoted 6 lines · H. Peter Anvin

Quoted 3 lines · Yinghai Lu

Quoted 3 lines

PG_reserved,
+ PG_uninitialized2mib, /* Is this the right spot? ntz - Yes - rmh */
PG_private, /* If pagecache, has fs-private data */

The comment here is WTF...

ntz: Nate Zimmer?
rmh: Robin Holt?

Quoted 5 lines · H. Peter Anvin

Quoted 3 lines · Yinghai Lu

Quoted 3 lines

PG_reserved,
+ PG_uninitialized2mib, /* Is this the right spot? ntz - Yes - rmh */
PG_private, /* If pagecache, has fs-private data */

The comment here is WTF...

ntz: Nate Zimmer?
rmh: Robin Holt?

Quoted 12 lines · Robin Holt

mm/page_alloc.c +6 -0

+
                page = pfn_to_page(pfn);
                __init_single_page(page, zone, nid, 1);
+
+               if (pfns > 1)
+                       SetPageUninitialized2Mib(page);
+
+               pfn += pfns;
        }
 }

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6305

Quoted 4 lines

mm/page_alloc.c +1 -0

@@ -6196,6 +6302,7 @@ static const struct trace_print_flags pageflag_names[] = {
        {1UL << PG_owner_priv_1,        "owner_priv_1"  },
        {1UL << PG_arch_1,              "arch_1"        },
        {1UL << PG_reserved,            "reserved"      },
+       {1UL << PG_uninitialized2mib,   "Uninit_2MiB"   },

PG_uninitialized_2m ?

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6308

Quoted 3 lines

mm/page_alloc.c +0 -0

        {1UL << PG_private,             "private"       },
        {1UL << PG_private_2,           "private_2"     },
        {1UL << PG_writeback,           "writeback"     },

Yinghai

I hadn't actually been very happy with having a PG_uninitialized2mib flag. It implies if we want to jump to 1Gb pages we would need a second flag, PG_uninitialized1gb, for that. I was thinking of changing it to PG_uninitialized and setting page->private to the correct order. Thoughts?

Quoted 7 lines · Nathan Zimmer

I hadn't actually been very happy with having a PG_uninitialized2mib flag. It implies if we want to jump to 1Gb pages we would need a second flag, PG_uninitialized1gb, for that. I was thinking of changing it to PG_uninitialized and setting page->private to the correct order. Thoughts?

Seems straightforward. The bigger issue is the amount of overhead we cause by having to check upstack for the initialization status of the superpages.

I'm concerned, obviously, about lingering overhead that is "forever". That being said, in the absolutely worst case we could have a counter to the number of uninitialized pages which when it hits zero we do a static switch and switch out the initialization code (would have to be undone on memory hotplug, of course.)

Is there a fairly cheap way to determine definitively that the struct page is not initialized?

I think this patch set can change fairly drastically if we have that. I think I will start working up those changes and code a heavy-handed check until I hear of an alternative way to cheaply check.

Quoted 2 lines · Robin Holt

Is there a fairly cheap way to determine definitively that the struct page is not initialized?

By definition I would assume no. The only way I can think of would be to unmap the memory associated with the struct page in the TLB and initialize the struct pages at trap time.

Quoted 8 lines · H. Peter Anvin

Quoted 2 lines · Robin Holt

Is there a fairly cheap way to determine definitively that the struct page is not initialized?

By definition I would assume no. The only way I can think of would be to unmap the memory associated with the struct page in the TLB and initialize the struct pages at trap time.

But ... the only fastpath impact I can see of delayed initialization right now is this piece of logic in prep_new_page():

(quoted diff) +4 -0

@@ -903,6 +964,10 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
+
+               if (PageUninitialized2Mib(p))
+                       expand_page_initialization(page);
+
                if (unlikely(check_new_page(p)))
                        return 1;
 

That is where I think it can be made zero overhead in the already-initialized case, because page-flags are already used in check_new_page():

static inline int check_new_page(struct page *page)

{
        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (page->flags & PAGE_FLAGS_CHECK_AT_PREP) |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);
                return 1;

see that PAGE_FLAGS_CHECK_AT_PREP flag? That always gets checked for every struct page on allocation.

We can micro-optimize that low overhead to zero-overhead, by integrating the PageUninitialized2Mib() check into check_new_page(). This can be done by adding PG_uninitialized2mib to PAGE_FLAGS_CHECK_AT_PREP and doing:

	if (unlikely(page->flags & PAGE_FLAGS_CHECK_AT_PREP)) {
		if (PageUninitialized2Mib(p))
			expand_page_initialization(page);
		...
	}

        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);

                return 1;

this will result in making it essentially zero-overhead, the expand_page_initialization() logic is now in a slowpath.

Am I missing anything here?

Thanks,

Ingo

Quoted 26 lines · Ingo Molnar

Quoted 8 lines · H. Peter Anvin

Quoted 2 lines · Robin Holt

Is there a fairly cheap way to determine definitively that the struct page is not initialized?

By definition I would assume no. The only way I can think of would be to unmap the memory associated with the struct page in the TLB and initialize the struct pages at trap time.

But ... the only fastpath impact I can see of delayed initialization right now is this piece of logic in prep_new_page():

(quoted diff) +4 -0

@@ -903,6 +964,10 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
+
+               if (PageUninitialized2Mib(p))
+                       expand_page_initialization(page);
+
                if (unlikely(check_new_page(p)))
                        return 1;
 

Quoted 3 lines

That is where I think it can be made zero overhead in the already-initialized case, because page-flags are already used in check_new_page():

The problem I see here is that the page flags we need to check for the uninitialized flag are in the "other" page for the page aligned at the 2MiB virtual address, not the page currently being referenced.

Let me try a version of the patch where we set the PG_unintialized_2m flag on all pages, including the aligned pages and see what that does to performance.

Robin

Quoted 41 lines

static inline int check_new_page(struct page *page)

{
        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (page->flags & PAGE_FLAGS_CHECK_AT_PREP) |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);
                return 1;

see that PAGE_FLAGS_CHECK_AT_PREP flag? That always gets checked for every struct page on allocation.

We can micro-optimize that low overhead to zero-overhead, by integrating the PageUninitialized2Mib() check into check_new_page(). This can be done by adding PG_uninitialized2mib to PAGE_FLAGS_CHECK_AT_PREP and doing:

	if (unlikely(page->flags & PAGE_FLAGS_CHECK_AT_PREP)) {
		if (PageUninitialized2Mib(p))
			expand_page_initialization(page);
		...
	}

        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);

                return 1;

this will result in making it essentially zero-overhead, the expand_page_initialization() logic is now in a slowpath.

Am I missing anything here?

Thanks,

Ingo

I think the other critical path which is affected is in expand(). There, we just call ensure_page_is_initialized() blindly which does the check against the other page. The below is a nearly zero addition. Sorry for the confusion. My morning coffee has not kicked in yet.

Robin

Quoted 82 lines · Robin Holt

Quoted 26 lines · Ingo Molnar

Quoted 8 lines · H. Peter Anvin

Quoted 2 lines · Robin Holt

Is there a fairly cheap way to determine definitively that the struct page is not initialized?

By definition I would assume no. The only way I can think of would be to unmap the memory associated with the struct page in the TLB and initialize the struct pages at trap time.

But ... the only fastpath impact I can see of delayed initialization right now is this piece of logic in prep_new_page():

(quoted diff) +4 -0

@@ -903,6 +964,10 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
+
+               if (PageUninitialized2Mib(p))
+                       expand_page_initialization(page);
+
                if (unlikely(check_new_page(p)))
                        return 1;
 

Quoted 3 lines

That is where I think it can be made zero overhead in the already-initialized case, because page-flags are already used in check_new_page():

The problem I see here is that the page flags we need to check for the uninitialized flag are in the "other" page for the page aligned at the 2MiB virtual address, not the page currently being referenced.

Let me try a version of the patch where we set the PG_unintialized_2m flag on all pages, including the aligned pages and see what that does to performance.

Robin

Quoted 41 lines

static inline int check_new_page(struct page *page)

{
        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (page->flags & PAGE_FLAGS_CHECK_AT_PREP) |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);
                return 1;

see that PAGE_FLAGS_CHECK_AT_PREP flag? That always gets checked for every struct page on allocation.

We can micro-optimize that low overhead to zero-overhead, by integrating the PageUninitialized2Mib() check into check_new_page(). This can be done by adding PG_uninitialized2mib to PAGE_FLAGS_CHECK_AT_PREP and doing:

	if (unlikely(page->flags & PAGE_FLAGS_CHECK_AT_PREP)) {
		if (PageUninitialized2Mib(p))
			expand_page_initialization(page);
		...
	}

        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);

                return 1;

this will result in making it essentially zero-overhead, the expand_page_initialization() logic is now in a slowpath.

Am I missing anything here?

Thanks,

Ingo

Quoted 4 lines · Robin Holt

I think the other critical path which is affected is in expand(). There, we just call ensure_page_is_initialized() blindly which does the check against the other page. The below is a nearly zero addition. Sorry for the confusion. My morning coffee has not kicked in yet.

I don't have access to the 16TiB system until Thursday unless the other testing on it fails early. I did boot a 2TiB system with the a change which set the Unitialized_2m flag on all pages in that 2MiB range during memmap_init_zone. That makes the expand check test against the referenced page instead of having to go back to the 2MiB page. It appears to have added less than a second to the 2TiB boot so I hope it has equally little impact to the 16TiB boot.

I will clean up this patch some more and resend the currently untested set later today.

Thanks,
Robin

Quoted 86 lines

Robin

Quoted 82 lines · Robin Holt

Quoted 26 lines · Ingo Molnar

Quoted 8 lines · H. Peter Anvin

Quoted 2 lines · Robin Holt

Is there a fairly cheap way to determine definitively that the struct page is not initialized?

By definition I would assume no. The only way I can think of would be to unmap the memory associated with the struct page in the TLB and initialize the struct pages at trap time.

But ... the only fastpath impact I can see of delayed initialization right now is this piece of logic in prep_new_page():

(quoted diff) +4 -0

@@ -903,6 +964,10 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
+
+               if (PageUninitialized2Mib(p))
+                       expand_page_initialization(page);
+
                if (unlikely(check_new_page(p)))
                        return 1;
 

Quoted 3 lines

That is where I think it can be made zero overhead in the already-initialized case, because page-flags are already used in check_new_page():

The problem I see here is that the page flags we need to check for the uninitialized flag are in the "other" page for the page aligned at the 2MiB virtual address, not the page currently being referenced.

Let me try a version of the patch where we set the PG_unintialized_2m flag on all pages, including the aligned pages and see what that does to performance.

Robin

Quoted 41 lines

static inline int check_new_page(struct page *page)

{
        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (page->flags & PAGE_FLAGS_CHECK_AT_PREP) |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);
                return 1;

see that PAGE_FLAGS_CHECK_AT_PREP flag? That always gets checked for every struct page on allocation.

We can micro-optimize that low overhead to zero-overhead, by integrating the PageUninitialized2Mib() check into check_new_page(). This can be done by adding PG_uninitialized2mib to PAGE_FLAGS_CHECK_AT_PREP and doing:

	if (unlikely(page->flags & PAGE_FLAGS_CHECK_AT_PREP)) {
		if (PageUninitialized2Mib(p))
			expand_page_initialization(page);
		...
	}

        if (unlikely(page_mapcount(page) |
                (page->mapping != NULL)  |
                (atomic_read(&page->_count) != 0)  |
                (mem_cgroup_bad_page_check(page)))) {
                bad_page(page);

                return 1;

this will result in making it essentially zero-overhead, the expand_page_initialization() logic is now in a slowpath.

Am I missing anything here?

Thanks,

Ingo

Quoted 90 lines · Robin Holt

During boot of large memory machines, a significant portion of boot is spent initializing the struct page array. The vast majority of those pages are not referenced during boot.

Change this over to only initializing the pages when they are actually allocated.

Besides the advantage of boot speed, this allows us the chance to use normal performance monitoring tools to determine where the bulk of time is spent during page initialization.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>

---
 include/linux/mm.h         |  11 +++++
 include/linux/page-flags.h |   5 +-
 mm/nobootmem.c             |   5 ++
 mm/page_alloc.c            | 117 +++++++++++++++++++++++++++++++++++++++++++--
 4 files changed, 132 insertions(+), 6 deletions(-)

include/linux/mm.h +11 -0

index e0c8528..3de08b5 100644
@@ -1330,8 +1330,19 @@ static inline void __free_reserved_page(struct page *page)
        __free_page(page);
 }
 
+extern void __reserve_bootmem_region(phys_addr_t start, phys_addr_t end);
+
+static inline void __reserve_bootmem_page(struct page *page)
+{
+       phys_addr_t start = page_to_pfn(page) << PAGE_SHIFT;
+       phys_addr_t end = start + PAGE_SIZE;
+
+       __reserve_bootmem_region(start, end);
+}
+
 static inline void free_reserved_page(struct page *page)
 {
+       __reserve_bootmem_page(page);
        __free_reserved_page(page);
        adjust_managed_page_count(page, 1);
 }

include/linux/page-flags.h +4 -1

index 6d53675..79e8eb7 100644
@@ -83,6 +83,7 @@ enum pageflags {
        PG_owner_priv_1,        /* Owner use. If pagecache, fs may use*/
        PG_arch_1,
        PG_reserved,
+       PG_uninitialized2mib,   /* Is this the right spot? ntz - Yes - rmh */
        PG_private,             /* If pagecache, has fs-private data */
        PG_private_2,           /* If pagecache, has fs aux data */
        PG_writeback,           /* Page is under writeback */

@@ -211,6 +212,8 @@ PAGEFLAG(SwapBacked, swapbacked) __CLEARPAGEFLAG(SwapBacked, swapbacked)
 
 __PAGEFLAG(SlobFree, slob_free)
 
+PAGEFLAG(Uninitialized2Mib, uninitialized2mib)
+
 /*
  * Private page markings that may be used by the filesystem that owns the page
  * for its own purposes.

@@ -499,7 +502,7 @@ static inline void ClearPageSlabPfmemalloc(struct page *page)
 #define PAGE_FLAGS_CHECK_AT_FREE \
        (1 << PG_lru     | 1 << PG_locked    | \
         1 << PG_private | 1 << PG_private_2 | \
-        1 << PG_writeback | 1 << PG_reserved | \
+        1 << PG_writeback | 1 << PG_reserved | 1 << PG_uninitialized2mib | \
         1 << PG_slab    | 1 << PG_swapcache | 1 << PG_active | \
         1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON | \
         __PG_COMPOUND_LOCK)

mm/nobootmem.c +0 -0

index 3b512ca..e3a386d 100644

mm/nobootmem.c static unsigned long __init free_low_memory_core_early(void) · line 131

Quoted 6 lines

mm/nobootmem.c +3 -0

@@ -126,6 +126,9 @@ static unsigned long __init free_low_memory_core_early(void)
        phys_addr_t start, end, size;
        u64 i;
 
+       for_each_reserved_mem_region(i, &start, &end)
+               __reserve_bootmem_region(start, end);
+

How about holes that is not in memblock.reserved?

before this patch:
free_area_init_node/free_area_init_core/memmap_init_zone will mark all page in node range to Reserved in struct page, at first.

but those holes is not mapped via kernel low mapping. so it should be ok not touch "struct page" for them.

Now you only mark reserved for memblock.reserved at first, and later mark {memblock.memory} - { memblock.reserved} to be available. And that is ok.

but should split that change to another patch and add some comment and change log for the change.
in case there is some user like UEFI etc do weird thing.

mm/nobootmem.c unsigned long __init free_all_bootmem(void) · line 169

Quoted 9 lines

mm/nobootmem.c +2 -0

        for_each_free_mem_range(i, MAX_NUMNODES, &start, &end, NULL)
                count += __free_memory_core(start, end);
 

@@ -162,6 +165,8 @@ unsigned long __init free_all_bootmem(void)
 {
        struct pglist_data *pgdat;
 
+       memblock_dump_all();
+

Not needed.

Quoted 48 lines

mm/nobootmem.c +0 -0

        for_each_online_pgdat(pgdat)
                reset_node_lowmem_managed_pages(pgdat);
 

mm/page_alloc.c +37 -0

index 635b131..fe51eb9 100644
@@ -740,6 +740,54 @@ static void __init_single_page(struct page *page, unsigned long zone, int nid, i
 #endif
 }
 
+static void expand_page_initialization(struct page *basepage)
+{
+       unsigned long pfn = page_to_pfn(basepage);
+       unsigned long end_pfn = pfn + PTRS_PER_PMD;
+       unsigned long zone = page_zonenum(basepage);
+       int reserved = PageReserved(basepage);
+       int nid = page_to_nid(basepage);
+
+       ClearPageUninitialized2Mib(basepage);
+
+       for( pfn++; pfn < end_pfn; pfn++ )
+               __init_single_page(pfn_to_page(pfn), zone, nid, reserved);
+}
+
+void ensure_pages_are_initialized(unsigned long start_pfn,
+                                 unsigned long end_pfn)
+{
+       unsigned long aligned_start_pfn = start_pfn & ~(PTRS_PER_PMD - 1);
+       unsigned long aligned_end_pfn;
+       struct page *page;
+
+       aligned_end_pfn = end_pfn & ~(PTRS_PER_PMD - 1);
+       aligned_end_pfn += PTRS_PER_PMD;
+       while (aligned_start_pfn < aligned_end_pfn) {
+               if (pfn_valid(aligned_start_pfn)) {
+                       page = pfn_to_page(aligned_start_pfn);
+
+                       if(PageUninitialized2Mib(page))
+                               expand_page_initialization(page);
+               }
+
+               aligned_start_pfn += PTRS_PER_PMD;
+       }
+}
+
+void __reserve_bootmem_region(phys_addr_t start, phys_addr_t end)
+{

mm/page_alloc.c static void __init_single_page(struct page *page, unsigned long zone, int nid, i · line 784

Quoted 5 lines

mm/page_alloc.c +5 -0

+       unsigned long start_pfn = PFN_DOWN(start);
+       unsigned long end_pfn = PFN_UP(end);
+
+       ensure_pages_are_initialized(start_pfn, end_pfn);
+}

that name is confusing, actually it is setting to struct page to Reserved only. maybe __reserve_pages_bootmem() to be aligned to free_pages_bootmem ?

mm/page_alloc.c static void __init_single_page(struct page *page, unsigned long zone, int nid, i · line 789

Quoted 5 lines

mm/page_alloc.c +5 -0

+
+static inline void ensure_page_is_initialized(struct page *page)
+{
+       __reserve_bootmem_page(page);
+}

how about use __reserve_page_bootmem directly and add comment in callers site ?

Quoted 81 lines

mm/page_alloc.c +24 -2

+
 static bool free_pages_prepare(struct page *page, unsigned int order)
 {
        int i;

@@ -751,7 +799,10 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
        if (PageAnon(page))
                page->mapping = NULL;
        for (i = 0; i < (1 << order); i++)
-               bad += free_pages_check(page + i);
+               if (PageUninitialized2Mib(page + i))
+                       i += PTRS_PER_PMD - 1;
+               else
+                       bad += free_pages_check(page + i);
        if (bad)
                return false;
 

@@ -795,13 +846,22 @@ void __meminit __free_pages_bootmem(struct page *page, unsigned int order)
        unsigned int loop;
 
        prefetchw(page);
-       for (loop = 0; loop < nr_pages; loop++) {
+       for (loop = 0; loop < nr_pages; ) {
                struct page *p = &page[loop];
 
                if (loop + 1 < nr_pages)
                        prefetchw(p + 1);
+
+               if ((PageUninitialized2Mib(p)) &&
+                   ((loop + PTRS_PER_PMD) > nr_pages))
+                       ensure_page_is_initialized(p);
+
                __ClearPageReserved(p);
                set_page_count(p, 0);
+               if (PageUninitialized2Mib(p))
+                       loop += PTRS_PER_PMD;
+               else
+                       loop += 1;
        }
 
        page_zone(page)->managed_pages += 1 << order;

@@ -856,6 +916,7 @@ static inline void expand(struct zone *zone, struct page *page,
                area--;
                high--;
                size >>= 1;
+               ensure_page_is_initialized(page);
                VM_BUG_ON(bad_range(zone, &page[size]));
 
 #ifdef CONFIG_DEBUG_PAGEALLOC

@@ -903,6 +964,10 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
+
+               if (PageUninitialized2Mib(p))
+                       expand_page_initialization(page);
+
                if (unlikely(check_new_page(p)))
                        return 1;
        }

@@ -985,6 +1050,7 @@ int move_freepages(struct zone *zone,
        unsigned long order;
        int pages_moved = 0;
 
+       ensure_pages_are_initialized(page_to_pfn(start_page), page_to_pfn(end_page));
 #ifndef CONFIG_HOLES_IN_ZONE
        /*
         * page_zone is not safe to call in this context when

@@ -3859,6 +3925,9 @@ static int pageblock_is_reserved(unsigned long start_pfn, unsigned long end_pfn)
        for (pfn = start_pfn; pfn < end_pfn; pfn++) {
                if (!pfn_valid_within(pfn) || PageReserved(pfn_to_page(pfn)))
                        return 1;
+
+               if (PageUninitialized2Mib(pfn_to_page(pfn)))
+                       pfn += PTRS_PER_PMD;
        }
        return 0;
 }

@@ -3947,6 +4016,29 @@ static void setup_zone_migrate_reserve(struct zone *zone)
        }
 }
 

mm/page_alloc.c static void setup_zone_migrate_reserve(struct zone *zone) · line 4020

Quoted 2 lines

mm/page_alloc.c +2 -0

+int __meminit pfn_range_init_avail(unsigned long pfn, unsigned long end_pfn,
+                                  unsigned long size, int nid)

why not use static ? it seems there is not outside user.

Quoted 41 lines

mm/page_alloc.c +25 -2

+{
+       unsigned long validate_end_pfn = pfn + size;
+
+       if (pfn & (size - 1))
+               return 1;
+
+       if (pfn + size >= end_pfn)
+               return 1;
+
+       while (pfn < validate_end_pfn)
+       {
+               if (!early_pfn_valid(pfn))
+                       return 1;
+               if (!early_pfn_in_nid(pfn, nid))
+                       return 1;
+               pfn++;
+       }
+
+       return size;
+}
+
 /*
  * Initially all pages are reserved - free ones are freed
  * up by free_all_bootmem() once the early boot process is

@@ -3964,20 +4056,34 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
                highest_memmap_pfn = end_pfn - 1;
 
        z = &NODE_DATA(nid)->node_zones[zone];
-       for (pfn = start_pfn; pfn < end_pfn; pfn++) {
+       for (pfn = start_pfn; pfn < end_pfn; ) {
                /*
                 * There can be holes in boot-time mem_map[]s
                 * handed to this function.  They do not
                 * exist on hotplugged memory.
                 */
+               int pfns = 1;
                if (context == MEMMAP_EARLY) {
-                       if (!early_pfn_valid(pfn))
+                       if (!early_pfn_valid(pfn)) {
+                               pfn++;
                                continue;

mm/page_alloc.c void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, · line 4078

Quoted 10 lines

mm/page_alloc.c +7 -1

-                       if (!early_pfn_in_nid(pfn, nid))
+                       }
+                       if (!early_pfn_in_nid(pfn, nid)) {
+                               pfn++;
                                continue;
+                       }
+
+                       pfns = pfn_range_init_avail(pfn, end_pfn,
+                                                   PTRS_PER_PMD, nid);
                }

maybe could update memmap_init_zone() only iterate {memblock.memory} - {memblock.reserved}, so you do need to check avail ....

as memmap_init_zone do not need to handle holes to mark reserve for them.

Quoted 12 lines

mm/page_alloc.c +6 -0

+
                page = pfn_to_page(pfn);
                __init_single_page(page, zone, nid, 1);
+
+               if (pfns > 1)
+                       SetPageUninitialized2Mib(page);
+
+               pfn += pfns;
        }
 }
 

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6305

Quoted 5 lines

mm/page_alloc.c +1 -0

@@ -6196,6 +6302,7 @@ static const struct trace_print_flags pageflag_names[] = {
        {1UL << PG_owner_priv_1,        "owner_priv_1"  },
        {1UL << PG_arch_1,              "arch_1"        },
        {1UL << PG_reserved,            "reserved"      },
+       {1UL << PG_uninitialized2mib,   "Uninit_2MiB"   },

PG_uninitialized_2m ?

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6308

Quoted 3 lines

mm/page_alloc.c +0 -0

        {1UL << PG_private,             "private"       },
        {1UL << PG_private_2,           "private_2"     },
        {1UL << PG_writeback,           "writeback"     },

Yinghai

Quoted 90 lines · Robin Holt

During boot of large memory machines, a significant portion of boot is spent initializing the struct page array. The vast majority of those pages are not referenced during boot.

Change this over to only initializing the pages when they are actually allocated.

Besides the advantage of boot speed, this allows us the chance to use normal performance monitoring tools to determine where the bulk of time is spent during page initialization.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>

---
 include/linux/mm.h         |  11 +++++
 include/linux/page-flags.h |   5 +-
 mm/nobootmem.c             |   5 ++
 mm/page_alloc.c            | 117 +++++++++++++++++++++++++++++++++++++++++++--
 4 files changed, 132 insertions(+), 6 deletions(-)

include/linux/mm.h +11 -0

index e0c8528..3de08b5 100644
@@ -1330,8 +1330,19 @@ static inline void __free_reserved_page(struct page *page)
        __free_page(page);
 }
 
+extern void __reserve_bootmem_region(phys_addr_t start, phys_addr_t end);
+
+static inline void __reserve_bootmem_page(struct page *page)
+{
+       phys_addr_t start = page_to_pfn(page) << PAGE_SHIFT;
+       phys_addr_t end = start + PAGE_SIZE;
+
+       __reserve_bootmem_region(start, end);
+}
+
 static inline void free_reserved_page(struct page *page)
 {
+       __reserve_bootmem_page(page);
        __free_reserved_page(page);
        adjust_managed_page_count(page, 1);
 }

include/linux/page-flags.h +4 -1

index 6d53675..79e8eb7 100644
@@ -83,6 +83,7 @@ enum pageflags {
        PG_owner_priv_1,        /* Owner use. If pagecache, fs may use*/
        PG_arch_1,
        PG_reserved,
+       PG_uninitialized2mib,   /* Is this the right spot? ntz - Yes - rmh */
        PG_private,             /* If pagecache, has fs-private data */
        PG_private_2,           /* If pagecache, has fs aux data */
        PG_writeback,           /* Page is under writeback */

@@ -211,6 +212,8 @@ PAGEFLAG(SwapBacked, swapbacked) __CLEARPAGEFLAG(SwapBacked, swapbacked)
 
 __PAGEFLAG(SlobFree, slob_free)
 
+PAGEFLAG(Uninitialized2Mib, uninitialized2mib)
+
 /*
  * Private page markings that may be used by the filesystem that owns the page
  * for its own purposes.

@@ -499,7 +502,7 @@ static inline void ClearPageSlabPfmemalloc(struct page *page)
 #define PAGE_FLAGS_CHECK_AT_FREE \
        (1 << PG_lru     | 1 << PG_locked    | \
         1 << PG_private | 1 << PG_private_2 | \
-        1 << PG_writeback | 1 << PG_reserved | \
+        1 << PG_writeback | 1 << PG_reserved | 1 << PG_uninitialized2mib | \
         1 << PG_slab    | 1 << PG_swapcache | 1 << PG_active | \
         1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON | \
         __PG_COMPOUND_LOCK)

mm/nobootmem.c +0 -0

index 3b512ca..e3a386d 100644

mm/nobootmem.c static unsigned long __init free_low_memory_core_early(void) · line 131

Quoted 6 lines

mm/nobootmem.c +3 -0

@@ -126,6 +126,9 @@ static unsigned long __init free_low_memory_core_early(void)
        phys_addr_t start, end, size;
        u64 i;
 
+       for_each_reserved_mem_region(i, &start, &end)
+               __reserve_bootmem_region(start, end);
+

How about holes that is not in memblock.reserved?

before this patch:
free_area_init_node/free_area_init_core/memmap_init_zone will mark all page in node range to Reserved in struct page, at first.

but those holes is not mapped via kernel low mapping. so it should be ok not touch "struct page" for them.

Now you only mark reserved for memblock.reserved at first, and later mark {memblock.memory} - { memblock.reserved} to be available. And that is ok.

but should split that change to another patch and add some comment and change log for the change.
in case there is some user like UEFI etc do weird thing.

mm/nobootmem.c unsigned long __init free_all_bootmem(void) · line 169

Quoted 10 lines

mm/nobootmem.c +2 -0

        for_each_free_mem_range(i, MAX_NUMNODES, &start, &end, NULL)
                count += __free_memory_core(start, end);
 

@@ -162,6 +165,8 @@ unsigned long __init free_all_bootmem(void)
 {
        struct pglist_data *pgdat;
 
+       memblock_dump_all();
+

Not needed.

Quoted 43 lines

mm/nobootmem.c +0 -0

        for_each_online_pgdat(pgdat)
                reset_node_lowmem_managed_pages(pgdat);
 

mm/page_alloc.c +32 -0

index 635b131..fe51eb9 100644
@@ -740,6 +740,54 @@ static void __init_single_page(struct page *page, unsigned long zone, int nid, i
 #endif
 }
 
+static void expand_page_initialization(struct page *basepage)
+{
+       unsigned long pfn = page_to_pfn(basepage);
+       unsigned long end_pfn = pfn + PTRS_PER_PMD;
+       unsigned long zone = page_zonenum(basepage);
+       int reserved = PageReserved(basepage);
+       int nid = page_to_nid(basepage);
+
+       ClearPageUninitialized2Mib(basepage);
+
+       for( pfn++; pfn < end_pfn; pfn++ )
+               __init_single_page(pfn_to_page(pfn), zone, nid, reserved);
+}
+
+void ensure_pages_are_initialized(unsigned long start_pfn,
+                                 unsigned long end_pfn)
+{
+       unsigned long aligned_start_pfn = start_pfn & ~(PTRS_PER_PMD - 1);
+       unsigned long aligned_end_pfn;
+       struct page *page;
+
+       aligned_end_pfn = end_pfn & ~(PTRS_PER_PMD - 1);
+       aligned_end_pfn += PTRS_PER_PMD;
+       while (aligned_start_pfn < aligned_end_pfn) {
+               if (pfn_valid(aligned_start_pfn)) {
+                       page = pfn_to_page(aligned_start_pfn);
+
+                       if(PageUninitialized2Mib(page))
+                               expand_page_initialization(page);
+               }
+
+               aligned_start_pfn += PTRS_PER_PMD;

mm/page_alloc.c static void __init_single_page(struct page *page, unsigned long zone, int nid, i · line 784

Quoted 10 lines

mm/page_alloc.c +10 -0

+       }
+}
+
+void __reserve_bootmem_region(phys_addr_t start, phys_addr_t end)
+{
+       unsigned long start_pfn = PFN_DOWN(start);
+       unsigned long end_pfn = PFN_UP(end);
+
+       ensure_pages_are_initialized(start_pfn, end_pfn);
+}

that name is confusing, actually it is setting to struct page to Reserved only. maybe __reserve_pages_bootmem() to be aligned to free_pages_bootmem ?

mm/page_alloc.c static void __init_single_page(struct page *page, unsigned long zone, int nid, i · line 789

Quoted 5 lines

mm/page_alloc.c +5 -0

+
+static inline void ensure_page_is_initialized(struct page *page)
+{
+       __reserve_bootmem_page(page);
+}

how about use __reserve_page_bootmem directly and add comment in callers site ?

Quoted 78 lines

mm/page_alloc.c +24 -2

+
 static bool free_pages_prepare(struct page *page, unsigned int order)
 {
        int i;

@@ -751,7 +799,10 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
        if (PageAnon(page))
                page->mapping = NULL;
        for (i = 0; i < (1 << order); i++)
-               bad += free_pages_check(page + i);
+               if (PageUninitialized2Mib(page + i))
+                       i += PTRS_PER_PMD - 1;
+               else
+                       bad += free_pages_check(page + i);
        if (bad)
                return false;
 

@@ -795,13 +846,22 @@ void __meminit __free_pages_bootmem(struct page *page, unsigned int order)
        unsigned int loop;
 
        prefetchw(page);
-       for (loop = 0; loop < nr_pages; loop++) {
+       for (loop = 0; loop < nr_pages; ) {
                struct page *p = &page[loop];
 
                if (loop + 1 < nr_pages)
                        prefetchw(p + 1);
+
+               if ((PageUninitialized2Mib(p)) &&
+                   ((loop + PTRS_PER_PMD) > nr_pages))
+                       ensure_page_is_initialized(p);
+
                __ClearPageReserved(p);
                set_page_count(p, 0);
+               if (PageUninitialized2Mib(p))
+                       loop += PTRS_PER_PMD;
+               else
+                       loop += 1;
        }
 
        page_zone(page)->managed_pages += 1 << order;

@@ -856,6 +916,7 @@ static inline void expand(struct zone *zone, struct page *page,
                area--;
                high--;
                size >>= 1;
+               ensure_page_is_initialized(page);
                VM_BUG_ON(bad_range(zone, &page[size]));
 
 #ifdef CONFIG_DEBUG_PAGEALLOC

@@ -903,6 +964,10 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
+
+               if (PageUninitialized2Mib(p))
+                       expand_page_initialization(page);
+
                if (unlikely(check_new_page(p)))
                        return 1;
        }

@@ -985,6 +1050,7 @@ int move_freepages(struct zone *zone,
        unsigned long order;
        int pages_moved = 0;
 
+       ensure_pages_are_initialized(page_to_pfn(start_page), page_to_pfn(end_page));
 #ifndef CONFIG_HOLES_IN_ZONE
        /*
         * page_zone is not safe to call in this context when

@@ -3859,6 +3925,9 @@ static int pageblock_is_reserved(unsigned long start_pfn, unsigned long end_pfn)
        for (pfn = start_pfn; pfn < end_pfn; pfn++) {
                if (!pfn_valid_within(pfn) || PageReserved(pfn_to_page(pfn)))
                        return 1;
+
+               if (PageUninitialized2Mib(pfn_to_page(pfn)))
+                       pfn += PTRS_PER_PMD;
        }
        return 0;
 }

mm/page_alloc.c static void setup_zone_migrate_reserve(struct zone *zone) · line 4020

Quoted 5 lines

mm/page_alloc.c +2 -0

@@ -3947,6 +4016,29 @@ static void setup_zone_migrate_reserve(struct zone *zone)
        }
 }
 
+int __meminit pfn_range_init_avail(unsigned long pfn, unsigned long end_pfn,
+                                  unsigned long size, int nid)

why not use static ? it seems there is not outside user.

Quoted 41 lines

mm/page_alloc.c +25 -2

+{
+       unsigned long validate_end_pfn = pfn + size;
+
+       if (pfn & (size - 1))
+               return 1;
+
+       if (pfn + size >= end_pfn)
+               return 1;
+
+       while (pfn < validate_end_pfn)
+       {
+               if (!early_pfn_valid(pfn))
+                       return 1;
+               if (!early_pfn_in_nid(pfn, nid))
+                       return 1;
+               pfn++;
+       }
+
+       return size;
+}
+
 /*
  * Initially all pages are reserved - free ones are freed
  * up by free_all_bootmem() once the early boot process is

@@ -3964,20 +4056,34 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
                highest_memmap_pfn = end_pfn - 1;
 
        z = &NODE_DATA(nid)->node_zones[zone];
-       for (pfn = start_pfn; pfn < end_pfn; pfn++) {
+       for (pfn = start_pfn; pfn < end_pfn; ) {
                /*
                 * There can be holes in boot-time mem_map[]s
                 * handed to this function.  They do not
                 * exist on hotplugged memory.
                 */
+               int pfns = 1;
                if (context == MEMMAP_EARLY) {
-                       if (!early_pfn_valid(pfn))
+                       if (!early_pfn_valid(pfn)) {
+                               pfn++;
                                continue;

mm/page_alloc.c void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, · line 4078

Quoted 10 lines

mm/page_alloc.c +7 -1

-                       if (!early_pfn_in_nid(pfn, nid))
+                       }
+                       if (!early_pfn_in_nid(pfn, nid)) {
+                               pfn++;
                                continue;
+                       }
+
+                       pfns = pfn_range_init_avail(pfn, end_pfn,
+                                                   PTRS_PER_PMD, nid);
                }

maybe could update memmap_init_zone() only iterate {memblock.memory} - {memblock.reserved}, so you do need to check avail ....

as memmap_init_zone do not need to handle holes to mark reserve for them.

Quoted 12 lines

mm/page_alloc.c +6 -0

+
                page = pfn_to_page(pfn);
                __init_single_page(page, zone, nid, 1);
+
+               if (pfns > 1)
+                       SetPageUninitialized2Mib(page);
+
+               pfn += pfns;
        }
 }
 

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6305

Quoted 4 lines

mm/page_alloc.c +1 -0

@@ -6196,6 +6302,7 @@ static const struct trace_print_flags pageflag_names[] = {
        {1UL << PG_owner_priv_1,        "owner_priv_1"  },
        {1UL << PG_arch_1,              "arch_1"        },
        {1UL << PG_reserved,            "reserved"      },
+       {1UL << PG_uninitialized2mib,   "Uninit_2MiB"   },

PG_uninitialized_2m ?

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6308

Quoted 3 lines

mm/page_alloc.c +0 -0

        {1UL << PG_private,             "private"       },
        {1UL << PG_private_2,           "private_2"     },
        {1UL << PG_writeback,           "writeback"     },

Yinghai

Quoted 10 lines

How about holes that is not in memblock.reserved?

before this patch:
free_area_init_node/free_area_init_core/memmap_init_zone will mark all page in node range to Reserved in struct page, at first.

but those holes is not mapped via kernel low mapping. so it should be ok not touch "struct page" for them.

Quoted 7 lines

Now you only mark reserved for memblock.reserved at first, and later mark {memblock.memory} - { memblock.reserved} to be available. And that is ok.

but should split that change to another patch and add some comment and change log for the change.
in case there is some user like UEFI etc do weird thing.

Nate and I talked this over today. Sorry for the delay, but it was the first time we were both free. Neither of us quite understands what you are asking for here. My interpretation is that you would like us to change the use of the PageReserved flag such that during boot, we do not set the flag at all from memmap_init_zone, and then only set it on pages which, at the time of free_all_bootmem, have been allocated/reserved in the memblock allocator. Is that correct? I will start to work that up on the assumption that is what you are asking for.

Quoted 18 lines · Robin Holt

Quoted 10 lines

How about holes that is not in memblock.reserved?

before this patch:
free_area_init_node/free_area_init_core/memmap_init_zone will mark all page in node range to Reserved in struct page, at first.

but those holes is not mapped via kernel low mapping. so it should be ok not touch "struct page" for them.

Quoted 7 lines

Now you only mark reserved for memblock.reserved at first, and later mark {memblock.memory} - { memblock.reserved} to be available. And that is ok.

but should split that change to another patch and add some comment and change log for the change.
in case there is some user like UEFI etc do weird thing.

Quoted 8 lines

Nate and I talked this over today. Sorry for the delay, but it was the first time we were both free. Neither of us quite understands what you are asking for here. My interpretation is that you would like us to change the use of the PageReserved flag such that during boot, we do not set the flag at all from memmap_init_zone, and then only set it on pages which, at the time of free_all_bootmem, have been allocated/reserved in the memblock allocator. Is that correct? I will start to work that up on the assumption that is what you are asking for.

Not exactly.

your change should be right, but there is some subtle change about holes handling.

before mem holes between memory ranges in memblock.memory, get struct page, and initialized with to Reserved in memmap_init_zone.
Those holes is not in memory.reserved, with your patches, those hole's struct page
will still have all 0.

Please separate change about set page to reserved according to memory.reserved to another patch.

Quoted 33 lines · Yinghai Lu

Quoted 18 lines · Robin Holt

Quoted 10 lines

How about holes that is not in memblock.reserved?

before this patch:
free_area_init_node/free_area_init_core/memmap_init_zone will mark all page in node range to Reserved in struct page, at first.

but those holes is not mapped via kernel low mapping. so it should be ok not touch "struct page" for them.

Quoted 7 lines

Now you only mark reserved for memblock.reserved at first, and later mark {memblock.memory} - { memblock.reserved} to be available. And that is ok.

but should split that change to another patch and add some comment and change log for the change.
in case there is some user like UEFI etc do weird thing.

Quoted 8 lines

Nate and I talked this over today. Sorry for the delay, but it was the first time we were both free. Neither of us quite understands what you are asking for here. My interpretation is that you would like us to change the use of the PageReserved flag such that during boot, we do not set the flag at all from memmap_init_zone, and then only set it on pages which, at the time of free_all_bootmem, have been allocated/reserved in the memblock allocator. Is that correct? I will start to work that up on the assumption that is what you are asking for.

Not exactly.

your change should be right, but there is some subtle change about holes handling.

Quoted 8 lines

before mem holes between memory ranges in memblock.memory, get struct page, and initialized with to Reserved in memmap_init_zone.
Those holes is not in memory.reserved, with your patches, those hole's struct page
will still have all 0.

Please separate change about set page to reserved according to memory.reserved to another patch.

Just want to make sure this is where you want me to go. Here is my currently untested patch. Is that what you were expecting to have done? One thing I don't like about this patch is it seems to slow down boot in my simulator environment. I think I am going to look at restructuring things a bit to see if I can eliminate that performance penalty. Otherwise, I think I am following your directions.

From bdd2fefa59af18f283af6f066bc644ddfa5c7da8 Mon Sep 17 00:00:00 2001
From: Robin Holt <[email protected]>
Date: Thu, 25 Jul 2013 04:25:15 -0500
Subject: [RFC -v2-pre2 4/5] ZZZ Only SegPageReserved() on memblock reserved
pages.

---
 include/linux/mm.h |  2 ++
 mm/nobootmem.c     |  3 +++
 mm/page_alloc.c    | 18 +++++++++++-------
 3 files changed, 16 insertions(+), 7 deletions(-)

include/linux/mm.h +2 -0

index e0c8528..b264a26 100644
@@ -1322,6 +1322,8 @@ static inline void adjust_managed_page_count(struct page *page, long count)
        totalram_pages += count;
 }
 
+extern void reserve_bootmem_region(unsigned long start, unsigned long end);
+
 /* Free the reserved page into the buddy system, so it gets managed. */
 static inline void __free_reserved_page(struct page *page)
 {

mm/nobootmem.c +3 -0

index 2159e68..0840af2 100644
@@ -117,6 +117,9 @@ static unsigned long __init free_low_memory_core_early(void)
        phys_addr_t start, end, size;
        u64 i;
 
+       for_each_reserved_mem_region(i, &start, &end)
+               reserve_bootmem_region(start, end);
+
        for_each_free_mem_range(i, MAX_NUMNODES, &start, &end, NULL)
                count += __free_memory_core(start, end);
 

mm/page_alloc.c +11 -8

index 048e166..3aa30b7 100644
@@ -698,7 +698,7 @@ static void free_one_page(struct zone *zone, struct page *page, int order,
 }
 
 static void __init_single_page(unsigned long pfn, unsigned long zone,
-                              int nid, int reserved)
+                              int nid, int page_count)
 {
        struct page *page = pfn_to_page(pfn);
        struct zone *z = &NODE_DATA(nid)->node_zones[zone];

@@ -708,12 +708,9 @@ static void __init_single_page(unsigned long pfn, unsigned long zone,
        init_page_count(page);
        page_mapcount_reset(page);
        page_nid_reset_last(page);
-       if (reserved) {
-               SetPageReserved(page);
-       } else {
-               ClearPageReserved(page);
-               set_page_count(page, 0);
-       }
+       ClearPageReserved(page);
+       set_page_count(page, page_count);
+
        /*
         * Mark the block movable so that blocks are reserved for
         * movable at startup. This will force kernel allocations

@@ -741,6 +738,13 @@ static void __init_single_page(unsigned long pfn, unsigned long zone,
 #endif
 }
 
+void reserve_bootmem_region(unsigned long start, unsigned long end)
+{
+       for (; start < end; start++)
+               if (pfn_valid(start))
+                       SetPageReserved(pfn_to_page(start));
+}
+
 static bool free_pages_prepare(struct page *page, unsigned int order)
 {
        int i;
--

1.8.2.1

During boot of large memory machines, a significant portion of boot is spent initializing the struct page array. The vast majority of those pages are not referenced during boot.

Change this over to only initializing the pages when they are actually allocated.

Besides the advantage of boot speed, this allows us the chance to use normal performance monitoring tools to determine where the bulk of time is spent during page initialization.

...

include/linux/mm.h +0 -0

@@ -1330,8 +1330,19 @@ static inline void __free_reserved_page(struct page *page)
 	__free_page(page);
 }

include/linux/mm.h +1 -0

 static inline void free_reserved_page(struct page *page)
 {
+	__reserve_bootmem_page(page);
 	__free_reserved_page(page);
 	adjust_managed_page_count(page, 1);
 }

include/linux/page-flags.h +0 -0

index 6d53675..79e8eb7 100644

...

mm/page_alloc.c +6 -0

@@ -740,6 +740,54 @@ static void __init_single_page(struct page *page, unsigned long zone, int nid, i
 #endif
 }
 
+static void expand_page_initialization(struct page *basepage)
+{
+	unsigned long pfn = page_to_pfn(basepage);
+	unsigned long end_pfn = pfn + PTRS_PER_PMD;
+	unsigned long zone = page_zonenum(basepage);
+	int reserved = PageReserved(basepage);

...

mm/page_alloc.c +12 -0

+int __meminit pfn_range_init_avail(unsigned long pfn, unsigned long end_pfn,
+				   unsigned long size, int nid)
+{
+	unsigned long validate_end_pfn = pfn + size;
+
+	if (pfn & (size - 1))
+		return 1;
+
+	if (pfn + size >= end_pfn)
+		return 1;
+
+	while (pfn < validate_end_pfn)

Quoted 19 lines · On Thu, 11 Jul 2013 21:03:55 -0500 Robin Holt <[email protected]> wrote:

During boot of large memory machines, a significant portion of boot is spent initializing the struct page array. The vast majority of those pages are not referenced during boot.

Change this over to only initializing the pages when they are actually allocated.

Besides the advantage of boot speed, this allows us the chance to use normal performance monitoring tools to determine where the bulk of time is spent during page initialization.

...

include/linux/mm.h +0 -0

@@ -1330,8 +1330,19 @@ static inline void __free_reserved_page(struct page *page)
 	__free_page(page);
 }
 

include/linux/mm.h static inline void __free_reserved_page(struct page *page) · line 1341

Quoted 9 lines

include/linux/mm.h +9 -0

+extern void __reserve_bootmem_region(phys_addr_t start, phys_addr_t end);
+
+static inline void __reserve_bootmem_page(struct page *page)
+{
+	phys_addr_t start = page_to_pfn(page) << PAGE_SHIFT;
+	phys_addr_t end = start + PAGE_SIZE;
+
+	__reserve_bootmem_region(start, end);
+}

It isn't obvious that this needed to be inlined?

Quoted 12 lines

include/linux/mm.h +1 -0

 static inline void free_reserved_page(struct page *page)
 {
+	__reserve_bootmem_page(page);
 	__free_reserved_page(page);
 	adjust_managed_page_count(page, 1);
 }

include/linux/page-flags.h +0 -0

index 6d53675..79e8eb7 100644

include/linux/page-flags.h enum pageflags { · line 86

Quoted 4 lines

include/linux/page-flags.h +1 -0

@@ -83,6 +83,7 @@ enum pageflags {
 	PG_owner_priv_1,	/* Owner use. If pagecache, fs may use*/
 	PG_arch_1,
 	PG_reserved,
+	PG_uninitialized2mib,	/* Is this the right spot? ntz - Yes - rmh */

"mib" creeps me out too. And it makes me think of SNMP, which I'd prefer not to think about.

We've traditionally had fears of running out of page flags, but I've lost track of how close we are to that happening. IIRC the answer depends on whether you believe there is such a thing as a 32-bit NUMA system.

Can this be avoided anyway? I suspect there's some idiotic combination of flags we could use to indicate the state. PG_reserved|PG_lru or something.

"2MB" sounds terribly arch-specific. Shouldn't we make it more generic for when the hexagon64 port wants to use 4MB?

That conversational code comment was already commented on, but it's still there?

Quoted 20 lines

...

mm/page_alloc.c +11 -0

@@ -740,6 +740,54 @@ static void __init_single_page(struct page *page, unsigned long zone, int nid, i
 #endif
 }
 
+static void expand_page_initialization(struct page *basepage)
+{
+	unsigned long pfn = page_to_pfn(basepage);
+	unsigned long end_pfn = pfn + PTRS_PER_PMD;
+	unsigned long zone = page_zonenum(basepage);
+	int reserved = PageReserved(basepage);
+	int nid = page_to_nid(basepage);
+
+	ClearPageUninitialized2Mib(basepage);
+
+	for( pfn++; pfn < end_pfn; pfn++ )

mm/page_alloc.c static void __init_single_page(struct page *page, unsigned long zone, int nid, i · line 758

Quoted 5 lines

mm/page_alloc.c +5 -0

+		__init_single_page(pfn_to_page(pfn), zone, nid, reserved);
+}
+
+void ensure_pages_are_initialized(unsigned long start_pfn,
+				  unsigned long end_pfn)

I think this can be made static. I hope so, as it's a somewhat odd-sounding identifier for a global.

Quoted 6 lines

mm/page_alloc.c +6 -0

+{
+	unsigned long aligned_start_pfn = start_pfn & ~(PTRS_PER_PMD - 1);
+	unsigned long aligned_end_pfn;
+	struct page *page;
+
+	aligned_end_pfn = end_pfn & ~(PTRS_PER_PMD - 1);

mm/page_alloc.c static void __init_single_page(struct page *page, unsigned long zone, int nid, i · line 770

Quoted 6 lines

mm/page_alloc.c +6 -0

+	aligned_end_pfn += PTRS_PER_PMD;
+	while (aligned_start_pfn < aligned_end_pfn) {
+		if (pfn_valid(aligned_start_pfn)) {
+			page = pfn_to_page(aligned_start_pfn);
+
+			if(PageUninitialized2Mib(page))

checkpatch them, please.

mm/page_alloc.c static void __init_single_page(struct page *page, unsigned long zone, int nid, i · line 776

Quoted 6 lines

mm/page_alloc.c +6 -0

+				expand_page_initialization(page);
+		}
+
+		aligned_start_pfn += PTRS_PER_PMD;
+	}
+}

Some nice code comments for the above two functions would be helpful.

Quoted 20 lines

...

mm/page_alloc.c +17 -0

+int __meminit pfn_range_init_avail(unsigned long pfn, unsigned long end_pfn,
+				   unsigned long size, int nid)
+{
+	unsigned long validate_end_pfn = pfn + size;
+
+	if (pfn & (size - 1))
+		return 1;
+
+	if (pfn + size >= end_pfn)
+		return 1;
+
+	while (pfn < validate_end_pfn)
+	{
+		if (!early_pfn_valid(pfn))
+			return 1;
+		if (!early_pfn_in_nid(pfn, nid))
+			return 1;

mm/page_alloc.c static void setup_zone_migrate_reserve(struct zone *zone) · line 4040

Quoted 5 lines

mm/page_alloc.c +5 -0

+		pfn++;
+ 	}
+
+	return size;
+}

Document it, please. The return value semantics look odd, so don't forget to explain all that as well.

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6305

Quoted 8 lines

...

mm/page_alloc.c +1 -0

@@ -6196,6 +6302,7 @@ static const struct trace_print_flags pageflag_names[] = {
 	{1UL << PG_owner_priv_1,	"owner_priv_1"	},
 	{1UL << PG_arch_1,		"arch_1"	},
 	{1UL << PG_reserved,		"reserved"	},
+	{1UL << PG_uninitialized2mib,	"Uninit_2MiB"	},

It would be better if the name which is visible in procfs matches the name in the kernel source code.

mm/page_alloc.c static const struct trace_print_flags pageflag_names[] = { · line 6308

Quoted 3 lines

mm/page_alloc.c +0 -0

 	{1UL << PG_private,		"private"	},
 	{1UL << PG_private_2,		"private_2"	},
 	{1UL << PG_writeback,		"writeback"	},

[...]

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

Robert Richter and Boris Petkov are working on 'persistent events' support for perf, which will eventually allow boot time profiling - I'm not sure if the patches and the tooling support is ready enough yet for your purposes.

Robert, Boris, the following workflow would be pretty intuitive:

- kernel developer sets boot flag: perf=boot,freq=1khz,size=16MB

- we'd get a single (cycles?) event running once the perf subsystem is up and running, with a sampling frequency of 1 KHz, sending profiling trace events to a sufficiently sized profiling buffer of 16 MB per CPU.

- once the system reaches SYSTEM_RUNNING, profiling is stopped either automatically - or the user stops it via a new tooling command.

- the profiling buffer is extracted into a regular perf.data via a special 'perf record' call or some other, new perf tooling solution/variant.

[ Alternatively the kernel could attempt to construct a 'virtual'
perf.data from the persistent buffer, available via /sys/debug or elsewhere in /sys - just like the kernel constructs a 'virtual' /proc/kcore, etc. That file could be copied or used directly. ]

- from that point on this workflow joins the regular profiling workflow:
perf report, perf script et al can be used to analyze the resulting boot profile.

Quoted 4 lines · Ingo Molnar

Robert Richter and Boris Petkov are working on 'persistent events' support for perf, which will eventually allow boot time profiling - I'm not sure if the patches and the tooling support is ready enough yet for your purposes.

Nope, not yet but we're getting there.

Quoted 3 lines

Robert, Boris, the following workflow would be pretty intuitive:

- kernel developer sets boot flag: perf=boot,freq=1khz,size=16MB

What does perf=boot mean? I assume boot tracing.

Quoted 11 lines · Borislav Petkov

Quoted 4 lines · Ingo Molnar

Robert Richter and Boris Petkov are working on 'persistent events' support for perf, which will eventually allow boot time profiling - I'm not sure if the patches and the tooling support is ready enough yet for your purposes.

Nope, not yet but we're getting there.

Quoted 3 lines

Robert, Boris, the following workflow would be pretty intuitive:

- kernel developer sets boot flag: perf=boot,freq=1khz,size=16MB

What does perf=boot mean? I assume boot tracing.

In this case it would mean boot profiling - i.e. a cycles hardware-PMU event collecting into a perf trace buffer as usual.

Essentially a 'perf record -a' work-alike, just one that gets activated as early as practical, and which would allow the profiling of memory initialization.

Now, one extra complication here is that to be able to profile buddy allocator this persistent event would have to work before the buddy allocator is active :-/ So this sort of profiling would have to use memblock_alloc().

Just wanted to highlight this usecase, we might eventually want to support it.

[ Note that this is different from boot tracing of one or more trace
events - but it's a conceptually pretty close cousin. ]

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 20 lines · Ingo Molnar

Quoted 11 lines · Borislav Petkov

Quoted 4 lines · Ingo Molnar

Robert Richter and Boris Petkov are working on 'persistent events' support for perf, which will eventually allow boot time profiling - I'm not sure if the patches and the tooling support is ready enough yet for your purposes.

Nope, not yet but we're getting there.

Quoted 3 lines

Robert, Boris, the following workflow would be pretty intuitive:

- kernel developer sets boot flag: perf=boot,freq=1khz,size=16MB

What does perf=boot mean? I assume boot tracing.

In this case it would mean boot profiling - i.e. a cycles hardware-PMU event collecting into a perf trace buffer as usual.

Essentially a 'perf record -a' work-alike, just one that gets activated as early as practical, and which would allow the profiling of memory initialization.

Quoted 4 lines

Now, one extra complication here is that to be able to profile buddy allocator this persistent event would have to work before the buddy allocator is active :-/ So this sort of profiling would have to use memblock_alloc().

Could perf=boot be used to sample the performance of memblock subsystem? I think the perf subsystem is too late to be initialized and monitor this.

Quoted 7 lines · Robin Holt

[...]

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

That's pretty impressive.

It's still a 15x speedup instead of a 512x speedup, so I'd say there's something else being the current bottleneck, besides page init granularity.

Can you boot with just a few gigs of RAM and stuff the rest into hotplug memory, and then hot-add that memory? That would allow easy profiling of remaining overhead.

Side note:

Robert Richter and Boris Petkov are working on 'persistent events' support for perf, which will eventually allow boot time profiling - I'm not sure if the patches and the tooling support is ready enough yet for your purposes.

Robert, Boris, the following workflow would be pretty intuitive:

- kernel developer sets boot flag: perf=boot,freq=1khz,size=16MB

- we'd get a single (cycles?) event running once the perf subsystem is up and running, with a sampling frequency of 1 KHz, sending profiling trace events to a sufficiently sized profiling buffer of 16 MB per CPU.

- once the system reaches SYSTEM_RUNNING, profiling is stopped either automatically - or the user stops it via a new tooling command.

- the profiling buffer is extracted into a regular perf.data via a special 'perf record' call or some other, new perf tooling solution/variant.

[ Alternatively the kernel could attempt to construct a 'virtual'
perf.data from the persistent buffer, available via /sys/debug or elsewhere in /sys - just like the kernel constructs a 'virtual' /proc/kcore, etc. That file could be copied or used directly. ]

- from that point on this workflow joins the regular profiling workflow:
perf report, perf script et al can be used to analyze the resulting boot profile.

Quoted 2 lines · Robin Holt

[...]

Quoted 5 lines

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

That's pretty impressive.

It's still a 15x speedup instead of a 512x speedup, so I'd say there's something else being the current bottleneck, besides page init granularity.

Can you boot with just a few gigs of RAM and stuff the rest into hotplug memory, and then hot-add that memory? That would allow easy profiling of remaining overhead.

Quoted 7 lines · Robin Holt

[...]

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

That's pretty impressive.

It's still a 15x speedup instead of a 512x speedup, so I'd say there's something else being the current bottleneck, besides page init granularity.

Can you boot with just a few gigs of RAM and stuff the rest into hotplug memory, and then hot-add that memory? That would allow easy profiling of remaining overhead.

Side note:

Robert Richter and Boris Petkov are working on 'persistent events' support for perf, which will eventually allow boot time profiling - I'm not sure if the patches and the tooling support is ready enough yet for your purposes.

Robert, Boris, the following workflow would be pretty intuitive:

- kernel developer sets boot flag: perf=boot,freq=1khz,size=16MB

- we'd get a single (cycles?) event running once the perf subsystem is up and running, with a sampling frequency of 1 KHz, sending profiling trace events to a sufficiently sized profiling buffer of 16 MB per CPU.

- once the system reaches SYSTEM_RUNNING, profiling is stopped either automatically - or the user stops it via a new tooling command.

- the profiling buffer is extracted into a regular perf.data via a special 'perf record' call or some other, new perf tooling solution/variant.

[ Alternatively the kernel could attempt to construct a 'virtual'
perf.data from the persistent buffer, available via /sys/debug or elsewhere in /sys - just like the kernel constructs a 'virtual' /proc/kcore, etc. That file could be copied or used directly. ]

How about executing a perf in usermodehelper and collecting output in tmpfs? Using this approach, we can start a perf after rootfs initialization,

Quoted 3 lines · Joonsoo Kim

How about executing a perf in usermodehelper and collecting output in tmpfs? Using this approach, we can start a perf after rootfs initialization,

What for if we can start logging to buffers much earlier? *Reading* from those buffers can be done much later, at our own leisure with full userspace up.

We have been working on this since we returned from shutdown and have something to discuss now. We restricted ourselves to 2MiB initialization to keep the patch set a little smaller and more clear.

First, I think I want to propose getting rid of the page flag. If I knew of a concrete way to determine that the page has not been initialized, this patch series would look different. If there is no definitive way to determine that the struct page has been initialized aside from checking the entire page struct is zero, then I think I would suggest we change the page flag to indicate the page has been initialized.

We have been working on this since we returned from shutdown and have something to discuss now. We restricted ourselves to 2MiB initialization to keep the patch set a little smaller and more clear.

First, I think I want to propose getting rid of the page flag. If I knew of a concrete way to determine that the page has not been initialized, this patch series would look different. If there is no definitive way to determine that the struct page has been initialized aside from checking the entire page struct is zero, then I think I would suggest we change the page flag to indicate the page has been initialized.

The heart of the problem as I see it comes from expand(). We nearly always see a first reference to a struct page which is in the middle of the 2MiB region. Due to that access, the unlikely() check that was originally proposed really ends up referencing a different page entirely. We actually did not introduce an unlikely and refactor the patches to make that unlikely inside a static inline function. Also, given the strong warning at the head of expand(), we did not feel experienced enough to refactor it to make things always reference the 2MiB page first.

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

I am on vacation for the next three days so I am sorry in advance for my infrequent or non-existant responses.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 21 lines · Robin Holt

We have been working on this since we returned from shutdown and have something to discuss now. We restricted ourselves to 2MiB initialization to keep the patch set a little smaller and more clear.

First, I think I want to propose getting rid of the page flag. If I knew of a concrete way to determine that the page has not been initialized, this patch series would look different. If there is no definitive way to determine that the struct page has been initialized aside from checking the entire page struct is zero, then I think I would suggest we change the page flag to indicate the page has been initialized.

The heart of the problem as I see it comes from expand(). We nearly always see a first reference to a struct page which is in the middle of the 2MiB region. Due to that access, the unlikely() check that was originally proposed really ends up referencing a different page entirely. We actually did not introduce an unlikely and refactor the patches to make that unlikely inside a static inline function. Also, given the strong warning at the head of expand(), we did not feel experienced enough to refactor it to make things always reference the 2MiB page first.

Quoted 5 lines

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

How to test how much time spend on free_all_bootmem?

Quoted 16 lines

I am on vacation for the next three days so I am sorry in advance for my infrequent or non-existant responses.

Signed-off-by Robin Holt <[email protected]>

Signed-off-by Nate Zimmer <[email protected]>

To: "H. Peter Anvin" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Linux Kernel <[email protected]>
Cc: Linux MM <[email protected]>
Cc: Rob Landley <[email protected]>
Cc: Mike Travis <[email protected]>
Cc: Daniel J Blueman <[email protected]>
Cc: Andrew Morton <[email protected]>

Quoted 8 lines

Cc: Greg KH <[email protected]>
Cc: Yinghai Lu <[email protected]>
Cc: Mel Gorman <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 6 lines · Sam Ben

Quoted 5 lines

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

Quoted 1 line

How to test how much time spend on free_all_bootmem?

We had put a pr_emerg at the beginning and end of free_all_bootmem and then used a modified version of script which record the time in uSecs at the beginning of each line of output.

Quoted 7 lines · Robin Holt

Quoted 6 lines · Sam Ben

Quoted 5 lines

With this patch, we did boot a 16TiB machine. Without the patches, the v3.10 kernel with the same configuration took 407 seconds for free_all_bootmem. With the patches and operating on 2MiB pages instead of 1GiB, it took 26 seconds so performance was improved. I have no feel for how the 1GiB chunk size will perform.

Quoted 5 lines

Quoted 1 line

How to test how much time spend on free_all_bootmem?

We had put a pr_emerg at the beginning and end of free_all_bootmem and then used a modified version of script which record the time in uSecs at the beginning of each line of output.

used two patches, found 3TiB system will take 100s before slub is ready.

about three portions:
1. sparse vmemap buf allocation, it is with bootmem wrapper, so clear those struct page area take about 30s.
2. memmap_init_zone: take about 25s
3. mem_init/free_all_bootmem about 30s.

so still wonder why 16TiB will need hours.

also your patches looks like only address 2 and 3.