mm/hugetlb_vmemmap: Fix usemap overrun on nodes with multiple ranges

sparse_init_nid() allocates one usemap slot (struct mem_section_usage) per present section in [pnum_begin, pnum_end), and sparse_init_early_section() hands them out in order, with a BUG_ON() when they run out. hugetlb_vmemmap_init_early(nid) however initializes the sections of every bootmem gigantic page on the node, including pages outside the range being initialized. If a node's memory is split into more than one range, sparse_init() calls sparse_init_nid() once per range, and this takes more slots than were allocated. The same happens if a gigantic page straddles a node boundary.

For example, with this SRAT layout every node has a small range below 4G and a large range above 4G:

 Node 0 PXM 0 [mem 0x00000000-0x2fffffff]          768 MB  (low)
 Node 0 PXM 0 [mem 0x100000000-0x80cfffffff]               (high)
 Node 1 PXM 1 [mem 0x30000000-0x5fffffff]          768 MB  (low)

...

In 128 MB memory sections (x86_64 SECTION_SIZE_BITS = 27), node 0's low range is sections 0x00-0x05 (6 present sections) and its high range is sections 0x20-0x1019. The low ranges of the other nodes are 4 or 6 sections, all smaller than the 8 sections of a 1G page.

Node 0's first range has 6 sections, so 6 slots. With "hugepagesz=1G hugepages=400", hugetlb_vmemmap_init_early(0) then initializes node 0's 1G page at 0x67c0000000 (sections 0xcf8-0xcff, in its high range), and the 7th section hits the BUG_ON() (seen on v7.2.6):

kernel BUG at mm/sparse.c:268!

 RIP: 0010:sparse_init_early_section+0x9f/0xb0
 RAX: 0000000000200000 RBX: 0000000000000cfe RCX: 0000000000000020
 RDX: 0000000000000cfe RSI: ffd7e1d09fc00000 RDI: ff252cb28ffb3f50
 R10: ff252cb28ffb3d40 R13: 0000000000000d00
 Call Trace:
  hugetlb_vmemmap_init_early+0xe1/0x140
  sparse_init_nid+0x56/0x370
  sparse_init+0x192/0x2b0
  free_area_init+0x7f/0x690
  start_kernel+0x64/0x540

Here RDX is the section number, and R10/RDI are 6 slots apart: the start of the usemap buffer and its current position, which is the end.

Pass the section range to hugetlb_vmemmap_init_early() and only pre-HVO a page whose sections are all present and within it. Ranges are disjoint, so each page is handled once, in its own range.

Skipped pages are not marked HUGE_BOOTMEM_HVO. Their sections get a regular memmap from sparse_init_nid(), and the pages are optimized later by hugetlb_vmemmap_optimize_bootmem_folios().

Quoted 43 lines · Tim Chen

sparse_init_nid() allocates one usemap slot (struct mem_section_usage) per present section in [pnum_begin, pnum_end), and sparse_init_early_section() hands them out in order, with a BUG_ON() when they run out. hugetlb_vmemmap_init_early(nid) however initializes the sections of every bootmem gigantic page on the node, including pages outside the range being initialized. If a node's memory is split into more than one range, sparse_init() calls sparse_init_nid() once per range, and this takes more slots than were allocated. The same happens if a gigantic page straddles a node boundary.

For example, with this SRAT layout every node has a small range below 4G and a large range above 4G:

 Node 0 PXM 0 [mem 0x00000000-0x2fffffff]          768 MB  (low)
 Node 0 PXM 0 [mem 0x100000000-0x80cfffffff]               (high)
 Node 1 PXM 1 [mem 0x30000000-0x5fffffff]          768 MB  (low)

...

In 128 MB memory sections (x86_64 SECTION_SIZE_BITS = 27), node 0's low range is sections 0x00-0x05 (6 present sections) and its high range is sections 0x20-0x1019. The low ranges of the other nodes are 4 or 6 sections, all smaller than the 8 sections of a 1G page.

Node 0's first range has 6 sections, so 6 slots. With "hugepagesz=1G hugepages=400", hugetlb_vmemmap_init_early(0) then initializes node 0's 1G page at 0x67c0000000 (sections 0xcf8-0xcff, in its high range), and the 7th section hits the BUG_ON() (seen on v7.2.6):

kernel BUG at mm/sparse.c:268!

 RIP: 0010:sparse_init_early_section+0x9f/0xb0
 RAX: 0000000000200000 RBX: 0000000000000cfe RCX: 0000000000000020
 RDX: 0000000000000cfe RSI: ffd7e1d09fc00000 RDI: ff252cb28ffb3f50
 R10: ff252cb28ffb3d40 R13: 0000000000000d00
 Call Trace:
  hugetlb_vmemmap_init_early+0xe1/0x140
  sparse_init_nid+0x56/0x370
  sparse_init+0x192/0x2b0
  free_area_init+0x7f/0x690
  start_kernel+0x64/0x540

Quoted 2 lines

Here RDX is the section number, and R10/RDI are 6 slots apart: the start of the usemap buffer and its current position, which is the end.

A detailed analysis.

Quoted 8 lines

Pass the section range to hugetlb_vmemmap_init_early() and only pre-HVO a page whose sections are all present and within it. Ranges are disjoint, so each page is handled once, in its own range.

Skipped pages are not marked HUGE_BOOTMEM_HVO. Their sections get a regular memmap from sparse_init_nid(), and the pages are optimized later by hugetlb_vmemmap_optimize_bootmem_folios().

Thanks for fixing this.

However, the master branch has already switched to section-based vmemmap optimization, it has been already fixed. So this fix should target the stable kernels (6.18, 7.2 and 7.3).

The change LGTM. So:

Reviewed-by Muchun Song <[email protected]>

Thanks.

sparse_init_nid() allocates one usemap slot (struct mem_section_usage) per present section in [pnum_begin, pnum_end), and sparse_init_early_section() hands them out in order, with a BUG_ON() when they run out. hugetlb_vmemmap_init_early(nid) however initializes the sections of every bootmem gigantic page on the node, including pages outside the range being initialized. If a node's memory is split into more than one range, sparse_init() calls sparse_init_nid() once per range, and this takes more slots than were allocated. The same happens if a gigantic page straddles a node boundary.

For example, with this SRAT layout every node has a small range below 4G and a large range above 4G:

  Node 0 PXM 0 [mem 0x00000000-0x2fffffff]          768 MB  (low)
  Node 0 PXM 0 [mem 0x100000000-0x80cfffffff]               (high)
  Node 1 PXM 1 [mem 0x30000000-0x5fffffff]          768 MB  (low)

...

In 128 MB memory sections (x86_64 SECTION_SIZE_BITS = 27), node 0's low range is sections 0x00-0x05 (6 present sections) and its high range is sections 0x20-0x1019. The low ranges of the other nodes are 4 or 6 sections, all smaller than the 8 sections of a 1G page.

Node 0's first range has 6 sections, so 6 slots. With "hugepagesz=1G hugepages=400", hugetlb_vmemmap_init_early(0) then initializes node 0's 1G page at 0x67c0000000 (sections 0xcf8-0xcff, in its high range), and the 7th section hits the BUG_ON() (seen on v7.2.6):

kernel BUG at mm/sparse.c:268!

  RIP: 0010:sparse_init_early_section+0x9f/0xb0
  RAX: 0000000000200000 RBX: 0000000000000cfe RCX: 0000000000000020
  RDX: 0000000000000cfe RSI: ffd7e1d09fc00000 RDI: ff252cb28ffb3f50
  R10: ff252cb28ffb3d40 R13: 0000000000000d00
  Call Trace:
   hugetlb_vmemmap_init_early+0xe1/0x140
   sparse_init_nid+0x56/0x370
   sparse_init+0x192/0x2b0
   free_area_init+0x7f/0x690
   start_kernel+0x64/0x540

Here RDX is the section number, and R10/RDI are 6 slots apart: the start of the usemap buffer and its current position, which is the end.

Pass the section range to hugetlb_vmemmap_init_early() and only pre-HVO a page whose sections are all present and within it. Ranges are disjoint, so each page is handled once, in its own range.

Skipped pages are not marked HUGE_BOOTMEM_HVO. Their sections get a regular memmap from sparse_init_nid(), and the pages are optimized later by hugetlb_vmemmap_optimize_bootmem_folios().

Fixes b1222550fbf7 ("mm/hugetlb: do pre-HVO for bootmem allocated pages")

Cc [email protected]

Reported-by Mohini Narkhede <[email protected]>

Tested-by Pooja Jain <[email protected]>

Signed-off-by Tim Chen <[email protected]>

---
 include/linux/mmzone.h |  9 ++++++---
 mm/hugetlb_vmemmap.c   | 38 ++++++++++++++++++++++++++++++++++----
 mm/hugetlb_vmemmap.h   |  7 +++++--
 mm/sparse-vmemmap.c    | 16 +++++++++-------
 mm/sparse.c            |  2 +-
 5 files changed, 55 insertions(+), 17 deletions(-)

include/linux/mmzone.h +1 -1

index 94f9c3ff54160..da17dae8b057b 100644
@@ -2167,13 +2167,16 @@ static inline int preinited_vmemmap_section(const struct mem_section *section)
 		(section->section_mem_map & SECTION_IS_VMEMMAP_PREINIT));
 }
 
-void sparse_vmemmap_init_nid_early(int nid);
+void sparse_vmemmap_init_nid_early(int nid, unsigned long pnum_begin,

Quoted 72 lines · Tim Chen

sparse_init_nid() allocates one usemap slot (struct mem_section_usage) per present section in [pnum_begin, pnum_end), and sparse_init_early_section() hands them out in order, with a BUG_ON() when they run out. hugetlb_vmemmap_init_early(nid) however initializes the sections of every bootmem gigantic page on the node, including pages outside the range being initialized. If a node's memory is split into more than one range, sparse_init() calls sparse_init_nid() once per range, and this takes more slots than were allocated. The same happens if a gigantic page straddles a node boundary.

For example, with this SRAT layout every node has a small range below 4G and a large range above 4G:

  Node 0 PXM 0 [mem 0x00000000-0x2fffffff]          768 MB  (low)
  Node 0 PXM 0 [mem 0x100000000-0x80cfffffff]               (high)
  Node 1 PXM 1 [mem 0x30000000-0x5fffffff]          768 MB  (low)

...

In 128 MB memory sections (x86_64 SECTION_SIZE_BITS = 27), node 0's low range is sections 0x00-0x05 (6 present sections) and its high range is sections 0x20-0x1019. The low ranges of the other nodes are 4 or 6 sections, all smaller than the 8 sections of a 1G page.

Node 0's first range has 6 sections, so 6 slots. With "hugepagesz=1G hugepages=400", hugetlb_vmemmap_init_early(0) then initializes node 0's 1G page at 0x67c0000000 (sections 0xcf8-0xcff, in its high range), and the 7th section hits the BUG_ON() (seen on v7.2.6):

kernel BUG at mm/sparse.c:268!

  RIP: 0010:sparse_init_early_section+0x9f/0xb0
  RAX: 0000000000200000 RBX: 0000000000000cfe RCX: 0000000000000020
  RDX: 0000000000000cfe RSI: ffd7e1d09fc00000 RDI: ff252cb28ffb3f50
  R10: ff252cb28ffb3d40 R13: 0000000000000d00
  Call Trace:
   hugetlb_vmemmap_init_early+0xe1/0x140
   sparse_init_nid+0x56/0x370
   sparse_init+0x192/0x2b0
   free_area_init+0x7f/0x690
   start_kernel+0x64/0x540

Here RDX is the section number, and R10/RDI are 6 slots apart: the start of the usemap buffer and its current position, which is the end.

Pass the section range to hugetlb_vmemmap_init_early() and only pre-HVO a page whose sections are all present and within it. Ranges are disjoint, so each page is handled once, in its own range.

Skipped pages are not marked HUGE_BOOTMEM_HVO. Their sections get a regular memmap from sparse_init_nid(), and the pages are optimized later by hugetlb_vmemmap_optimize_bootmem_folios().

Fixes b1222550fbf7 ("mm/hugetlb: do pre-HVO for bootmem allocated pages")

Cc [email protected]

Reported-by Mohini Narkhede <[email protected]>

Tested-by Pooja Jain <[email protected]>

Signed-off-by Tim Chen <[email protected]>

---
 include/linux/mmzone.h |  9 ++++++---
 mm/hugetlb_vmemmap.c   | 38 ++++++++++++++++++++++++++++++++++----
 mm/hugetlb_vmemmap.h   |  7 +++++--
 mm/sparse-vmemmap.c    | 16 +++++++++-------
 mm/sparse.c            |  2 +-
 5 files changed, 55 insertions(+), 17 deletions(-)

include/linux/mmzone.h +0 -0

index 94f9c3ff54160..da17dae8b057b 100644
@@ -2167,13 +2167,16 @@ static inline int preinited_vmemmap_section(const struct mem_section *section)
 		(section->section_mem_map & SECTION_IS_VMEMMAP_PREINIT));
 }
 

include/linux/mmzone.h static inline int preinited_vmemmap_section(const struct mem_section *section) · line 2179

Quoted 12 lines

include/linux/mmzone.h +5 -2

-void sparse_vmemmap_init_nid_early(int nid);
+void sparse_vmemmap_init_nid_early(int nid, unsigned long pnum_begin,
+				   unsigned long pnum_end);
 #else
 static inline int preinited_vmemmap_section(const struct mem_section *section)
 {
 	return 0;
 }
-static inline void sparse_vmemmap_init_nid_early(int nid)
+static inline void sparse_vmemmap_init_nid_early(int nid,
+						 unsigned long pnum_begin,
+						 unsigned long pnum_end)

Two tab indent please in all cases here that you are touching, thanks.