Skip to contentSkip to conversation

hugetlb: parallelize hugetlb page init on boot

PATCH RESEND Superseded Gang Li 4 participants 30 messages Jan 18, 2024 – Jan 25, 2024

Subsystems (5)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Gang Li

author
#0 lore raw

v4 cover hugetlb: parallelize hugetlb page init on boot

Hi all, hugetlb init parallelization has now been updated to v4.

This version is tested on next-20240112.

Update Summary:
- Make padata_do_multithreaded dispatch all jobs with a global iterator
- Revise commit message
- Rename some functions
- Collect Tested-by and Reviewed-by

# Introduction
Hugetlb initialization during boot takes up a considerable amount of time. For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2 seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel host takes more than 1 minute[1]. This is a noteworthy figure.

Inspired by [2] and [3], hugetlb initialization can also be accelerated through parallelization. Kernel already has infrastructure like padata_do_multithreaded, this patch uses it to achieve effective results by minimal modifications.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/

# Test result
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------

  256c2t(4 node) 1G           4745          2024   57.34%
  128c1t(2 node) 1G           3358          1712   49.02%
      12t        1G          77000         18300   76.23%
  256c2t(4 node) 2M           3336          1051   68.52%
  128c1t(2 node) 2M           1943           716   63.15%

# Change log
Changes in v4:
- Make padata_do_multithreaded dispatch all jobs with a global iterator
- Revise commit message
- Rename some functions
- Collect Tested-by and Reviewed-by

Changes in v3:
- https://lore.kernel.org/all/[email protected]/
- Select CONFIG_PADATA as we use padata_do_multithreaded
- Fix a race condition in h->next_nid_to_alloc
- Fix local variable initialization issues
- Remove RFC tag

Changes in v2:
- https://lore.kernel.org/all/[email protected]/
- Reduce complexity with padata_do_multithreaded
- Support 1G hugetlb

v1:
- https://lore.kernel.org/all/[email protected]/
- parallelize 2M hugetlb initialization with workqueue

Gang Li (7):
hugetlb: code clean for hugetlb_hstate_alloc_pages
hugetlb: split hugetlb_hstate_alloc_pages
padata: dispatch works on different nodes
hugetlb: pass *next_nid_to_alloc directly to
for_each_node_mask_to_alloc
hugetlb: have CONFIG_HUGETLBFS select CONFIG_PADATA
hugetlb: parallelize 2M hugetlb allocation and initialization
hugetlb: parallelize 1G hugetlb initialization

 fs/Kconfig              |   1 +
 include/linux/hugetlb.h |   2 +-
 include/linux/padata.h  |   3 +
 kernel/padata.c         |  14 ++-
 mm/hugetlb.c            | 227 ++++++++++++++++++++++++++--------------
 mm/mm_init.c            |   1 +
 6 files changed, 168 insertions(+), 80 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v4 1/7 hugetlb: code clean for hugetlb_hstate_alloc_pages

The readability of hugetlb_hstate_alloc_pages is poor. By cleaning the code, its readability can be improved, facilitating future modifications.

This patch extracts two functions to reduce the complexity of hugetlb_hstate_alloc_pages and has no functional changes.

- hugetlb_hstate_alloc_pages_node_specific() to handle iterates through each online node and performs allocation if necessary.
- hugetlb_hstate_alloc_pages_report() report error during allocation. And the value of h->max_huge_pages is updated accordingly.

---
 mm/hugetlb.c | 46 +++++++++++++++++++++++++++++-----------------
 1 file changed, 29 insertions(+), 17 deletions(-)
mm/hugetlb.c +29 -17
index ed1581b670d4..b8e4a6adefd6 100644
@@ -3482,6 +3482,33 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
 	h->max_huge_pages_node[nid] = i;
 }
 
+static bool __init hugetlb_hstate_alloc_pages_specific_nodes(struct hstate *h)
+{
+	int i;
+	bool node_specific_alloc = false;
+
+	for_each_online_node(i) {
+		if (h->max_huge_pages_node[i] > 0) {
+			hugetlb_hstate_alloc_pages_onenode(h, i);
+			node_specific_alloc = true;
+		}
+	}
+
+	return node_specific_alloc;
+}
+
+static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, struct hstate *h)
+{
+	if (allocated < h->max_huge_pages) {
+		char buf[32];
+
+		string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32);
+		pr_warn("HugeTLB: allocating %lu of page size %s failed.  Only allocated %lu hugepages.\n",
+			h->max_huge_pages, buf, allocated);
+		h->max_huge_pages = allocated;
+	}
+}
+
 /*
  * NOTE: this routine is called in different contexts for gigantic and
  * non-gigantic pages.
@@ -3499,7 +3526,6 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
 	struct folio *folio;
 	LIST_HEAD(folio_list);
 	nodemask_t *node_alloc_noretry;
-	bool node_specific_alloc = false;
 
 	/* skip gigantic hugepages allocation if hugetlb_cma enabled */
 	if (hstate_is_gigantic(h) && hugetlb_cma_size) {
@@ -3508,14 +3534,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
 	}
 
 	/* do node specific alloc */
-	for_each_online_node(i) {
-		if (h->max_huge_pages_node[i] > 0) {
-			hugetlb_hstate_alloc_pages_onenode(h, i);
-			node_specific_alloc = true;
-		}
-	}
-
-	if (node_specific_alloc)
+	if (hugetlb_hstate_alloc_pages_specific_nodes(h))
 		return;
 
 	/* below will do all node balanced alloc */
@@ -3558,14 +3577,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
 	/* list will be empty if hstate_is_gigantic */
 	prep_and_add_allocated_folios(h, &folio_list);
 
-	if (i < h->max_huge_pages) {
-		char buf[32];
-
-		string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32);
-		pr_warn("HugeTLB: allocating %lu of page size %s failed.  Only allocated %lu hugepages.\n",
-			h->max_huge_pages, buf, i);
-		h->max_huge_pages = i;
-	}
+	hugetlb_hstate_alloc_pages_errcheck(i, h);
 	kfree(node_alloc_noretry);
 }
 
Signature · 1 line
2.20.1
author
#2 lore raw

PATCH v4 2/7 hugetlb: split hugetlb_hstate_alloc_pages

In reply to Gang Li

1G and 2M huge pages have different allocation and initialization logic, which leads to subtle differences in parallelization. Therefore, it is appropriate to split hugetlb_hstate_alloc_pages into gigantic and non-gigantic.

This patch has no functional changes.

---
 mm/hugetlb.c | 87 ++++++++++++++++++++++++++--------------------------
 1 file changed, 43 insertions(+), 44 deletions(-)
mm/hugetlb.c +43 -44
index b8e4a6adefd6..98ae108e1fac 100644
@@ -3509,6 +3509,43 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated,
 	}
 }
 
+static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h)
+{
+	unsigned long i;
+
+	for (i = 0; i < h->max_huge_pages; ++i) {
+		if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
+			break;
+		cond_resched();
+	}
+
+	return i;
+}
+
+static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
+{
+	unsigned long i;
+	struct folio *folio;
+	LIST_HEAD(folio_list);
+	nodemask_t node_alloc_noretry;
+
+	/* Bit mask controlling how hard we retry per-node allocations.*/
+	nodes_clear(node_alloc_noretry);
+
+	for (i = 0; i < h->max_huge_pages; ++i) {
+		folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
+						&node_alloc_noretry);
+		if (!folio)
+			break;
+		list_add(&folio->lru, &folio_list);
+		cond_resched();
+	}
+
+	prep_and_add_allocated_folios(h, &folio_list);
+
+	return i;
+}
+
 /*
  * NOTE: this routine is called in different contexts for gigantic and
  * non-gigantic pages.
@@ -3522,10 +3559,7 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated,
  */
 static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
 {
-	unsigned long i;
-	struct folio *folio;
-	LIST_HEAD(folio_list);
-	nodemask_t *node_alloc_noretry;
+	unsigned long allocated;
 
 	/* skip gigantic hugepages allocation if hugetlb_cma enabled */
 	if (hstate_is_gigantic(h) && hugetlb_cma_size) {
@@ -3538,47 +3572,12 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
 		return;
 
 	/* below will do all node balanced alloc */
-	if (!hstate_is_gigantic(h)) {
-		/*
-		 * Bit mask controlling how hard we retry per-node allocations.
-		 * Ignore errors as lower level routines can deal with
-		 * node_alloc_noretry == NULL.  If this kmalloc fails at boot
-		 * time, we are likely in bigger trouble.
-		 */
-		node_alloc_noretry = kmalloc(sizeof(*node_alloc_noretry),
-						GFP_KERNEL);
-	} else {
-		/* allocations done at boot time */
-		node_alloc_noretry = NULL;
-	}
-
-	/* bit mask controlling how hard we retry per-node allocations */
-	if (node_alloc_noretry)
-		nodes_clear(*node_alloc_noretry);
-
-	for (i = 0; i < h->max_huge_pages; ++i) {
-		if (hstate_is_gigantic(h)) {
-			/*
-			 * gigantic pages not added to list as they are not
-			 * added to pools now.
-			 */
-			if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
-				break;
-		} else {
-			folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
-							node_alloc_noretry);
-			if (!folio)
-				break;
-			list_add(&folio->lru, &folio_list);
-		}
-		cond_resched();
-	}
-
-	/* list will be empty if hstate_is_gigantic */
-	prep_and_add_allocated_folios(h, &folio_list);
+	if (hstate_is_gigantic(h))
+		allocated = hugetlb_gigantic_pages_alloc_boot(h);
+	else
+		allocated = hugetlb_pages_alloc_boot(h);
 
-	hugetlb_hstate_alloc_pages_errcheck(i, h);
-	kfree(node_alloc_noretry);
+	hugetlb_hstate_alloc_pages_errcheck(allocated, h);
 }
 
 static void __init hugetlb_init_hstates(void)
Signature · 1 line
2.20.1
maintainer Reviewed-by on v4 2/7
#3 lore raw

Re: [PATCH v4 2/7] hugetlb: split hugetlb_hstate_alloc_pages

Quoted 10 lines · Gang Li

1G and 2M huge pages have different allocation and initialization logic, which leads to subtle differences in parallelization. Therefore, it is appropriate to split hugetlb_hstate_alloc_pages into gigantic and non-gigantic.

This patch has no functional changes.

Thanks.

author
#4 lore raw

PATCH v4 3/7 padata: dispatch works on different nodes

In reply to Gang Li

When a group of tasks that access different nodes are scheduled on the same node, they may encounter bandwidth bottlenecks and access latency.

Thus, numa_aware flag is introduced here, allowing tasks to be distributed across different nodes to fully utilize the advantage of multi-node systems.

---
 include/linux/padata.h |  3 +++
 kernel/padata.c        | 14 ++++++++++++--
 mm/mm_init.c           |  1 +
 3 files changed, 16 insertions(+), 2 deletions(-)
include/linux/padata.h +3 -0
index 495b16b6b4d7..f79ccd50e7f4 100644
@@ -137,6 +137,8 @@ struct padata_shell {
  *             appropriate for one worker thread to do at once.
  * @max_threads: Max threads to use for the job, actual number may be less
  *               depending on task size and minimum chunk size.
+ * @numa_aware: Dispatch jobs to different nodes. If a node only has memory but
+ *              no CPU, dispatch its jobs to a random CPU.
  */
 struct padata_mt_job {
 	void (*thread_fn)(unsigned long start, unsigned long end, void *arg);
@@ -146,6 +148,7 @@ struct padata_mt_job {
 	unsigned long		align;
 	unsigned long		min_chunk;
 	int			max_threads;
+	bool			numa_aware;
 };
 
 /**
kernel/padata.c +12 -2
index 179fb1518070..10eae3f59203 100644
@@ -485,7 +485,8 @@ void __init padata_do_multithreaded(struct padata_mt_job *job)
 	struct padata_work my_work, *pw;
 	struct padata_mt_job_state ps;
 	LIST_HEAD(works);
-	int nworks;
+	int nworks, nid;
+	static atomic_t last_used_nid = ATOMIC_INIT(0);
 
 	if (job->size == 0)
 		return;
@@ -517,7 +518,16 @@ void __init padata_do_multithreaded(struct padata_mt_job *job)
 	ps.chunk_size = roundup(ps.chunk_size, job->align);
 
 	list_for_each_entry(pw, &works, pw_list)
-		queue_work(system_unbound_wq, &pw->pw_work);
+		if (job->numa_aware) {
+			int old_node = atomic_read(&last_used_nid);
+
+			do {
+				nid = next_node_in(old_node, node_states[N_CPU]);
+			} while (!atomic_try_cmpxchg(&last_used_nid, &old_node, nid));
+			queue_work_node(nid, system_unbound_wq, &pw->pw_work);
+		} else {
+			queue_work(system_unbound_wq, &pw->pw_work);
+		}
 
 	/* Use the current thread, which saves starting a workqueue worker. */
 	padata_work_init(&my_work, padata_mt_helper, &ps, PADATA_WORK_ONSTACK);
mm/mm_init.c +1 -0
index 2c19f5515e36..549e76af8f82 100644
@@ -2231,6 +2231,7 @@ static int __init deferred_init_memmap(void *data)
 			.align       = PAGES_PER_SECTION,
 			.min_chunk   = PAGES_PER_SECTION,
 			.max_threads = max_threads,
+			.numa_aware  = false,
 		};
 
 		padata_do_multithreaded(&job);
Signature · 1 line
2.20.1
#5 lore raw

Re: [PATCH v4 3/7] padata: dispatch works on different nodes

Quoted 20 lines · Gang Li

When a group of tasks that access different nodes are scheduled on the same node, they may encounter bandwidth bottlenecks and access latency.

Thus, numa_aware flag is introduced here, allowing tasks to be distributed across different nodes to fully utilize the advantage of multi-node systems.

---
 include/linux/padata.h |  3 +++
 kernel/padata.c        | 14 ++++++++++++--
 mm/mm_init.c           |  1 +
 3 files changed, 16 insertions(+), 2 deletions(-)
include/linux/padata.h +0 -0
index 495b16b6b4d7..f79ccd50e7f4 100644

include/linux/padata.h struct padata_shell { · line 141

Quoted 5 lines
include/linux/padata.h +2 -0
@@ -137,6 +137,8 @@ struct padata_shell {
  *             appropriate for one worker thread to do at once.
  * @max_threads: Max threads to use for the job, actual number may be less
  *               depending on task size and minimum chunk size.
+ * @numa_aware: Dispatch jobs to different nodes. If a node only has memory but
+ *              no CPU, dispatch its jobs to a random CPU.

Suggest:
Distribute jobs to different nodes with CPU in a round robin fashion.

Quoted 2 lines
include/linux/padata.h +0 -0
  */
 struct padata_mt_job {
author
#6 lore raw

Re: [PATCH v4 3/7] padata: dispatch works on different nodes

Quoted 16 lines · Tim Chen
Quoted 15 lines · Gang Li

When a group of tasks that access different nodes are scheduled on the same node, they may encounter bandwidth bottlenecks and access latency.

Thus, numa_aware flag is introduced here, allowing tasks to be distributed across different nodes to fully utilize the advantage of multi-node systems.

---
  include/linux/padata.h |  3 +++
  kernel/padata.c        | 14 ++++++++++++--
  mm/mm_init.c           |  1 +
  3 files changed, 16 insertions(+), 2 deletions(-)
Quoted 14 lines

include/linux/padata.h struct padata_shell { · line 141

Quoted 10 lines
include/linux/padata.h +2 -0
index 495b16b6b4d7..f79ccd50e7f4 100644
@@ -137,6 +137,8 @@ struct padata_shell {
   *             appropriate for one worker thread to do at once.
   * @max_threads: Max threads to use for the job, actual number may be less
   *               depending on task size and minimum chunk size.
+ * @numa_aware: Dispatch jobs to different nodes. If a node only has memory but
+ *              no CPU, dispatch its jobs to a random CPU.

Suggest:
Distribute jobs to different nodes with CPU in a round robin fashion.

Good idea.
Thanks.

Quoted 3 lines
Quoted 2 lines
include/linux/padata.h +0 -0
   */
  struct padata_mt_job {
maintainer Reviewed-by on v4 3/7
#7 lore raw

Re: [PATCH v4 3/7] padata: dispatch works on different nodes

In reply to Gang Li
Quoted 41 lines · Gang Li

When a group of tasks that access different nodes are scheduled on the same node, they may encounter bandwidth bottlenecks and access latency.

Thus, numa_aware flag is introduced here, allowing tasks to be distributed across different nodes to fully utilize the advantage of multi-node systems.

---
  include/linux/padata.h |  3 +++
  kernel/padata.c        | 14 ++++++++++++--
  mm/mm_init.c           |  1 +
  3 files changed, 16 insertions(+), 2 deletions(-)
include/linux/padata.h +3 -0
index 495b16b6b4d7..f79ccd50e7f4 100644
@@ -137,6 +137,8 @@ struct padata_shell {
   *             appropriate for one worker thread to do at once.
   * @max_threads: Max threads to use for the job, actual number may be less
   *               depending on task size and minimum chunk size.
+ * @numa_aware: Dispatch jobs to different nodes. If a node only has memory but
+ *              no CPU, dispatch its jobs to a random CPU.
   */
  struct padata_mt_job {
  	void (*thread_fn)(unsigned long start, unsigned long end, void *arg);
@@ -146,6 +148,7 @@ struct padata_mt_job {
  	unsigned long		align;
  	unsigned long		min_chunk;
  	int			max_threads;
+	bool			numa_aware;
  };
 
  /**
kernel/padata.c +0 -0
index 179fb1518070..10eae3f59203 100644

kernel/padata.c void __init padata_do_multithreaded(struct padata_mt_job *job) · line 489

Quoted 6 lines
kernel/padata.c +2 -1
@@ -485,7 +485,8 @@ void __init padata_do_multithreaded(struct padata_mt_job *job)
  	struct padata_work my_work, *pw;
  	struct padata_mt_job_state ps;
  	LIST_HEAD(works);
-	int nworks;
+	int nworks, nid;
+	static atomic_t last_used_nid = ATOMIC_INIT(0);

last_used_nid is only used during boot time so it could be __init_data. Otherwise, LGTM.

Quoted 33 lines
kernel/padata.c +10 -1
  	if (job->size == 0)
  		return;
@@ -517,7 +518,16 @@ void __init padata_do_multithreaded(struct padata_mt_job *job)
  	ps.chunk_size = roundup(ps.chunk_size, job->align);
 
  	list_for_each_entry(pw, &works, pw_list)
-		queue_work(system_unbound_wq, &pw->pw_work);
+		if (job->numa_aware) {
+			int old_node = atomic_read(&last_used_nid);
+
+			do {
+				nid = next_node_in(old_node, node_states[N_CPU]);
+			} while (!atomic_try_cmpxchg(&last_used_nid, &old_node, nid));
+			queue_work_node(nid, system_unbound_wq, &pw->pw_work);
+		} else {
+			queue_work(system_unbound_wq, &pw->pw_work);
+		}
 
  	/* Use the current thread, which saves starting a workqueue worker. */
  	padata_work_init(&my_work, padata_mt_helper, &ps, PADATA_WORK_ONSTACK);
mm/mm_init.c +1 -0
index 2c19f5515e36..549e76af8f82 100644
@@ -2231,6 +2231,7 @@ static int __init deferred_init_memmap(void *data)
  			.align       = PAGES_PER_SECTION,
  			.min_chunk   = PAGES_PER_SECTION,
  			.max_threads = max_threads,
+			.numa_aware  = false,
  		};
 
  		padata_do_multithreaded(&job);
author
#8 lore raw

Re: [PATCH v4 3/7] padata: dispatch works on different nodes

Quoted 14 lines · Muchun Song

kernel/padata.c void __init padata_do_multithreaded(struct · line 490

Quoted 10 lines · Gang Li
kernel/padata.c +2 -1
@@ -485,7 +485,8 @@ void __init padata_do_multithreaded(struct 
 padata_mt_job *job)
       struct padata_work my_work, *pw;
       struct padata_mt_job_state ps;
       LIST_HEAD(works);
-    int nworks;
+    int nworks, nid;
+    static atomic_t last_used_nid = ATOMIC_INIT(0);

last_used_nid is only used during boot time so it could be __init_data. Otherwise, LGTM.

Quoted 2 lines

OK, thanks.

author
#9 lore raw

PATCH v4 4/7 hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc

In reply to Gang Li

With parallelization of hugetlb allocation across different threads, each thread works on a differnet node to allocate pages from, instead of all allocating from a common node h->next_nid_to_alloc. To address this, it's necessary to assign a separate next_nid_to_alloc for each thread.

Consequently, the hstate_next_node_to_alloc and for_each_node_mask_to_alloc have been modified to directly accept a *next_nid_to_alloc parameter, ensuring thread-specific allocation and avoiding concurrent access issues.

---
 mm/hugetlb.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
mm/hugetlb.c +12 -10
index 98ae108e1fac..effe5539e545 100644
@@ -1464,15 +1464,15 @@ static int get_valid_node_allowed(int nid, nodemask_t *nodes_allowed)
  * next node from which to allocate, handling wrap at end of node
  * mask.
  */
-static int hstate_next_node_to_alloc(struct hstate *h,
+static int hstate_next_node_to_alloc(int *next_node,
 					nodemask_t *nodes_allowed)
 {
 	int nid;
 
 	VM_BUG_ON(!nodes_allowed);
 
-	nid = get_valid_node_allowed(h->next_nid_to_alloc, nodes_allowed);
-	h->next_nid_to_alloc = next_node_allowed(nid, nodes_allowed);
+	nid = get_valid_node_allowed(*next_node, nodes_allowed);
+	*next_node = next_node_allowed(nid, nodes_allowed);
 
 	return nid;
 }
@@ -1495,10 +1495,10 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed)
 	return nid;
 }
 
-#define for_each_node_mask_to_alloc(hs, nr_nodes, node, mask)		\
+#define for_each_node_mask_to_alloc(next_node, nr_nodes, node, mask)		\
 	for (nr_nodes = nodes_weight(*mask);				\
 		nr_nodes > 0 &&						\
-		((node = hstate_next_node_to_alloc(hs, mask)) || 1);	\
+		((node = hstate_next_node_to_alloc(next_node, mask)) || 1);	\
 		nr_nodes--)
 
 #define for_each_node_mask_to_free(hs, nr_nodes, node, mask)		\
@@ -2350,12 +2350,13 @@ static void prep_and_add_allocated_folios(struct hstate *h,
  */
 static struct folio *alloc_pool_huge_folio(struct hstate *h,
 					nodemask_t *nodes_allowed,
-					nodemask_t *node_alloc_noretry)
+					nodemask_t *node_alloc_noretry,
+					int *next_node)
 {
 	gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
 	int nr_nodes, node;
 
-	for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
+	for_each_node_mask_to_alloc(next_node, nr_nodes, node, nodes_allowed) {
 		struct folio *folio;
 
 		folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node,
@@ -3310,7 +3311,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid)
 		goto found;
 	}
 	/* allocate from next node when distributing huge pages */
-	for_each_node_mask_to_alloc(h, nr_nodes, node, &node_states[N_MEMORY]) {
+	for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_MEMORY]) {
 		m = memblock_alloc_try_nid_raw(
 				huge_page_size(h), huge_page_size(h),
 				0, MEMBLOCK_ALLOC_ACCESSIBLE, node);
@@ -3679,7 +3680,7 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed,
 	VM_BUG_ON(delta != -1 && delta != 1);
 
 	if (delta < 0) {
-		for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
+		for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, nodes_allowed) {
 			if (h->surplus_huge_pages_node[node])
 				goto found;
 		}
@@ -3794,7 +3795,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,
 		cond_resched();
 
 		folio = alloc_pool_huge_folio(h, nodes_allowed,
-						node_alloc_noretry);
+						node_alloc_noretry,
+						&h->next_nid_to_alloc);
 		if (!folio) {
 			prep_and_add_allocated_folios(h, &page_list);
 			spin_lock_irq(&hugetlb_lock);
Signature · 1 line
2.20.1
Reviewed-by on v4 4/7
#10 lore raw

Re: [PATCH v4 4/7] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc

Quoted 11 lines · Gang Li

With parallelization of hugetlb allocation across different threads, each thread works on a differnet node to allocate pages from, instead of all allocating from a common node h->next_nid_to_alloc. To address this, it's necessary to assign a separate next_nid_to_alloc for each thread.

Consequently, the hstate_next_node_to_alloc and for_each_node_mask_to_alloc have been modified to directly accept a *next_nid_to_alloc parameter, ensuring thread-specific allocation and avoiding concurrent access issues.

Quoted 1 line

---

maintainer Reviewed-by on v4 4/7
#11 lore raw

Re: [PATCH v4 4/7] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc

In reply to Gang Li
Quoted 9 lines · Gang Li

With parallelization of hugetlb allocation across different threads, each thread works on a differnet node to allocate pages from, instead of all allocating from a common node h->next_nid_to_alloc. To address this, it's necessary to assign a separate next_nid_to_alloc for each thread.

Quoted 6 lines

Consequently, the hstate_next_node_to_alloc and for_each_node_mask_to_alloc have been modified to directly accept a *next_nid_to_alloc parameter, ensuring thread-specific allocation and avoiding concurrent access issues.

Thanks.

maintainer
#12 lore raw

Re: [PATCH v4 4/7] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc

In reply to Gang Li
Quoted 54 lines · Gang Li

With parallelization of hugetlb allocation across different threads, each thread works on a differnet node to allocate pages from, instead of all allocating from a common node h->next_nid_to_alloc. To address this, it's necessary to assign a separate next_nid_to_alloc for each thread.

Consequently, the hstate_next_node_to_alloc and for_each_node_mask_to_alloc have been modified to directly accept a *next_nid_to_alloc parameter, ensuring thread-specific allocation and avoiding concurrent access issues.

---
  mm/hugetlb.c | 22 ++++++++++++----------
  1 file changed, 12 insertions(+), 10 deletions(-)
mm/hugetlb.c +5 -5
index 98ae108e1fac..effe5539e545 100644
@@ -1464,15 +1464,15 @@ static int get_valid_node_allowed(int nid, nodemask_t *nodes_allowed)
   * next node from which to allocate, handling wrap at end of node
   * mask.
   */
-static int hstate_next_node_to_alloc(struct hstate *h,
+static int hstate_next_node_to_alloc(int *next_node,
  					nodemask_t *nodes_allowed)
  {
  	int nid;
 
  	VM_BUG_ON(!nodes_allowed);
 
-	nid = get_valid_node_allowed(h->next_nid_to_alloc, nodes_allowed);
-	h->next_nid_to_alloc = next_node_allowed(nid, nodes_allowed);
+	nid = get_valid_node_allowed(*next_node, nodes_allowed);
+	*next_node = next_node_allowed(nid, nodes_allowed);
 
  	return nid;
  }
@@ -1495,10 +1495,10 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed)
  	return nid;
  }
 
-#define for_each_node_mask_to_alloc(hs, nr_nodes, node, mask)		\
+#define for_each_node_mask_to_alloc(next_node, nr_nodes, node, mask)		\
  	for (nr_nodes = nodes_weight(*mask);				\
  		nr_nodes > 0 &&						\
-		((node = hstate_next_node_to_alloc(hs, mask)) || 1);	\
+		((node = hstate_next_node_to_alloc(next_node, mask)) || 1);	\
  		nr_nodes--)
 
  #define for_each_node_mask_to_free(hs, nr_nodes, node, mask)		\
@@ -2350,12 +2350,13 @@ static void prep_and_add_allocated_folios(struct hstate *h,
   */
  static struct folio *alloc_pool_huge_folio(struct hstate *h,

mm/hugetlb.c static void prep_and_add_allocated_folios(struct hstate *h, · line 2359

Quoted 10 lines
mm/hugetlb.c +3 -2
  					nodemask_t *nodes_allowed,
-					nodemask_t *node_alloc_noretry)
+					nodemask_t *node_alloc_noretry,
+					int *next_node)
  {
  	gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
  	int nr_nodes, node;
 
-	for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
+	for_each_node_mask_to_alloc(next_node, nr_nodes, node, nodes_allowed) {

A small question here, why not pass h->next_nid_to_alloc to for_each_node_mask_to_alloc()? What's the purpose of the third parameter of alloc_pool_huge_folio()?

Thanks.

author
#13 lore raw

Re: [PATCH v4 4/7] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc

Quoted 14 lines · Muchun Song
Quoted 11 lines · Gang Li

  static struct folio *alloc_pool_huge_folio(struct hstate *h, nodemask_t *nodes_allowed,
-                    nodemask_t *node_alloc_noretry)
+                    nodemask_t *node_alloc_noretry,
+                    int *next_node)
  {
      gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; int nr_nodes, node;
-    for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { +    for_each_node_mask_to_alloc(next_node, nr_nodes, node, nodes_allowed) {

Quoted 6 lines

A small question here, why not pass h->next_nid_to_alloc to for_each_node_mask_to_alloc()? What's the purpose of the third parameter of alloc_pool_huge_folio()?

Thanks.

In hugetlb_alloc_node->alloc_pool_huge_folio, hugetlb is initialized in parallel at boot time, then it needs each thread to have its own
next_nid, and can't use the global h->next_nid_to_alloc. so an extra parameter is added.

And h->next_nid_to_alloc in set_max_huge_pages->alloc_pool_huge_folio can not be removed. Because if the user calls set_max_huge_pages frequently and only adds 1 page at a time, that would result in each request being made on the same node if local variables are used.

maintainer
#14 lore raw

Re: [PATCH v4 4/7] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc

Quoted 16 lines · Gang Li
Quoted 11 lines · Muchun Song
Quoted 10 lines · Gang Li

static struct folio *alloc_pool_huge_folio(struct hstate *h,
nodemask_t *nodes_allowed,

mm/hugetlb.c +3 -2
-                    nodemask_t *node_alloc_noretry)
+                    nodemask_t *node_alloc_noretry,
+                    int *next_node)
  {
      gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
      int nr_nodes, node;
-    for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
+    for_each_node_mask_to_alloc(next_node, nr_nodes, node, nodes_allowed) {
Quoted 8 lines
Quoted 4 lines

A small question here, why not pass h->next_nid_to_alloc to for_each_node_mask_to_alloc()? What's the purpose of the third parameter of alloc_pool_huge_folio()?
Thanks.

In hugetlb_alloc_node->alloc_pool_huge_folio, hugetlb is initialized in parallel at boot time, then it needs each thread to have its own
next_nid, and can't use the global h->next_nid_to_alloc. so an extra parameter is added.

Yes. When I read your patch 6, I realized this.

Quoted 5 lines

And h->next_nid_to_alloc in set_max_huge_pages->alloc_pool_huge_folio can not be removed. Because if the user calls set_max_huge_pages frequently and only adds 1 page at a time, that would result in each request being made on the same node if local variables are used.

author
#15 lore raw

PATCH v4 5/7 hugetlb: have CONFIG_HUGETLBFS select CONFIG_PADATA

In reply to Gang Li

Allow hugetlb use padata_do_multithreaded for parallel initialization. Select CONFIG_PADATA in this case.

---
 fs/Kconfig | 1 +
 1 file changed, 1 insertion(+)
fs/Kconfig +1 -0
index 89fdbefd1075..a57d6e6c41e6 100644
@@ -262,6 +262,7 @@ menuconfig HUGETLBFS
 	depends on X86 || SPARC64 || ARCH_SUPPORTS_HUGETLBFS || BROKEN
 	depends on (SYSFS || SYSCTL)
 	select MEMFD_CREATE
+	select PADATA
 	help
 	  hugetlbfs is a filesystem backing for HugeTLB pages, based on
 	  ramfs. For architectures that support it, say Y here and read
Signature · 1 line
2.20.1
author
#16 lore raw

PATCH v4 6/7 hugetlb: parallelize 2M hugetlb allocation and initialization

In reply to Gang Li

By distributing both the allocation and the initialization tasks across multiple threads, the initialization of 2M hugetlb will be faster, thereby improving the boot speed.

Here are some test results:
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------

  256c2t(4 node) 2M           3336          1051   68.52%
  128c1t(2 node) 2M           1943           716   63.15%
---
 mm/hugetlb.c | 70 ++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 52 insertions(+), 18 deletions(-)
mm/hugetlb.c +52 -18
index effe5539e545..9b348ba418f5 100644
@@ -35,6 +35,7 @@
 #include <linux/delayacct.h>
 #include <linux/memory.h>
 #include <linux/mm_inline.h>
+#include <linux/padata.h>
 
 #include <asm/page.h>
 #include <asm/pgalloc.h>
@@ -3510,43 +3511,76 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated,
 	}
 }
 
-static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h)
+static void __init hugetlb_alloc_node(unsigned long start, unsigned long end, void *arg)
 {
-	unsigned long i;
+	struct hstate *h = (struct hstate *)arg;
+	int i, num = end - start;
+	nodemask_t node_alloc_noretry;
+	unsigned long flags;
+	int next_node = 0;
 
-	for (i = 0; i < h->max_huge_pages; ++i) {
-		if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
+	/* Bit mask controlling how hard we retry per-node allocations.*/
+	nodes_clear(node_alloc_noretry);
+
+	for (i = 0; i < num; ++i) {
+		struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
+						&node_alloc_noretry, &next_node);
+		if (!folio)
 			break;
+		spin_lock_irqsave(&hugetlb_lock, flags);
+		__prep_account_new_huge_page(h, folio_nid(folio));
+		enqueue_hugetlb_folio(h, folio);
+		spin_unlock_irqrestore(&hugetlb_lock, flags);
 		cond_resched();
 	}
+}
 
-	return i;
+static void __init hugetlb_vmemmap_optimize_node(unsigned long start, unsigned long end, void *arg)
+{
+	struct hstate *h = (struct hstate *)arg;
+	int nid = start;
+
+	hugetlb_vmemmap_optimize_folios(h, &h->hugepage_freelists[nid]);
 }
 
-static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
+static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h)
 {
 	unsigned long i;
-	struct folio *folio;
-	LIST_HEAD(folio_list);
-	nodemask_t node_alloc_noretry;
-
-	/* Bit mask controlling how hard we retry per-node allocations.*/
-	nodes_clear(node_alloc_noretry);
 
 	for (i = 0; i < h->max_huge_pages; ++i) {
-		folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
-						&node_alloc_noretry);
-		if (!folio)
+		if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
 			break;
-		list_add(&folio->lru, &folio_list);
 		cond_resched();
 	}
 
-	prep_and_add_allocated_folios(h, &folio_list);
-
 	return i;
 }
 
+static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
+{
+	struct padata_mt_job job = {
+		.fn_arg		= h,
+		.align		= 1,
+		.numa_aware	= true
+	};
+
+	job.thread_fn	= hugetlb_alloc_node;
+	job.start	= 0;
+	job.size	= h->max_huge_pages;
+	job.min_chunk	= h->max_huge_pages / num_node_state(N_MEMORY) / 2;
+	job.max_threads	= num_node_state(N_MEMORY) * 2;
+	padata_do_multithreaded(&job);
+
+	job.thread_fn	= hugetlb_vmemmap_optimize_node;
+	job.start	= 0;
+	job.size	= num_node_state(N_MEMORY);
+	job.min_chunk	= 1;
+	job.max_threads	= num_node_state(N_MEMORY);
+	padata_do_multithreaded(&job);
+
+	return h->nr_huge_pages;
+}
+
 /*
  * NOTE: this routine is called in different contexts for gigantic and
  * non-gigantic pages.
Signature · 1 line
2.20.1
maintainer
#17 lore raw

Re: [PATCH v4 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization

Quoted 31 lines · Gang Li

By distributing both the allocation and the initialization tasks across multiple threads, the initialization of 2M hugetlb will be faster, thereby improving the boot speed.

Here are some test results:
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------

   256c2t(4 node) 2M           3336          1051   68.52%
   128c1t(2 node) 2M           1943           716   63.15%
---
  mm/hugetlb.c | 70 ++++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 52 insertions(+), 18 deletions(-)
mm/hugetlb.c +1 -0
index effe5539e545..9b348ba418f5 100644
@@ -35,6 +35,7 @@
  #include <linux/delayacct.h>
  #include <linux/memory.h>
  #include <linux/mm_inline.h>
+#include <linux/padata.h>
 
  #include <asm/page.h>
  #include <asm/pgalloc.h>
@@ -3510,43 +3511,76 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated,
  	}
  }

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, · line 3520

Quoted 10 lines
mm/hugetlb.c +6 -2
 
-static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h)
+static void __init hugetlb_alloc_node(unsigned long start, unsigned long end, void *arg)
  {
-	unsigned long i;
+	struct hstate *h = (struct hstate *)arg;
+	int i, num = end - start;
+	nodemask_t node_alloc_noretry;
+	unsigned long flags;
+	int next_node = 0;

This should be first_online_node which may be not zero.

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, · line 3530

Quoted 12 lines
mm/hugetlb.c +8 -2
-	for (i = 0; i < h->max_huge_pages; ++i) {
-		if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
+	/* Bit mask controlling how hard we retry per-node allocations.*/
+	nodes_clear(node_alloc_noretry);
+
+	for (i = 0; i < num; ++i) {
+		struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
+						&node_alloc_noretry, &next_node);
+		if (!folio)
  			break;
+		spin_lock_irqsave(&hugetlb_lock, flags);

I suspect there will more contention on this lock when parallelizing. I want to know why you chose to drop prep_and_add_allocated_folios() call in the original hugetlb_pages_alloc_boot()?

Quoted 45 lines
mm/hugetlb.c +15 -14
+		__prep_account_new_huge_page(h, folio_nid(folio));
+		enqueue_hugetlb_folio(h, folio);
+		spin_unlock_irqrestore(&hugetlb_lock, flags);
  		cond_resched();
  	}
+}
 
-	return i;
+static void __init hugetlb_vmemmap_optimize_node(unsigned long start, unsigned long end, void *arg)
+{
+	struct hstate *h = (struct hstate *)arg;
+	int nid = start;
+
+	hugetlb_vmemmap_optimize_folios(h, &h->hugepage_freelists[nid]);
  }
 
-static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
+static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h)
  {
  	unsigned long i;
-	struct folio *folio;
-	LIST_HEAD(folio_list);
-	nodemask_t node_alloc_noretry;
-
-	/* Bit mask controlling how hard we retry per-node allocations.*/
-	nodes_clear(node_alloc_noretry);
 
  	for (i = 0; i < h->max_huge_pages; ++i) {
-		folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
-						&node_alloc_noretry);
-		if (!folio)
+		if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
  			break;
-		list_add(&folio->lru, &folio_list);
  		cond_resched();
  	}
 
-	prep_and_add_allocated_folios(h, &folio_list);
-
  	return i;
  }
 
+static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
+{
+	struct padata_mt_job job = {

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, · line 3571

Quoted 10 lines
mm/hugetlb.c +10 -0
+		.fn_arg		= h,
+		.align		= 1,
+		.numa_aware	= true
+	};
+
+	job.thread_fn	= hugetlb_alloc_node;
+	job.start	= 0;
+	job.size	= h->max_huge_pages;
+	job.min_chunk	= h->max_huge_pages / num_node_state(N_MEMORY) / 2;
+	job.max_threads	= num_node_state(N_MEMORY) * 2;

I am curious the magic number of 2 used in assignments of ->min_chunk and ->max_threads, does it from your experiment? I thinke it should be a comment here.

And I am also sceptical about the optimization for a small amount of allocation of hugepages. Given 4 hugepags needed to be allocated on UMA system, job.min_chunk will be 2, job.max_threads will be 2. Then, 2 workers will be scheduled, however each worker will just allocate 2 pages, how much the cost of scheduling? What if allocate 4 pages in single worker? Do you have any numbers on parallelism vs non-parallelism in a small allocation case? If we cannot gain from this case, I think we shold assign a reasonable value to ->min_chunk based on experiment.

Thanks.

Quoted 15 lines
mm/hugetlb.c +12 -0
+	padata_do_multithreaded(&job);
+
+	job.thread_fn	= hugetlb_vmemmap_optimize_node;
+	job.start	= 0;
+	job.size	= num_node_state(N_MEMORY);
+	job.min_chunk	= 1;
+	job.max_threads	= num_node_state(N_MEMORY);
+	padata_do_multithreaded(&job);
+
+	return h->nr_huge_pages;
+}
+
  /*
   * NOTE: this routine is called in different contexts for gigantic and
   * non-gigantic pages.
author
#18 lore raw

Re: [PATCH v4 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization

Quoted 10 lines · Gang Li
Quoted 9 lines

+static void __init hugetlb_alloc_node(unsigned long start, unsigned long end, void *arg)
  {
-    unsigned long i;
+    struct hstate *h = (struct hstate *)arg;
+    int i, num = end - start;
+    nodemask_t node_alloc_noretry;
+    unsigned long flags;
+    int next_node = 0;

Quoted 2 lines

This should be first_online_node which may be not zero.

That's right. Thanks!

Quoted 5 lines
Quoted 5 lines

-    for (i = 0; i < h->max_huge_pages; ++i) {
-        if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
+    /* Bit mask controlling how hard we retry per-node allocations.*/ +    nodes_clear(node_alloc_noretry);
+

Quoted 8 lines
Quoted 8 lines

+    for (i = 0; i < num; ++i) {
+        struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
+                        &node_alloc_noretry, &next_node); +        if (!folio)
              break;
+        spin_lock_irqsave(&hugetlb_lock, flags);
I suspect there will more contention on this lock when parallelizing.

In the worst case, there are only 'numa node number' of threads in contention. And in my testing, it doesn't degrade performance, but rather improves performance due to the reduced granularity.

Quoted 2 lines

I want to know why you chose to drop prep_and_add_allocated_folios() call in the original hugetlb_pages_alloc_boot()?

Splitting him to parallelize hugetlb_vmemmap_optimize_folios.

Quoted 9 lines
Quoted 9 lines
mm/hugetlb.c +9 -0
+static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
+{
+    struct padata_mt_job job = {
+        .fn_arg        = h,
+        .align        = 1,
+        .numa_aware    = true
+    };
+
+    job.thread_fn    = hugetlb_alloc_node;
Quoted 9 lines
Quoted 4 lines

+    job.start    = 0;
+    job.size    = h->max_huge_pages;
+    job.min_chunk    = h->max_huge_pages / num_node_state(N_MEMORY) / 2; +    job.max_threads    = num_node_state(N_MEMORY) * 2;

I am curious the magic number of 2 used in assignments of ->min_chunk and ->max_threads, does it from your experiment? I thinke it should be a comment here.

This is tested and I can perform more detailed tests and provide data.

Quoted 11 lines

And I am also sceptical about the optimization for a small amount of allocation of hugepages. Given 4 hugepags needed to be allocated on UMA system, job.min_chunk will be 2, job.max_threads will be 2. Then, 2 workers will be scheduled, however each worker will just allocate 2 pages, how much the cost of scheduling? What if allocate 4 pages in single worker? Do you have any numbers on parallelism vs non-parallelism in a small allocation case? If we cannot gain from this case, I think we shold assign a reasonable value to ->min_chunk based on experiment.

Thanks.

That's a good suggestion, I'll run some tests and choose the best values.

maintainer
#19 lore raw

Re: [PATCH v4 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization

Quoted 30 lines · Gang Li
Quoted 9 lines · Gang Li

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, · line 3520

Quoted 8 lines
mm/hugetlb.c +6 -1
+static void __init hugetlb_alloc_node(unsigned long start, unsigned long end, void *arg)
  {
-    unsigned long i;
+    struct hstate *h = (struct hstate *)arg;
+    int i, num = end - start;
+    nodemask_t node_alloc_noretry;
+    unsigned long flags;
+    int next_node = 0;

This should be first_online_node which may be not zero.

That's right. Thanks!

Quoted 4 lines
Quoted 4 lines
mm/hugetlb.c +2 -2
-    for (i = 0; i < h->max_huge_pages; ++i) {
-        if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
+    /* Bit mask controlling how hard we retry per-node allocations.*/
+    nodes_clear(node_alloc_noretry);
Quoted 8 lines
Quoted 8 lines
mm/hugetlb.c +6 -0
+
+    for (i = 0; i < num; ++i) {
+        struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
+                        &node_alloc_noretry, &next_node);
+        if (!folio)
              break;
+        spin_lock_irqsave(&hugetlb_lock, flags);

I suspect there will more contention on this lock when parallelizing.

Quoted 3 lines

In the worst case, there are only 'numa node number' of threads in contention. And in my testing, it doesn't degrade performance, but rather improves performance due to the reduced granularity.

So, the performance does not change if you move the lock out of loop?

Quoted 5 lines
Quoted 3 lines

I want to know why you chose to drop prep_and_add_allocated_folios() call in the original hugetlb_pages_alloc_boot()?

Splitting him to parallelize hugetlb_vmemmap_optimize_folios.

Unfortunately, HVO should be enabled before pages go to the pool list.

Quoted 35 lines
Quoted 17 lines
Quoted 4 lines
mm/hugetlb.c +3 -0
+static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
+{
+    struct padata_mt_job job = {

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, · line 3571

Quoted 10 lines
mm/hugetlb.c +10 -0
+        .fn_arg        = h,
+        .align        = 1,
+        .numa_aware    = true
+    };
+
+    job.thread_fn    = hugetlb_alloc_node;
+    job.start    = 0;
+    job.size    = h->max_huge_pages;
+    job.min_chunk    = h->max_huge_pages / num_node_state(N_MEMORY) / 2;
+    job.max_threads    = num_node_state(N_MEMORY) * 2;

I am curious the magic number of 2 used in assignments of ->min_chunk and ->max_threads, does it from your experiment? I thinke it should be a comment here.

This is tested and I can perform more detailed tests and provide data.

Quoted 10 lines

And I am also sceptical about the optimization for a small amount of allocation of hugepages. Given 4 hugepags needed to be allocated on UMA system, job.min_chunk will be 2, job.max_threads will be 2. Then, 2 workers will be scheduled, however each worker will just allocate 2 pages, how much the cost of scheduling? What if allocate 4 pages in single worker? Do you have any numbers on parallelism vs non-parallelism in a small allocation case? If we cannot gain from this case, I think we shold assign a reasonable value to ->min_chunk based on experiment. Thanks.

That's a good suggestion, I'll run some tests and choose the best values.

author
#20 lore raw

Re: [PATCH v4 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization

Quoted 28 lines · Muchun Song
Quoted 28 lines · Gang Li
Quoted 9 lines · Gang Li

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, · line 3520

Quoted 8 lines
mm/hugetlb.c +6 -1
+static void __init hugetlb_alloc_node(unsigned long start, unsigned long end, void *arg)
   {
-    unsigned long i;
+    struct hstate *h = (struct hstate *)arg;
+    int i, num = end - start;
+    nodemask_t node_alloc_noretry;
+    unsigned long flags;
+    int next_node = 0;

This should be first_online_node which may be not zero.

That's right. Thanks!

Quoted 12 lines
Quoted 12 lines
mm/hugetlb.c +8 -2
-    for (i = 0; i < h->max_huge_pages; ++i) {
-        if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
+    /* Bit mask controlling how hard we retry per-node allocations.*/
+    nodes_clear(node_alloc_noretry);
+
+    for (i = 0; i < num; ++i) {
+        struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
+                        &node_alloc_noretry, &next_node);
+        if (!folio)
               break;
+        spin_lock_irqsave(&hugetlb_lock, flags);

I suspect there will more contention on this lock when parallelizing.

Quoted 7 lines
Quoted 3 lines

In the worst case, there are only 'numa node number' of threads in contention. And in my testing, it doesn't degrade performance, but rather improves performance due to the reduced granularity.

So, the performance does not change if you move the lock out of loop?

If we move the lock out of loop, then multi-threading becomes single-threading, which definitely reduces performance.

```
+       spin_lock_irqsave(&hugetlb_lock, flags);
        for (i = 0; i < num; ++i) {
                struct folio *folio = alloc_pool_huge_folio(h, 
&node_states[N_MEMORY],
                                                &node_alloc_noretry, 
&next_node);
                if (!folio)
                        break;
-               spin_lock_irqsave(&hugetlb_lock, flags);
                __prep_account_new_huge_page(h, folio_nid(folio));
                enqueue_hugetlb_folio(h, folio);
-               spin_unlock_irqrestore(&hugetlb_lock, flags);
                cond_resched();
        }
+       spin_unlock_irqrestore(&hugetlb_lock, flags);
 }
```
maintainer
#21 lore raw

Re: [PATCH v4 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization

Quoted 33 lines · Gang Li
Quoted 28 lines · Muchun Song
Quoted 28 lines · Gang Li
Quoted 9 lines · Gang Li

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, · line 3520

Quoted 8 lines
mm/hugetlb.c +6 -1
+static void __init hugetlb_alloc_node(unsigned long start, unsigned long end, void *arg)
  {
-    unsigned long i;
+    struct hstate *h = (struct hstate *)arg;
+    int i, num = end - start;
+    nodemask_t node_alloc_noretry;
+    unsigned long flags;
+    int next_node = 0;

This should be first_online_node which may be not zero.

That's right. Thanks!

Quoted 12 lines
Quoted 12 lines
mm/hugetlb.c +8 -2
-    for (i = 0; i < h->max_huge_pages; ++i) {
-        if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
+    /* Bit mask controlling how hard we retry per-node allocations.*/
+    nodes_clear(node_alloc_noretry);
+
+    for (i = 0; i < num; ++i) {
+        struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
+                        &node_alloc_noretry, &next_node);
+        if (!folio)
              break;
+        spin_lock_irqsave(&hugetlb_lock, flags);

I suspect there will more contention on this lock when parallelizing.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

In the worst case, there are only 'numa node number' of threads in contention. And in my testing, it doesn't degrade performance, but rather improves performance due to the reduced granularity.

So, the performance does not change if you move the lock out of loop?

If we move the lock out of loop, then multi-threading becomes single-threading, which definitely reduces performance.

No. I mean batching the pages into pool list just like prep_and_add_allocated_folios does.

Quoted 17 lines

```

(quoted diff) +2 -2
+       spin_lock_irqsave(&hugetlb_lock, flags);
       for (i = 0; i < num; ++i) {
               struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
                                               &node_alloc_noretry, &next_node);
               if (!folio)
                       break;
-               spin_lock_irqsave(&hugetlb_lock, flags);
               __prep_account_new_huge_page(h, folio_nid(folio));
               enqueue_hugetlb_folio(h, folio);
-               spin_unlock_irqrestore(&hugetlb_lock, flags);
               cond_resched();
       }
+       spin_unlock_irqrestore(&hugetlb_lock, flags);

}
```

author
#22 lore raw

PATCH v4 7/7 hugetlb: parallelize 1G hugetlb initialization

In reply to Gang Li

Optimizing the initialization speed of 1G huge pages through parallelization.

1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in gather_bootmem_prealloc.

Here are some test results:
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------

  256c2t(4 node) 1G           4745          2024   57.34%
  128c1t(2 node) 1G           3358          1712   49.02%
      12t        1G          77000         18300   76.23%
---
 include/linux/hugetlb.h |  2 +-
 mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++--------
 2 files changed, 35 insertions(+), 9 deletions(-)
include/linux/hugetlb.h +1 -1
index c1ee640d87b1..77b30a8c6076 100644
@@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
 
 extern int sysctl_hugetlb_shm_group;
-extern struct list_head huge_boot_pages;
+extern struct list_head huge_boot_pages[MAX_NUMNODES];
 
 /* arch callbacks */
 
mm/hugetlb.c +34 -8

98 lines not on this page · raw message

Signature · 1 line
2.20.1
#23 lore raw

Re: [PATCH v4 7/7] hugetlb: parallelize 1G hugetlb initialization

Quoted 27 lines · Gang Li

Optimizing the initialization speed of 1G huge pages through parallelization.

1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in gather_bootmem_prealloc.

Here are some test results:
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------

   256c2t(4 node) 1G           4745          2024   57.34%
   128c1t(2 node) 1G           3358          1712   49.02%
       12t        1G          77000         18300   76.23%
---
  include/linux/hugetlb.h |  2 +-
  mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++--------
  2 files changed, 35 insertions(+), 9 deletions(-)
include/linux/hugetlb.h +0 -0
index c1ee640d87b1..77b30a8c6076 100644

include/linux/hugetlb.h pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, · line 181

Quoted 5 lines
include/linux/hugetlb.h +1 -1
@@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
  struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
 
  extern int sysctl_hugetlb_shm_group;
-extern struct list_head huge_boot_pages;
+extern struct list_head huge_boot_pages[MAX_NUMNODES];

Hi, please update arch/powerpc/mm/hugetlbpage.c too, since it was used in ppc.

Quoted 112 lines
include/linux/hugetlb.h +0 -0
  /* arch callbacks */
 
mm/hugetlb.c +34 -8
index 9b348ba418f5..2f4b77630ada 100644
@@ -69,7 +69,7 @@ static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
  #endif
  static unsigned long hugetlb_cma_size __initdata;
 
-__initdata LIST_HEAD(huge_boot_pages);
+__initdata struct list_head huge_boot_pages[MAX_NUMNODES];
 
  /* for command line parsing */
  static struct hstate * __initdata parsed_hstate;
@@ -3301,7 +3301,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid)
  int __alloc_bootmem_huge_page(struct hstate *h, int nid)
  {
  	struct huge_bootmem_page *m = NULL; /* initialize for clang */
-	int nr_nodes, node;
+	int nr_nodes, node = nid;
 
  	/* do node specific alloc */
  	if (nid != NUMA_NO_NODE) {
@@ -3339,7 +3339,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid)
  		huge_page_size(h) - PAGE_SIZE);
  	/* Put them into a private list first because mem_map is not up yet */
  	INIT_LIST_HEAD(&m->list);
-	list_add(&m->list, &huge_boot_pages);
+	list_add(&m->list, &huge_boot_pages[node]);
  	m->hstate = h;
  	return 1;
  }
@@ -3390,8 +3390,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
  	/* Send list for bulk vmemmap optimization processing */
  	hugetlb_vmemmap_optimize_folios(h, folio_list);
 
-	/* Add all new pool pages to free lists in one lock cycle */
-	spin_lock_irqsave(&hugetlb_lock, flags);
  	list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
  		if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
  			/*
@@ -3404,23 +3402,27 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
  					HUGETLB_VMEMMAP_RESERVE_PAGES,
  					pages_per_huge_page(h));
  		}
+		/* Subdivide locks to achieve better parallel performance */
+		spin_lock_irqsave(&hugetlb_lock, flags);
  		__prep_account_new_huge_page(h, folio_nid(folio));
  		enqueue_hugetlb_folio(h, folio);
+		spin_unlock_irqrestore(&hugetlb_lock, flags);
  	}
-	spin_unlock_irqrestore(&hugetlb_lock, flags);
  }
 
  /*
   * Put bootmem huge pages into the standard lists after mem_map is up.
   * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages.
   */
-static void __init gather_bootmem_prealloc(void)
+static void __init __gather_bootmem_prealloc(unsigned long start, unsigned long end, void *arg)
+
  {
+	int nid = start;
  	LIST_HEAD(folio_list);
  	struct huge_bootmem_page *m;
  	struct hstate *h = NULL, *prev_h = NULL;
 
-	list_for_each_entry(m, &huge_boot_pages, list) {
+	list_for_each_entry(m, &huge_boot_pages[nid], list) {
  		struct page *page = virt_to_page(m);
  		struct folio *folio = (void *)page;
 
@@ -3453,6 +3455,22 @@ static void __init gather_bootmem_prealloc(void)
  	prep_and_add_bootmem_folios(h, &folio_list);
  }
 
+static void __init gather_bootmem_prealloc(void)
+{
+	struct padata_mt_job job = {
+		.thread_fn	= __gather_bootmem_prealloc,
+		.fn_arg		= NULL,
+		.start		= 0,
+		.size		= num_node_state(N_MEMORY),
+		.align		= 1,
+		.min_chunk	= 1,
+		.max_threads	= num_node_state(N_MEMORY),
+		.numa_aware	= true,
+	};
+
+	padata_do_multithreaded(&job);
+}
+
  static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
  {
  	unsigned long i;
@@ -3602,6 +3620,14 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
  		return;
  	}
 
+	/* hugetlb_hstate_alloc_pages will be called many times, init huge_boot_pages once*/
+	if (huge_boot_pages[0].next == NULL) {
+		int i = 0;
+
+		for (i = 0; i < MAX_NUMNODES; i++)
+			INIT_LIST_HEAD(&huge_boot_pages[i]);
+	}
+
  	/* do node specific alloc */
  	if (hugetlb_hstate_alloc_pages_specific_nodes(h))
  		return;
author
#24 lore raw

Re: [PATCH v4 7/7] hugetlb: parallelize 1G hugetlb initialization

Quoted 5 lines · Kefeng Wang
Quoted 4 lines · Gang Li

  include/linux/hugetlb.h |  2 +-
  mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++--------

  2 files changed, 35 insertions(+), 9 deletions(-)
Quoted 14 lines

include/linux/hugetlb.h pte_t *huge_pmd_share(struct mm_struct *mm, struct · line 181

Quoted 10 lines
include/linux/hugetlb.h +1 -1
index c1ee640d87b1..77b30a8c6076 100644
@@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct 
 vm_area_struct *vma,
   struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
   extern int sysctl_hugetlb_shm_group;
-extern struct list_head huge_boot_pages;
+extern struct list_head huge_boot_pages[MAX_NUMNODES];

Hi, please update arch/powerpc/mm/hugetlbpage.c too, since it was used in ppc.

Thanks, will do

maintainer
#25 lore raw

Re: [PATCH v4 7/7] hugetlb: parallelize 1G hugetlb initialization

In reply to Gang Li
Quoted 8 lines · Gang Li

Optimizing the initialization speed of 1G huge pages through parallelization.

1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in gather_bootmem_prealloc.

Quoted 4 lines

Here are some test results:
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------
256c2t(4 node) 1G 4745 2024 57.34%

What does "256c2t" mean?

Quoted 37 lines
   128c1t(2 node) 1G           3358          1712   49.02%
       12t        1G          77000         18300   76.23%
---
  include/linux/hugetlb.h |  2 +-
  mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++--------
  2 files changed, 35 insertions(+), 9 deletions(-)
include/linux/hugetlb.h +1 -1
index c1ee640d87b1..77b30a8c6076 100644
@@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
  struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
 
  extern int sysctl_hugetlb_shm_group;
-extern struct list_head huge_boot_pages;
+extern struct list_head huge_boot_pages[MAX_NUMNODES];
 
  /* arch callbacks */
 
mm/hugetlb.c +1 -1
index 9b348ba418f5..2f4b77630ada 100644
@@ -69,7 +69,7 @@ static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
  #endif
  static unsigned long hugetlb_cma_size __initdata;
 
-__initdata LIST_HEAD(huge_boot_pages);
+__initdata struct list_head huge_boot_pages[MAX_NUMNODES];
 
  /* for command line parsing */
  static struct hstate * __initdata parsed_hstate;

mm/hugetlb.c int alloc_bootmem_huge_page(struct hstate *h, int nid) · line 3304

Quoted 5 lines
mm/hugetlb.c +1 -1
@@ -3301,7 +3301,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid)
  int __alloc_bootmem_huge_page(struct hstate *h, int nid)
  {
  	struct huge_bootmem_page *m = NULL; /* initialize for clang */
-	int nr_nodes, node;
+	int nr_nodes, node = nid;

Why not use nid directly in the following list_add()?

Quoted 30 lines
mm/hugetlb.c +4 -3
  	/* do node specific alloc */
  	if (nid != NUMA_NO_NODE) {
@@ -3339,7 +3339,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid)
  		huge_page_size(h) - PAGE_SIZE);
  	/* Put them into a private list first because mem_map is not up yet */
  	INIT_LIST_HEAD(&m->list);
-	list_add(&m->list, &huge_boot_pages);
+	list_add(&m->list, &huge_boot_pages[node]);
  	m->hstate = h;
  	return 1;
  }
@@ -3390,8 +3390,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
  	/* Send list for bulk vmemmap optimization processing */
  	hugetlb_vmemmap_optimize_folios(h, folio_list);
 
-	/* Add all new pool pages to free lists in one lock cycle */
-	spin_lock_irqsave(&hugetlb_lock, flags);
  	list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
  		if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
  			/*
@@ -3404,23 +3402,27 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
  					HUGETLB_VMEMMAP_RESERVE_PAGES,
  					pages_per_huge_page(h));
  		}
+		/* Subdivide locks to achieve better parallel performance *
+		spin_lock_irqsave(&hugetlb_lock, flags);
  		__prep_account_new_huge_page(h, folio_nid(folio));
  		enqueue_hugetlb_folio(h, folio);
+		spin_unlock_irqrestore(&hugetlb_lock, flags);

mm/hugetlb.c static void __init prep_and_add_bootmem_folios(struct hstate *h, · line 3417

Quoted 10 lines
mm/hugetlb.c +1 -2
  	}
-	spin_unlock_irqrestore(&hugetlb_lock, flags);
  }
 
  /*
   * Put bootmem huge pages into the standard lists after mem_map is up.
   * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages.
   */
-static void __init gather_bootmem_prealloc(void)
+static void __init __gather_bootmem_prealloc(unsigned long start, unsigned long end, void *arg)

This function name could be gather_bootmem_prealloc_node.

Quoted 36 lines
mm/hugetlb.c +19 -1
+
  {
+	int nid = start;
  	LIST_HEAD(folio_list);
  	struct huge_bootmem_page *m;
  	struct hstate *h = NULL, *prev_h = NULL;
 
-	list_for_each_entry(m, &huge_boot_pages, list) {
+	list_for_each_entry(m, &huge_boot_pages[nid], list) {
  		struct page *page = virt_to_page(m);
  		struct folio *folio = (void *)page;
 
@@ -3453,6 +3455,22 @@ static void __init gather_bootmem_prealloc(void)
  	prep_and_add_bootmem_folios(h, &folio_list);
  }
 
+static void __init gather_bootmem_prealloc(void)
+{
+	struct padata_mt_job job = {
+		.thread_fn	= __gather_bootmem_prealloc,
+		.fn_arg		= NULL,
+		.start		= 0,
+		.size		= num_node_state(N_MEMORY),
+		.align		= 1,
+		.min_chunk	= 1,
+		.max_threads	= num_node_state(N_MEMORY),
+		.numa_aware	= true,
+	};
+
+	padata_do_multithreaded(&job);
+}
+
  static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
  {
  	unsigned long i;

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages(struct hstate *h) · line 3623

Quoted 4 lines
mm/hugetlb.c +1 -0
@@ -3602,6 +3620,14 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
  		return;
  	}
 
+	/* hugetlb_hstate_alloc_pages will be called many times, init huge_boot_pages once*/

s/init/initialize/g

And you miss a black right before "*/".

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages(struct hstate *h) · line 3624

Quoted 1 line
mm/hugetlb.c +1 -0
+	if (huge_boot_pages[0].next == NULL) {

It it not intuitive. I'd like to use a 'initialied' variable to indicate whether it has been initialized. BTW, it can be marked as __initdata.

Quoted 9 lines
mm/hugetlb.c +6 -0
+		int i = 0;
+
+		for (i = 0; i < MAX_NUMNODES; i++)
+			INIT_LIST_HEAD(&huge_boot_pages[i]);
+	}
+
  	/* do node specific alloc */
  	if (hugetlb_hstate_alloc_pages_specific_nodes(h))
  		return;
author
#26 lore raw

Re: [PATCH v4 7/7] hugetlb: parallelize 1G hugetlb initialization

Quoted 9 lines · Muchun Song
Quoted 8 lines · Gang Li

Optimizing the initialization speed of 1G huge pages through parallelization.

1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in gather_bootmem_prealloc.

Quoted 6 lines
Quoted 4 lines

Here are some test results:
         test          no patch(ms)   patched(ms)   saved ------------------- -------------- ------------- -------- 256c2t(4 node) 1G           4745          2024   57.34%

What does "256c2t" mean?

A machine with 256 core and 2T memory.

Quoted 35 lines
Quoted 35 lines

   128c1t(2 node) 1G           3358          1712   49.02% 12t        1G          77000         18300   76.23%

---
  include/linux/hugetlb.h |  2 +-
  mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++--------

  2 files changed, 35 insertions(+), 9 deletions(-)
include/linux/hugetlb.h +1 -1
index c1ee640d87b1..77b30a8c6076 100644
@@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct 
 vm_area_struct *vma,
   struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
   extern int sysctl_hugetlb_shm_group;
-extern struct list_head huge_boot_pages;
+extern struct list_head huge_boot_pages[MAX_NUMNODES];
   /* arch callbacks */
mm/hugetlb.c +1 -1
index 9b348ba418f5..2f4b77630ada 100644
@@ -69,7 +69,7 @@ static bool hugetlb_cma_folio(struct folio *folio, 
 unsigned int order)
   #endif
   static unsigned long hugetlb_cma_size __initdata;
-__initdata LIST_HEAD(huge_boot_pages);
+__initdata struct list_head huge_boot_pages[MAX_NUMNODES];
   /* for command line parsing */
   static struct hstate * __initdata parsed_hstate;
@@ -3301,7 +3301,7 @@ int alloc_bootmem_huge_page(struct hstate *h, 
Quoted 8 lines
Quoted 6 lines
mm/hugetlb.c +0 -0
 int nid)
   int __alloc_bootmem_huge_page(struct hstate *h, int nid)
   {

      struct huge_bootmem_page *m = NULL; /* initialize for clang */ -    int nr_nodes, node;
+    int nr_nodes, node = nid;

Why not use nid directly in the following list_add()?

node may be changed in for_each_node_mask_to_alloc.

Quoted 36 lines
Quoted 36 lines

      /* do node specific alloc */
      if (nid != NUMA_NO_NODE) {

mm/hugetlb.c +1 -1
@@ -3339,7 +3339,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, 
 int nid)
           huge_page_size(h) - PAGE_SIZE);
       /* Put them into a private list first because mem_map is not up yet */
       INIT_LIST_HEAD(&m->list);
-    list_add(&m->list, &huge_boot_pages);
+    list_add(&m->list, &huge_boot_pages[node]);
       m->hstate = h;
       return 1;

  }

mm/hugetlb.c +3 -3
@@ -3390,8 +3390,6 @@ static void __init 
 prep_and_add_bootmem_folios(struct hstate *h,
       /* Send list for bulk vmemmap optimization processing */
       hugetlb_vmemmap_optimize_folios(h, folio_list);
-    /* Add all new pool pages to free lists in one lock cycle */
-    spin_lock_irqsave(&hugetlb_lock, flags);
       list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
           if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
               /*
@@ -3404,23 +3402,27 @@ static void __init 
 prep_and_add_bootmem_folios(struct hstate *h,
                       HUGETLB_VMEMMAP_RESERVE_PAGES,
                       pages_per_huge_page(h));
           }
+        /* Subdivide locks to achieve better parallel performance *
+        spin_lock_irqsave(&hugetlb_lock, flags);
           __prep_account_new_huge_page(h, folio_nid(folio));
           enqueue_hugetlb_folio(h, folio);
+        spin_unlock_irqrestore(&hugetlb_lock, flags);
       }
-    spin_unlock_irqrestore(&hugetlb_lock, flags);
   }
   /*
Quoted 9 lines

mm/hugetlb.c static void __init prep_and_add_bootmem_folios(struct hstate *h,

Quoted 6 lines
mm/hugetlb.c +1 -1
    * Put bootmem huge pages into the standard lists after mem_map is up.
    * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages.
    */
-static void __init gather_bootmem_prealloc(void)
+static void __init __gather_bootmem_prealloc(unsigned long start, 
 unsigned long end, void *arg)

This function name could be gather_bootmem_prealloc_node.

LGTM.

Quoted 41 lines
Quoted 37 lines
mm/hugetlb.c +20 -1
+
   {
+    int nid = start;
       LIST_HEAD(folio_list);
       struct huge_bootmem_page *m;
       struct hstate *h = NULL, *prev_h = NULL;
-    list_for_each_entry(m, &huge_boot_pages, list) {
+    list_for_each_entry(m, &huge_boot_pages[nid], list) {
           struct page *page = virt_to_page(m);
           struct folio *folio = (void *)page;
@@ -3453,6 +3455,22 @@ static void __init gather_bootmem_prealloc(void)
       prep_and_add_bootmem_folios(h, &folio_list);
   }
+static void __init gather_bootmem_prealloc(void)
+{
+    struct padata_mt_job job = {
+        .thread_fn    = __gather_bootmem_prealloc,
+        .fn_arg        = NULL,
+        .start        = 0,
+        .size        = num_node_state(N_MEMORY),
+        .align        = 1,
+        .min_chunk    = 1,
+        .max_threads    = num_node_state(N_MEMORY),
+        .numa_aware    = true,
+    };
+
+    padata_do_multithreaded(&job);
+}
+
   static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
   {
       unsigned long i;
@@ -3602,6 +3620,14 @@ static void __init 
 hugetlb_hstate_alloc_pages(struct hstate *h)
           return;
       }
+    /* hugetlb_hstate_alloc_pages will be called many times, init 
Quoted 1 line

huge_boot_pages once*/

s/init/initialize/g

Quoted 1 line

And you miss a black right before "*/".

OK

Quoted 4 lines
Quoted 2 lines

+    if (huge_boot_pages[0].next == NULL) {

It it not intuitive. I'd like to use a 'initialied' variable

Would it make the code look a bit redundant?

Quoted 3 lines

to indicate whether it has been initialized. BTW, it can be marked as __initdata.

OK

maintainer
#27 lore raw

Re: [PATCH v4 7/7] hugetlb: parallelize 1G hugetlb initialization

Quoted 14 lines · Gang Li
Quoted 9 lines · Muchun Song
Quoted 8 lines · Gang Li

Optimizing the initialization speed of 1G huge pages through parallelization.

1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in gather_bootmem_prealloc.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

Here are some test results:
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------
256c2t(4 node) 1G 4745 2024 57.34%

What does "256c2t" mean?

A machine with 256 core and 2T memory.

A little confusing. I thought 256c2 is a number in hexadecimal. The unit of memory should be capital T. We should add a simple explanation about this.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
   128c1t(2 node) 1G           3358          1712   49.02%
       12t        1G          77000         18300   76.23%

I am curious how many NUMA nodes does this system have? I suspect it should not be one.

Quoted 37 lines
Quoted 28 lines
Quoted 28 lines
---
  include/linux/hugetlb.h |  2 +-
  mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++--------
  2 files changed, 35 insertions(+), 9 deletions(-)
include/linux/hugetlb.h +1 -1
index c1ee640d87b1..77b30a8c6076 100644
@@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
  struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
  extern int sysctl_hugetlb_shm_group;
-extern struct list_head huge_boot_pages;
+extern struct list_head huge_boot_pages[MAX_NUMNODES];
  /* arch callbacks */
mm/hugetlb.c +1 -1
index 9b348ba418f5..2f4b77630ada 100644
@@ -69,7 +69,7 @@ static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
  #endif
  static unsigned long hugetlb_cma_size __initdata;
-__initdata LIST_HEAD(huge_boot_pages);
+__initdata struct list_head huge_boot_pages[MAX_NUMNODES];
  /* for command line parsing */
Quoted 8 lines

mm/hugetlb.c int alloc_bootmem_huge_page(struct hstate *h, int nid) · line 3304

Quoted 7 lines
mm/hugetlb.c +1 -1
  static struct hstate * __initdata parsed_hstate;
@@ -3301,7 +3301,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid)
  int __alloc_bootmem_huge_page(struct hstate *h, int nid)
  {
      struct huge_bootmem_page *m = NULL; /* initialize for clang */
-    int nr_nodes, node;
+    int nr_nodes, node = nid;

Why not use nid directly in the following list_add()?

Quoted 1 line

node may be changed in for_each_node_mask_to_alloc.

Got it.

Quoted 81 lines
Quoted 31 lines
Quoted 31 lines
mm/hugetlb.c +4 -4
      /* do node specific alloc */
      if (nid != NUMA_NO_NODE) {
@@ -3339,7 +3339,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid)
          huge_page_size(h) - PAGE_SIZE);
      /* Put them into a private list first because mem_map is not up yet */
      INIT_LIST_HEAD(&m->list);
-    list_add(&m->list, &huge_boot_pages);
+    list_add(&m->list, &huge_boot_pages[node]);
      m->hstate = h;
      return 1;
  }
@@ -3390,8 +3390,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
      /* Send list for bulk vmemmap optimization processing */
      hugetlb_vmemmap_optimize_folios(h, folio_list);
-    /* Add all new pool pages to free lists in one lock cycle */
-    spin_lock_irqsave(&hugetlb_lock, flags);
      list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
          if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
              /*
@@ -3404,23 +3402,27 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
                      HUGETLB_VMEMMAP_RESERVE_PAGES,
                      pages_per_huge_page(h));
          }
+        /* Subdivide locks to achieve better parallel performance *
+        spin_lock_irqsave(&hugetlb_lock, flags);
          __prep_account_new_huge_page(h, folio_nid(folio));
          enqueue_hugetlb_folio(h, folio);
+        spin_unlock_irqrestore(&hugetlb_lock, flags);
      }
-    spin_unlock_irqrestore(&hugetlb_lock, flags);
Quoted 8 lines

mm/hugetlb.c static void __init prep_and_add_bootmem_folios(struct hstate *h, · line 3417

Quoted 7 lines
mm/hugetlb.c +1 -1
  }
  /*
   * Put bootmem huge pages into the standard lists after mem_map is up.
   * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages.
   */
-static void __init gather_bootmem_prealloc(void)
+static void __init __gather_bootmem_prealloc(unsigned long start, unsigned long end, void *arg)

This function name could be gather_bootmem_prealloc_node.

LGTM.

Quoted 30 lines
Quoted 30 lines
mm/hugetlb.c +19 -1
+
  {
+    int nid = start;
      LIST_HEAD(folio_list);
      struct huge_bootmem_page *m;
      struct hstate *h = NULL, *prev_h = NULL;
-    list_for_each_entry(m, &huge_boot_pages, list) {
+    list_for_each_entry(m, &huge_boot_pages[nid], list) {
          struct page *page = virt_to_page(m);
          struct folio *folio = (void *)page;
@@ -3453,6 +3455,22 @@ static void __init gather_bootmem_prealloc(void)
      prep_and_add_bootmem_folios(h, &folio_list);
  }
+static void __init gather_bootmem_prealloc(void)
+{
+    struct padata_mt_job job = {
+        .thread_fn    = __gather_bootmem_prealloc,
+        .fn_arg        = NULL,
+        .start        = 0,
+        .size        = num_node_state(N_MEMORY),
+        .align        = 1,
+        .min_chunk    = 1,
+        .max_threads    = num_node_state(N_MEMORY),
+        .numa_aware    = true,
+    };
+
+    padata_do_multithreaded(&job);
+}
+
  static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
Quoted 8 lines

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages(struct hstate *h) · line 3622

Quoted 6 lines
mm/hugetlb.c +1 -0
  {
      unsigned long i;
@@ -3602,6 +3620,14 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
          return;
      }
+    /* hugetlb_hstate_alloc_pages will be called many times, init huge_boot_pages once*/

s/init/initialize/g
And you miss a black right before "*/".

Quoted 6 lines

OK

Quoted 2 lines

mm/hugetlb.c static void __init hugetlb_hstate_alloc_pages(struct hstate *h) · line 3624

Quoted 1 line
mm/hugetlb.c +1 -0
+    if (huge_boot_pages[0].next == NULL) {

It it not intuitive. I'd like to use a 'initialied' variable

Would it make the code look a bit redundant?

What is redundant?

Quoted 6 lines
Quoted 4 lines

to indicate whether it has been initialized. BTW, it can be marked as __initdata.

OK

author
#28 lore raw

Re: [PATCH v4 7/7] hugetlb: parallelize 1G hugetlb initialization

On 2024/1/25 10:48, Muchun Song wrote:>>>> 12t 1G
77000 18300 76.23%

Quoted 4 lines

I am curious how many NUMA nodes does this system have? I suspect it should not be one.

Hi David Rientjes,

I'm also curious. Can you show us some details about this machine? Such as lscpu.

Thanks!

author
#29 lore raw

Re: [PATCH v4 7/7] hugetlb: parallelize 1G hugetlb initialization

In reply to Muchun Song
Quoted 7 lines · Muchun Song
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

+ if (huge_boot_pages[0].next == NULL) {

It it not intuitive. I'd like to use a 'initialied' variable

Would it make the code look a bit redundant?

What is redundant?

I was thinking of adding a global variable at first, but it's possible to add a local static variable, which is more concise.