Dynamic huge page pool resizing

PATCH In review Adam Litke 5 participants 12 messages Sep 17, 2007 – Sep 21, 2007

Subsystems (3)

Memory management HugeTLB Proc sysctl

lore

Share & tools

lore

Adam Litke

author

2007-09-17 16:39 UTC #0 lore raw

cover Dynamic huge page pool resizing

*** Series updated to remove locked_vm accounting
The upper bound on pool growth is governed by per-filesystem quotas which maintains the global nature of huge page usage limits. Per process accounting of hugepages as locked memory has been pulled out of this patch series as it is logically separate, and will be pushed separately.
***

In most real-world scenarios, configuring the size of the hugetlb pool correctly is a difficult task. If too few pages are allocated to the pool, applications using MAP_SHARED may fail to mmap() a hugepage region and applications using MAP_PRIVATE may receive SIGBUS. Isolating too much memory in the hugetlb pool means it is not available for other uses, especially those programs not using huge pages.

The obvious answer is to let the hugetlb pool grow and shrink in response to the runtime demand for huge pages. The work Mel Gorman has been doing to establish a memory zone for movable memory allocations makes dynamically resizing the hugetlb pool reliable within the limits of that zone. This patch series implements dynamic pool resizing for private and shared mappings while being careful to maintain existing semantics. Please reply with your comments and feedback; even just to say whether it would be a useful feature to you. Thanks.

How it works
============

Upon depletion of the hugetlb pool, rather than reporting an error immediately, first try and allocate the needed huge pages directly from the buddy allocator. Care must be taken to avoid unbounded growth of the hugetlb pool, so the hugetlb filesystem quota is used to limit overall pool size.

The real work begins when we decide there is a shortage of huge pages. What happens next depends on whether the pages are for a private or shared mapping. Private mappings are straightforward. At fault time, if alloc_huge_page() fails, we allocate a page from the buddy allocator and increment the source node's surplus_huge_pages counter. When free_huge_page() is called for a page on a node with a surplus, the page is freed directly to the buddy allocator instead of the hugetlb pool.

Because shared mappings require all of the pages to be reserved up front, some additional work must be done at mmap() to support them. We determine the reservation shortage and allocate the required number of pages all at once. These pages are then added to the hugetlb pool and marked reserved. Where that is not possible the mmap() will fail. As with private mappings, the appropriate surplus counters are updated. Since reserved huge pages won't necessarily be used by the process, we can't be sure that free_huge_page() will always be called to return surplus pages to the buddy allocator. To prevent the huge page pool from bloating, we must free unused surplus pages when their reservation has ended.

Controlling it
==============

With the entire patch series applied, pool resizing is off by default so unless specific action is taken, the semantics are unchanged.

To take advantage of the flexibility afforded by this patch series one must tolerate a change in semantics. To control hugetlb pool growth, the following techniques can be employed:

* A sysctl tunable to enable/disable the feature entirely
* The size= mount option for hugetlbfs filesystems to limit pool size

Future Improvements
===================

I am aware of the following issues and plan to address then in the future as separate changes since they are not critical and to keep this patch series as small as possible.

* Modifying the pool size (via sysctl) while there is a pool surplus could be made smarter. Right now the surplus is ignored which can result in unneeded alloc_fresh_huge_page() calls.

Changelog
=========

9/17/2007 - Version 3
- fixed gather_surplus_pages 'needed' check
- Removed 'locked_vm' changes from this series since that is really a separate logical change

9/12/2007 - Version 2
- Integrated the cpuset_mems_nr() best-effort reservation check
- Surplus pool allocations respect cpuset and numa policy restrictions
- Unused surplus pages that are part of a reservation are freed when the reservation is released

7/13/2007 - Initial Post

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Adam Litke

author

2007-09-17 16:39 UTC #1 lore raw

PATCH 1/4 hugetlb: Move update_and_free_page

This patch simply moves update_and_free_page() so that it can be reused later in this patch series. The implementation is not changed.

Signed-off-by Adam Litke <[email protected]>

Acked-by Andy Whitcroft <[email protected]>

---

 mm/hugetlb.c |   30 +++++++++++++++---------------
 1 files changed, 15 insertions(+), 15 deletions(-)

mm/hugetlb.c +15 ~~-15~~

index de4cf45..eb5b9f4 100644
@@ -90,6 +90,21 @@ static struct page *dequeue_huge_page(struct vm_area_struct *vma,
 	return page;
 }
 
+static void update_and_free_page(struct page *page)
+{
+	int i;
+	nr_huge_pages--;
+	nr_huge_pages_node[page_to_nid(page)]--;
+	for (i = 0; i < (HPAGE_SIZE / PAGE_SIZE); i++) {
+		page[i].flags &= ~(1 << PG_locked | 1 << PG_error | 1 << PG_referenced |
+				1 << PG_dirty | 1 << PG_active | 1 << PG_reserved |
+				1 << PG_private | 1<< PG_writeback);
+	}
+	set_compound_page_dtor(page, NULL);
+	set_page_refcounted(page);
+	__free_pages(page, HUGETLB_PAGE_ORDER);
+}
+
 static void free_huge_page(struct page *page)
 {
 	BUG_ON(page_count(page));

@@ -199,21 +214,6 @@ static unsigned int cpuset_mems_nr(unsigned int *array)
 }
 
 #ifdef CONFIG_SYSCTL
-static void update_and_free_page(struct page *page)
-{
-	int i;
-	nr_huge_pages--;
-	nr_huge_pages_node[page_to_nid(page)]--;
-	for (i = 0; i < (HPAGE_SIZE / PAGE_SIZE); i++) {
-		page[i].flags &= ~(1 << PG_locked | 1 << PG_error | 1 << PG_referenced |
-				1 << PG_dirty | 1 << PG_active | 1 << PG_reserved |
-				1 << PG_private | 1<< PG_writeback);
-	}
-	set_compound_page_dtor(page, NULL);
-	set_page_refcounted(page);
-	__free_pages(page, HUGETLB_PAGE_ORDER);
-}
-
 #ifdef CONFIG_HIGHMEM
 static void try_to_free_low(unsigned long count)
 {

Adam Litke

author

2007-09-17 16:40 UTC #2 lore raw

PATCH 2/4 hugetlb: Try to grow hugetlb pool for MAP_PRIVATE mappings

In reply to Adam Litke

Because we overcommit hugepages for MAP_PRIVATE mappings, it is possible that the hugetlb pool will be exhausted or completely reserved when a hugepage is needed to satisfy a page fault. Before killing the process in this situation, try to allocate a hugepage directly from the buddy allocator.

The explicitly configured pool size becomes a low watermark. When dynamically grown, the allocated huge pages are accounted as a surplus over the watermark. As huge pages are freed on a node, surplus pages are released to the buddy allocator so that the pool will shrink back to the watermark.

Signed-off-by Adam Litke <[email protected]>

Signed-off-by Mel Gorman <[email protected]>

Acked-by Andy Whitcroft <[email protected]>

---

 mm/hugetlb.c |   43 +++++++++++++++++++++++++++++++++++++++----
 1 files changed, 39 insertions(+), 4 deletions(-)

mm/hugetlb.c +39 -4

index eb5b9f4..63abd31 100644
@@ -27,6 +27,7 @@ unsigned long max_huge_pages;
 static struct list_head hugepage_freelists[MAX_NUMNODES];
 static unsigned int nr_huge_pages_node[MAX_NUMNODES];
 static unsigned int free_huge_pages_node[MAX_NUMNODES];
+static unsigned int surplus_huge_pages_node[MAX_NUMNODES];
 static gfp_t htlb_alloc_mask = GFP_HIGHUSER;
 unsigned long hugepages_treat_as_movable;

@@ -107,12 +108,18 @@ static void update_and_free_page(struct page *page)
 
 static void free_huge_page(struct page *page)
 {
-	BUG_ON(page_count(page));
+	int nid = page_to_nid(page);
 
+	BUG_ON(page_count(page));
 	INIT_LIST_HEAD(&page->lru);
 
 	spin_lock(&hugetlb_lock);
-	enqueue_huge_page(page);
+	if (surplus_huge_pages_node[nid]) {
+		update_and_free_page(page);
+		surplus_huge_pages_node[nid]--;
+	} else {
+		enqueue_huge_page(page);
+	}
 	spin_unlock(&hugetlb_lock);
 }

@@ -148,10 +155,29 @@ static int alloc_fresh_huge_page(void)
 	return 0;
 }
 
+static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma,
+						unsigned long address)
+{
+	struct page *page;
+
+	page = alloc_pages(htlb_alloc_mask|__GFP_COMP|__GFP_NOWARN,
+					HUGETLB_PAGE_ORDER);
+	if (page) {
+		set_compound_page_dtor(page, free_huge_page);
+		spin_lock(&hugetlb_lock);
+		nr_huge_pages++;
+		nr_huge_pages_node[page_to_nid(page)]++;
+		surplus_huge_pages_node[page_to_nid(page)]++;
+		spin_unlock(&hugetlb_lock);
+	}
+
+	return page;
+}
+
 static struct page *alloc_huge_page(struct vm_area_struct *vma,
 				    unsigned long addr)
 {
-	struct page *page;
+	struct page *page = NULL;
 
 	spin_lock(&hugetlb_lock);
 	if (vma->vm_flags & VM_MAYSHARE)

@@ -171,7 +197,16 @@ fail:
 	if (vma->vm_flags & VM_MAYSHARE)
 		resv_huge_pages++;
 	spin_unlock(&hugetlb_lock);
-	return NULL;
+
+	/*
+	 * Private mappings do not use reserved huge pages so the allocation
+	 * may have failed due to an undersized hugetlb pool.  Try to grab a
+	 * surplus huge page from the buddy allocator.
+	 */
+	if (!(vma->vm_flags & VM_MAYSHARE))
+		page = alloc_buddy_huge_page(vma, addr);
+
+	return page;
 }
 
 static int __init hugetlb_init(void)

Adam Litke

author

2007-09-17 16:40 UTC #3 lore raw

PATCH 3/4 hugetlb: Try to grow hugetlb pool for MAP_SHARED mappings

In reply to Adam Litke

Shared mappings require special handling because the huge pages needed to fully populate the VMA must be reserved at mmap time. If not enough pages are available when making the reservation, allocate all of the shortfall at once from the buddy allocator and add the pages directly to the hugetlb pool. If they cannot be allocated, then fail the mapping. The page surplus is accounted for in the same way as for private mappings; faulted surplus pages will be freed at unmap time. Reserved, surplus pages that have not been used must be freed separately when their reservation has been released.

Signed-off-by Adam Litke <[email protected]>

Acked-by Andy Whitcroft <[email protected]>

---

 mm/hugetlb.c |  161 ++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 files changed, 138 insertions(+), 23 deletions(-)

mm/hugetlb.c +138 ~~-23~~

index 63abd31..1e78574 100644
@@ -28,6 +28,7 @@ static struct list_head hugepage_freelists[MAX_NUMNODES];
 static unsigned int nr_huge_pages_node[MAX_NUMNODES];
 static unsigned int free_huge_pages_node[MAX_NUMNODES];
 static unsigned int surplus_huge_pages_node[MAX_NUMNODES];
+static unsigned long unused_surplus_pages;
 static gfp_t htlb_alloc_mask = GFP_HIGHUSER;
 unsigned long hugepages_treat_as_movable;

@@ -85,6 +86,10 @@ static struct page *dequeue_huge_page(struct vm_area_struct *vma,
 			list_del(&page->lru);
 			free_huge_pages--;
 			free_huge_pages_node[nid]--;
+			if (vma && vma->vm_flags & VM_MAYSHARE) {
+				resv_huge_pages--;
+				unused_surplus_pages--;
+			}
 			break;
 		}
 	}

@@ -174,15 +179,120 @@ static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma,
 	return page;
 }
 
+/*
+ * Increase the hugetlb pool such that it can accomodate a reservation
+ * of size 'delta'.
+ */
+static int gather_surplus_pages(int delta)
+{
+	struct list_head surplus_list;
+	struct page *page, *tmp;
+	int ret, i;
+	int needed, allocated;
+
+	needed = (resv_huge_pages + delta) - free_huge_pages;
+	if (needed <= 0)
+		return 0;
+
+	allocated = 0;
+	INIT_LIST_HEAD(&surplus_list);
+
+	ret = -ENOMEM;
+retry:
+	spin_unlock(&hugetlb_lock);
+	for (i = 0; i < needed; i++) {
+		page = alloc_buddy_huge_page(NULL, 0);
+		if (!page) {
+			/*
+			 * We were not able to allocate enough pages to
+			 * satisfy the entire reservation so we free what
+			 * we've allocated so far.
+			 */
+			spin_lock(&hugetlb_lock);
+			needed = 0;
+			goto free;
+		}
+
+		list_add(&page->lru, &surplus_list);
+	}
+	allocated += needed;
+
+	/*
+	 * After retaking hugetlb_lock, we need to recalculate 'needed'
+	 * because either resv_huge_pages or free_huge_pages may have changed.
+	 */
+	spin_lock(&hugetlb_lock);
+	needed = (resv_huge_pages + delta) - (free_huge_pages + allocated);
+	if (needed > 0)
+		goto retry;
+
+	/*
+	 * The surplus_list now contains _at_least_ the number of extra pages
+	 * needed to accomodate the reservation.  Add the appropriate number
+	 * of pages to the hugetlb pool and free the extras back to the buddy
+	 * allocator.
+	 *
+	 * Those pages that get added to the pool may never be allocated and
+	 * subsequently freed so keep track of them in unused_surplus_pages
+	 * so they can be freed again when a reservation is released.
+	 */
+	needed += allocated;
+	unused_surplus_pages += needed;
+	ret = 0;
+free:
+	list_for_each_entry_safe(page, tmp, &surplus_list, lru) {
+		list_del(&page->lru);
+		if ((--needed) >= 0)
+			enqueue_huge_page(page);
+		else
+			update_and_free_page(page);
+	}
+
+	return ret;
+}
+
+/*
+ * When releasing a reservation, free all unused, surplus huge pages that are
+ * no longer reserved.
+ */
+void return_unused_surplus_pages(void)
+{
+	static int nid = -1;
+	int delta;
+	struct page *page;
+
+	delta = unused_surplus_pages - resv_huge_pages;
+
+	while (delta) {
+		nid = next_node(nid, node_online_map);
+		if (nid == MAX_NUMNODES)
+			nid = first_node(node_online_map);
+
+		if (!surplus_huge_pages_node[nid])
+			continue;
+
+		if (!list_empty(&hugepage_freelists[nid])) {
+			page = list_entry(hugepage_freelists[nid].next,
+					  struct page, lru);
+			list_del(&page->lru);
+			update_and_free_page(page);
+			free_huge_pages--;
+			free_huge_pages_node[nid]--;
+			surplus_huge_pages_node[nid]--;
+			unused_surplus_pages--;
+			delta--;
+		}
+	}
+}
+
 static struct page *alloc_huge_page(struct vm_area_struct *vma,
 				    unsigned long addr)
 {
 	struct page *page = NULL;
+	int use_reserved_page = vma->vm_flags & VM_MAYSHARE;
 
 	spin_lock(&hugetlb_lock);
-	if (vma->vm_flags & VM_MAYSHARE)
-		resv_huge_pages--;
-	else if (free_huge_pages <= resv_huge_pages)
+	if (!use_reserved_page && (free_huge_pages <= resv_huge_pages))
 		goto fail;
 
 	page = dequeue_huge_page(vma, addr);

@@ -194,8 +304,6 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
 	return page;
 
 fail:
-	if (vma->vm_flags & VM_MAYSHARE)
-		resv_huge_pages++;
 	spin_unlock(&hugetlb_lock);
 
 	/*

@@ -203,7 +311,7 @@ fail:
 	 * may have failed due to an undersized hugetlb pool.  Try to grab a
 	 * surplus huge page from the buddy allocator.
 	 */
-	if (!(vma->vm_flags & VM_MAYSHARE))
+	if (!use_reserved_page)
 		page = alloc_buddy_huge_page(vma, addr);
 
 	return page;

@@ -876,21 +984,6 @@ static int hugetlb_acct_memory(long delta)
 	int ret = -ENOMEM;
 
 	spin_lock(&hugetlb_lock);
-	if ((delta + resv_huge_pages) <= free_huge_pages) {
-		resv_huge_pages += delta;
-		ret = 0;
-	}
-	spin_unlock(&hugetlb_lock);
-	return ret;
-}
-
-int hugetlb_reserve_pages(struct inode *inode, long from, long to)
-{
-	long ret, chg;
-
-	chg = region_chg(&inode->i_mapping->private_list, from, to);
-	if (chg < 0)
-		return chg;
 	/*
 	 * When cpuset is configured, it breaks the strict hugetlb page
 	 * reservation as the accounting is done on a global variable. Such

@@ -908,8 +1001,30 @@ int hugetlb_reserve_pages(struct inode *inode, long from, long to)
 	 * a best attempt and hopefully to minimize the impact of changing
 	 * semantics that cpuset has.
 	 */
-	if (chg > cpuset_mems_nr(free_huge_pages_node))
-		return -ENOMEM;
+	if (delta > 0) {
+		if (gather_surplus_pages(delta) < 0)
+			goto out;
+
+		if (delta > cpuset_mems_nr(free_huge_pages_node))
+			goto out;
+	}
+
+	ret = 0;
+	resv_huge_pages += delta;
+	if (delta <= 0)
+		return_unused_surplus_pages();
+out:
+	spin_unlock(&hugetlb_lock);
+	return ret;
+}
+
+int hugetlb_reserve_pages(struct inode *inode, long from, long to)
+{
+	long ret, chg;
+
+	chg = region_chg(&inode->i_mapping->private_list, from, to);
+	if (chg < 0)
+		return chg;
 
 	ret = hugetlb_acct_memory(chg);
 	if (ret < 0)

Mika Penttilä

2007-09-17 17:22 UTC #4 lore raw

Re: [PATCH 3/4] hugetlb: Try to grow hugetlb pool for MAP_SHARED mappings

mm/hugetlb.c static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma, · line 266

Quoted 10 lines

mm/hugetlb.c +9 -0

+void return_unused_surplus_pages(void)
+{
+	static int nid = -1;
+	int delta;
+	struct page *page;
+
+	delta = unused_surplus_pages - resv_huge_pages;
+
+	while (delta) {

Shouldn't this be while (delta >= 0) ?

Quoted 11 lines

mm/hugetlb.c +11 -0

+		nid = next_node(nid, node_online_map);
+		if (nid == MAX_NUMNODES)
+			nid = first_node(node_online_map);
+
+		if (!surplus_huge_pages_node[nid])
+			continue;
+
+		if (!list_empty(&hugepage_freelists[nid])) {
+			page = list_entry(hugepage_freelists[nid].next,
+					  struct page, lru);
+			list_del(&page->lru);

mm/hugetlb.c static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma, · line 287

Quoted 10 lines

mm/hugetlb.c +10 -0

+			update_and_free_page(page);
+			free_huge_pages--;
+			free_huge_pages_node[nid]--;
+			surplus_huge_pages_node[nid]--;
+			unused_surplus_pages--;
+			delta--;
+		}
+	}
+}
+

--Mika

Adam Litke

author

2007-09-17 17:42 UTC #5 lore raw

Re: [PATCH 3/4] hugetlb: Try to grow hugetlb pool for MAP_SHARED mappings

Quoted 11 lines · Mika Penttila

mm/hugetlb.c static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma, · line 266
Quoted 10 lines
mm/hugetlb.c +9 -0
+void return_unused_surplus_pages(void)
+{
+	static int nid = -1;
+	int delta;
+	struct page *page;
+
+	delta = unused_surplus_pages - resv_huge_pages;
+
+	while (delta) {
Shouldn't this be while (delta >= 0) ?

unused_surplus_pages is always >= resv_huge_pages so delta cannot go negative. But for clarity it makes sense to apply the change you suggest. Thanks for responding.

Quoted 25 lines

Quoted 11 lines

mm/hugetlb.c +11 -0

+		nid = next_node(nid, node_online_map);
+		if (nid == MAX_NUMNODES)
+			nid = first_node(node_online_map);
+
+		if (!surplus_huge_pages_node[nid])
+			continue;
+
+		if (!list_empty(&hugepage_freelists[nid])) {
+			page = list_entry(hugepage_freelists[nid].next,
+					  struct page, lru);
+			list_del(&page->lru);

mm/hugetlb.c static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma, · line 287

Quoted 10 lines

mm/hugetlb.c +10 -0

+			update_and_free_page(page);
+			free_huge_pages--;
+			free_huge_pages_node[nid]--;
+			surplus_huge_pages_node[nid]--;
+			unused_surplus_pages--;
+			delta--;
+		}
+	}
+}
+

--Mika

Signature · 8 lines

Adam Litke - (agl at us.ibm.com)
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Mika Penttilä

2007-09-17 18:42 UTC #6 lore raw

Re: [PATCH 3/4] hugetlb: Try to grow hugetlb pool for MAP_SHARED mappings

Quoted 16 lines · Adam Litke

Quoted 4 lines · Mika Penttila
Quoted 4 lines
mm/hugetlb.c +4 -0
+void return_unused_surplus_pages(void)
+{
+	static int nid = -1;
+	int delta;
Quoted 9 lines
mm/hugetlb.c static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma, · line 266
Quoted 7 lines
mm/hugetlb.c +5 -0
+	struct page *page;
+
+	delta = unused_surplus_pages - resv_huge_pages;
+
+	while (delta) {
Shouldn't this be while (delta >= 0) ?

Quoted 5 lines

unused_surplus_pages is always >= resv_huge_pages so delta cannot go negative. But for clarity it makes sense to apply the change you suggest. Thanks for responding.

I think unused_surplus_pages accounting isn't quite right. It gets always decremented in dequeue_huge_page() but incremented only if we haven't enough free pages at reserve time.

--Mika

Adam Litke

author

2007-09-17 20:07 UTC #7 lore raw

Re: [PATCH 3/4] hugetlb: Try to grow hugetlb pool for MAP_SHARED mappings

Quoted 17 lines · Mika Penttila

Quoted 16 lines · Adam Litke
Quoted 13 lines · Mika Penttila
mm/hugetlb.c static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma, · line 266
Quoted 11 lines
mm/hugetlb.c +9 -0
+void return_unused_surplus_pages(void)
+{
+	static int nid = -1;
+	int delta;
+	struct page *page;
+
+	delta = unused_surplus_pages - resv_huge_pages;
+
+	while (delta) {
Shouldn't this be while (delta >= 0) ?

Quoted 8 lines

Quoted 5 lines
unused_surplus_pages is always >= resv_huge_pages so delta cannot go negative. But for clarity it makes sense to apply the change you suggest. Thanks for responding.
I think unused_surplus_pages accounting isn't quite right. It gets always decremented in dequeue_huge_page() but incremented only if we haven't enough free pages at reserve time.

Ahh yes, good catch. The solution is to only decrement unused_surplus_pages in dequeue_huge_page() until it becomes zero. Once that condition is true we know that return_unused_surplus_pages() will have no work to do. Thank you for your careful review. I am now testing this case specifically.

Signature · 8 lines

Adam Litke - (agl at us.ibm.com)
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Adam Litke

author

2007-09-17 16:41 UTC #8 lore raw

PATCH 4/4 hugetlb: Add hugetlb_dynamic_pool sysctl

In reply to Adam Litke

Allowing the hugetlb pool to grow dynamically changes the semantics of the system by permitting more system memory to be used for huge pages than has been explicitly dedicated to the pool.

This patch introduces a sysctl which must be enabled to turn on the dynamic pool resizing feature. This will avoid an involuntary change in behavior.

When hugetlb pool growth is enabled via the hugetlb_dynamic_pool sysctl, an upper-bound on huge page allocation can be set by constraining the size of the hugetlb filesystem via the 'size' mount option.

Signed-off-by Adam Litke <[email protected]>

Acked-by Andy Whitcroft <[email protected]>

---

 include/linux/hugetlb.h |    1 +
 kernel/sysctl.c         |    8 ++++++++
 mm/hugetlb.c            |    5 +++++
 3 files changed, 14 insertions(+), 0 deletions(-)

include/linux/hugetlb.h +1 -0

index 3a19b03..ea0f50b 100644
@@ -33,6 +33,7 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed);
 
 extern unsigned long max_huge_pages;
 extern unsigned long hugepages_treat_as_movable;
+extern int hugetlb_dynamic_pool;
 extern const unsigned long hugetlb_zero, hugetlb_infinity;
 extern int sysctl_hugetlb_shm_group;

kernel/sysctl.c +8 -0

index 6ace893..45e8045 100644
@@ -889,6 +889,14 @@ static ctl_table vm_table[] = {
 		.mode		= 0644,
 		.proc_handler	= &hugetlb_treat_movable_handler,
 	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "hugetlb_dynamic_pool",
+		.data		= &hugetlb_dynamic_pool,
+		.maxlen		= sizeof(hugetlb_dynamic_pool),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
 #endif
 	{
 		.ctl_name	= VM_LOWMEM_RESERVE_RATIO,

mm/hugetlb.c +5 -0

index 1e78574..cd8f710 100644
@@ -31,6 +31,7 @@ static unsigned int surplus_huge_pages_node[MAX_NUMNODES];
 static unsigned long unused_surplus_pages;
 static gfp_t htlb_alloc_mask = GFP_HIGHUSER;
 unsigned long hugepages_treat_as_movable;
+int hugetlb_dynamic_pool;
 
 /*
  * Protects updates to hugepage_freelists, nr_huge_pages, and free_huge_pages

@@ -165,6 +166,10 @@ static struct page *alloc_buddy_huge_page(struct vm_area_struct *vma,
 {
 	struct page *page;
 
+	/* Check if the dynamic pool is enabled */
+	if (!hugetlb_dynamic_pool)
+		return NULL;
+
 	page = alloc_pages(htlb_alloc_mask|__GFP_COMP|__GFP_NOWARN,
 					HUGETLB_PAGE_ORDER);
 	if (page) {

Dave McCracken

Acked-by on the v1 series

2007-09-17 17:37 UTC #9 lore raw

In reply to Adam Litke

Quoted 7 lines · Adam Litke

In most real-world scenarios, configuring the size of the hugetlb pool correctly is a difficult task. A If too few pages are allocated to the pool, applications using MAP_SHARED may fail to mmap() a hugepage region and applications using MAP_PRIVATE may receive SIGBUS. A Isolating too much memory in the hugetlb pool means it is not available for other uses, especially those programs not using huge pages.

Quoted 8 lines

The obvious answer is to let the hugetlb pool grow and shrink in response to the runtime demand for huge pages. A The work Mel Gorman has been doing to establish a memory zone for movable memory allocations makes dynamically resizing the hugetlb pool reliable within the limits of that zone. A This patch series implements dynamic pool resizing for private and shared mappings while being careful to maintain existing semantics. A Please reply with your comments and feedback; even just to say whether it would be a useful feature to you. Thanks.

Now that we have Mel's mobility patches to make it feasible to dynamically allocate huge pages, I'd say it's definitely time to get this patch in. Users will really appreciate not having to preallocate huge pages for all possible workloads.

Dave

Acked-by Dave McCracken <[email protected]>

Andrew Hastings

2007-09-17 18:07 UTC #10 lore raw

In reply to Adam Litke

Quoted 7 lines · Adam Litke

The obvious answer is to let the hugetlb pool grow and shrink in response to the runtime demand for huge pages. The work Mel Gorman has been doing to establish a memory zone for movable memory allocations makes dynamically resizing the hugetlb pool reliable within the limits of that zone. This patch series implements dynamic pool resizing for private and shared mappings while being careful to maintain existing semantics. Please reply with your comments and feedback; even just to say whether it would be a useful feature to you.

Thanks, this will be extremely useful for our customers' workloads.

-Andrew Hastings
Cray Inc.

Avi Kivity

2007-09-21 05:16 UTC #11 lore raw

In reply to Adam Litke

Quoted 14 lines · Adam Litke

*** Series updated to remove locked_vm accounting
The upper bound on pool growth is governed by per-filesystem quotas which maintains the global nature of huge page usage limits. Per process accounting of hugepages as locked memory has been pulled out of this patch series as it is logically separate, and will be pushed separately.
***
In most real-world scenarios, configuring the size of the hugetlb pool correctly is a difficult task. If too few pages are allocated to the pool, applications using MAP_SHARED may fail to mmap() a hugepage region and applications using MAP_PRIVATE may receive SIGBUS. Isolating too much memory in the hugetlb pool means it is not available for other uses, especially those programs not using huge pages.

Quoted 10 lines

The obvious answer is to let the hugetlb pool grow and shrink in response to the runtime demand for huge pages. The work Mel Gorman has been doing to establish a memory zone for movable memory allocations makes dynamically resizing the hugetlb pool reliable within the limits of that zone. This patch series implements dynamic pool resizing for private and shared mappings while being careful to maintain existing semantics. Please reply with your comments and feedback; even just to say whether it would be a useful feature to you. Thanks.

kvm with newer hardware (that supports nested paging) stands to benefit greatly from this. backing guest memory with huge pages significantly decreases tlb miss costs, and allows using huge tlb entries for guest kernel mappings.