hugepage allocator cleanup
Subsystems (2)
PATCH 1/1 hugepage allocator cleanup
Hi,
This is a slight rework of the mechanism for allocating "fresh" hugepages. Comments?
Thanks,
Nick
--
Insert "fresh" huge pages into the hugepage allocator by the same means as they are freed back into it. This reduces code size and allows enqueue_huge_page to be inlined into the hugepage free fastpath.
Eliminate occurances of hugepages on the free list with non-zero refcount. This can allow stricter refcount checks in future. Also required for lockless pagecache.
mm/hugetlb.c +8 -16
@@ -64,7 +64,7 @@ static struct page *dequeue_huge_page(streturn page; } -static struct page *alloc_fresh_huge_page (void ) +static int alloc_fresh_huge_page (void ) {static int nid =0 ;struct page *page;
@@ -72,12 +72,15 @@ static struct page *alloc_fresh_huge_pag HUGETLB_PAGE_ORDER); nid = (nid +1 ) %num_online_nodes ();if (page) { + page[1 ].mapping = (void *)free_huge_page;spin_lock (&hugetlb_lock); nr_huge_pages++; nr_huge_pages_node[page_to_nid (page)]++;spin_unlock (&hugetlb_lock); +put_page (page);/* free it into the hugepage allocator */ +return 1 ; } -return page; +return 0 ; }void free_huge_page (struct page *page)
@@ -85,7 +88,6 @@ void free_huge_page(struct page *page)BUG_ON (page_count (page));INIT_LIST_HEAD (&page->lru); - page[1 ].mapping =NULL ;spin_lock (&hugetlb_lock);enqueue_huge_page (page);
@@ -105,7 +107,6 @@ struct page *alloc_huge_page(struct vm_a }spin_unlock (&hugetlb_lock);set_page_count (page,1 ); - page[1 ].mapping = (void *)free_huge_page;for (i =0 ; i < (HPAGE_SIZE /PAGE_SIZE ); ++i)clear_highpage (&page[i]);return page;
@@ -114,7 +115,6 @@ struct page *alloc_huge_page(struct vm_astatic int __init hugetlb_init (void ) {unsigned long i; -struct page *page;if (HPAGE_SHIFT ==0 )return 0 ;
@@ -123,12 +123,8 @@ static int __init hugetlb_init(void)INIT_LIST_HEAD (&hugepage_freelists[i]);for (i =0 ; i < max_huge_pages; ++i) { - page =alloc_fresh_huge_page (); -if (!page) +if (!alloc_fresh_huge_page ())break ; -spin_lock (&hugetlb_lock); -enqueue_huge_page (page); -spin_unlock (&hugetlb_lock); } max_huge_pages = free_huge_pages = nr_huge_pages = i;printk ("Total HugeTLB memory allocated, %ld\n" , free_huge_pages);
@@ -154,8 +150,8 @@ static void update_and_free_page(struct page[i].flags &= ~(1 << PG_locked |1 << PG_error |1 << PG_referenced |1 << PG_dirty |1 << PG_active |1 << PG_reserved |1 << PG_private |1 << PG_writeback); -set_page_count (&page[i],0 ); } + page[1 ].mapping =NULL ;set_page_count (page,1 );__free_pages (page,HUGETLB_PAGE_ORDER ); }
@@ -188,12 +184,8 @@ static inline void try_to_free_low(unsigstatic unsigned long set_max_huge_pages (unsigned long count) {while (count > nr_huge_pages) { -struct page *page =alloc_fresh_huge_page (); -if (!page) +if (!alloc_fresh_huge_page ())return nr_huge_pages; -spin_lock (&hugetlb_lock); -enqueue_huge_page (page); -spin_unlock (&hugetlb_lock); }if (count >= nr_huge_pages)return nr_huge_pages;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Nick Piggin
This is a slight rework of the mechanism for allocating "fresh" hugepages. Comments?
--
Insert "fresh" huge pages into the hugepage allocator by the same means as they are freed back into it. This reduces code size and allowsenqueue_huge_pageto be inlined into the hugepage free fastpath.
Eliminate occurances of hugepages on the free list with non-zero refcount. This can allow stricter refcount checks in future. Also required for lockless pagecache.
I don't really see any particular benefit to the rearrangement for hugetlb's own sake. Explaining more about how it it's needed for the lockless pagecache might help.
-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines · William Lee Irwin III
Quoted 10 lines · Nick Piggin
This is a slight rework of the mechanism for allocating "fresh" hugepages. Comments?
--
Insert "fresh" huge pages into the hugepage allocator by the same means as they are freed back into it. This reduces code size and allowsenqueue_huge_pageto be inlined into the hugepage free fastpath.
Eliminate occurances of hugepages on the free list with non-zero refcount. This can allow stricter refcount checks in future. Also required for lockless pagecache.
Quoted 2 lines
I don't really see any particular benefit to the rearrangement for hugetlb's own sake.
I like the fact that freelist pages always have a zero refcount now. And I thought it was quite neat to set up the page for the new allocator then free straight into it... but you're right, there is no *particular* benefit ;)
Quoted 3 lines
Explaining more about how it it's needed for the
lockless pagecache might help.
OK. Though obviously I don't want to introduce any ugliness or hackery to core code just for lockless pagecache (which may not ever go in itself).
Basically with lockless pagecache it becomes possible that any page taken out of the page allocator may have its refcount raised by another thread.
That other thread is looking for a pagecache page, if it takes a reused one, it will quickly drop the refcount again.
So it is important not to muddle the count. A 1->0 transition (as currently when allocating fresh pages for the first time) needs to be done with a dec-and-test rather than a plain set.
Thanks,
Nick
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Nick Piggin
OK. Though obviously I don't want to introduce any ugliness or hackery to core code just for lockless pagecache (which may not ever go in itself).
Basically with lockless pagecache it becomes possible that any page taken out of the page allocator may have its refcount raised by another thread.
That other thread is looking for a pagecache page, if it takes a reused one, it will quickly drop the refcount again. So it is important not to muddle the count. A 1->0 transition (as currently when allocating fresh pages for the first time) needs to be done with a dec-and-test rather than a plain set.
Preparatory cleanups are fine by me barring where things get to the point of churn, which isn't a concern here.
It appears the crucial component of this update_and_free_page(). It shouldn't be necessary as disciplined page->_count references are redirected to the head of the hugepage, but it's trying to clean up the page->_counts in tail pages of the hugepage in preparation for freeing. Arguably 1->0 transition logic shouldn't be triggered, but the locking protocol envisioned may not allow unconditionally setting page->_count.
Just yanking the page refcount affairs out of update_and_free_page() should suffice. Could I get things trimmed down to that?
-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · William Lee Irwin III
Preparatory cleanups are fine by me barring where things get to the point of churn, which isn't a concern here.
Cool.
Quoted 4 lines
It appears the crucial component of this
update_and_free_page(). It shouldn't be necessary as disciplinedpage->_countreferences are redirected to the head of the hugepage, but it's trying to clean up thepage->_countsin tail pages of the hugepage in preparation for freeing.
Yes, that is the crucial part.
Quoted 3 lines
Arguably 1->0 transition logic shouldn't be triggered, but the locking protocol envisioned may not allow unconditionally setting
page->_count.
Unfortunately yes I wasn't clear I guess. Any page with a nonzero refcount must not be unconditionally set.
Quoted 3 lines
Just yanking the page refcount affairs out of
update_and_free_page()should suffice. Could I get things trimmed down to that?
I could remove the first set_page_count, and make the second conditional on the page having a zero refcount... for a 3-liner. But that's kind of ugly (if less intrusive), and it is adds seemingly nonsense code if one doesn't have the context of my out-of-tree patches.
Hmm... it's obviously not 2.6.16 material so there is no rush to think it over. It is even simple enough that I don't mind carrying with my patchset indefinitely.
Thanks,
Nick
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines
Just yanking the page refcount affairs out of
update_and_free_page()should suffice. Could I get things trimmed down to that?
Quoted 7 lines · Nick Piggin
I could remove the first
set_page_count, and make the second conditional on the page having a zero refcount... for a 3-liner. But that's kind of ugly (if less intrusive), and it is adds seemingly nonsense code if one doesn't have the context of my out-of-tree patches.
Hmm... it's obviously not 2.6.16 material so there is no rush to think it over. It is even simple enough that I don't mind carrying with my patchset indefinitely.
After I thought about it, alloc_fresh_huge_page() does enqueue pages with refcounts of 1, where free_huge_page() (called from the freeing hook in page[1].mapping) enqueues pages with refcounts of 0, so it would actually make sense (and possibly prevent leaks) to take the whole patch as-is.
-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · William Lee Irwin III
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines
Just yanking the page refcount affairs out of
update_and_free_page()should suffice. Could I get things trimmed down to that?Quoted 7 lines · Nick Piggin
I could remove the first
set_page_count, and make the second conditional on the page having a zero refcount... for a 3-liner. But that's kind of ugly (if less intrusive), and it is adds seemingly nonsense code if one doesn't have the context of my out-of-tree patches.
Hmm... it's obviously not 2.6.16 material so there is no rush to think it over. It is even simple enough that I don't mind carrying with my patchset indefinitely.
Quoted 3 lines
After I thought about it,
alloc_fresh_huge_page()does enqueue pages with refcounts of 1, wherefree_huge_page()(called from the freeing hook in page[1].mapping) enqueues pages with refcounts of 0, so it
Yep.
Quoted 4 lines
would actually make sense (and possibly prevent leaks) to take the whole patch as-is.
Yeah, you could add a bad_page-like check to verify the refcount hasn't been mucked with... and of course the regular allocator can now verify refcounts are alright when update_and_free_page returns them.
Nick
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Nick Piggin
This is a slight rework of the mechanism for allocating "fresh" hugepages. Comments?
--
Insert "fresh" huge pages into the hugepage allocator by the same means as they are freed back into it. This reduces code size and allowsenqueue_huge_pageto be inlined into the hugepage free fastpath.
Eliminate occurances of hugepages on the free list with non-zero refcount. This can allow stricter refcount checks in future. Also required for lockless pagecache.Signed-off-by Nick Piggin <[email protected]>
This patch also eliminates a leak "cleaned up" by re-clobbering the refcount on every allocation from the hugepage freelists. With respect to the lockless pagecache, the crucial aspect is to eliminate unconditional set_page_count() to 0 on pages with potentially nonzero refcounts, though closer inspection suggests the assignments removed are entirely spurious.
-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>