mm: hugetlbfs: Add some VM_BUG_ON()s to catch non-hugetlbfs pages
Subsystems (4)
PATCH 1/2 mm: hugetlbfs: Add some VM_BUG_ON()s to catch non-hugetlbfs pages
Dave Jiang reported that he was seeing oopses when running NUMA systems and default_hugepagesz=1G. I traced the issue down to migrate_page_copy() trying to use the same code for hugetlb pages and transparent hugepages. It should not have been trying to pass thp pages in there.
So, add some VM_BUG_ON()s for the next hapless VM developer that tries the same thing.
---
linux.git-davehans/include/linux/hugetlb.h | 1 + linux.git-davehans/mm/hugetlb.c | 1 + 2 files changed, 2 insertions(+)
include/linux/hugetlb.h renamed from include/linux/hugetlb.h~bug-not-hugetlbfs-in-copy_huge_page +1 -0
@@ -355,6 +355,7 @@ static inline pte_t arch_make_huge_pte(pstatic inline struct hstate *page_hstate (struct page *page) { +VM_BUG_ON (!PageHuge (page));return size_to_hstate (PAGE_SIZE <<compound_order (page)); }
mm/hugetlb.c renamed from mm/hugetlb.c~bug-not-hugetlbfs-in-copy_huge_page +1 -0
@@ -498,6 +498,7 @@ void copy_huge_page(struct page *dst, stint i;struct hstate *h =page_hstate (src); +VM_BUG_ON (!h);if (unlikely (pages_per_huge_page (h) >MAX_ORDER_NR_PAGES )) {copy_gigantic_page (dst, src);return ;
_
PATCH 2/2 mm: thp: give transparent hugepage code a separate copy_page
From: Dave Hansen <[email protected]>
Right now, the migration code in migrate_page_copy() uses copy_huge_page() for hugetlbfs and thp pages:
if (PageHuge(page) || PageTransHuge(page))
copy_huge_page(newpage, page);So, yay for code reuse. But:
void copy_huge_page(struct page *dst, struct page *src)
{
struct hstate *h = page_hstate(src);and a non-hugetlbfs page has no page_hstate(). This works 99% of the time because page_hstate() determines the hstate from the page order alone. Since the page order of a THP page matches the default hugetlbfs page order, it works.
But, if you change the default huge page size on the boot command-line (say default_hugepagesz=1G), then we might not even *have* a 2MB hstate so page_hstate() returns null and copy_huge_page() oopses pretty fast since copy_huge_page() dereferences the hstate:
void copy_huge_page(struct page *dst, struct page *src)
{
struct hstate *h = page_hstate(src);
if (unlikely(pages_per_huge_page(h) > MAX_ORDER_NR_PAGES)) {...
This patch creates a copy_high_order_page() which can be used on THP pages.
---
linux.git-davehans/include/linux/huge_mm.h | 16 ++++++++++++++++ linux.git-davehans/mm/huge_memory.c | 13 +++++++++++++ linux.git-davehans/mm/migrate.c | 4 +++- 3 files changed, 32 insertions(+), 1 deletion(-)
include/linux/huge_mm.h renamed from include/linux/huge_mm.h~copy-huge-separate-from-copy-transhuge +16 -0
@@ -177,6 +177,10 @@ static inline struct page *compound_tranextern int do_huge_pmd_numa_page (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pmd_t pmd,pmd_t *pmdp); +extern void copy_high_order_page (struct page *newpage, +struct page *oldpage, +int order); +#else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })#define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; })
@@ -227,6 +231,18 @@ static inline int do_huge_pmd_numa_page(return 0 ; } +/* +* The non-stub version of this code is probably usable +* generically but its only user is thp at the moment, +* so enforce that with a BUG() +*/ +static inline void copy_high_order_page (struct page *newpage, +struct page *oldpage, +int order) +{ +BUG (); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* _LINUX_HUGE_MM_H */
mm/huge_memory.c renamed from mm/huge_memory.c~copy-huge-separate-from-copy-transhuge +13 -0
@@ -2789,3 +2789,16 @@ void __vma_adjust_trans_huge(struct vm_asplit_huge_page_address (next->vm_mm, nstart); } } + +void copy_high_order_page (struct page *newpage, +struct page *oldpage, +int order) +{ +int i; + +might_sleep (); +for (i =0 ; i < (1 <<order); i++) { +cond_resched (); +copy_highpage (newpage + i, oldpage + i); + } +}
mm/migrate.c renamed from mm/migrate.c~copy-huge-separate-from-copy-transhuge +3 -1
@@ -443,8 +443,10 @@ int migrate_huge_page_move_mapping(struc*/ void migrate_page_copy (struct page *newpage,struct page *page) { -if (PageHuge (page) ||PageTransHuge (page)) +if (PageHuge (page))copy_huge_page (newpage, page); +else if (PageTransHuge (page)) +copy_high_order_page (newpage, page,HPAGE_PMD_ORDER );else copy_highpage (newpage, page);
_
Re: [PATCH 2/2] mm: thp: give transparent hugepage code a separate copy_page
Quoted 27 lines · Dave Hansen
From: Dave Hansen <[email protected]>
Right now, the migration code in
migrate_page_copy()usescopy_huge_page()for hugetlbfs and thp pages:if (PageHuge(page) || PageTransHuge(page)) copy_huge_page(newpage, page);So, yay for code reuse. But:
void
copy_huge_page(struct page *dst, struct page *src){ struct hstate *h = page_hstate(src);and a non-hugetlbfs page has no
page_hstate(). This works 99% of the time becausepage_hstate()determines the hstate from the page order alone. Since the page order of a THP page matches the default hugetlbfs page order, it works.But, if you change the default huge page size on the boot command-line (say default_hugepagesz=1G), then we might not even *have* a 2MB hstate so
page_hstate()returns null andcopy_huge_page()oopses pretty fast sincecopy_huge_page()dereferences the hstate:
Quoted 8 lines
void
copy_huge_page(struct page *dst, struct page *src){ struct hstate *h = page_hstate(src); if (unlikely(pages_per_huge_page(h) > MAX_ORDER_NR_PAGES)) {...
This patch creates a
copy_high_order_page()which can be used on THP pages.
We already have copy_user_huge_page() and copy_user_gigantic_page() in generic code (mm/memory.c). I think copy_gigantic_page() and copy_huge_page() should be moved there too.
BTW, I think pages_per_huge_page in copy_user_huge_page() is redunand:compound_order(page) should be enough, right?
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/2] mm: thp: give transparent hugepage code a separatecopy_page
Quoted 34 lines · Kirill A. Shutemov
Quoted 33 lines · Dave Hansen
From: Dave Hansen <[email protected]>
Right now, the migration code in
migrate_page_copy()usescopy_huge_page()for hugetlbfs and thp pages:if (PageHuge(page) || PageTransHuge(page)) copy_huge_page(newpage, page);So, yay for code reuse. But:
void
copy_huge_page(struct page *dst, struct page *src){ struct hstate *h = page_hstate(src);and a non-hugetlbfs page has no
page_hstate(). This works 99% of the time becausepage_hstate()determines the hstate from the page order alone. Since the page order of a THP page matches the default hugetlbfs page order, it works.But, if you change the default huge page size on the boot command-line (say default_hugepagesz=1G), then we might not even *have* a 2MB hstate so
page_hstate()returns null andcopy_huge_page()oopses pretty fast sincecopy_huge_page()dereferences the hstate:void
copy_huge_page(struct page *dst, struct page *src){ struct hstate *h = page_hstate(src); if (unlikely(pages_per_huge_page(h) > MAX_ORDER_NR_PAGES)) {...
Quoted 6 lines
Quoted 2 lines
This patch creates a
copy_high_order_page()which can be used on THP pages.We already have
copy_user_huge_page()andcopy_user_gigantic_page()in generic code (mm/memory.c). I thinkcopy_gigantic_page()andcopy_huge_page()should be moved there too.
I agree this.
Quoted 2 lines
BTW, I think
pages_per_huge_pageincopy_user_huge_page()is redunand:compound_order(page)should be enough, right?
I guess that thp code is very strict on performance, so developers chose to pass it as an argument instead of calculating compound_order in each call. I think the performance gain is small (maybe invisible),
but it's not a bad idea to me.
Thanks,
Naoya
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/2] mm: thp: give transparent hugepage code a separate copy_page
In reply to Kirill A. ShutemovQuoted 7 lines · Kirill A. Shutemov
Quoted 6 lines · Dave Hansen
void
copy_huge_page(struct page *dst, struct page *src){ struct hstate *h = page_hstate(src); if (unlikely(pages_per_huge_page(h) > MAX_ORDER_NR_PAGES)) {...
Quoted 6 lines
Quoted 2 lines
This patch creates a
copy_high_order_page()which can be used on THP pages.We already have
copy_user_huge_page()andcopy_user_gigantic_page()in generic code (mm/memory.c). I thinkcopy_gigantic_page()andcopy_huge_page()should be moved there too.
That would be fine I guesss... in another patch. :)
Quoted 2 lines
BTW, I think
pages_per_huge_pageincopy_user_huge_page()is redunand:compound_order(page)should be enough, right?
The way it is now, the compiler can optimize for the !HUGETLBFS case. Also, pages_per_huge_page() works for gigantic pages. compound_order() wouldn't work for those.
Re: [PATCH 2/2] mm: thp: give transparent hugepage code a separate copy_page
In reply to Dave Hansenmm/huge_memory.c void __vma_adjust_trans_huge(struct vm_a · line 2804
Quoted 13 lines · Dave Hansen
mm/huge_memory.c renamed from mm/huge_memory.c~copy-huge-separate-from-copy-transhuge +13
-0+ +void copy_high_order_page (struct page *newpage, +struct page *oldpage, +int order) +{ +int i; + +might_sleep (); +for (i =0 ; i < (1 <<order); i++) { +cond_resched (); +copy_highpage (newpage + i, oldpage + i); + } +}
Can we make no use of might_sleep here with cond_resched in loop?
Re: [PATCH 2/2] mm: thp: give transparent hugepage code a separate copy_page
Quoted 16 lines · Hillf Danton
mm/huge_memory.c void __vma_adjust_trans_huge(struct vm_a · line 2804
Quoted 13 lines · Dave Hansen
mm/huge_memory.c renamed from mm/huge_memory.c~copy-huge-separate-from-copy-transhuge +13
-0+ +void copy_high_order_page (struct page *newpage, +struct page *oldpage, +int order) +{ +int i; + +might_sleep (); +for (i =0 ; i < (1 <<order); i++) { +cond_resched (); +copy_highpage (newpage + i, oldpage + i); + } +}Can we make no use of
might_sleephere withcond_reschedin loop?
I'm not sure what you're saying.
Are you pointing out that cond_resched() actually calls might_sleep() so the might_sleep() is redundant?
Re: [PATCH 1/2] mm: hugetlbfs: Add some VM_BUG_ON()s to catchnon-hugetlbfs pages
In reply to Dave HansenQuoted 8 lines · Dave Hansen
Dave Jiang reported that he was seeing oopses when running NUMA systems and default_hugepagesz=1G. I traced the issue down to
migrate_page_copy()trying to use the same code for hugetlb pages and transparent hugepages. It should not have been trying to pass thp pages in there.So, add some VM_BUG_ON()s for the next hapless VM developer that
Looks good to me, thanks.
Quoted 31 lines
---
linux.git-davehans/include/linux/hugetlb.h | 1 + linux.git-davehans/mm/hugetlb.c | 1 + 2 files changed, 2 insertions(+)diff -puN
include/linux/hugetlb.h~bug-not-hugetlbfs-in-copy_huge_pageinclude/linux/hugetlb.hinclude/linux/hugetlb.h renamed from include/linux/hugetlb.h~bug-not-hugetlbfs-in-copy_huge_page +1
-0@@ -355,6 +355,7 @@ static inline pte_t arch_make_huge_pte(pstatic inline struct hstate *page_hstate (struct page *page) { +VM_BUG_ON (!PageHuge (page));return size_to_hstate (PAGE_SIZE <<compound_order (page)); }diff -puN
mm/hugetlb.c~bug-not-hugetlbfs-in-copy_huge_pagemm/hugetlb.cmm/hugetlb.c renamed from mm/hugetlb.c~bug-not-hugetlbfs-in-copy_huge_page +1
-0@@ -498,6 +498,7 @@ void copy_huge_page(struct page *dst, stint i;struct hstate *h =page_hstate (src); +VM_BUG_ON (!h);if (unlikely (pages_per_huge_page (h) >MAX_ORDER_NR_PAGES )) {copy_gigantic_page (dst, src);return ;_
Quoted 6 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>