Hugepage regression
Subsystems (3)
It seems commit fe1668ae5bf0145014c71797febd9ad5670d5d05 causes a hugepage regression. A git bisect points the finger at that commit for causing an oops in the 'alloc-instantiate-race' test from the libhugetlbfs testsuite.
Still looking to determine the reason it breaks things.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 7 lines · On Tue, 10 Oct 2006 18:47:48 +1000 David Gibson <[email protected]>…
It seems commit
fe1668ae5bf0145014c71797febd9ad5670d5d05causes a hugepage regression. A git bisect points the finger at that commit for causing an oops in the 'alloc-instantiate-race' test from the libhugetlbfs testsuite.Still looking to determine the reason it breaks things.
It's assuming that unmap_hugepage_range() is always freeing these pages. If the page is shared by another mapping, bad things will happen: the threads fight over page->lru.
Doing
+ if (page_count(page) == 1)
list_add(&page->lru, &page_list);
might help. But then we miss the tlb flush in rare racy conditions.
Quoted 20 lines · Andrew Morton
Quoted 7 lines · On Tue, 10 Oct 2006 18:47:48 +1000 David Gibson <[email protected]>…
It seems commit
fe1668ae5bf0145014c71797febd9ad5670d5d05causes a hugepage regression. A git bisect points the finger at that commit for causing an oops in the 'alloc-instantiate-race' test from the libhugetlbfs testsuite.Still looking to determine the reason it breaks things.
It's assuming that
unmap_hugepage_range()is always freeing these pages. If the page is shared by another mapping, bad things will happen: the threads fight overpage->lru.Doing
(quoted diff) +1
-0+ if (page_count(page) == 1)list_add(&page->lru, &page_list);
Quoted 1 line
might help. But then we miss the tlb flush in rare racy conditions.
Well, there'd need to be an else doing a put_page(), too.
Looks like the fundamental problem is that a list is not a suitable data structure for gathering here, since it's not truly local. We should probably change it to a small array, like in the normal tlb gather structure. If we run out of space we can force the tlb flush and keep going.
Or we could just give up on lazy tlb flush for hugepages.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
David Gibson wrote on Tuesday, October 10, 2006 2:16 AM
Quoted 21 lines
Quoted 17 lines
Quoted 7 lines
It seems commit
fe1668ae5bf0145014c71797febd9ad5670d5d05causes a hugepage regression. A git bisect points the finger at that commit for causing an oops in the 'alloc-instantiate-race' test from the libhugetlbfs testsuite.Still looking to determine the reason it breaks things.
It's assuming that
unmap_hugepage_range()is always freeing these pages. If the page is shared by another mapping, bad things will happen: the threads fight overpage->lru.Doing
(quoted diff) +1
-0+ if (page_count(page) == 1)list_add(&page->lru, &page_list);Quoted 1 line
might help. But then we miss the tlb flush in rare racy conditions.
Well, there'd need to be an else doing a
put_page(), too.
Quoted 5 lines
Looks like the fundamental problem is that a list is not a suitable data structure for gathering here, since it's not truly local. We should probably change it to a small array, like in the normal tlb gather structure. If we run out of space we can force the tlb flush and keep going.
With the pending shared page table for hugetlb currently sitting in -mm, we serialize the all hugetlb unmap with a per file i_mmap_lock. This race could well be solved by that pending patch?
- Ken
Quoted 29 lines · Kenneth W
David Gibson wrote on Tuesday, October 10, 2006 2:16 AM
Quoted 21 lines
Quoted 17 lines
Quoted 7 lines
It seems commit
fe1668ae5bf0145014c71797febd9ad5670d5d05causes a hugepage regression. A git bisect points the finger at that commit for causing an oops in the 'alloc-instantiate-race' test from the libhugetlbfs testsuite.Still looking to determine the reason it breaks things.
It's assuming that
unmap_hugepage_range()is always freeing these pages. If the page is shared by another mapping, bad things will happen: the threads fight overpage->lru.Doing
(quoted diff) +1
-0+ if (page_count(page) == 1)list_add(&page->lru, &page_list);Quoted 1 line
might help. But then we miss the tlb flush in rare racy conditions.
Well, there'd need to be an else doing a
put_page(), too.Quoted 5 lines
Looks like the fundamental problem is that a list is not a suitable data structure for gathering here, since it's not truly local. We should probably change it to a small array, like in the normal tlb gather structure. If we run out of space we can force the tlb flush and keep going.
Quoted 7 lines
With the pending shared page table for hugetlb currently sitting in -mm, we serialize the all hugetlb unmap with a per file
i_mmap_lock. This race could well be solved by that pending patch?
We need something for 2.6.19 though. As David indicates, not using page->lru should fix it (pagevec_add, pagevec_release would suit).
Or just a separate TBL invalidation per page. Is that likely to be particularly expensive? It's the first one which hurts?
Quoted 6 lines · Kenneth W
With the pending shared page table for hugetlb currently sitting in -mm, we serialize the all hugetlb unmap with a per file
i_mmap_lock. This race could well be solved by that pending patch?
Hey, nice try, Ken! But I don't think we can let you sneak shared pagetables into 2.6.19 that way ;)
Sorry for not noticing this bug in your original TLB flush fix, which had looked good to me.
Yes, I'd expect your i_mmap_lock to solve the problem: and since you're headed in that direction anyway, it makes most sense to use that solution rather than get into defining arrays, or sacrificing the lazy flush, or risking page_count races.
So please extract the __unmap_hugepage_range mods from your shared pagetable patch, and use that to fix the bug. But again, I protest the "if (vma->vm_file)" in your unmap_hugepage_range - how would a hugepage area ever have NULL vma->vm_file?
Hugh
Hugh Dickins wrote on Tuesday, October 10, 2006 12:18 PM
Quoted 7 lines · Kenneth W
Quoted 6 lines
With the pending shared page table for hugetlb currently sitting in -mm, we serialize the all hugetlb unmap with a per file
i_mmap_lock. This race could well be solved by that pending patch?
Quoted 3 lines
Hey, nice try, Ken! But I don't think we can let you sneak shared pagetables into 2.6.19 that way ;)
It wasn't my intention to sneak in shared page table, though it does sort of look like so.
Quoted 7 lines
Yes, I'd expect your
i_mmap_lockto solve the problem: and since you're headed in that direction anyway, it makes most sense to use that solution rather than get into defining arrays, or sacrificing the lazy flush, or riskingpage_countraces.So please extract the
__unmap_hugepage_rangemods from your shared pagetable patch, and use that to fix the bug.
OK, I was about to do so too.
Quoted 3 lines
But again, I protest
the "if (vma->vm_file)" in yourunmap_hugepage_range- how would a hugepage area ever have NULLvma->vm_file?
It's coming from do_mmap_pgoff(), file->f_op->mmap can fail with error code (e.g. not enough hugetlb page) and in the error recovery path, it nulls out vma->vm_file first before calls down to unmap_region(). I asked that question before: can we reverse that order (call unmap_region and then nulls out vma->vmfile and fput)?
unmap_and_free_vma:
if (correct_wcount)
atomic_inc(&inode->i_writecount);
vma->vm_file = NULL;
fput(file);
/* Undo any partial mapping done by a device driver. */
unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);Quoted 9 lines · Kenneth W
Hugh Dickins wrote on Tuesday, October 10, 2006 12:18 PM
Quoted 3 lines
But again, I protest
the "if (vma->vm_file)" in yourunmap_hugepage_range- how would a hugepage area ever have NULLvma->vm_file?It's coming from
do_mmap_pgoff(),file->f_op->mmapcan fail with error code (e.g. not enough hugetlb page) and in the error recovery path, it nulls outvma->vm_filefirst before calls down tounmap_region().
I stand corrected: thanks.
Quoted 1 line
I asked that question before:
So you did, on Oct 2nd: sorry, that got lost amidst the overload in my mailbox, I've just forwarded it to myself again, to check later on what else I may have missed. (I am aware that I've still not scrutinized the patches you sent out a day or two later, now in -mm.)
Quoted 2 lines
can we reverse that order (call
unmap_region
and then nulls outvma->vmfileand fput)?
I'm pretty sure we cannot: the ordering is quite intentional, that if a driver ->mmap failed, then it'd be wrong to call down to driver in the unmap_region (if a driver is nicely behaved, that unmap_region shouldn't be unnecessary; but some do rely on us clearing ptes there).
Okay, last refuge of all who've made a fool of themselves:
may I ask you to add a comment in your unmap_hugepage_range, pointing to how the do_mmap_pgoff error case NULLifies vm_file?
(Or else change hugetlbfs_file_mmap to set VM_HUGETLB only once it's
succeeded: but that smacks of me refusing to accept I was wrong.)
Hugh
Quoted 9 lines
unmap_and_free_vma:if (correct_wcount) atomic_inc(&inode->i_writecount); vma->vm_file = NULL; fput(file); /* Undo any partial mapping done by a device driver. */ unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
Hugh Dickins wrote on Tuesday, October 10, 2006 1:10 PM
Quoted 7 lines
Quoted 2 lines
can we reverse that order (call
unmap_region
and then nulls outvma->vmfileand fput)?I'm pretty sure we cannot: the ordering is quite intentional, that if a driver ->mmap failed, then it'd be wrong to call down to driver in the
unmap_region(if a driver is nicely behaved, thatunmap_regionshouldn't be unnecessary; but some do rely on us clearing ptes there).
Even not something like the following? I believe you that nullifying vma->vm_file can not be done after unmap_region(), I just want to make sure we are talking the same thing. It looks OK to me to defer the fput in the do_mmap_pgoff clean up path.
mm/mmap.c renamed from mm/mmap.c.orig +3 -2
@@ -1159,11 +1159,12 @@ out: unmap_and_free_vma:if (correct_wcount)atomic_inc (&inode->i_writecount); - vma->vm_file =NULL ; -fput (file);/* Undo any partial mapping done by a device driver. */ unmap_region (mm, vma, prev, vma->vm_start, vma->vm_end); + + vma->vm_file =NULL ; +fput (file); charged =0 ; free_vma:kmem_cache_free (vm_area_cachep, vma);
Quoted 8 lines · Kenneth W
Hugh Dickins wrote on Tuesday, October 10, 2006 1:10 PM
Quoted 7 lines
Quoted 2 lines
can we reverse that order (call
unmap_region
and then nulls outvma->vmfileand fput)?I'm pretty sure we cannot: the ordering is quite intentional, that if a driver ->mmap failed, then it'd be wrong to call down to driver in the
unmap_region(if a driver is nicely behaved, thatunmap_regionshouldn't be unnecessary; but some do rely on us clearing ptes there).
Looking at it again, my explanation seems wrong: I can't see any danger of calling down to the _driver_ there (there's no remove_vma): rather, it's __remove_shared_vm_struct we're avoiding by setting vm_file NULL.
Quoted 3 lines
Even not something like the following? I believe you that nullifying
vma->vm_filecan not be done afterunmap_region(),
Yet in your patch below, you do nullify vm_file _after_ unmap_region.
Quoted 1 line
I just want to make sure we are talking the same thing.
So I'm not sure if we are!
Quoted 1 line
It looks OK to me to defer the fput in the
do_mmap_pgoffclean up path.
Yes, it would be quite okay to defer the fput until after the unmap_region; but there's no point in making that change, is there?
Hugh (sorry, I was off sick for a couple of days)
Quoted 20 lines
mm/mmap.c renamed from mm/mmap.c.orig +3
-2@@ -1159,11 +1159,12 @@ out: unmap_and_free_vma:if (correct_wcount)atomic_inc (&inode->i_writecount); - vma->vm_file =NULL ; -fput (file);/* Undo any partial mapping done by a device driver. */ unmap_region (mm, vma, prev, vma->vm_start, vma->vm_end); + + vma->vm_file =NULL ; +fput (file); charged =0 ; free_vma:kmem_cache_free (vm_area_cachep, vma);
Hugh Dickins wrote on Tuesday, October 10, 2006 12:18 PM
Quoted 7 lines
Yes, I'd expect your
i_mmap_lockto solve the problem: and since you're headed in that direction anyway, it makes most sense to use that solution rather than get into defining arrays, or sacrificing the lazy flush, or riskingpage_countraces.So please extract the
__unmap_hugepage_rangemods from your shared pagetable patch, and use that to fix the bug.
OK, here is a bug fix patch fixing earlier "bug fix" patch :-(
[patch] hugetlb: fix linked list corruption in unmap_hugepage_range
commit fe1668ae5bf0145014c71797febd9ad5670d5d05 causes kernel to oops with libhugetlbfs test suite. The problem is that hugetlb pages can be shared by multiple mappings. Multiple threads can fight over page->lru in the unmap path and bad things happen. We now serialize __unmap_hugepage_range to void concurrent linked list manipulation. Such serialization is also needed for shared page table page on hugetlb area. This patch will fixed the bug and also serve as a prepatch for shared page table.
fs/hugetlbfs/inode.c renamed from fs/hugetlbfs/inode.c.orig +1 -1
@@ -293,7 +293,7 @@ hugetlb_vmtruncate_list(struct prio_treeif (h_vm_pgoff >= h_pgoff) v_offset =0 ; -unmap_hugepage_range (vma, +__unmap_hugepage_range (vma, vma->vm_start + v_offset, vma->vm_end); } }
mm/hugetlb.c renamed from mm/hugetlb.c.orig +20 -2
@@ -356,8 +356,8 @@ nomem:return -ENOMEM ; } -void unmap_hugepage_range (struct vm_area_struct *vma,unsigned long start, -unsigned long end) +void __unmap_hugepage_range (struct vm_area_struct *vma,unsigned long start, +unsigned long end) {struct mm_struct *mm = vma->vm_mm;unsigned long address;
@@ -398,6 +398,24 @@ void unmap_hugepage_range(struct vm_area } } +void unmap_hugepage_range (struct vm_area_struct *vma,unsigned long start, +unsigned long end) +{ +/* +* It is undesirable to test vma->vm_file as it should be non-null +* for valid hugetlb area. However, vm_file will be NULL in the error +* cleanup path of do_mmap_pgoff. When hugetlbfs ->mmap method fails, +* do_mmap_pgoff() nullifies vma->vm_file before calling this function +* to clean up. Since no pte has actually been setup, it is safe to +* do nothing in this case. +*/ +if (vma->vm_file) { +spin_lock (&vma->vm_file->f_mapping->i_mmap_lock); +__unmap_hugepage_range (vma, start, end); +spin_unlock (&vma->vm_file->f_mapping->i_mmap_lock); + } +} +static int hugetlb_cow (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *ptep,pte_t pte) {
include/linux/hugetlb.h renamed from include/linux/hugetlb.h.orig +0 -0
@@ -17,6 +17,7 @@ int hugetlb_sysctl_handler(struct ctl_taint copy_hugetlb_page_range (struct mm_struct *,struct mm_struct *,struct vm_area_struct *); int follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, struct vm_area_struct **, unsigned long *, int
*, int);
void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long);
+void __unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long);
int hugetlb_prefault(struct address_space *, struct vm_area_struct *); int hugetlb_report_meminfo(char *);
int hugetlb_report_node_meminfo(int, char *);
Quoted 15 lines · Kenneth W
Hugh Dickins wrote on Tuesday, October 10, 2006 12:18 PM
Quoted 7 lines
Yes, I'd expect your
i_mmap_lockto solve the problem: and since you're headed in that direction anyway, it makes most sense to use that solution rather than get into defining arrays, or sacrificing the lazy flush, or riskingpage_countraces.So please extract the
__unmap_hugepage_rangemods from your shared pagetable patch, and use that to fix the bug.OK, here is a bug fix patch fixing earlier "bug fix" patch :-(
[patch] hugetlb: fix linked list corruption in
unmap_hugepage_range
Quoted 7 lines
commit
fe1668ae5bf0145014c71797febd9ad5670d5d05causes kernel to oops with libhugetlbfs test suite. The problem is that hugetlb pages can be shared by multiple mappings. Multiple threads can fight overpage->lruin the unmap path and bad things happen. We now serialize__unmap_hugepage_rangeto void concurrent linked list manipulation. Such serialization is also needed for shared page table page on hugetlb area. This patch will fixed the bug and also serve as a prepatch for shared page table.
Can I suggest that you put a big comment on the linked list declaration itself saying that you're relying on serialization here. Otherwise I'm worried someone will try to de-serialize it again, and break it without realizing. Given the number of people who failed to spot the problem with the patch the first time around..
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
David Gibson wrote on Tuesday, October 10, 2006 6:18 PM
Quoted 5 lines
Can I suggest that you put a big comment on the linked list declaration itself saying that you're relying on serialization here. Otherwise I'm worried someone will try to de-serialize it again, and break it without realizing. Given the number of people who failed to spot the problem with the patch the first time around..
I'm not very good at writing comments, how about the following?
mm/hugetlb.c renamed from mm/hugetlb.c.orig +5 -0
@@ -365,6 +365,11 @@ void __unmap_hugepage_range(struct vm_arpte_t pte;struct page *page;struct page *tmp; +/* +* A page gathering list, protected by per file i_mmap_lock. The +* lock is used to avoid list corruption from multiple unmapping +* of the same page since we are using page->lru. +*/ LIST_HEAD (page_list);WARN_ON (!is_vm_hugetlb_page (vma));