Skip to contentSkip to conversation

mm/hugetlb: avoid looping to the same hugepage if !pages and !vmas

PATCH Superseded Zhigang Lu 2 participants 3 messages Aug 29, 2019 – Sep 4, 2019

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Zhigang Lu

author
#0 lore raw

PATCH v2 1/1 mm/hugetlb: avoid looping to the same hugepage if !pages and !vmas

From: Zhigang Lu <[email protected]>

When mmapping an existing hugetlbfs file with MAP_POPULATE, we find it is very time consuming. For example, mmapping a 128GB file takes about 50 milliseconds. Sampling with perfevent shows it spends 99% time in the same_page loop in follow_hugetlb_page().

samples: 205 of event 'cycles', Event count (approx.): 136686374
- 99.04% test_mmap_huget [kernel.kallsyms] [k] follow_hugetlb_page

        follow_hugetlb_page
        __get_user_pages
        __mlock_vma_pages_range
        __mm_populate
        vm_mmap_pgoff
        sys_mmap_pgoff
        sys_mmap
        system_call_fastpath
        __mmap64

follow_hugetlb_page() is called with pages=NULL and vmas=NULL, so for each hugepage, we run into the same_page loop for pages_per_huge_page() times, but doing nothing. With this change, it takes less then 1 millisecond to mmap a 128GB file in hugetlbfs.

---
 mm/hugetlb.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/hugetlb.c +11 -0
index 6d7296d..2df941a 100644
@@ -4391,6 +4391,17 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 				break;
 			}
 		}
+
+		if (!pages && !vmas && !pfn_offset &&
+		    (vaddr + huge_page_size(h) < vma->vm_end) &&
+		    (remainder >= pages_per_huge_page(h))) {
+			vaddr += huge_page_size(h);
+			remainder -= pages_per_huge_page(h);
+			i += pages_per_huge_page(h);
+			spin_unlock(ptl);
+			continue;
+		}
+
 same_page:
 		if (pages) {
 			pages[i] = mem_map_offset(page, pfn_offset);
Signature · 1 line
1.8.3.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 19 lines · Zhigang Lu

From: Zhigang Lu <[email protected]>

When mmapping an existing hugetlbfs file with MAP_POPULATE, we find it is very time consuming. For example, mmapping a 128GB file takes about 50 milliseconds. Sampling with perfevent shows it spends 99% time in the same_page loop in follow_hugetlb_page().

samples: 205 of event 'cycles', Event count (approx.): 136686374
- 99.04% test_mmap_huget [kernel.kallsyms] [k] follow_hugetlb_page

        follow_hugetlb_page
        __get_user_pages
        __mlock_vma_pages_range
        __mm_populate
        vm_mmap_pgoff
        sys_mmap_pgoff
        sys_mmap
        system_call_fastpath
        __mmap64
Quoted 4 lines

follow_hugetlb_page() is called with pages=NULL and vmas=NULL, so for each hugepage, we run into the same_page loop for pages_per_huge_page() times, but doing nothing. With this change, it takes less then 1 millisecond to mmap a 128GB file in hugetlbfs.

Thanks for the analysis!

Just curious, do you have an application that does this (mmap(MAP_POPULATE) for an existing hugetlbfs file), or was this part of some test suite or debug code?

Quoted 13 lines
---
 mm/hugetlb.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
mm/hugetlb.c +0 -0
index 6d7296d..2df941a 100644

mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 4393

Quoted 3 lines
mm/hugetlb.c +0 -0
@@ -4391,6 +4391,17 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 				break;
 			}
 		}

It might be helpful to add a comment here to help readers of the code. Something like:

		/*
		 * If subpage information not requested, update counters
		 * and skip the same_page loop below.
		 */
Quoted 5 lines
mm/hugetlb.c +4 -0
+
+		if (!pages && !vmas && !pfn_offset &&
+		    (vaddr + huge_page_size(h) < vma->vm_end) &&
+		    (remainder >= pages_per_huge_page(h))) {

mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 4407

Quoted 10 lines
mm/hugetlb.c +7 -0
+			vaddr += huge_page_size(h);
+			remainder -= pages_per_huge_page(h);
+			i += pages_per_huge_page(h);
+			spin_unlock(ptl);
+			continue;
+		}
+
 same_page:
 		if (pages) {
 			pages[i] = mem_map_offset(page, pfn_offset);

With a comment added to the code,

Signature · 1 line
Mike Kravetz
author
#2 lore raw
Quoted 26 lines · Mike Kravetz <[email protected]> 于2019年9月4日周三 上午5:26写道:
Quoted 19 lines · Zhigang Lu

From: Zhigang Lu <[email protected]>

When mmapping an existing hugetlbfs file with MAP_POPULATE, we find it is very time consuming. For example, mmapping a 128GB file takes about 50 milliseconds. Sampling with perfevent shows it spends 99% time in the same_page loop in follow_hugetlb_page().

samples: 205 of event 'cycles', Event count (approx.): 136686374
- 99.04% test_mmap_huget [kernel.kallsyms] [k] follow_hugetlb_page

        follow_hugetlb_page
        __get_user_pages
        __mlock_vma_pages_range
        __mm_populate
        vm_mmap_pgoff
        sys_mmap_pgoff
        sys_mmap
        system_call_fastpath
        __mmap64
Quoted 4 lines

follow_hugetlb_page() is called with pages=NULL and vmas=NULL, so for each hugepage, we run into the same_page loop for pages_per_huge_page() times, but doing nothing. With this change, it takes less then 1 millisecond to mmap a 128GB file in hugetlbfs.

Quoted 5 lines

Thanks for the analysis!

Just curious, do you have an application that does this (mmap(MAP_POPULATE) for an existing hugetlbfs file), or was this part of some test suite or debug code?

Yes, DPDK and SPDK actually do this in vhost_user.c. vhost_setup_mem_table() {
...
mmap_size = RTE_ALIGN_CEIL(mmap_size, alignment);

mmap_addr = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, fd, 0);
...
}

Quoted 9 lines
Quoted 9 lines
---
 mm/hugetlb.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
Quoted 12 lines

mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 4393

Quoted 8 lines
mm/hugetlb.c +0 -0
index 6d7296d..2df941a 100644
@@ -4391,6 +4391,17 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
                              break;
                      }
              }

It might be helpful to add a comment here to help readers of the code. Something like:

Thanks, I will add this comment and send a new version.

Quoted 24 lines

/*
* If subpage information not requested, update counters
* and skip the same_page loop below.
*/

Quoted 5 lines
mm/hugetlb.c +4 -0
+
+             if (!pages && !vmas && !pfn_offset &&
+                 (vaddr + huge_page_size(h) < vma->vm_end) &&
+                 (remainder >= pages_per_huge_page(h))) {

mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 4407

Quoted 10 lines
mm/hugetlb.c +7 -0
+                     vaddr += huge_page_size(h);
+                     remainder -= pages_per_huge_page(h);
+                     i += pages_per_huge_page(h);
+                     spin_unlock(ptl);
+                     continue;
+             }
+
 same_page:
              if (pages) {
                      pages[i] = mem_map_offset(page, pfn_offset);

With a comment added to the code,

--
Mike Kravetz