hugetlbfs: Take read_lock on i_mmap for PMD sharing

A customer with large SMP systems (up to 16 sockets) with application that uses large amount of static hugepages (~500-1500GB) are experiencing random multisecond delays. These delays was caused by the long time it took to scan the VMA interval tree with mmap_sem held.

The sharing of huge PMD does not require changes to the i_mmap at all. Therefore, we can just take the read lock and let other threads searching for the right VMA to share it in parallel. Once the right VMA is found, either the PMD lock (2M huge page for x86-64) or the mm->page_table_lock will be acquired to perform the actual PMD sharing.

Lock contention, if present, will happen in the spinlock. That is much better than contention in the rwsem where the time needed to scan the the interval tree is indeterminate.

With this patch applied, the customer is seeing significant performance improvement over the unpatched kernel.

---
 mm/hugetlb.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index b45a95363a84..f78891f92765 100644
@@ -4842,7 +4842,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
 	if (!vma_shareable(vma, addr))
 		return (pte_t *)pmd_alloc(mm, pud, addr);
 
-	i_mmap_lock_write(mapping);
+	i_mmap_lock_read(mapping);
 	vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) {
 		if (svma == vma)
 			continue;

@@ -4872,7 +4872,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
 	spin_unlock(ptl);
 out:
 	pte = (pte_t *)pmd_alloc(mm, pud, addr);
-	i_mmap_unlock_write(mapping);
+	i_mmap_unlock_read(mapping);
 	return pte;
 }
 

2.18.1

Could you give more details here?

Thanks,
Davidlohr

Red Hat has a customer that is running a transactional database workload. In this particular case, about ~500-1500GB of static hugepages are allocated.  The database then allocates a single large shared memory segment in those hugepages to use primarily as a database buffer for 8kB blocks from disk (there are also other database structures in that shared memory, but it's mostly for buffer).  Then thousands of separate processes reference and load data into that buffer. They were seeing multi-second pauses when starting up the database.

I first gave them a patched kernel that disabled PMD sharing. That fixed their problem. After that, I gave them another test kernel that contained this patch. They said there were significant improved compared with the unpatched kernel. There is still some degradation compared to the kernel with huge shared pmd disabled entirely, but they're pretty close in performance.

Cheer,
Longman