mlock: release mmap_sem every 256 faulted pages

Hi,

I'd like to sollicit comments on this proposal:

Currently mlock() holds mmap_sem in exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time.

Quoted 8 lines

Hi,

I'd like to sollicit comments on this proposal:

Currently mlock() holds mmap_sem in exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time.

A more compelling description of why this problem needs addressing would help things along.

mm/mlock.c static int do_mlock(unsigned long start, size_t len, int on) · line 504

Quoted 6 lines

mm/mlock.c +5 -0

+		/*
+		 * Limit batch size to 256 pages in order to reduce
+		 * mmap_sem hold time.
+		 */
+		nfault = nstart + 256 * PAGE_SIZE;

It would be nicer if there was an rwsem API to ask if anyone is currently blocked in down_read() or down_write(). That wouldn't be too hard to do. It wouldn't detect people polling down_read_trylock() or down_write_trylock() though.

Quoted 5 lines · On Mon, 22 Nov 2010 21:00:52 -0800 Michel Lespinasse <[email protected]> wrote:

I'd like to sollicit comments on this proposal:

Currently mlock() holds mmap_sem in exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time.

A more compelling description of why this problem needs addressing would help things along.

Quoted 5 lines

+             /*
+              * Limit batch size to 256 pages in order to reduce
+              * mmap_sem hold time.
+              */
+             nfault = nstart + 256 * PAGE_SIZE;

It would be nicer if there was an rwsem API to ask if anyone is currently blocked in down_read() or down_write().  That wouldn't be too hard to do.  It wouldn't detect people polling down_read_trylock() or down_write_trylock() though.

Quoted 2 lines

A more compelling description of why this problem needs addressing would help things along.

Oh my. It's probably not too useful for desktops, where such large mlocks are hopefully uncommon.

At google we have many applications that serve data from memory and don't want to allow for disk latencies. Some of the simpler ones use mlock (though there are other ways - anon memory running with swap disabled is a surprisingly popular choice).

Kosaki is also showing interest in mlock, though I'm not sure what his use case is.

Hi,

I'd like to sollicit comments on this proposal:

Currently mlock() holds mmap_sem in exclusive mode while the pages get faulted in. In the case of a large mlock, this can potentially take a very long time.

I propose that mlock() could release mmap_sem after the VM_LOCKED bits have been set in all appropriate VMAs. Then a second pass could be done to actually mlock the pages, in small batches, never holding mmap_sem for longer than it takes to process one single batch. We need to recheck the vma flags whenever we re-acquire mmap_sem, but this is not difficult.

This is only an RFC rather than an actual submission, as I think this could / should be completed to handle more than the mlock() and mlockall() cases (there are many call sites to mlock_vma_pages_range() that should ideally be converted as well), and maybe use the fault retry mechanism to drop mmap_sem when blocking on disk access rather than using an arbitrary page batch size.

Patch is against v2.6.36, but should apply to linus tree too.

------------------------------- 8< -----------------------------

Let mlock / mlockall release mmap_sem after the vmas have been marked as VM_LOCKED. Then, mark the vmas as mlocked in small batches. For every batch, we need to grab mmap_sem in read mode, check that the vma has not been munlocked, and mlock the pages.

In the case where a vma has been munlocked before mlock completes, pages that were already marked as PageMlocked() are handled by the munlock() call, and mlock() is careful to not mark new page batches as PageMlocked() after the munlock() call has cleared the VM_LOCKED vma flags. So, the end result will be identical to what'd happen if munlock() had executed after the mlock() call.

Signed-off-by Michel Lespinasse <[email protected]>

---
 mm/mlock.c |   79 +++++++++++++++++++++++++++++++++++++++++++++++------------
 1 files changed, 63 insertions(+), 16 deletions(-)

mm/mlock.c +5 -15

index b70919c..0aa4df5 100644
@@ -373,17 +373,11 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,
 	int lock = newflags & VM_LOCKED;
 
 	if (newflags == vma->vm_flags ||
-			(vma->vm_flags & (VM_IO | VM_PFNMAP)))
+	    (vma->vm_flags & (VM_IO | VM_PFNMAP |
+			      VM_DONTEXPAND | VM_RESERVED)) ||
+	    is_vm_hugetlb_page(vma) || vma == get_gate_vma(current))
 		goto out;	/* don't set VM_LOCKED,  don't count */
 
-	if ((vma->vm_flags & (VM_DONTEXPAND | VM_RESERVED)) ||
-			is_vm_hugetlb_page(vma) ||
-			vma == get_gate_vma(current)) {
-		if (lock)
-			make_pages_present(start, end);
-		goto out;	/* don't set VM_LOCKED,  don't count */
-	}
-
 	pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);
 	*prev = vma_merge(mm, *prev, start, end, newflags, vma->anon_vma,
 			  vma->vm_file, pgoff, vma_policy(vma));

@@ -419,14 +413,10 @@ success:
 	 * set VM_LOCKED, __mlock_vma_pages_range will bring it back.
 	 */
 
-	if (lock) {
+	if (lock)
 		vma->vm_flags = newflags;
-		ret = __mlock_vma_pages_range(vma, start, end);
-		if (ret < 0)
-			ret = __mlock_posix_error_return(ret);
-	} else {
+	else
 		munlock_vma_pages_range(vma, start, end);
-	}
 
 out:
 	*prev = vma;

mm/mlock.c +7 -0

 	end = start + len;
 	if (end < start)
 		return -EINVAL;

@@ -482,6 +473,58 @@ static int do_mlock(unsigned long start, size_t len, int on)
 	return error;
 }
 
+static int do_mlock_pages(unsigned long start, size_t len)
+{
+	struct mm_struct *mm = current->mm;
+	unsigned long end, nstart, nend, nfault;
+	struct vm_area_struct *vma;
+	int error = 0;
+

mm/mlock.c +6 -0

+	}
+	return 0;
+}
+
 SYSCALL_DEFINE2(mlock, unsigned long, start, size_t, len)
 {
 	unsigned long locked;

@@ -507,6 +550,8 @@ SYSCALL_DEFINE2(mlock, unsigned long, start, size_t, len)
 	if ((locked <= lock_limit) || capable(CAP_IPC_LOCK))
 		error = do_mlock(start, len, 1);
 	up_write(&current->mm->mmap_sem);
+	if (!error)
+		error = do_mlock_pages(start, len);
 	return error;
 }
 

mm/mlock.c +12 -0

+
+		/*
+		 * Now fault in a batch of pages. We need to check the vma
+		 * flags again, as we've not been holding mmap_sem.
+		 */
+		if ((vma->vm_flags & (VM_DONTEXPAND | VM_RESERVED)) ||
+		    is_vm_hugetlb_page(vma) || vma == get_gate_vma(current)) {
+			if (nfault < nend)
+				nend = nfault;
+			make_pages_present(nstart, nend);
+		} else if (vma->vm_flags & VM_LOCKED) {
+			if (nfault < nend)