Skip to contentSkip to conversation

mm: ksm: add MAP_MERGEABLE mmap() as a KSM shortcut

RFC PATCH New Dave Hansen 1 participant 1 message Feb 12, 2014

Subsystems (5)
Memory management Documentation Generic include/asm header files Memory mapping +1 more subsystems
lore
Share & tools
lore

Dave Hansen

author
#0 lore raw

RFC PATCH 1/1 mm: ksm: add MAP_MERGEABLE mmap() as a KSM shortcut

We are starting to see substantial amounts (seconds) of latency being incurred by users of mmap_sem in the worst case. It is very common to see them spike up in to the tens-of-ms range. Any acquisition, especially for write, is a potential problem.

The aggravating factor here is that we have been encouraging folks to be "polite" to the VM and do things like: call MADV_DONTNEED, unmap when you're done using things, and use KSM. All of these things take mmap_sem(). JVMs are starting to put nuggets like this in their generic malloc() functions:

	addr = mmap(foo_bytes, ...);
	madvise(MADV_MERGABLE, addr, foo_bytes);

That means that every single malloc() call does at _least_ two write acquisitions of mmap_sem. We can try to batch these things in userspace more, of course, but this is becoming a very common pattern. We should allow a shortcut.

I'm a little concerned that we might be in the middle of constructing the VMA when we make the decision to set VM_MERGEABLE and miss one of the "bad" flags. I've sprinkled a few VM_BUG_ON()s to watch out for any cases where we've missed something. I turned this on for _every_ VMA to test it, and it hasn't blown up yet.

There are probably some other ways to do this. We could have prctl, or some kind of boot option, or even something analogous to the transparent-huge-page 'always' option (as opposed to madvise()). We could even extend madvise() for this kind of thing. We could allow MADV_MERGEABLE to be specified for unmapped areas in _advance_ for when brk() or mmap() is called on them.

Applying transactional memory to mmap_sem would probably also help out here a lot.

Cc [email protected]
Cc Andi Kleen <[email protected]>

---

 b/Documentation/vm/ksm.txt        |    5 +++-
 b/include/linux/ksm.h             |   13 +++++++++++
 b/include/uapi/asm-generic/mman.h |    1 
 b/mm/ksm.c                        |   43 +++++++++++++++++++++++---------------
 b/mm/mmap.c                       |   13 +++++++++++
 5 files changed, 58 insertions(+), 17 deletions(-)
include/uapi/asm-generic/mman.h renamed from include/uapi/asm-generic/mman.h~mmap-flag-for-ksm +1 -0
diff -puN include/uapi/asm-generic/mman.h~mmap-flag-for-ksm include/uapi/asm-generic/mman.h
@@ -12,6 +12,7 @@
 #define MAP_NONBLOCK	0x10000		/* do not block on IO */
 #define MAP_STACK	0x20000		/* give out an address that is best suited for process/thread stacks */
 #define MAP_HUGETLB	0x40000		/* create a huge page mapping */
+#define MAP_MERGEABLE	0x80000		/* mark mapping as mergeable by KSM */
 
 /* Bits [26:31] are reserved, see mman-common.h for MAP_HUGETLB usage */
mm/mmap.c renamed from mm/mmap.c~mmap-flag-for-ksm +13 -0
diff -puN mm/mmap.c~mmap-flag-for-ksm mm/mmap.c
@@ -36,6 +36,7 @@
 #include <linux/sched/sysctl.h>
 #include <linux/notifier.h>
 #include <linux/memory.h>
+#include <linux/ksm.h>
 
 #include <asm/uaccess.h>
 #include <asm/cacheflush.h>

@@ -1361,6 +1362,18 @@ unsigned long do_mmap_pgoff(struct file
 			vm_flags |= VM_NORESERVE;
 	}
 
+	/*
+	 * This *must* happen after all the other vm_flags have
+	 * been set, but before we make the decision about
+	 * whether this vma can be merged with another.
+	 */
+	if ((flags & MAP_MERGEABLE) && ksm_can_handle_vma(vm_flags)) {
+		int err = ksm_enter_if_new(mm);
+		if (err)
+			return err;
+		vm_flags |= VM_MERGEABLE;
+	}
+
 	addr = mmap_region(file, addr, len, vm_flags, pgoff);
 	if (!IS_ERR_VALUE(addr) &&
 	    ((vm_flags & VM_LOCKED) ||
mm/ksm.c renamed from mm/ksm.c~mmap-flag-for-ksm +27 -16

81 lines not on this page · raw message

include/linux/ksm.h renamed from include/linux/ksm.h~mmap-flag-for-ksm +13 -0
diff -puN include/linux/ksm.h~mmap-flag-for-ksm include/linux/ksm.h
@@ -29,6 +29,13 @@ static inline int ksm_fork(struct mm_str
 	return 0;
 }
 
+static inline int ksm_enter_if_new(struct mm_struct *mm)
+{
+	if (test_bit(!MMF_VM_MERGEABLE, &mm->flags))
+		return __ksm_enter(mm);
+	return 0;
+}
+
 static inline void ksm_exit(struct mm_struct *mm)
 {
 	if (test_bit(MMF_VM_MERGEABLE, &mm->flags))

@@ -75,6 +82,7 @@ struct page *ksm_might_need_to_copy(stru
 
 int rmap_walk_ksm(struct page *page, struct rmap_walk_control *rwc);
 void ksm_migrate_page(struct page *newpage, struct page *oldpage);
+int ksm_can_handle_vma(unsigned long vm_flags);
 
 #else  /* !CONFIG_KSM */
 

@@ -91,6 +99,11 @@ static inline int PageKsm(struct page *p
 {
 	return 0;
 }
+
+static inline int ksm_can_handle_vma(unsigned long vm_flags)
+{
+	return 0;
+}
 
 #ifdef CONFIG_MMU
 static inline int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
Documentation/vm/ksm.txt renamed from Documentation/vm/ksm.txt~mmap-flag-for-ksm +4 -1
diff -puN Documentation/vm/ksm.txt~mmap-flag-for-ksm Documentation/vm/ksm.txt
@@ -22,7 +22,10 @@ are swapped back in: ksmd must rediscove
 
 KSM only operates on those areas of address space which an application
 has advised to be likely candidates for merging, by using the madvise(2)
-system call: int madvise(addr, length, MADV_MERGEABLE).
+system call: int madvise(addr, length, MADV_MERGEABLE).  As a shortcut,
+an application may also specify MAP_MERGEABLE to the mmap() system call.
+This will have the same effect that calling mmap() followed by madvise()
+would have had.
 
 The app may call int madvise(addr, length, MADV_UNMERGEABLE) to cancel
 that advice and restore unshared pages: whereupon KSM unmerges whatever

_