Support volatile for anonymous range
Subsystems (17)
v5 cover Support volatile for anonymous range
This is still RFC because we need more input from user-space people, more stress test, design discussion about interface/reclaim policy of volatile pages and I want to expand this concept to tmpfs volatile range if it is possbile without big performance drop of anonymous volatile range.
(Let's define our term. anon volatile VS tmpfs volatile? John?)
I hope more inputs from user-space allocator people and test patch with their allocator because it might need design change of arena management for getting real vaule.
TODO
* Improve volatile range scanning speed
* Aware of NUMA policy with vma's mempolicy
* Add direct reclaim hook for discarding volatile pages first
* Support tmpfs-volatile
Changelog from v5 - There are many changes.
* Support CONFIG_VOLATILE_PAGE
* Working with THP/KSM
* Remove vma hacking logic in m[no]volatile system call
* Discard page without swap cache
* Kswapd discard volatile page so we can discard volatile pages although we don't have swap.
Changelog from v4
* Add new system call mvolatile/mnovolatile
* Add sigbus when user try to access volatile range
* Rebased on v3.7
* Applied bug fix from John Stultz, Thanks!
Changelog from v3
* Removing madvise(addr, length, MADV_NOVOLATILE).
* add vmstat about the number of discarded volatile pages
* discard volatile pages without promotion in reclaim path
This is based on v3.7
- What's the mvolatile(addr, length)?
It's a hint that user deliver to kernel so kernel can *discard* pages in a range anytime.
- What happens if user access page(ie, virtual address) discarded by kernel?
The user can encounter SIGBUS.
- What should user do for avoding SIGBUS?
He should call mnovolatie(addr, length) before accessing the range which was called by mvolatile.
- What happens if user access page(ie, virtual address) doesn't discarded by kernel?
The user can see old data without page fault.
- What's different with madvise(DONTNEED)?
System call semantic
DONTNEED makes sure user always can see zero-fill pages after he calls madvise while mvolatile can see old data or encounter SIGBUS.
Internal implementation
The madvise(DONTNEED) should zap all mapped pages in range so overhead is increased linearly with the number of mapped pages. Even, if user access zapped pages as write mode, page fault + page allocation + memset should be happened.
The mvolatile just marks the flag in a range(ie, VMA) instead of zapping all of pte in the vma so it doesn't touch ptes any more.
- What's the benefit compared to DONTNEED?
1. The system call overhead is smaller because mvolatile just marks the flag to VMA instead of zapping all the page in a range so overhead should be very small.
2. It has a chance to eliminate overheads (ex, zapping pte + page fault
+ page allocation + memset(PAGE_SIZE)) if memory pressure isn't severe.
3. It has a potential to zap all ptes and free the pages if memory pressure is severe so reclaim overhead could be disappear - TODO
- Isn't there any drawback?
Madvise(DONTNEED) doesn't need exclusive mmap_sem so concurrent page fault of other threads could be allowed. But m[no]volatile needs exclusive mmap_sem so other thread would be blocked if they try to access not-yet-mapped pages. That's why I design m[no]volatile overhead should be small as far as possible.
It could suffer from max rss usage increasement because madvise(DONTNEED) deallocates pages instantly when the system call is issued while mvoatile delays it until memory pressure happens so if memory pressure is severe by max rss incresement, system would suffer. First of all, allocator needs some balance logic for that or kernel might handle it by zapping pages although user calls mvolatile if memory pressure is severe. The problem is how we know memory pressure is severe.
One of solution is to see kswapd is active or not. Another solution is Anton's mempressure so allocator can handle it.
- What's for targetting?
Firstly, user-space allocator like ptmalloc, tcmalloc or heap management of virtual machine like Dalvik. Also, it comes in handy for embedded which doesn't have swap device so they can't reclaim anonymous pages. By discarding instead of swapout, it could be used in the non-swap system.
- Stupid performance test
I attach test program/script which are utter crap and I don't expect current smart allocator never have done it so we need more practical data with real allocator.
KVM - 8 core, 2G
VOLATILE test
13.16user 7.58system 0:06.04elapsed 343%CPU (0avgtext+0avgdata 2624096maxresident)k 0inputs+0outputs (0major+164050minor)pagefaults 0swaps
DONTNEED test
23.30user 228.92system 0:33.10elapsed 762%CPU (0avgtext+0avgdata 213088maxresident)k 0inputs+0outputs (0major+16384210minor)pagefaults 0swaps
x86-64 - 12 core, 2G
VOLATILE test
33.38user 0.44system 0:02.87elapsed 1178%CPU (0avgtext+0avgdata 3935008maxresident)k 0inputs+0outputs (0major+245989minor)pagefaults 0swaps
DONTNEED test
28.02user 41.25system 0:05.80elapsed 1192%CPU (0avgtext+0avgdata 387776maxresident)k
[1] 74e3f3c3, vmscan: prevent background aging of anon page in no swap system
Any comments are welcome!
Minchan Kim (8):
Introduce new system call mvolatile Don't allow volatile attribute on THP and KSM bail out when the page is in VOLATILE vma add page_locked parameter in free_swap_and_cache Discard volatile page
add PGVOLATILE vmstat count
add volatile page discard hook to kswapd extend PGVOLATILE vmstat to kswapd
arch/x86/syscalls/syscall_64.tbl | 2 + fs/exec.c | 4 +- include/linux/memory.h | 2 + include/linux/mm.h | 6 +- include/linux/mm_types.h | 4 + include/linux/mvolatile.h | 63 +++ include/linux/rmap.h | 2 + include/linux/sched.h | 1 + include/linux/swap.h | 6 +- include/linux/syscalls.h | 2 + include/linux/vm_event_item.h | 4 + kernel/fork.c | 2 + mm/Kconfig | 11 + mm/Makefile | 2 +- mm/fremap.c | 2 +- mm/huge_memory.c | 9 +- mm/internal.h | 2 + mm/ksm.c | 3 +- mm/madvise.c | 2 +- mm/memory.c | 12 +- mm/mempolicy.c | 2 +- mm/mlock.c | 7 +- mm/mmap.c | 62 ++- mm/mprotect.c | 3 +- mm/mremap.c | 2 +- mm/mvolatile.c | 813 ++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 11 +- mm/shmem.c | 2 +- mm/swapfile.c | 7 +- mm/vmscan.c | 57 ++- mm/vmstat.c | 4 + 31 files changed, 1065 insertions(+), 46 deletions(-) create mode 100644 include/linux/mvolatile.h create mode 100644 mm/mvolatile.c
================== 8< =============================
#define _GNU_SOURCE
#include <stdio.h>
#include <pthread.h>
#include <sched.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/syscall.h>
#define SYS_mvolatile 313
#define SYS_mnovolatile 314
#define ALLOC_SIZE (8 << 20)
#define MAP_SIZE (ALLOC_SIZE * 10)
#define PAGE_SIZE (1 << 12)
#define RETRY 100
pthread_barrier_t barrier;
int mode;
#define VOLATILE_MODE 1
static int mvolatile(void *addr, size_t length)
{
return syscall(SYS_mvolatile, addr, length);
}static int mnovolatile(void *addr, size_t length)
{
return syscall(SYS_mnovolatile, addr, length);
}void *thread_entry(void *data)
{
unsigned long i;
cpu_set_t set;
int cpu = *(int*)data;
void *mmap_area;
int retry = RETRY;
CPU_ZERO(&set);
CPU_SET(cpu, &set);
sched_setaffinity(0, sizeof(set), &set);
mmap(NULL, PAGE_SIZE, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0);
mmap_area = mmap(NULL, MAP_SIZE, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, 0, 0);
if (mmap_area == MAP_FAILED) {
fprintf(stderr, "Fail to mmap [%d]\n", *(int*)data);
exit(1);
}
pthread_barrier_wait(&barrier);
while(retry--) {
if (mode == VOLATILE_MODE) {
mvolatile(mmap_area, MAP_SIZE);
for (i = 0; i < MAP_SIZE; i+= ALLOC_SIZE) {
mnovolatile(mmap_area + i, ALLOC_SIZE);
memset(mmap_area + i, i, ALLOC_SIZE);
mvolatile(mmap_area + i, ALLOC_SIZE);
}
} else {
for (i = 0; i < MAP_SIZE; i += ALLOC_SIZE) {
memset(mmap_area + i, i, ALLOC_SIZE);
madvise(mmap_area + i, ALLOC_SIZE, MADV_DONTNEED);
}
}
}
return NULL;
}int main(int argc, char *argv[])
{
int i, nr_thread;
int *data;
if (argc < 3)
return 1;
nr_thread = atoi(argv[1]);
mode = atoi(argv[2]);
pthread_t *thread = malloc(sizeof(pthread_t) * nr_thread);
data = malloc(sizeof(int) * nr_thread);
pthread_barrier_init(&barrier, NULL, nr_thread);
for (i = 0; i < nr_thread; i++) {
data[i] = i;
if (pthread_create(&thread[i], NULL, thread_entry, &data[i])) {
perror("Fail to create thread\n");
exit(1);
}
}
for (i = 0; i < nr_thread; i++) {
if (pthread_join(thread[i], NULL))
perror("Fail to join thread\n");
printf("[%d] thread done\n", i);
}
return 0;
}Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 1/8 Introduce new system call mvolatile
This patch adds new system call m[no]volatile.
If someone asks is_volatile system call, it could be added, too.
The reason why I introduced new system call instead of madvise is m[no]volatile vma handling is totally different with madvise's vma handling.
1) The m[no]volatile should be successful although the range includes unmapped or non-volatile range. It just skips such range without stopping with returning error although it encounters invalid range. It makes user convenient without calling several system call of small range - Suggested by John Stultz
2) The purged state of volatile range should be propagated out to user although the range is merged with adjacent non-volatile range when user calls mnovolatile.
3) mvolatile's interface could be changed with madvise
in future discussion. For example, I feel needs movlatile(start, len, mode).
'mode' means FULL_VOLATILE or PARTIAL_VOLATILE.
FULL volatile means that if VM decide to reclaim the range, it would reclaim all of pages in the range but in case of PARTIAL_VOLATILE, VM could reclaim just a few number of pages in the range. In case of tmpfs-volatile, user may regenerate all images data once one of page in the range is discarded so there is pointless that VM discard a page in the range when memory pressure is severe. In case of anon-volatile, too excess discarding cause too many minor fault for the allocator so it would be better to discard part of the range.
3) The mvolatile system call's return value is quite different with madvise. Look at below semantic explanation.
So I want to separate mvolatile from madvise.
mvolatile(start, len)'s semantics
1) It makes range(start, len) as volatile although the range includes unmapped area, speacial mapping and mlocked area which are just skipped.
Return -EINVAL if range doesn't include a right vma at all. Return -ENOMEM with interrupting range opeartion if memory is not enough to merge/split vmas. In this case, some ranges would be volatile and others not so user may recall mvolatile after he cancel all range by mnovolatile.
Return 0 if range consists of only proper vmas.
Return 1 if part of range includes hole/huge/ksm/mlock/special area.
2) If user calls mvolatile to the range which was already volatile VMA and even purged state, VOLATILE attributes still remains but purged state is reset. I expect some user want to split volatile vma into smaller ranges. Although he can do it for mnovlatile(whole range) and serveral calling with movlatile(smaller range), this function can avoid mnovolatile if he doesn't care purged state. I'm not sure we really need this function so I hope listen opinions. Unfortunately, current implemenation doesn't split volatile VMA with new range in this case. I forgot implementing it in this version but decide to send it to listen opinions because implementing is rather trivial if we decided.
mnovolatile(start, len)'s semantics is following as.
1) It makes range(start, len) as non-volatile although the range includes unmapped area, speacial mapping and non-volatile range which are just skipped.
2) If the range is purged, it will return 1 regardless of including invalid range.
3) It returns -ENOMEM if system doesn't have enough memory for vma operation.
4) It returns -EINVAL if range doesn't include a right vma at all.
5) If user try to access purged range without mnovoatile call, it encounters SIGBUS which would show up next patch.
--- arch/x86/syscalls/syscall_64.tbl | 2 + fs/exec.c | 4 +- include/linux/mm.h | 6 +- include/linux/mm_types.h | 4 + include/linux/mvolatile.h | 30 ++++ include/linux/syscalls.h | 2 + mm/Kconfig | 11 ++ mm/Makefile | 2 +- mm/madvise.c | 2 +- mm/mempolicy.c | 2 +- mm/mlock.c | 7 +- mm/mmap.c | 62 ++++++-- mm/mprotect.c | 3 +- mm/mremap.c | 2 +- mm/mvolatile.c | 312 ++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 2 + 16 files changed, 427 insertions(+), 26 deletions(-) create mode 100644 include/linux/mvolatile.h create mode 100644 mm/mvolatile.c
arch/x86/syscalls/syscall_64.tbl +2 -0
@@ -319,6 +319,8 @@ 310 64 process_vm_readv sys_process_vm_readv 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp +313 common mvolatile sys_mvolatile +314 common mnovolatile sys_mnovolatile # # x32-specific system call numbers start at 512 to avoid cache impact
fs/exec.c +2 -2
@@ -594,7 +594,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)/* * cover the whole range: [new_start, old_end) */ -if (vma_adjust (vma, new_start, old_end, vma->vm_pgoff,NULL )) +if (vma_adjust (vma, new_start, old_end, vma->vm_pgoff,NULL ,NULL ))return -ENOMEM ;/*
@@ -628,7 +628,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)/* * Shrink the vma to just the new range. Always succeeds. */ -vma_adjust (vma, new_start, new_end, vma->vm_pgoff,NULL ); +vma_adjust (vma, new_start, new_end, vma->vm_pgoff,NULL ,NULL );return 0 ; }
include/linux/mm.h +4 -2
@@ -87,6 +87,7 @@ extern unsigned int kobjsize(const void *objp);#define VM_PFNMAP 0x00000400/* Page-ranges managed without "struct page", just pure PFN */ #define VM_DENYWRITE 0x00000800/* ETXTBSY on write attempts.. */ +#define VM_VOLATILE 0x00001000/* Pages could be discarede without swapout */ #define VM_LOCKED 0x00002000#define VM_IO 0x00004000/* Memory mapped I/O or similar */
@@ -1411,11 +1412,12 @@ void anon_vma_interval_tree_verify(struct anon_vma_chain *node);/* mmap.c */ extern int __vm_enough_memory (struct mm_struct *mm,long pages,int cap_sys_admin);extern int vma_adjust (struct vm_area_struct *vma,unsigned long start, -unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert); +unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert, +bool *purged);extern struct vm_area_struct *vma_merge (struct mm_struct *,struct vm_area_struct *prev,unsigned long addr,unsigned long end,unsigned long vm_flags,struct anon_vma *,struct file *,pgoff_t , -struct mempolicy *); +struct mempolicy *,bool *purged);extern struct anon_vma *find_mergeable_anon_vma (struct vm_area_struct *);extern int split_vma (struct mm_struct *,struct vm_area_struct *,unsigned long addr,int new_below);
include/linux/mm_types.h +4 -0
@@ -275,6 +275,10 @@ struct vm_area_struct {#ifdef CONFIG_NUMA struct mempolicy *vm_policy;/* NUMA policy for the VMA */ #endif +#ifdef CONFIG_VOLATILE_PAGE +/* True if more than a page in this vma is reclaimed. */ +bool purged;/* Serialized by mmap_sem and anon_vma's mutex */ +#endif };struct core_thread {
include/linux/mvolatile.h new file +30 -0
@@ -0,0 +1,30 @@ +#ifndef __LINUX_MVOLATILE_H +#define __LINUX_MVOLATILE_H + +#include <linux/syscalls.h> + +#ifdef CONFIG_VOLATILE_PAGE +static inline bool vma_purged (struct vm_area_struct *vma) +{ +return vma->purged; +} + +static inline void vma_purge_copy (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ + dst->purged = src->purged; +} +#else +static inline bool vma_purged (struct vm_area_struct *vma) +{ +return false; +} + +static inline void vma_purge_copy (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ + +} +#endif +#endif +
include/linux/syscalls.h +2 -0
@@ -470,6 +470,8 @@ asmlinkage long sys_munlock(unsigned long start, size_t len);asmlinkage long sys_mlockall (int flags);asmlinkage long sys_munlockall (void );asmlinkage long sys_madvise (unsigned long start,size_t len,int behavior); +asmlinkage long sys_mvolatile (unsigned long start,size_t len); +asmlinkage long sys_mnovolatile (unsigned long start,size_t len);asmlinkage long sys_mincore (unsigned long start,size_t len,unsigned char __user * vec);
mm/Kconfig +11 -0
@@ -355,6 +355,17 @@ choice benefit. endchoice +config VOLATILE_PAGE + bool "Volatile Page Support" + depends on MMU + help + Enabling this option adds the system calls mvolatile and mnovolatile + which are for giving user's address space range to kernel so VM + can discard pages of the range anytime instead swapout. This feature + can enhance performance to certain application(ex, memory allocator, + web browser's tmpfs pages) by reduce the number of minor fault and + swap out. + config CROSS_MEMORY_ATTACH bool "Cross Memory Support" depends on MMU
mm/Makefile +1 -1
@@ -5,7 +5,7 @@ mmu-y := nommu.o mmu-$(CONFIG_MMU) := fremap.o highmem.o madvise.o memory.o mincore.o \ mlock.o mmap.o mprotect.o mremap.o msync.o rmap.o \ - vmalloc.o pagewalk.o pgtable-generic.o + mvolatile.o vmalloc.o pagewalk.o pgtable-generic.o ifdef CONFIG_CROSS_MEMORY_ATTACH mmu-$(CONFIG_MMU) += process_vm_access.o
mm/madvise.c +1 -1
@@ -99,7 +99,7 @@ static long madvise_behavior(struct vm_area_struct * vma, pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, new_flags, vma->anon_vma, - vma->vm_file, pgoff,vma_policy (vma)); + vma->vm_file, pgoff,vma_policy (vma),NULL );if (*prev) { vma = *prev; goto success;
mm/mempolicy.c +1 -1
@@ -675,7 +675,7 @@ static int mbind_range(struct mm_struct *mm, unsigned long start, ((vmstart - vma->vm_start) >>PAGE_SHIFT ); prev =vma_merge (mm, prev, vmstart, vmend, vma->vm_flags, vma->anon_vma, vma->vm_file, pgoff, - new_pol); + new_pol,NULL );if (prev) { vma = prev; next = vma->vm_next;
mm/mlock.c +4 -3
@@ -316,13 +316,14 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,int ret =0 ;int lock = !!(newflags &VM_LOCKED ); -if (newflags == vma->vm_flags || (vma->vm_flags &VM_SPECIAL ) || -is_vm_hugetlb_page (vma) || vma ==get_gate_vma (current->mm)) +if (newflags == vma->vm_flags || (vma->vm_flags & + (VM_SPECIAL |VM_VOLATILE )) ||is_vm_hugetlb_page (vma) || + vma ==get_gate_vma (current->mm)) goto out;/* don't set VM_LOCKED, don't count */ pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, newflags, vma->anon_vma, - vma->vm_file, pgoff,vma_policy (vma)); + vma->vm_file, pgoff,vma_policy (vma),NULL );if (*prev) { vma = *prev; goto success;
mm/mmap.c +48 -14
153 lines not on this page · raw message
mm/mprotect.c +2 -1
@@ -179,7 +179,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,*/ pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *pprev =vma_merge (mm, *pprev, start, end, newflags, - vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma)); + vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), +NULL );if (*pprev) { vma = *pprev; goto success;
mm/mremap.c +1 -1
@@ -512,7 +512,7 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,int pages = (new_len - old_len) >>PAGE_SHIFT ;if (vma_adjust (vma, vma->vm_start, addr + new_len, - vma->vm_pgoff,NULL )) { + vma->vm_pgoff,NULL ,NULL )) { ret = -ENOMEM ; goto out; }
mm/mvolatile.c new file +312 -0
312 lines not on this page · raw message
mm/rmap.c +2 -0
@@ -57,6 +57,7 @@#include <linux/migrate.h> #include <linux/hugetlb.h> #include <linux/backing-dev.h> +#include <linux/mvolatile.h> #include <asm/tlbflush.h>
@@ -308,6 +309,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) vma->anon_vma = anon_vma;anon_vma_lock (anon_vma);anon_vma_chain_link (vma, avc, anon_vma); +vma_purge_copy (vma, pvma);anon_vma_unlock (anon_vma);return 0 ;
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 1/8] Introduce new system call mvolatile
Quoted 17 lines · Minchan Kim
This patch adds new system call m[no]volatile.
If someone asksis_volatilesystem call, it could be added, too.The reason why I introduced new system call instead of madvise is m[no]volatile vma handling is totally different with madvise's vma handling.
1) The m[no]volatile should be successful although the range includes unmapped or non-volatile range. It just skips such range without stopping with returning error although it encounters invalid range. It makes user convenient without calling several system call of small range - Suggested by John Stultz
2) The purged state of volatile range should be propagated out to user although the range is merged with adjacent non-volatile range when user calls mnovolatile.
Quoted 13 lines
3) mvolatile's interface could be changed with madvise
in future discussion. For example, I feel needsmovlatile(start, len, mode).
'mode' meansFULL_VOLATILEorPARTIAL_VOLATILE.
FULL volatile means that if VM decide to reclaim the range, it would reclaim all of pages in the range but in case ofPARTIAL_VOLATILE, VM could reclaim just a few number of pages in the range. In case of tmpfs-volatile, user may regenerate all images data once one of page in the range is discarded so there is pointless that VM discard a page in the range when memory pressure is severe. In case of anon-volatile, too excess discarding cause too many minor fault for the allocator so it would be better to discard part of the range.
I don't understand point 3).
Are you saying that using mvolatile in conjuction with madvise could allow mvolatile behavior to be tweaked in the future? Or are you suggesting adding an extra parameter in the future(what would that have to do with madvise)?
4) Having a new system call makes it easier for userspace apps to detect kernels without this functionality.
I really like the proposed interface. I like the suggestion of having explicit FULL|PARTIAL_VOLATILE. Why not include PARTIAL_VOLATILE as a required 3rd param in first version with expectation that FULL_VOLATILE will be added later(and returning some not-supported error in meantime)?
Quoted 30 lines
3) The mvolatile system call's return value is quite different with madvise. Look at below semantic explanation.
So I want to separate mvolatile from madvise.
mvolatile(start, len)'s semantics1) It makes
range(start, len)as volatile although the range includes unmapped area, speacial mapping and mlocked area which are just skipped.Return
-EINVALif range doesn't include a right vma at all. Return-ENOMEMwith interrupting range opeartion if memory is not enough to merge/split vmas. In this case, some ranges would be volatile and others not so user may recall mvolatile after he cancel all range by mnovolatile.
Return 0 if range consists of only proper vmas.
Return 1 if part of range includes hole/huge/ksm/mlock/special area.2) If user calls mvolatile to the range which was already volatile VMA and even purged state, VOLATILE attributes still remains but purged state is reset. I expect some user want to split volatile vma into smaller ranges. Although he can do it for mnovlatile(whole range) and serveral calling with movlatile(smaller range), this function can avoid mnovolatile if he doesn't care purged state. I'm not sure we really need this function so I hope listen opinions. Unfortunately, current implemenation doesn't split volatile VMA with new range in this case. I forgot implementing it in this version but decide to send it to listen opinions because implementing is rather trivial if we decided.
Quoted 8 lines
mnovolatile(start, len)'s semantics is following as.1) It makes
range(start, len)as non-volatile although the range includes unmapped area, speacial mapping and non-volatile range which are just skipped.2) If the range is purged, it will return 1 regardless of including invalid range.
If I understand this correctly:mvolatile(0, 10);
//then range [9,10] is purged by kernelmnovolatile(0,4) will fail?
that seems counterintuitive.
One of the uses for mnovolatile is to atomicly lock the pages(vs a racy proposed is_volatile) syscall. Above situation would make it less effective.
Quoted 788 lines
3) It returns
-ENOMEMif system doesn't have enough memory for vma operation.4) It returns
-EINVALif range doesn't include a right vma at all.5) If user try to access purged range without mnovoatile call, it encounters SIGBUS which would show up next patch.
Cc Michael Kerrisk <[email protected]>Cc Arun Sharma <[email protected]>Cc Paul Turner <[email protected]>CC David Rientjes <[email protected]>Cc John Stultz <[email protected]>Cc Andrew Morton <[email protected]>Cc Christoph Lameter <[email protected]>Cc Android Kernel Team <[email protected]>Cc Robert Love <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Dave Hansen <[email protected]>Cc Rik van Riel <[email protected]>Cc Dave Chinner <[email protected]>Cc Neil Brown <[email protected]>Cc Mike Hommey <[email protected]>Cc Taras Glek <[email protected]>Cc KOSAKI Motohiro <[email protected]>Cc KAMEZAWA Hiroyuki <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- arch/x86/syscalls/syscall_64.tbl | 2 + fs/exec.c | 4 +- include/linux/mm.h | 6 +- include/linux/mm_types.h | 4 + include/linux/mvolatile.h | 30 ++++ include/linux/syscalls.h | 2 + mm/Kconfig | 11 ++ mm/Makefile | 2 +- mm/madvise.c | 2 +- mm/mempolicy.c | 2 +- mm/mlock.c | 7 +- mm/mmap.c | 62 ++++++-- mm/mprotect.c | 3 +- mm/mremap.c | 2 +- mm/mvolatile.c | 312 ++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 2 + 16 files changed, 427 insertions(+), 26 deletions(-)create mode 100644
include/linux/mvolatile.h
create mode 100644mm/mvolatile.carch/x86/syscalls/syscall_64.tbl +2
-0@@ -319,6 +319,8 @@ 310 64 process_vm_readv sys_process_vm_readv 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp +313 common mvolatile sys_mvolatile +314 common mnovolatile sys_mnovolatile # # x32-specific system call numbers start at 512 to avoid cache impactfs/exec.c +2
-2@@ -594,7 +594,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)/* * cover the whole range: [new_start, old_end) */ -if (vma_adjust (vma, new_start, old_end, vma->vm_pgoff,NULL )) +if (vma_adjust (vma, new_start, old_end, vma->vm_pgoff,NULL ,NULL ))return -ENOMEM ;/* @@ -628,7 +628,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)/* * Shrink the vma to just the new range. Always succeeds. */ -vma_adjust (vma, new_start, new_end, vma->vm_pgoff,NULL ); +vma_adjust (vma, new_start, new_end, vma->vm_pgoff,NULL ,NULL );return 0 ; }include/linux/mm.h +4
-2@@ -87,6 +87,7 @@ extern unsigned int kobjsize(const void *objp);#define VM_PFNMAP 0x00000400/* Page-ranges managed without "struct page", just pure PFN */ #define VM_DENYWRITE 0x00000800/* ETXTBSY on write attempts.. */ +#define VM_VOLATILE 0x00001000/* Pages could be discarede without swapout */ #define VM_LOCKED 0x00002000#define VM_IO 0x00004000/* Memory mapped I/O or similar */ @@ -1411,11 +1412,12 @@ void anon_vma_interval_tree_verify(struct anon_vma_chain *node);/* mmap.c */ extern int __vm_enough_memory (struct mm_struct *mm,long pages,int cap_sys_admin);extern int vma_adjust (struct vm_area_struct *vma,unsigned long start, -unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert); +unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert, +bool *purged);extern struct vm_area_struct *vma_merge (struct mm_struct *,struct vm_area_struct *prev,unsigned long addr,unsigned long end,unsigned long vm_flags,struct anon_vma *,struct file *,pgoff_t , -struct mempolicy *); +struct mempolicy *,bool *purged);extern struct anon_vma *find_mergeable_anon_vma (struct vm_area_struct *);extern int split_vma (struct mm_struct *,struct vm_area_struct *,unsigned long addr,int new_below);include/linux/mm_types.h +4
-0@@ -275,6 +275,10 @@ struct vm_area_struct {#ifdef CONFIG_NUMA struct mempolicy *vm_policy;/* NUMA policy for the VMA */ #endif +#ifdef CONFIG_VOLATILE_PAGE +/* True if more than a page in this vma is reclaimed. */ +bool purged;/* Serialized by mmap_sem and anon_vma's mutex */ +#endif };struct core_thread {include/linux/mvolatile.h new file +30
-0@@ -0,0 +1,30 @@ +#ifndef __LINUX_MVOLATILE_H +#define __LINUX_MVOLATILE_H + +#include <linux/syscalls.h> + +#ifdef CONFIG_VOLATILE_PAGE +static inline bool vma_purged (struct vm_area_struct *vma) +{ +return vma->purged; +} + +static inline void vma_purge_copy (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ + dst->purged = src->purged; +} +#else +static inline bool vma_purged (struct vm_area_struct *vma) +{ +return false; +} + +static inline void vma_purge_copy (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ + +} +#endif +#endif +include/linux/syscalls.h +2
-0@@ -470,6 +470,8 @@ asmlinkage long sys_munlock(unsigned long start, size_t len);asmlinkage long sys_mlockall (int flags);asmlinkage long sys_munlockall (void );asmlinkage long sys_madvise (unsigned long start,size_t len,int behavior); +asmlinkage long sys_mvolatile (unsigned long start,size_t len); +asmlinkage long sys_mnovolatile (unsigned long start,size_t len);asmlinkage long sys_mincore (unsigned long start,size_t len,unsigned char __user * vec);mm/Kconfig +11
-0@@ -355,6 +355,17 @@ choice benefit. endchoice +config VOLATILE_PAGE + bool "Volatile Page Support" + depends on MMU + help + Enabling this option adds the system calls mvolatile and mnovolatile + which are for giving user's address space range to kernel so VM + can discard pages of the range anytime instead swapout. This feature + can enhance performance to certain application(ex, memory allocator, + web browser's tmpfs pages) by reduce the number of minor fault and + swap out. + config CROSS_MEMORY_ATTACH bool "Cross Memory Support" depends on MMUmm/Makefile +1
-1@@ -5,7 +5,7 @@ mmu-y := nommu.o mmu-$(CONFIG_MMU) := fremap.o highmem.o madvise.o memory.o mincore.o \ mlock.o mmap.o mprotect.o mremap.o msync.o rmap.o \ - vmalloc.o pagewalk.o pgtable-generic.o + mvolatile.o vmalloc.o pagewalk.o pgtable-generic.o ifdef CONFIG_CROSS_MEMORY_ATTACH mmu-$(CONFIG_MMU) += process_vm_access.omm/madvise.c +1
-1@@ -99,7 +99,7 @@ static long madvise_behavior(struct vm_area_struct * vma, pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, new_flags, vma->anon_vma, - vma->vm_file, pgoff,vma_policy (vma)); + vma->vm_file, pgoff,vma_policy (vma),NULL );if (*prev) { vma = *prev; goto success;mm/mempolicy.c +1
-1@@ -675,7 +675,7 @@ static int mbind_range(struct mm_struct *mm, unsigned long start, ((vmstart - vma->vm_start) >>PAGE_SHIFT ); prev =vma_merge (mm, prev, vmstart, vmend, vma->vm_flags, vma->anon_vma, vma->vm_file, pgoff, - new_pol); + new_pol,NULL );if (prev) { vma = prev; next = vma->vm_next;mm/mlock.c +4
-3@@ -316,13 +316,14 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,int ret =0 ;int lock = !!(newflags &VM_LOCKED ); -if (newflags == vma->vm_flags || (vma->vm_flags &VM_SPECIAL ) || -is_vm_hugetlb_page (vma) || vma ==get_gate_vma (current->mm)) +if (newflags == vma->vm_flags || (vma->vm_flags & + (VM_SPECIAL |VM_VOLATILE )) ||is_vm_hugetlb_page (vma) || + vma ==get_gate_vma (current->mm)) goto out;/* don't set VM_LOCKED, don't count */ pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, newflags, vma->anon_vma, - vma->vm_file, pgoff,vma_policy (vma)); + vma->vm_file, pgoff,vma_policy (vma),NULL );if (*prev) { vma = *prev; goto success;mm/mmap.c +48
-14@@ -31,6 +31,7 @@#include <linux/audit.h> #include <linux/khugepaged.h> #include <linux/uprobes.h> +#include <linux/mvolatile.h> #include <asm/uaccess.h> #include <asm/cacheflush.h> @@ -516,7 +517,8 @@ __vma_unlink(struct mm_struct *mm, struct vm_area_struct *vma,* before we drop the necessary locks. */ int vma_adjust (struct vm_area_struct *vma,unsigned long start, -unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert) +unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert, +bool *purged) {struct mm_struct *mm = vma->vm_mm;struct vm_area_struct *next = vma->vm_next;@@ -527,10 +529,9 @@ int vma_adjust(struct vm_area_struct *vma, unsigned long start,struct file *file = vma->vm_file;long adjust_next =0 ;int remove_next =0 ; +struct vm_area_struct *exporter =NULL ;if (next && !insert) { -struct vm_area_struct *exporter =NULL ; -if (end >= next->vm_end) {/* * vma expands, overlapping all the next, and @@ -621,6 +622,15 @@ again: remove_next = 1 + (end > next->vm_end);if (adjust_next) { next->vm_start += adjust_next <<PAGE_SHIFT ; next->vm_pgoff += adjust_next; +/* +* Look at mm/mvolatile.c for knowing terminology. +* V4. NNPPVV -> NNNPVV +*/ +if (purged) { + *purged =vma_purged (next); +if (exporter == vma)/* V5. VVPPNN -> VVPNNN */ + *purged =vma_purged (vma); + } }if (root) {@@ -651,6 +661,13 @@ again: remove_next = 1 + (end > next->vm_end);anon_vma_interval_tree_post_update_vma (vma);if (adjust_next)anon_vma_interval_tree_post_update_vma (next); +/* +* Look at mm/mvolatile.c for knowing terminology. +* V7. VVPPVV -> VVNPVV +* V8. VVPPVV -> VVPNVV +*/ +if (insert) +vma_purge_copy (insert, vma);anon_vma_unlock (anon_vma); }if (mapping)@@ -670,6 +687,20 @@ again: remove_next = 1 + (end > next->vm_end); }if (next->anon_vma)anon_vma_merge (vma, next); + +/* +* next is detatched from anon vma chain so purged isn't +* raced any more. +* Look at mm/mvolatile.c for knowing terminology. +* +* V1. NNPPVV -> NNNNVV +* V2. VVPPNN -> VVNNNN +* V3. NNPPNN -> NNNNNN +*/ +if (purged) { + *purged |=vma_purged (vma);/* case V2 */ + *purged |=vma_purged (next);/* case V1,V3 */ + } mm->map_count--;mpol_put (vma_policy (next));kmem_cache_free (vm_area_cachep, next);@@ -798,7 +829,7 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, struct vm_area_struct *prev, unsigned long addr, unsigned long end, unsigned long vm_flags, struct anon_vma *anon_vma, struct file *file, - pgoff_t pgoff, struct mempolicy *policy) + pgoff_t pgoff, struct mempolicy *policy, bool *purged) { pgoff_t pglen = (end - addr) >> PAGE_SHIFT;struct vm_area_struct *area, *next;@@ -808,7 +839,7 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm,* We later require that vma->vm_flags == vm_flags, * so this tests vma->vm_flags & VM_SPECIAL, too. */ -if (vm_flags &VM_SPECIAL ) +if (vm_flags & (VM_SPECIAL |VM_VOLATILE ))return NULL ;if (prev)@@ -837,10 +868,10 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, next->anon_vma, NULL)) {/* cases 1, 6 */ err =vma_adjust (prev, prev->vm_start, - next->vm_end, prev->vm_pgoff,NULL ); + next->vm_end, prev->vm_pgoff,NULL , purged); } else /* cases 2, 5, 7 */ err =vma_adjust (prev, prev->vm_start, - end, prev->vm_pgoff,NULL ); + end, prev->vm_pgoff,NULL , purged);if (err)return NULL ;khugepaged_enter_vma_merge (prev);@@ -856,10 +887,10 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, anon_vma, file, pgoff+pglen)) {if (prev && addr < prev->vm_end)/* case 4 */ err =vma_adjust (prev, prev->vm_start, - addr, prev->vm_pgoff,NULL ); + addr, prev->vm_pgoff,NULL , purged);else /* cases 3, 8 */ err =vma_adjust (area, addr, next->vm_end, - next->vm_pgoff - pglen,NULL ); + next->vm_pgoff - pglen,NULL , purged);if (err)return NULL ;khugepaged_enter_vma_merge (area);@@ -1292,7 +1323,8 @@ munmap_back:/* * Can we just expand an old mapping? */ - vma =vma_merge (mm, prev, addr, addr + len, vm_flags,NULL , file, pgoff,NULL ); + vma =vma_merge (mm, prev, addr, addr + len, vm_flags,NULL , file, + pgoff,NULL ,NULL );if (vma) goto out;@@ -2025,9 +2057,10 @@ static int __split_vma(struct mm_struct * mm, struct vm_area_struct * vma,if (new_below) err =vma_adjust (vma, addr, vma->vm_end, vma->vm_pgoff + - ((addr - new->vm_start) >>PAGE_SHIFT ), new); + ((addr - new->vm_start) >>PAGE_SHIFT ), new,NULL );else - err =vma_adjust (vma, vma->vm_start, addr, vma->vm_pgoff, new); + err =vma_adjust (vma, vma->vm_start, addr, vma->vm_pgoff, + new,NULL );/* Success. */ if (!err)@@ -2240,7 +2273,7 @@ static unsigned long do_brk(unsigned long addr, unsigned long len)/* Can we just expand an old private anonymous mapping? */ vma =vma_merge (mm, prev, addr, addr + len, flags, -NULL ,NULL , pgoff,NULL ); +NULL ,NULL , pgoff,NULL ,NULL );if (vma) goto out;@@ -2396,7 +2429,8 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,if (find_vma_links (mm, addr, addr + len, &prev, &rb_link, &rb_parent))return NULL ;/* should never get here */ new_vma =vma_merge (mm, prev, addr, addr + len, vma->vm_flags, - vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma)); + vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), +NULL );if (new_vma) {/* * Source vma may have been merged into new_vma mm/mprotect.c +2
-1@@ -179,7 +179,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,*/ pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *pprev =vma_merge (mm, *pprev, start, end, newflags, - vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma)); + vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), +NULL );if (*pprev) { vma = *pprev; goto success;mm/mremap.c +1
-1@@ -512,7 +512,7 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,int pages = (new_len - old_len) >>PAGE_SHIFT ;if (vma_adjust (vma, vma->vm_start, addr + new_len, - vma->vm_pgoff,NULL )) { + vma->vm_pgoff,NULL ,NULL )) { ret = -ENOMEM ; goto out; }mm/mvolatile.c new file +312
-0@@ -0,0 +1,312 @@ +/* +* linux/mm/mvolatile.c +* +* Copyright 2012 Minchan Kim +* +* This work is licensed under the terms of the GNU GPL, version 2. See +* the COPYING file in the top-level directory. +*/ + +#include <linux/mvolatile.h> +#include <linux/mm_types.h> +#include <linux/mm.h> +#include <linux/rmap.h> +#include <linux/mempolicy.h> + +#ifndef CONFIG_VOLATILE_PAGE +SYSCALL_DEFINE2(mnovolatile, unsigned long, start, size_t, len) +{ +return -EINVAL ; +} + +SYSCALL_DEFINE2(mvolatile, unsigned long, start, size_t, len) +{ +return -EINVAL ; +} +#else + +#define NO_PURGED 0 +#define PURGED 1 + +/* +* N: Normal VMA +* V: Volatile VMA +* P: Purged volatile VMA +* +* Assume that each VMA has two block so case 1-8 consists of three VMA. +* For example, NNPPVV means VMA1 has normal VMA, VMA2 has purged volailte VMA, +* and VMA3 has volatile VMA. With another example, NNPVVV means VMA1 has +* normal VMA, VMA2-1 has purged volatile VMA, VMA2-2 has volatile VMA. +* +* Case 7,8 create a new VMA and we call it VMA4 which can be loated before VMA2 +* or after. +* +* Notice: The merge between volatile VMAs shouldn't happen. +* If we call mnovolatile(VMA2), +* +* Case 1 NNPPVV -> NNNNVV +* Case 2 VVPPNN -> VVNNNN +* Case 3 NNPPNN -> NNNNNN +* Case 4 NNPPVV -> NNNPVV +* case 5 VVPPNN -> VVPNNN +* case 6 VVPPVV -> VVNNVV +* case 7 VVPPVV -> VVNPVV +* case 8 VVPPVV -> VVPNVV +*/ +static int do_mnovolatile (struct vm_area_struct *vma, +struct vm_area_struct **prev,unsigned long start, +unsigned long end,bool *is_purged) +{ +unsigned long new_flags; +int error =0 ; +struct mm_struct *mm = vma->vm_mm; +pgoff_t pgoff; +bool purged = false; + + new_flags = vma->vm_flags & ~VM_VOLATILE ; +if (new_flags == vma->vm_flags) { + *prev = vma; + goto success; + } + + + pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); + *prev =vma_merge (mm, *prev, start, end, new_flags, vma->anon_vma, + vma->vm_file, pgoff,vma_policy (vma), &purged); +if (*prev) { + vma = *prev; + goto success; + } + + *prev = vma; + +if (start != vma->vm_start) { + error =split_vma (mm, vma, start,1 ); +if (error) + goto out; + } + +if (end != vma->vm_end) { + error =split_vma (mm, vma, end,0 ); +if (error) + goto out; + } + +success: +/* V6. VVPPVV -> VVNNVV */ +vma_lock_anon_vma (vma); + *is_purged |= (vma->purged|purged); +vma_unlock_anon_vma (vma); + + vma->vm_flags = new_flags; + vma->purged = false; +return 0 ; +out: +return error; +} + +/* I didn't look into KSM/Hugepage so disalbed them */ +#define VM_NO_VOLATILE (VM_SPECIAL|VM_MIXEDMAP|VM_HUGETLB|\ + VM_MERGEABLE|VM_HUGEPAGE|VM_LOCKED) + +static int do_mvolatile (struct vm_area_struct *vma, +struct vm_area_struct **prev,unsigned long start,unsigned long end) +{ +int error = -EINVAL ; +vm_flags_t new_flags = vma->vm_flags; +struct mm_struct *mm = vma->vm_mm; + + new_flags |=VM_VOLATILE ; + +/* Note : Current version doesn't support file vma volatile */ +if (vma->vm_file) { + *prev = vma; + goto out; + } + +if (vma->vm_flags &VM_NO_VOLATILE || + (vma ==get_gate_vma (current->mm))) { + *prev = vma; + goto out; + } +/* +* In case of calling MADV_VOLATILE again, +* We just reset purged state. +*/ +if (new_flags == vma->vm_flags) { + *prev = vma; +vma_lock_anon_vma (vma); + vma->purged = false; +vma_unlock_anon_vma (vma); + error =0 ; + goto out; + } + + *prev = vma; + +if (start != vma->vm_start) { + error =split_vma (mm, vma, start,1 ); +if (error) + goto out; + } + +if (end != vma->vm_end) { + error =split_vma (mm, vma, end,0 ); +if (error) + goto out; + } + + error =0 ; + +vma_lock_anon_vma (vma); + vma->vm_flags = new_flags; +vma_unlock_anon_vma (vma); +out: +return error; +} + +/* +* Return -EINVAL if range doesn't include a right vma at all. +* Return -ENOMEM with interrupting range opeartion if memory is not enough to +* merge/split vmas. +* Return 0 if range consists of only proper vmas. +* Return 1 if part of range includes inavlid area(ex, hole/huge/ksm/mlock/ +* special area) +*/ +SYSCALL_DEFINE2(mvolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +bool invalid = false; +int error = -EINVAL ; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + invalid = true; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mvolatile (vma, &prev, start, tmp); +if (error == -ENOMEM ) { +up_write (¤t->mm->mmap_sem); +return error; + } +if (error == -EINVAL ) + invalid = true; +else + error =0 ; + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; + + vma = prev->vm_next; +if (!vma) +break ; + } +out: +up_write (¤t->mm->mmap_sem); +return invalid ?1 :0 ; +} +/* +* Return -ENOMEM with interrupting range opeartion if memory is not enough +* to merge/split vmas. +* Return 1 if part of range includes purged's one, otherwise, return 0 +*/ +SYSCALL_DEFINE2(mnovolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +int ret, error = -EINVAL ; +bool is_purged = false; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mnovolatile (vma, &prev, start, tmp, &is_purged); +if (error) { +WARN_ON (error != -ENOMEM ); + goto out; + } + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; + + vma = prev->vm_next; +if (!vma) +break ; + } +out: +up_write (¤t->mm->mmap_sem); + +if (error) + ret = error; +else if (is_purged) + ret =PURGED ; +else + ret =NO_PURGED ; + +return ret; +} +#endif mm/rmap.c +1
-0@@ -57,6 +57,7 @@#include <linux/migrate.h> #include <linux/hugetlb.h> #include <linux/backing-dev.h> +#include <linux/mvolatile.h> #include <asm/tlbflush.h>
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 315
Quoted 7 lines
mm/rmap.c +1
-0@@ -308,6 +309,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) vma->anon_vma = anon_vma;anon_vma_lock (anon_vma);anon_vma_chain_link (vma, avc, anon_vma); +vma_purge_copy (vma, pvma);anon_vma_unlock (anon_vma);return 0 ;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 1/8] Introduce new system call mvolatile
Hi,
Quoted 31 lines · Taras Glek
Quoted 17 lines · Minchan Kim
This patch adds new system call m[no]volatile.
If someone asksis_volatilesystem call, it could be added, too.The reason why I introduced new system call instead of madvise is m[no]volatile vma handling is totally different with madvise's vma handling.
1) The m[no]volatile should be successful although the range includes unmapped or non-volatile range. It just skips such range without stopping with returning error although it encounters invalid range. It makes user convenient without calling several system call of small range - Suggested by John Stultz
2) The purged state of volatile range should be propagated out to user although the range is merged with adjacent non-volatile range when user calls mnovolatile.
Quoted 13 lines
3) mvolatile's interface could be changed with madvise
in future discussion. For example, I feel needsmovlatile(start, len, mode).
'mode' meansFULL_VOLATILEorPARTIAL_VOLATILE.
FULL volatile means that if VM decide to reclaim the range, it would reclaim all of pages in the range but in case ofPARTIAL_VOLATILE, VM could reclaim just a few number of pages in the range. In case of tmpfs-volatile, user may regenerate all images data once one of page in the range is discarded so there is pointless that VM discard a page in the range when memory pressure is severe. In case of anon-volatile, too excess discarding cause too many minor fault for the allocator so it would be better to discard part of the range.
Quoted 5 lines
I don't understand point 3).
Are you saying that using mvolatile in conjuction with madvise could allow mvolatile behavior to be tweaked in the future? Or are you suggesting adding an extra parameter in the future(what would that have to do with madvise)?
I meant I might want to expand mvolatile's interface like below during discussion.
int mvolatile(start, len, mode);
Quoted 3 lines
4) Having a new system call makes it easier for userspace apps to detect kernels without this functionality.
I coudn't understand your claim.
Now mvolatile just return EINVAL on !CONFIG_VOLATILE_PAGE system. Why is it easy compared to returning EINVAL when we call madvise(VOLATILE) on !CONFIG_VOLATILE_PAGE?
Quoted 2 lines
I really like the proposed interface. I like the suggestion of
Thanks.
Quoted 4 lines
having explicit FULL|PARTIAL_VOLATILE. Why not include
PARTIAL_VOLATILEas a required 3rd param in first version with expectation thatFULL_VOLATILEwill be added later(and returning some not-supported error in meantime)?
I just wanted to discuss about needs of it.
The reason I need PARTIAL_VOLATILE is that avoids many minor fault for allocator. Is it useful for tmpfs-volatile, too?
Thanks for the feedback, Taras.
Quoted 825 lines
Quoted 30 lines
3) The mvolatile system call's return value is quite different with madvise. Look at below semantic explanation.
So I want to separate mvolatile from madvise.
mvolatile(start, len)'s semantics1) It makes
range(start, len)as volatile although the range includes unmapped area, speacial mapping and mlocked area which are just skipped.Return
-EINVALif range doesn't include a right vma at all. Return-ENOMEMwith interrupting range opeartion if memory is not enough to merge/split vmas. In this case, some ranges would be volatile and others not so user may recall mvolatile after he cancel all range by mnovolatile.
Return 0 if range consists of only proper vmas.
Return 1 if part of range includes hole/huge/ksm/mlock/special area.2) If user calls mvolatile to the range which was already volatile VMA and even purged state, VOLATILE attributes still remains but purged state is reset. I expect some user want to split volatile vma into smaller ranges. Although he can do it for mnovlatile(whole range) and serveral calling with movlatile(smaller range), this function can avoid mnovolatile if he doesn't care purged state. I'm not sure we really need this function so I hope listen opinions. Unfortunately, current implemenation doesn't split volatile VMA with new range in this case. I forgot implementing it in this version but decide to send it to listen opinions because implementing is rather trivial if we decided.
Quoted 8 lines
mnovolatile(start, len)'s semantics is following as.1) It makes
range(start, len)as non-volatile although the range includes unmapped area, speacial mapping and non-volatile range which are just skipped.2) If the range is purged, it will return 1 regardless of including invalid range.
If I understand this correctly:
mvolatile(0, 10);
//then range [9,10] is purged by kernelmnovolatile(0,4)will fail?
that seems counterintuitive.One of the uses for mnovolatile is to atomicly lock the
pages(vs a racy proposed is_volatile)syscall. Above situation would make it less effective.Quoted 764 lines
3) It returns
-ENOMEMif system doesn't have enough memory for vma operation.4) It returns
-EINVALif range doesn't include a right vma at all.5) If user try to access purged range without mnovoatile call, it encounters SIGBUS which would show up next patch.
Cc Michael Kerrisk <[email protected]>Cc Arun Sharma <[email protected]>Cc Paul Turner <[email protected]>CC David Rientjes <[email protected]>Cc John Stultz <[email protected]>Cc Andrew Morton <[email protected]>Cc Christoph Lameter <[email protected]>Cc Android Kernel Team <[email protected]>Cc Robert Love <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Dave Hansen <[email protected]>Cc Rik van Riel <[email protected]>Cc Dave Chinner <[email protected]>Cc Neil Brown <[email protected]>Cc Mike Hommey <[email protected]>Cc Taras Glek <[email protected]>Cc KOSAKI Motohiro <[email protected]>Cc KAMEZAWA Hiroyuki <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- arch/x86/syscalls/syscall_64.tbl | 2 + fs/exec.c | 4 +- include/linux/mm.h | 6 +- include/linux/mm_types.h | 4 + include/linux/mvolatile.h | 30 ++++ include/linux/syscalls.h | 2 + mm/Kconfig | 11 ++ mm/Makefile | 2 +- mm/madvise.c | 2 +- mm/mempolicy.c | 2 +- mm/mlock.c | 7 +- mm/mmap.c | 62 ++++++-- mm/mprotect.c | 3 +- mm/mremap.c | 2 +- mm/mvolatile.c | 312 ++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 2 + 16 files changed, 427 insertions(+), 26 deletions(-) create mode 100644 include/linux/mvolatile.h create mode 100644 mm/mvolatile.carch/x86/syscalls/syscall_64.tbl +2
-0@@ -319,6 +319,8 @@ 310 64 process_vm_readv sys_process_vm_readv 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp +313 common mvolatile sys_mvolatile +314 common mnovolatile sys_mnovolatile # # x32-specific system call numbers start at 512 to avoid cache impactfs/exec.c +2
-2@@ -594,7 +594,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)/* * cover the whole range: [new_start, old_end) */ -if (vma_adjust (vma, new_start, old_end, vma->vm_pgoff,NULL )) +if (vma_adjust (vma, new_start, old_end, vma->vm_pgoff,NULL ,NULL ))return -ENOMEM ;/* @@ -628,7 +628,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)/* * Shrink the vma to just the new range. Always succeeds. */ -vma_adjust (vma, new_start, new_end, vma->vm_pgoff,NULL ); +vma_adjust (vma, new_start, new_end, vma->vm_pgoff,NULL ,NULL );return 0 ; }include/linux/mm.h +4
-2@@ -87,6 +87,7 @@ extern unsigned int kobjsize(const void *objp);#define VM_PFNMAP 0x00000400/* Page-ranges managed without "struct page", just pure PFN */ #define VM_DENYWRITE 0x00000800/* ETXTBSY on write attempts.. */ +#define VM_VOLATILE 0x00001000/* Pages could be discarede without swapout */ #define VM_LOCKED 0x00002000#define VM_IO 0x00004000/* Memory mapped I/O or similar */ @@ -1411,11 +1412,12 @@ void anon_vma_interval_tree_verify(struct anon_vma_chain *node);/* mmap.c */ extern int __vm_enough_memory (struct mm_struct *mm,long pages,int cap_sys_admin);extern int vma_adjust (struct vm_area_struct *vma,unsigned long start, -unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert); +unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert, +bool *purged);extern struct vm_area_struct *vma_merge (struct mm_struct *,struct vm_area_struct *prev,unsigned long addr,unsigned long end,unsigned long vm_flags,struct anon_vma *,struct file *,pgoff_t , -struct mempolicy *); +struct mempolicy *,bool *purged);extern struct anon_vma *find_mergeable_anon_vma (struct vm_area_struct *);extern int split_vma (struct mm_struct *,struct vm_area_struct *,unsigned long addr,int new_below);include/linux/mm_types.h +4
-0@@ -275,6 +275,10 @@ struct vm_area_struct {#ifdef CONFIG_NUMA struct mempolicy *vm_policy;/* NUMA policy for the VMA */ #endif +#ifdef CONFIG_VOLATILE_PAGE +/* True if more than a page in this vma is reclaimed. */ +bool purged;/* Serialized by mmap_sem and anon_vma's mutex */ +#endif };struct core_thread {include/linux/mvolatile.h new file +30
-0@@ -0,0 +1,30 @@ +#ifndef __LINUX_MVOLATILE_H +#define __LINUX_MVOLATILE_H + +#include <linux/syscalls.h> + +#ifdef CONFIG_VOLATILE_PAGE +static inline bool vma_purged (struct vm_area_struct *vma) +{ +return vma->purged; +} + +static inline void vma_purge_copy (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ + dst->purged = src->purged; +} +#else +static inline bool vma_purged (struct vm_area_struct *vma) +{ +return false; +} + +static inline void vma_purge_copy (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ + +} +#endif +#endif +include/linux/syscalls.h +2
-0@@ -470,6 +470,8 @@ asmlinkage long sys_munlock(unsigned long start, size_t len);asmlinkage long sys_mlockall (int flags);asmlinkage long sys_munlockall (void );asmlinkage long sys_madvise (unsigned long start,size_t len,int behavior); +asmlinkage long sys_mvolatile (unsigned long start,size_t len); +asmlinkage long sys_mnovolatile (unsigned long start,size_t len);asmlinkage long sys_mincore (unsigned long start,size_t len,unsigned char __user * vec);mm/Kconfig +11
-0@@ -355,6 +355,17 @@ choice benefit. endchoice +config VOLATILE_PAGE + bool "Volatile Page Support" + depends on MMU + help + Enabling this option adds the system calls mvolatile and mnovolatile + which are for giving user's address space range to kernel so VM + can discard pages of the range anytime instead swapout. This feature + can enhance performance to certain application(ex, memory allocator, + web browser's tmpfs pages) by reduce the number of minor fault and + swap out. + config CROSS_MEMORY_ATTACH bool "Cross Memory Support" depends on MMUmm/Makefile +1
-1@@ -5,7 +5,7 @@ mmu-y := nommu.o mmu-$(CONFIG_MMU) := fremap.o highmem.o madvise.o memory.o mincore.o \ mlock.o mmap.o mprotect.o mremap.o msync.o rmap.o \ - vmalloc.o pagewalk.o pgtable-generic.o + mvolatile.o vmalloc.o pagewalk.o pgtable-generic.o ifdef CONFIG_CROSS_MEMORY_ATTACH mmu-$(CONFIG_MMU) += process_vm_access.omm/madvise.c +1
-1@@ -99,7 +99,7 @@ static long madvise_behavior(struct vm_area_struct * vma, pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, new_flags, vma->anon_vma, - vma->vm_file, pgoff,vma_policy (vma)); + vma->vm_file, pgoff,vma_policy (vma),NULL );if (*prev) { vma = *prev; goto success;mm/mempolicy.c +1
-1@@ -675,7 +675,7 @@ static int mbind_range(struct mm_struct *mm, unsigned long start, ((vmstart - vma->vm_start) >>PAGE_SHIFT ); prev =vma_merge (mm, prev, vmstart, vmend, vma->vm_flags, vma->anon_vma, vma->vm_file, pgoff, - new_pol); + new_pol,NULL );if (prev) { vma = prev; next = vma->vm_next;mm/mlock.c +4
-3@@ -316,13 +316,14 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,int ret =0 ;int lock = !!(newflags &VM_LOCKED ); -if (newflags == vma->vm_flags || (vma->vm_flags &VM_SPECIAL ) || -is_vm_hugetlb_page (vma) || vma ==get_gate_vma (current->mm)) +if (newflags == vma->vm_flags || (vma->vm_flags & + (VM_SPECIAL |VM_VOLATILE )) ||is_vm_hugetlb_page (vma) || + vma ==get_gate_vma (current->mm)) goto out;/* don't set VM_LOCKED, don't count */ pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *prev =vma_merge (mm, *prev, start, end, newflags, vma->anon_vma, - vma->vm_file, pgoff,vma_policy (vma)); + vma->vm_file, pgoff,vma_policy (vma),NULL );if (*prev) { vma = *prev; goto success;mm/mmap.c +48
-14@@ -31,6 +31,7 @@#include <linux/audit.h> #include <linux/khugepaged.h> #include <linux/uprobes.h> +#include <linux/mvolatile.h> #include <asm/uaccess.h> #include <asm/cacheflush.h> @@ -516,7 +517,8 @@ __vma_unlink(struct mm_struct *mm, struct vm_area_struct *vma,* before we drop the necessary locks. */ int vma_adjust (struct vm_area_struct *vma,unsigned long start, -unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert) +unsigned long end,pgoff_t pgoff,struct vm_area_struct *insert, +bool *purged) {struct mm_struct *mm = vma->vm_mm;struct vm_area_struct *next = vma->vm_next;@@ -527,10 +529,9 @@ int vma_adjust(struct vm_area_struct *vma, unsigned long start,struct file *file = vma->vm_file;long adjust_next =0 ;int remove_next =0 ; +struct vm_area_struct *exporter =NULL ;if (next && !insert) { -struct vm_area_struct *exporter =NULL ; -if (end >= next->vm_end) {/* * vma expands, overlapping all the next, and @@ -621,6 +622,15 @@ again: remove_next = 1 + (end > next->vm_end);if (adjust_next) { next->vm_start += adjust_next <<PAGE_SHIFT ; next->vm_pgoff += adjust_next; +/* +* Look at mm/mvolatile.c for knowing terminology. +* V4. NNPPVV -> NNNPVV +*/ +if (purged) { + *purged =vma_purged (next); +if (exporter == vma)/* V5. VVPPNN -> VVPNNN */ + *purged =vma_purged (vma); + } }if (root) {@@ -651,6 +661,13 @@ again: remove_next = 1 + (end > next->vm_end);anon_vma_interval_tree_post_update_vma (vma);if (adjust_next)anon_vma_interval_tree_post_update_vma (next); +/* +* Look at mm/mvolatile.c for knowing terminology. +* V7. VVPPVV -> VVNPVV +* V8. VVPPVV -> VVPNVV +*/ +if (insert) +vma_purge_copy (insert, vma);anon_vma_unlock (anon_vma); }if (mapping)@@ -670,6 +687,20 @@ again: remove_next = 1 + (end > next->vm_end); }if (next->anon_vma)anon_vma_merge (vma, next); + +/* +* next is detatched from anon vma chain so purged isn't +* raced any more. +* Look at mm/mvolatile.c for knowing terminology. +* +* V1. NNPPVV -> NNNNVV +* V2. VVPPNN -> VVNNNN +* V3. NNPPNN -> NNNNNN +*/ +if (purged) { + *purged |=vma_purged (vma);/* case V2 */ + *purged |=vma_purged (next);/* case V1,V3 */ + } mm->map_count--;mpol_put (vma_policy (next));kmem_cache_free (vm_area_cachep, next);@@ -798,7 +829,7 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, struct vm_area_struct *prev, unsigned long addr, unsigned long end, unsigned long vm_flags, struct anon_vma *anon_vma, struct file *file, - pgoff_t pgoff, struct mempolicy *policy) + pgoff_t pgoff, struct mempolicy *policy, bool *purged) { pgoff_t pglen = (end - addr) >> PAGE_SHIFT;struct vm_area_struct *area, *next;@@ -808,7 +839,7 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm,* We later require that vma->vm_flags == vm_flags, * so this tests vma->vm_flags & VM_SPECIAL, too. */ -if (vm_flags &VM_SPECIAL ) +if (vm_flags & (VM_SPECIAL |VM_VOLATILE ))return NULL ;if (prev)@@ -837,10 +868,10 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, next->anon_vma, NULL)) {/* cases 1, 6 */ err =vma_adjust (prev, prev->vm_start, - next->vm_end, prev->vm_pgoff,NULL ); + next->vm_end, prev->vm_pgoff,NULL , purged); } else /* cases 2, 5, 7 */ err =vma_adjust (prev, prev->vm_start, - end, prev->vm_pgoff,NULL ); + end, prev->vm_pgoff,NULL , purged);if (err)return NULL ;khugepaged_enter_vma_merge (prev);@@ -856,10 +887,10 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, anon_vma, file, pgoff+pglen)) {if (prev && addr < prev->vm_end)/* case 4 */ err =vma_adjust (prev, prev->vm_start, - addr, prev->vm_pgoff,NULL ); + addr, prev->vm_pgoff,NULL , purged);else /* cases 3, 8 */ err =vma_adjust (area, addr, next->vm_end, - next->vm_pgoff - pglen,NULL ); + next->vm_pgoff - pglen,NULL , purged);if (err)return NULL ;khugepaged_enter_vma_merge (area);@@ -1292,7 +1323,8 @@ munmap_back:/* * Can we just expand an old mapping? */ - vma =vma_merge (mm, prev, addr, addr + len, vm_flags,NULL , file, pgoff,NULL ); + vma =vma_merge (mm, prev, addr, addr + len, vm_flags,NULL , file, + pgoff,NULL ,NULL );if (vma) goto out;@@ -2025,9 +2057,10 @@ static int __split_vma(struct mm_struct * mm, struct vm_area_struct * vma,if (new_below) err =vma_adjust (vma, addr, vma->vm_end, vma->vm_pgoff + - ((addr - new->vm_start) >>PAGE_SHIFT ), new); + ((addr - new->vm_start) >>PAGE_SHIFT ), new,NULL );else - err =vma_adjust (vma, vma->vm_start, addr, vma->vm_pgoff, new); + err =vma_adjust (vma, vma->vm_start, addr, vma->vm_pgoff, + new,NULL );/* Success. */ if (!err)@@ -2240,7 +2273,7 @@ static unsigned long do_brk(unsigned long addr, unsigned long len)/* Can we just expand an old private anonymous mapping? */ vma =vma_merge (mm, prev, addr, addr + len, flags, -NULL ,NULL , pgoff,NULL ); +NULL ,NULL , pgoff,NULL ,NULL );if (vma) goto out;@@ -2396,7 +2429,8 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,if (find_vma_links (mm, addr, addr + len, &prev, &rb_link, &rb_parent))return NULL ;/* should never get here */ new_vma =vma_merge (mm, prev, addr, addr + len, vma->vm_flags, - vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma)); + vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), +NULL );if (new_vma) {/* * Source vma may have been merged into new_vma mm/mprotect.c +2
-1@@ -179,7 +179,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,*/ pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); *pprev =vma_merge (mm, *pprev, start, end, newflags, - vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma)); + vma->anon_vma, vma->vm_file, pgoff,vma_policy (vma), +NULL );if (*pprev) { vma = *pprev; goto success;mm/mremap.c +1
-1@@ -512,7 +512,7 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,int pages = (new_len - old_len) >>PAGE_SHIFT ;if (vma_adjust (vma, vma->vm_start, addr + new_len, - vma->vm_pgoff,NULL )) { + vma->vm_pgoff,NULL ,NULL )) { ret = -ENOMEM ; goto out; }mm/mvolatile.c new file +312
-0@@ -0,0 +1,312 @@ +/* +* linux/mm/mvolatile.c +* +* Copyright 2012 Minchan Kim +* +* This work is licensed under the terms of the GNU GPL, version 2. See +* the COPYING file in the top-level directory. +*/ + +#include <linux/mvolatile.h> +#include <linux/mm_types.h> +#include <linux/mm.h> +#include <linux/rmap.h> +#include <linux/mempolicy.h> + +#ifndef CONFIG_VOLATILE_PAGE +SYSCALL_DEFINE2(mnovolatile, unsigned long, start, size_t, len) +{ +return -EINVAL ; +} + +SYSCALL_DEFINE2(mvolatile, unsigned long, start, size_t, len) +{ +return -EINVAL ; +} +#else + +#define NO_PURGED 0 +#define PURGED 1 + +/* +* N: Normal VMA +* V: Volatile VMA +* P: Purged volatile VMA +* +* Assume that each VMA has two block so case 1-8 consists of three VMA. +* For example, NNPPVV means VMA1 has normal VMA, VMA2 has purged volailte VMA, +* and VMA3 has volatile VMA. With another example, NNPVVV means VMA1 has +* normal VMA, VMA2-1 has purged volatile VMA, VMA2-2 has volatile VMA. +* +* Case 7,8 create a new VMA and we call it VMA4 which can be loated before VMA2 +* or after. +* +* Notice: The merge between volatile VMAs shouldn't happen. +* If we call mnovolatile(VMA2), +* +* Case 1 NNPPVV -> NNNNVV +* Case 2 VVPPNN -> VVNNNN +* Case 3 NNPPNN -> NNNNNN +* Case 4 NNPPVV -> NNNPVV +* case 5 VVPPNN -> VVPNNN +* case 6 VVPPVV -> VVNNVV +* case 7 VVPPVV -> VVNPVV +* case 8 VVPPVV -> VVPNVV +*/ +static int do_mnovolatile (struct vm_area_struct *vma, +struct vm_area_struct **prev,unsigned long start, +unsigned long end,bool *is_purged) +{ +unsigned long new_flags; +int error =0 ; +struct mm_struct *mm = vma->vm_mm; +pgoff_t pgoff; +bool purged = false; + + new_flags = vma->vm_flags & ~VM_VOLATILE ; +if (new_flags == vma->vm_flags) { + *prev = vma; + goto success; + } + + + pgoff = vma->vm_pgoff + ((start - vma->vm_start) >>PAGE_SHIFT ); + *prev =vma_merge (mm, *prev, start, end, new_flags, vma->anon_vma, + vma->vm_file, pgoff,vma_policy (vma), &purged); +if (*prev) { + vma = *prev; + goto success; + } + + *prev = vma; + +if (start != vma->vm_start) { + error =split_vma (mm, vma, start,1 ); +if (error) + goto out; + } + +if (end != vma->vm_end) { + error =split_vma (mm, vma, end,0 ); +if (error) + goto out; + } + +success: +/* V6. VVPPVV -> VVNNVV */ +vma_lock_anon_vma (vma); + *is_purged |= (vma->purged|purged); +vma_unlock_anon_vma (vma); + + vma->vm_flags = new_flags; + vma->purged = false; +return 0 ; +out: +return error; +} + +/* I didn't look into KSM/Hugepage so disalbed them */ +#define VM_NO_VOLATILE (VM_SPECIAL|VM_MIXEDMAP|VM_HUGETLB|\ + VM_MERGEABLE|VM_HUGEPAGE|VM_LOCKED) + +static int do_mvolatile (struct vm_area_struct *vma, +struct vm_area_struct **prev,unsigned long start,unsigned long end) +{ +int error = -EINVAL ; +vm_flags_t new_flags = vma->vm_flags; +struct mm_struct *mm = vma->vm_mm; + + new_flags |=VM_VOLATILE ; + +/* Note : Current version doesn't support file vma volatile */ +if (vma->vm_file) { + *prev = vma; + goto out; + } + +if (vma->vm_flags &VM_NO_VOLATILE || + (vma ==get_gate_vma (current->mm))) { + *prev = vma; + goto out; + } +/* +* In case of calling MADV_VOLATILE again, +* We just reset purged state. +*/ +if (new_flags == vma->vm_flags) { + *prev = vma; +vma_lock_anon_vma (vma); + vma->purged = false; +vma_unlock_anon_vma (vma); + error =0 ; + goto out; + } + + *prev = vma; + +if (start != vma->vm_start) { + error =split_vma (mm, vma, start,1 ); +if (error) + goto out; + } + +if (end != vma->vm_end) { + error =split_vma (mm, vma, end,0 ); +if (error) + goto out; + } + + error =0 ; + +vma_lock_anon_vma (vma); + vma->vm_flags = new_flags; +vma_unlock_anon_vma (vma); +out: +return error; +} + +/* +* Return -EINVAL if range doesn't include a right vma at all. +* Return -ENOMEM with interrupting range opeartion if memory is not enough to +* merge/split vmas. +* Return 0 if range consists of only proper vmas. +* Return 1 if part of range includes inavlid area(ex, hole/huge/ksm/mlock/ +* special area) +*/ +SYSCALL_DEFINE2(mvolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +bool invalid = false; +int error = -EINVAL ; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + invalid = true; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mvolatile (vma, &prev, start, tmp); +if (error == -ENOMEM ) { +up_write (¤t->mm->mmap_sem); +return error; + } +if (error == -EINVAL ) + invalid = true; +else + error =0 ; + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; + + vma = prev->vm_next; +if (!vma) +break ; + } +out: +up_write (¤t->mm->mmap_sem); +return invalid ?1 :0 ; +} +/* +* Return -ENOMEM with interrupting range opeartion if memory is not enough +* to merge/split vmas. +* Return 1 if part of range includes purged's one, otherwise, return 0 +*/ +SYSCALL_DEFINE2(mnovolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +int ret, error = -EINVAL ; +bool is_purged = false; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mnovolatile (vma, &prev, start, tmp, &is_purged); +if (error) { +WARN_ON (error != -ENOMEM ); + goto out; + } + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; + + vma = prev->vm_next; +if (!vma) +break ; + } +out: +up_write (¤t->mm->mmap_sem); + +if (error) + ret = error; +else if (is_purged) + ret =PURGED ; +else + ret =NO_PURGED ; + +return ret; +} +#endif mm/rmap.c +1
-0@@ -57,6 +57,7 @@#include <linux/migrate.h> #include <linux/hugetlb.h> #include <linux/backing-dev.h> +#include <linux/mvolatile.h> #include <asm/tlbflush.h> mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 314
Quoted 6 lines
mm/rmap.c +1
-0@@ -308,6 +309,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) vma->anon_vma = anon_vma;anon_vma_lock (anon_vma);anon_vma_chain_link (vma, avc, anon_vma); +vma_purge_copy (vma, pvma);anon_vma_unlock (anon_vma);return 0 ;--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 1/8] Introduce new system call mvolatile
In reply to Minchan KimQuoted 2 lines · Minchan Kim
This patch adds new system call m[no]volatile.
If someone asksis_volatilesystem call, it could be added, too.
So some nits below from my initial playing around with this patchset.
Quoted 62 lines
mm/mvolatile.c new file +62
-0+/* +* Return -EINVAL if range doesn't include a right vma at all. +* Return -ENOMEM with interrupting range opeartion if memory is not enough to +* merge/split vmas. +* Return 0 if range consists of only proper vmas. +* Return 1 if part of range includes inavlid area(ex, hole/huge/ksm/mlock/ +* special area) +*/ +SYSCALL_DEFINE2(mvolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +bool invalid = false; +int error = -EINVAL ; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + invalid = true; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mvolatile (vma, &prev, start, tmp); +if (error == -ENOMEM ) { +up_write (¤t->mm->mmap_sem); +return error; + } +if (error == -EINVAL ) + invalid = true; +else + error =0 ; + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end)
mm/mvolatile.c line 239
Quoted 10 lines
mm/mvolatile.c new file +10
-0+break ; + + vma = prev->vm_next; +if (!vma) +break ; + } +out: +up_write (¤t->mm->mmap_sem); +return invalid ?1 :0 ; +}
The error logic here is really strange. If any of the early error cases are triggered (ie: (start & ~PAGE_MASK), etc), then we jump to out and return 0 (instead of EINVAL). I don't think that's what you intended.
Quoted 50 lines
mm/mvolatile.c new file +50
-0+/* +* Return -ENOMEM with interrupting range opeartion if memory is not enough +* to merge/split vmas. +* Return 1 if part of range includes purged's one, otherwise, return 0 +*/ +SYSCALL_DEFINE2(mnovolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +int ret, error = -EINVAL ; +bool is_purged = false; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mnovolatile (vma, &prev, start, tmp, &is_purged); +if (error) { +WARN_ON (error != -ENOMEM ); + goto out; + }
mm/mvolatile.c line 299
Quoted 10 lines
mm/mvolatile.c new file +10
-0+ start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; + + vma = prev->vm_next; +if (!vma) +break ; + }
I'm still not sure how this logic improves over the madvise case. If we catch an error mid-way through setting a series of vmas to non-volatile, we end up exiting and losing state (ie: if only the first vma was purged, but half way through 10 vmas we get a ENOMEM error. So the first vma is now non-volatile, but we do not return the purged flag ).
If we're going to have a new syscall for this (which I'm not sure is the right approach), we should make use of multiple arguments so we can return if data was purged, even if we hit an error midway).
Alternatively, if we can find a way to allocate any necessary memory before we do any vma volatility state changes, then we can return ENOMEM then and be confident we won't end up with failed partial state change (this is the approach I used in my fallocate-volatile patches).
thanks
-john
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 1/8] Introduce new system call mvolatile
Quoted 75 lines · John Stultz
Quoted 2 lines · Minchan Kim
This patch adds new system call m[no]volatile.
If someone asksis_volatilesystem call, it could be added, too.So some nits below from my initial playing around with this patchset.
Quoted 69 lines
mm/mvolatile.c new file +69
-0+/* +* Return -EINVAL if range doesn't include a right vma at all. +* Return -ENOMEM with interrupting range opeartion if memory is not enough to +* merge/split vmas. +* Return 0 if range consists of only proper vmas. +* Return 1 if part of range includes inavlid area(ex, hole/huge/ksm/mlock/ +* special area) +*/ +SYSCALL_DEFINE2(mvolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +bool invalid = false; +int error = -EINVAL ; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + invalid = true; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mvolatile (vma, &prev, start, tmp); +if (error == -ENOMEM ) { +up_write (¤t->mm->mmap_sem); +return error; + } +if (error == -EINVAL ) + invalid = true; +else + error =0 ; + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; + + vma = prev->vm_next; +if (!vma) +break ; + } +out:
Quoted 8 lines
mm/mvolatile.c line 239
Quoted 3 lines
mm/mvolatile.c new file +3
-0+up_write (¤t->mm->mmap_sem); +return invalid ?1 :0 ; +}The error logic here is really strange. If any of the early error cases are triggered (ie: (start & ~PAGE_MASK), etc), then we jump to out and return 0 (instead of EINVAL). I don't think that's what you intended.
Need fixing.
Quoted 61 lines
Quoted 61 lines
mm/mvolatile.c new file +59
-0+/* +* Return -ENOMEM with interrupting range opeartion if memory is not enough +* to merge/split vmas. +* Return 1 if part of range includes purged's one, otherwise, return 0 +*/ +SYSCALL_DEFINE2(mnovolatile, unsigned long, start, size_t, len) +{ +unsigned long end, tmp; +struct vm_area_struct *vma, *prev; +int ret, error = -EINVAL ; +bool is_purged = false; + +down_write (¤t->mm->mmap_sem); +if (start & ~PAGE_MASK ) + goto out; + + len &=PAGE_MASK ; +if (!len) + goto out; + + end = start + len; +if (end < start) + goto out; + + vma =find_vma_prev (current->mm, start, &prev); +if (!vma) + goto out; + +if (start > vma->vm_start) + prev = vma; + +for (;;) { +/* Here start < (end|vma->vm_end). */ +if (start < vma->vm_start) { + start = vma->vm_start; +if (start >= end) + goto out; + } + +/* Here vma->vm_start <= start < (end|vma->vm_end) */ + tmp = vma->vm_end; +if (end < tmp) + tmp = end; + +/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ + error =do_mnovolatile (vma, &prev, start, tmp, &is_purged); +if (error) { +WARN_ON (error != -ENOMEM ); + goto out; + } + start = tmp; +if (prev && start < prev->vm_end) + start = prev->vm_end; +if (start >= end) +break ; + + vma = prev->vm_next; +if (!vma) +break ;
Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ }I'm still not sure how this logic improves over the madvise case. If we catch an error mid-way through setting a series of vmas to non-volatile, we end up exiting and losing state (ie: if only the first vma was purged, but half way through 10 vmas we get a ENOMEM error. So the first vma is now non-volatile, but we do not return the purged flag ).
Right.
Quoted 4 lines
If we're going to have a new syscall for this (which I'm not sure is the right approach), we should make use of multiple arguments so we can return if data was purged, even if we hit an error midway).
It would be easier method to achieve our goal than below suggestion in case of VMA-basd approach because it's hard to expect how many we need vmas with atomically.
Will do it in next version.
Quoted 6 lines
Alternatively, if we can find a way to allocate any necessary memory before we do any vma volatility state changes, then we can return ENOMEM then and be confident we won't end up with failed partial state change (this is the approach I used in my fallocate-volatile patches).
Thanks for the review, John.
Quoted 9 lines
thanks
-john--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 2/8 Don't allow volatile attribute on THP and KSM
In reply to Minchan KimVOLATILE imply the the pages in the range isn't working set any more so it's pointless that make them to THP/KSM.
--- mm/huge_memory.c | 9 +++++++-- mm/ksm.c | 3 ++- 2 files changed, 9 insertions(+), 3 deletions(-)
mm/huge_memory.c +7 -2
@@ -1477,7 +1477,8 @@ out:return ret; } -#define VM_NO_THP (VM_SPECIAL|VM_MIXEDMAP|VM_HUGETLB|VM_SHARED|VM_MAYSHARE) +#define VM_NO_THP (VM_SPECIAL|VM_MIXEDMAP|VM_HUGETLB|\ + VM_SHARED|VM_MAYSHARE|VM_VOLATILE)int hugepage_madvise (struct vm_area_struct *vma,unsigned long *vm_flags,int advice)
@@ -1641,6 +1642,8 @@ int khugepaged_enter_vma_merge(struct vm_area_struct *vma)* page fault if needed. */ return 0 ; +if (vma->vm_flags &VM_VOLATILE ) +return 0 ;if (vma->vm_ops)/* khugepaged not yet working on file or special mappings */ return 0 ;
@@ -1969,6 +1972,8 @@ static void collapse_huge_page(struct mm_struct *mm, goto out;if (is_vma_temporary_stack (vma)) goto out; +if (vma->vm_flags &VM_VOLATILE ) + goto out;VM_BUG_ON (vma->vm_flags &VM_NO_THP ); pgd =pgd_offset (mm, address);
@@ -2196,7 +2201,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages,if ((!(vma->vm_flags &VM_HUGEPAGE ) && !khugepaged_always ()) || - (vma->vm_flags &VM_NOHUGEPAGE )) { + (vma->vm_flags & (VM_NOHUGEPAGE |VM_VOLATILE ))) { skip: progress++;continue ;
mm/ksm.c +2 -1
@@ -1486,7 +1486,8 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start,*/ if (*vm_flags & (VM_MERGEABLE |VM_SHARED |VM_MAYSHARE |VM_PFNMAP |VM_IO |VM_DONTEXPAND | -VM_HUGETLB |VM_NONLINEAR |VM_MIXEDMAP )) +VM_HUGETLB |VM_NONLINEAR |VM_MIXEDMAP | +VM_VOLATILE ))return 0 ;/* just ignore the advice */ #ifdef VM_SAO
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 2/8] Don't allow volatile attribute on THP and KSM
Quoted 2 lines · Minchan Kim
VOLATILE imply the the pages in the range isn't working set any more so it's pointless that make them to THP/KSM.
One of the points of this implementation is that it be able to preserve memory contents when there is no pressure. If those contents happen to contain a THP/KSM page, and there's no pressure, it seems like the right thing to do is to leave that memory in place.
It might be a fair thing to do this in order to keep the implementation more sane at the moment. But, we should make sure there's some good text on that in the changelog.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 2/8] Don't allow volatile attribute on THP and KSM
Quoted 8 lines · Dave Hansen
Quoted 2 lines · Minchan Kim
VOLATILE imply the the pages in the range isn't working set any more so it's pointless that make them to THP/KSM.
One of the points of this implementation is that it be able to preserve memory contents when there is no pressure. If those contents happen to contain a THP/KSM page, and there's no pressure, it seems like the right thing to do is to leave that memory in place.
Indeed. I should have written more cleary,
Current implementation is following as
1. madvised-THP/KSM(1, 10) -> mvolatile(1, 10) -> fail
2. mvolatile(1, 10) -> madvised-THP/KSM(1, 10) -> fail
3. always-THP -> mvolatile -> success -> if memory pressure happens -> split_huge_page -> discard.
I think 2,3 makes sense to me but we need to fix 1 in further patches.
Quoted 4 lines
It might be a fair thing to do this in order to keep the implementation more sane at the moment. But, we should make sure there's some good text on that in the changelog.
Absolutely, Thanks for pointing out, Dave.
Quoted 6 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 3/8 bail out when the page is in VOLATILE vma
In reply to Minchan KimIf we found a page is in VOLATILE vma, hurry up discarding instead of access bit check because it's very unlikey working set.
Next patch will use it.
--- mm/rmap.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
mm/rmap.c +4 -2
@@ -695,10 +695,12 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,if (!pte) goto out; -if (vma->vm_flags &VM_LOCKED ) { +if ((vma->vm_flags &VM_LOCKED ) || + (vma->vm_flags &VM_VOLATILE )) {pte_unmap_unlock (pte, ptl); *mapcount =0 ;/* break early from loop */ - *vm_flags |=VM_LOCKED ; + *vm_flags |= (vma->vm_flags &VM_LOCKED ? +VM_LOCKED :VM_VOLATILE ); goto out; }
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 4/8 add page_locked parameter in free_swap_and_cache
In reply to Minchan KimAdd page_locked parameter for avoiding trylock_page. Next patch will use it.
--- include/linux/swap.h | 6 +++--- mm/fremap.c | 2 +- mm/memory.c | 2 +- mm/shmem.c | 2 +- mm/swapfile.c | 7 ++++--- 5 files changed, 10 insertions(+), 9 deletions(-)
include/linux/swap.h +3 -3
@@ -357,7 +357,7 @@ extern int swap_duplicate(swp_entry_t);extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t );extern void swapcache_free (swp_entry_t ,struct page *page); -extern int free_swap_and_cache (swp_entry_t ); +extern int free_swap_and_cache (swp_entry_t ,bool );extern int swap_type_of (dev_t ,sector_t ,struct block_device **);extern unsigned int count_swap_pages (int ,int );extern sector_t map_swap_page (struct page *,struct block_device **);
@@ -397,8 +397,8 @@ static inline void show_swap_cache_info(void) { } -#define free_swap_and_cache (swp) is_migration_entry(swp) -#define swapcache_prepare (swp) is_migration_entry(swp) +#define free_swap_and_cache (swp, page_locked) is_migration_entry(swp) +#define swapcache_prepare (swp) is_migration_entry(swp)static inline int add_swap_count_continuation (swp_entry_t swp,gfp_t gfp_mask) {
mm/fremap.c +1 -1
@@ -44,7 +44,7 @@ static void zap_pte(struct mm_struct *mm, struct vm_area_struct *vma, } }else {if (!pte_file (pte)) -free_swap_and_cache (pte_to_swp_entry (pte)); +free_swap_and_cache (pte_to_swp_entry (pte), false);pte_clear_not_present_full (mm, addr, ptep,0 ); } }
mm/memory.c +1 -1
@@ -1198,7 +1198,7 @@ again:else rss[MM_FILEPAGES ]--; } -if (unlikely (!free_swap_and_cache (entry))) +if (unlikely (!free_swap_and_cache (entry, false)))print_bad_pte (vma, addr, ptent,NULL ); }pte_clear_not_present_full (mm, addr, pte, tlb->fullmm);
mm/shmem.c +1 -1
@@ -391,7 +391,7 @@ static int shmem_free_swap(struct address_space *mapping, error =shmem_radix_tree_replace (mapping, index, radswap,NULL );spin_unlock_irq (&mapping->tree_lock);if (!error) -free_swap_and_cache (radix_to_swp_entry (radswap)); +free_swap_and_cache (radix_to_swp_entry (radswap), false);return error; }
mm/swapfile.c +4 -3
@@ -688,7 +688,7 @@ int try_to_free_swap(struct page *page)* Free the swap entry like above, but also try to * free the page cache entry if it is the last user. */ -int free_swap_and_cache (swp_entry_t entry) +int free_swap_and_cache (swp_entry_t entry,bool page_locked) {struct swap_info_struct *p;struct page *page =NULL ;
@@ -700,7 +700,7 @@ int free_swap_and_cache(swp_entry_t entry)if (p) {if (swap_entry_free (p, entry,1 ) ==SWAP_HAS_CACHE ) { page =find_get_page (&swapper_space, entry.val); -if (page && !trylock_page (page)) { +if (page && !page_locked && !trylock_page (page)) {page_cache_release (page); page =NULL ; }
@@ -717,7 +717,8 @@ int free_swap_and_cache(swp_entry_t entry)delete_from_swap_cache (page);SetPageDirty (page); } -unlock_page (page); +if (!page_locked) +unlock_page (page);page_cache_release (page); }return p !=NULL ;
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 5/8 Discard volatile page
In reply to Minchan KimVM don't need to swap out volatile pages. Instead, it just discards pages and set true to the vma's purge state so if user try to access purged vma without calling mnovolatile, it will encounter SIGBUS.
--- include/linux/memory.h | 2 + include/linux/mvolatile.h | 20 +++++ include/linux/rmap.h | 2 + mm/memory.c | 10 ++- mm/mvolatile.c | 185 ++++++++++++++++++++++++++++++++++++++++++++- mm/rmap.c | 3 +- mm/vmscan.c | 13 ++++ 7 files changed, 230 insertions(+), 5 deletions(-)
include/linux/memory.h +2 -0
@@ -150,5 +150,7 @@ struct memory_accessor {* can sleep. */ extern struct mutex text_mutex; +void print_bad_pte (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct page *page);#endif /* _LINUX_MEMORY_H_ */
include/linux/mvolatile.h +20 -0
@@ -2,8 +2,15 @@#define __LINUX_MVOLATILE_H#include <linux/syscalls.h> +#include <linux/rmap.h> #ifdef CONFIG_VOLATILE_PAGE + +static inline bool is_volatile_vma (struct vm_area_struct *vma) +{ +return vma->vm_flags &VM_VOLATILE ; +} +static inline bool vma_purged (struct vm_area_struct *vma) {return vma->purged;
@@ -14,6 +21,8 @@ static inline void vma_purge_copy(struct vm_area_struct *dst, { dst->purged = src->purged; } + +int discard_volatile_page (struct page *page,enum ttu_flags ttu_flags);#else static inline bool vma_purged (struct vm_area_struct *vma) {
@@ -25,6 +34,17 @@ static inline void vma_purge_copy(struct vm_area_struct *dst, { } + +static inline int discard_volatile_page (struct page *page, +enum ttu_flags ttu_flags) +{ +return 0 ; +} + +static inline bool is_volatile_vma (struct vm_area_struct *vma) +{ +return false; +}#endif #endif
include/linux/rmap.h +2 -0
@@ -223,6 +223,7 @@ int try_to_munlock(struct page *);struct anon_vma *page_lock_anon_vma (struct page *page);void page_unlock_anon_vma (struct anon_vma *anon_vma);int page_mapped_in_vma (struct page *page,struct vm_area_struct *vma); +unsigned long vma_address (struct page *page,struct vm_area_struct *vma);/* * Called by migrate.c to remove migration ptes, but might be used more later.
@@ -244,6 +245,7 @@ static inline int page_referenced(struct page *page, int is_locked,return 0 ; } +#define try_to_unmap (page, refs) SWAP_FAILstatic inline int page_mkclean (struct page *page)
mm/memory.c +8 -2
@@ -57,6 +57,7 @@#include <linux/swapops.h> #include <linux/elf.h> #include <linux/gfp.h> +#include <linux/mvolatile.h> #include <asm/io.h> #include <asm/pgalloc.h>
@@ -655,7 +656,7 @@ static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss)* * The calling function must still handle the error. */ -static void print_bad_pte (struct vm_area_struct *vma,unsigned long addr, +void print_bad_pte (struct vm_area_struct *vma,unsigned long addr,pte_t pte,struct page *page) {pgd_t *pgd =pgd_offset (vma->vm_mm, addr);
@@ -3459,6 +3460,8 @@ int handle_pte_fault(struct mm_struct *mm,return do_linear_fault (mm, vma, address, pte, pmd, flags, entry); } +if (unlikely (is_volatile_vma (vma))) +return VM_FAULT_SIGBUS ;return do_anonymous_page (mm, vma, address, pte, pmd, flags); }
@@ -3528,9 +3531,12 @@ retry:if (!pmd)return VM_FAULT_OOM ;if (pmd_none (*pmd) &&transparent_hugepage_enabled (vma)) { -if (!vma->vm_ops) +if (!vma->vm_ops) { +if (unlikely (is_volatile_vma (vma))) +return VM_FAULT_SIGBUS ;return do_huge_pmd_anonymous_page (mm, vma, address, pmd, flags); + } }else {pmd_t orig_pmd = *pmd;int ret;
mm/mvolatile.c +184 -1
198 lines not on this page · raw message
mm/rmap.c +1 -2
@@ -525,8 +525,7 @@ __vma_address(struct page *page, struct vm_area_struct *vma)return vma->vm_start + ((pgoff - vma->vm_pgoff) <<PAGE_SHIFT ); } -inline unsigned long -vma_address (struct page *page,struct vm_area_struct *vma) +unsigned long vma_address (struct page *page,struct vm_area_struct *vma) {unsigned long address =__vma_address (page, vma);
mm/vmscan.c +13 -0
@@ -42,6 +42,7 @@#include <linux/sysctl.h> #include <linux/oom.h> #include <linux/prefetch.h> +#include <linux/mvolatile.h> #include <asm/tlbflush.h> #include <asm/div64.h>
@@ -609,6 +610,7 @@ redo:enum page_references {PAGEREF_RECLAIM ,PAGEREF_RECLAIM_CLEAN , +PAGEREF_DISCARD ,PAGEREF_KEEP ,PAGEREF_ACTIVATE , };
@@ -627,9 +629,16 @@ static enum page_references page_check_references(struct page *page,* Mlock lost the isolation race with us. Let try_to_unmap() * move the page to the unevictable list. */ + +VM_BUG_ON ((vm_flags & (VM_LOCKED |VM_VOLATILE )) == + (VM_LOCKED |VM_VOLATILE )); +if (vm_flags &VM_LOCKED )return PAGEREF_RECLAIM ; +if (vm_flags &VM_VOLATILE ) +return PAGEREF_DISCARD ; +if (referenced_ptes) {if (PageSwapBacked (page))return PAGEREF_ACTIVATE ;
@@ -768,6 +777,10 @@ static unsigned long shrink_page_list(struct list_head *page_list, goto activate_locked;case PAGEREF_KEEP : goto keep_locked; +case PAGEREF_DISCARD : +if (discard_volatile_page (page, ttu_flags)) + goto free_it; +break ;case PAGEREF_RECLAIM :case PAGEREF_RECLAIM_CLEAN : ;/* try to reclaim the page below */
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 6/8 add PGVOLATILE vmstat count
In reply to Minchan KimThis patch add pgvolatile vmstat so admin can see how many of volatile pages are discarded by VM until now.
--- include/linux/vm_event_item.h | 3 +++ mm/mvolatile.c | 1 + mm/vmstat.c | 3 +++ 3 files changed, 7 insertions(+)
include/linux/vm_event_item.h +3 -0
@@ -25,6 +25,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,FOR_ALL_ZONES (PGALLOC ),PGFREE ,PGACTIVATE ,PGDEACTIVATE ,PGFAULT ,PGMAJFAULT , +#ifdef CONFIG_VOLATILE_PAGE +PGVOLATILE , +#endifFOR_ALL_ZONES (PGREFILL ),FOR_ALL_ZONES (PGSTEAL_KSWAPD ), FOR_ALL_ZONES(PGSTEAL_DIRECT),
mm/mvolatile.c +1 -0
@@ -201,6 +201,7 @@ int discard_volatile_page(struct page *page, enum ttu_flags ttu_flags)if (try_to_volatile_page (page, ttu_flags)) {if (page_freeze_refs (page,1 )) {unlock_page (page); +count_vm_event (PGVOLATILE );return 1 ; } }
mm/vmstat.c +3 -0
@@ -753,6 +753,9 @@ const char * const vmstat_text[] = {"pgfault" ,"pgmajfault" , +#ifdef CONFIG_VOLATILE_PAGE +"pgvolatile" , +#endif TEXTS_FOR_ZONES("pgrefill") TEXTS_FOR_ZONES("pgsteal_kswapd")TEXTS_FOR_ZONES ("pgsteal_direct" )
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 7/8 add volatile page discard hook to kswapd
In reply to Minchan KimThis patch adds volatile page discard hook to kswapd for minimizing eviction of working set and enable discarding volatile page although we don't turn on swap.
This patch is copied heavily from THP.
--- include/linux/mvolatile.h | 13 ++ include/linux/sched.h | 1 + kernel/fork.c | 2 + mm/internal.h | 2 + mm/mvolatile.c | 314 +++++++++++++++++++++++++++++++++++++++++++++ mm/vmscan.c | 44 ++++++- 6 files changed, 374 insertions(+), 2 deletions(-)
include/linux/mvolatile.h +13 -0
@@ -23,6 +23,9 @@ static inline void vma_purge_copy(struct vm_area_struct *dst, }int discard_volatile_page (struct page *page,enum ttu_flags ttu_flags); +unsigned int discard_volatile_pages (struct zone *zone,unsigned int nr_pages); +void mvolatile_exit (struct mm_struct *mm); +#else static inline bool vma_purged (struct vm_area_struct *vma) {
@@ -45,6 +48,16 @@ static inline bool is_volatile_vma(struct vm_area_struct *vma) {return false; } + +static inline unsigned int discard_volatile_pages (struct zone *zone, +unsigned int nr_pages) +{ +return 0 ; +} + +static inline void mvolatile_exit (struct mm_struct *mm) +{ +}#endif #endif
include/linux/sched.h +1 -0
@@ -408,6 +408,7 @@ extern int get_dumpable(struct mm_struct *mm);#define MMF_HAS_UPROBES 19/* has uprobes */ #define MMF_RECALC_UPROBES 20/* MMF_HAS_UPROBES can be wrong */ +#define MMF_VM_VOLATILE 21/* set when VM_VOLATILE is set on vma */ #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK)
kernel/fork.c +2 -0
@@ -70,6 +70,7 @@#include <linux/khugepaged.h> #include <linux/signalfd.h> #include <linux/uprobes.h> +#include <linux/mvolatile.h> #include <asm/pgtable.h> #include <asm/pgalloc.h>
@@ -612,6 +613,7 @@ void mmput(struct mm_struct *mm)uprobe_clear_state (mm);exit_aio (mm);ksm_exit (mm); +mvolatile_exit (mm);khugepaged_exit (mm);/* must run before exit_mmap */ exit_mmap (mm);set_mm_exe_file (mm,NULL );
mm/internal.h +2 -0
@@ -351,6 +351,8 @@ extern unsigned long vm_mmap_pgoff(struct file *, unsigned long, unsigned long, unsigned long);extern void set_pageblock_order (void ); +unsigned long discard_volatile_page_list (struct zone *zone, +struct list_head *page_list);unsigned long reclaim_clean_pages_from_list (struct zone *zone,struct list_head *page_list);/* The ALLOC_WMARK bits are used as an index to zone->watermark */
mm/mvolatile.c +314 -0
338 lines not on this page · raw message
mm/vmscan.c +42 -2
62 lines not on this page · raw message
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC v5 8/8 extend PGVOLATILE vmstat to kswapd
In reply to Minchan KimNow kswapd can discard volatile pages so let's cover it for vmstat.
--- include/linux/vm_event_item.h | 3 ++- mm/mvolatile.c | 5 ++++- mm/vmstat.c | 3 ++- 3 files changed, 8 insertions(+), 3 deletions(-)
include/linux/vm_event_item.h +2 -1
@@ -26,7 +26,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,PGFREE ,PGACTIVATE ,PGDEACTIVATE ,PGFAULT ,PGMAJFAULT , #ifdef CONFIG_VOLATILE_PAGE -PGVOLATILE , +PGVOLATILE_DIRECT , +PGVOLATILE_KSWAPD , #endifFOR_ALL_ZONES (PGREFILL ), FOR_ALL_ZONES(PGSTEAL_KSWAPD),
mm/mvolatile.c +4 -1
@@ -246,7 +246,10 @@ int discard_volatile_page(struct page *page, enum ttu_flags ttu_flags)if (try_to_volatile_page (page, ttu_flags)) {if (page_freeze_refs (page,1 )) {unlock_page (page); -count_vm_event (PGVOLATILE ); +if (current_is_kswapd ()) +count_vm_event (PGVOLATILE_KSWAPD ); +else +count_vm_event (PGVOLATILE_DIRECT );return 1 ; } }
mm/vmstat.c +2 -1
@@ -754,7 +754,8 @@ const char * const vmstat_text[] = {"pgmajfault" , #ifdef CONFIG_VOLATILE_PAGE -"pgvolatile" , +"pgvolatile_direct" , +"pgvolatile_kswapd" , #endif TEXTS_FOR_ZONES("pgrefill")TEXTS_FOR_ZONES ("pgsteal_kswapd" )
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 2 lines · Minchan Kim
This is still RFC because we need more input from user-space people, more stress test, design discussion about interface/reclaim
Speaking as one of the authors of tcmalloc, I don't see any particular need for this new system call for tcmalloc. We are fine using madvise(MADV_DONTNEED) and don't notice any significant performance issues caused by it. Background: we throttle how quickly we release memory back to the system (1-10MB/s), so we do not call madvise() very much, and we don't end up reusing madvise-ed away pages at a fast rate. My guess is that we won't see large enough application-level performance improvements to cause us to change tcmalloc to use this system call.
Quoted 7 lines
- What's different with madvise(DONTNEED)?
System call semantic
DONTNEED makes sure user always can see zero-fill pages after he calls madvise while mvolatile can see old data or encounter SIGBUS.
Do you need a new system call for this? Why not just a new flag to madvise with weaker guarantees than zero-filling? All of the implementation changes you point out below could be triggered from that flag.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hello,
Quoted 11 lines · Sanjay Ghemawat
Quoted 2 lines · Minchan Kim
This is still RFC because we need more input from user-space people, more stress test, design discussion about interface/reclaim
Speaking as one of the authors of tcmalloc, I don't see any particular need for this new system call for tcmalloc. We are fine using
madvise(MADV_DONTNEED)and don't notice any significant performance issues caused by it. Background: we throttle how quickly we release memory back to the system (1-10MB/s), so we do not callmadvise()very much, and we don't end up reusing madvise-ed away pages at a fast rate. My guess is that we won't
It means TCmalloc controls madvise's rate dynamically without user's intervention? Smart TCmalloc!
Let me ask some questions.
What is your policy for control of throttling of madvise? I guess policy is following as.
The madvise's frequent calling is bad because pte zap overhead of madvise + next page fault/memset + page access bit emulatation page fault in some architecture like ARM when reused the range. So we should call it fast rate only when memory pressure happens very carefully. Is it similar to your throttling logic?
If my assumption isn't totally wrong, how could a process know the memory pressure at the moment by just per-process view, NOT system view?
If your logic takes some mistake, (for instace, memory pressure is severe but it doesn't call madvise) working set could be reclaimed like file-backed pages, which could minimize your benefit via madvise throttling. I guess it's very fragile. It's more severe in embedded world because they don't use swap so system encounters OOM instead of swappout.
In this point, mvolatile's concept is light weight system call by just mark the flag in the vma and auto free when system suffers from memory pressure(about this, my plan is zap all pages if kswapd is active when movlatile system call is called) by reclaimer with preventing working set page eviction, otherwise enhance application's speed with removing (minor fault + page allocation + memset). Also, it would make allocator simple through removing control logic, which is less error-prone and even might make smart TCmalloc better than now althoug it doesn't have any significat performance issue.
Quoted 7 lines
see large enough application-level performance improvements to cause us to change tcmalloc to use this system call.
Quoted 4 lines
- What's different with madvise(DONTNEED)?
System call semantic
Quoted 7 lines
Quoted 3 lines
DONTNEED makes sure user always can see zero-fill pages after he calls madvise while mvolatile can see old data or encounter SIGBUS.
Do you need a new system call for this? Why not just a new flag to madvise with weaker guarantees than zero-filling? All of the implementation changes you point out below could be triggered from that flag.
Agreed and actually, I tried it but changed my mind because it required adding many hacky codes in madvise due to return value and error's semantic is totally different with normal madvise and needs three flags at least at the moment but not sure we need more flags during discussion.
MADV_VOLATILE, MADV_NOVOLATILE, MADV_[NO]VOLATILE|MADV_PARTIAL_DISCARD
I don't want to make madvise dirty and consume lots of new flags of madvise for a volatile feature. But if everybody want to fold into madivse, I can do it, too.
Thanks for the feedback, Sanjay!
Quoted 6 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>