madvise(MADV_USERFAULT) & sys_remap_anon_pages()
Subsystems (13)
cover madvise(MADV_USERFAULT) & sys_remap_anon_pages()
Hello everyone,
this is a patchset to implement two new kernel features:MADV_USERFAULT and remap_anon_pages.
The combination of the two features are what I would propose to implement postcopy live migration, and in general demand paging of remote memory, hosted in different cloud nodes with KSM. It might also be used without virt to offload parts of memory to different nodes using some userland library and a network memory manager.
Postcopy live migration is currently implemented using a chardevice, which remains open for the whole VM lifetime and all virtual memory then becomes owned by the chardevice and it's not anonymous anymore.
http://lists.gnu.org/archive/html/qemu-devel/2012-10/msg05274.html
The main cons of the chardevice design is that all nice Linux MM features (like swapping/THP/KSM/automatic-NUMA-balancing) are disabled if the guest physical memory doesn't remain in anonymous memory. This is entirely solved by this alternative kernel solution. In fact remap_anon_pages will move THP pages natively by just updating two pmd pointers if alignment and length permits without any THP split.
The other bonus is that MADV_USERFAULT and remap_anon_pages are implemented in the MM core and remap_anon_pages furthermore provides a functionality similar to what is already available for filebacked pages with remap_file_pages. That is usually more maintainable than having MM parts in a chardevice.
In addition to asking review of the internals, this also need review the user APIs, as both those features are userland visible changes.
MADV_USERFAULT is only enabled for anonymous mappings so far but it could be extended. To be strict, -EINVAL is returned if run on non anonymous mappings (where it would currently be a noop).
The remap_anon_pages syscall API is not vectored, as I expect it used for demand paging only (where there can be just one faulting range per fault) or for large ranges where vectoring isn't going to provide performance advantages.
The current behavior of remap_anon_pages is very strict to avoid any chance of memory corruption going unnoticed, and it will return -EFAULT at the first sign of something unexpected (like a page already mapped in the destination pmd/pte, potentially signaling an userland thread race condition with two threads userfaulting on the same destination address). mremap is not strict like that: it would drop the destination range silently and it would succeed in such a condition. So on the API side, I wonder if I should add a flag to remap_anon_pages to provide non-strict behavior more similar to mremap. OTOH not providing the permissive mremap behavior may actually be better to force userland to be strict and be sure it knows what it is doing (otherwise it should use mremap in the first place?).
Comments welcome, thanks!
Andrea
Andrea Arcangeli (4):
mm: madvise MADV_USERFAULT
mm: rmap preparation for remap_anon_pages
mm: swp_entry_swapcount
mm: sys_remap_anon_pages
arch/alpha/include/uapi/asm/mman.h | 3 + arch/mips/include/uapi/asm/mman.h | 3 + arch/parisc/include/uapi/asm/mman.h | 3 + arch/x86/syscalls/syscall_32.tbl | 1 + arch/x86/syscalls/syscall_64.tbl | 1 + arch/xtensa/include/uapi/asm/mman.h | 3 + include/linux/huge_mm.h | 6 + include/linux/mm.h | 1 + include/linux/mm_types.h | 2 +- include/linux/swap.h | 6 + include/linux/syscalls.h | 3 + include/uapi/asm-generic/mman-common.h | 3 + kernel/sys_ni.c | 1 + mm/fremap.c | 440 +++++++++++++++++++++++++++++++++ mm/huge_memory.c | 158 ++++++++++-- mm/madvise.c | 16 ++ mm/memory.c | 10 + mm/rmap.c | 9 + mm/swapfile.c | 13 + 19 files changed, 667 insertions(+), 15 deletions(-)
PATCH 1/4 mm: madvise MADV_USERFAULT
MADV_USERFAULT is a new madvise flag that will set VM_USERFAULT in the vma flags. Whenever VM_USERFAULT is set in an anonymous vma, if userland touches a still unmapped virtual address, a sigbus signal is sent instead of allocating a new page. The sigbus signal handler will then resolve the page fault in userland by calling the remap_anon_pages syscall.
This functionality is needed to reliably implement postcopy live migration in KVM (without having to use a special chardevice that would disable all advanced Linux VM features, like swapping, KSM, THP, automatic NUMA balancing, etc...).
MADV_USERFAULT could also be used to offload parts of anonymous memory regions to remote nodes or to implement network distributed shared memory.
Here I enlarged the vm_flags to 64bit as we run out of bits (noop on 64bit kernels). An alternative is to find some combination of flags that are mutually exclusive if set.
--- arch/alpha/include/uapi/asm/mman.h | 3 +++ arch/mips/include/uapi/asm/mman.h | 3 +++ arch/parisc/include/uapi/asm/mman.h | 3 +++ arch/xtensa/include/uapi/asm/mman.h | 3 +++ include/linux/mm.h | 1 + include/linux/mm_types.h | 2 +- include/uapi/asm-generic/mman-common.h | 3 +++ mm/huge_memory.c | 34 ++++++++++++++++++++++++---------- mm/madvise.c | 16 ++++++++++++++++ mm/memory.c | 10 ++++++++++ 10 files changed, 67 insertions(+), 11 deletions(-)
arch/alpha/include/uapi/asm/mman.h +3 -0
@@ -60,6 +60,9 @@overrides the coredump filter bits */ #define MADV_DODUMP 17/* Clear the MADV_NODUMP flag */ +#define MADV_USERFAULT 18/* Trigger user faults if not mapped */ +#define MADV_NOUSERFAULT 19/* Don't trigger user faults */ +/* compatibility flags */ #define MAP_FILE 0
arch/mips/include/uapi/asm/mman.h +3 -0
@@ -84,6 +84,9 @@overrides the coredump filter bits */ #define MADV_DODUMP 17/* Clear the MADV_NODUMP flag */ +#define MADV_USERFAULT 18/* Trigger user faults if not mapped */ +#define MADV_NOUSERFAULT 19/* Don't trigger user faults */ +/* compatibility flags */ #define MAP_FILE 0
arch/parisc/include/uapi/asm/mman.h +3 -0
@@ -66,6 +66,9 @@overrides the coredump filter bits */ #define MADV_DODUMP 70/* Clear the MADV_NODUMP flag */ +#define MADV_USERFAULT 71/* Trigger user faults if not mapped */ +#define MADV_NOUSERFAULT 72/* Don't trigger user faults */ +/* compatibility flags */ #define MAP_FILE 0#define MAP_VARIABLE 0
arch/xtensa/include/uapi/asm/mman.h +3 -0
@@ -90,6 +90,9 @@overrides the coredump filter bits */ #define MADV_DODUMP 17/* Clear the MADV_NODUMP flag */ +#define MADV_USERFAULT 18/* Trigger user faults if not mapped */ +#define MADV_NOUSERFAULT 19/* Don't trigger user faults */ +/* compatibility flags */ #define MAP_FILE 0
include/linux/mm.h +1 -0
@@ -110,6 +110,7 @@ extern unsigned int kobjsize(const void *objp);#define VM_HUGEPAGE 0x20000000/* MADV_HUGEPAGE marked this vma */ #define VM_NOHUGEPAGE 0x40000000/* MADV_NOHUGEPAGE marked this vma */ #define VM_MERGEABLE 0x80000000/* KSM may merge identical pages */ +#define VM_USERFAULT 0x100000000/* Trigger user faults if not mapped */ #if defined(CONFIG_X86 )# define VM_PAT VM_ARCH_1/* PAT reserves whole VMA at once (x86) */
include/linux/mm_types.h +1 -1
@@ -198,7 +198,7 @@ struct page_frag {#endif }; -typedef unsigned long __nocast vm_flags_t ; +typedef unsigned long long __nocast vm_flags_t ;/* * A region containing a mapping of a non-memory backed file under NOMMU
include/uapi/asm-generic/mman-common.h +3 -0
@@ -52,6 +52,9 @@overrides the coredump filter bits */ #define MADV_DODUMP 17/* Clear the MADV_NODUMP flag */ +#define MADV_USERFAULT 18/* Trigger user faults if not mapped */ +#define MADV_NOUSERFAULT 19/* Don't trigger user faults */ +/* compatibility flags */ #define MAP_FILE 0
mm/huge_memory.c +24 -10
@@ -727,6 +727,11 @@ static int __do_huge_pmd_anonymous_page(struct mm_struct *mm,pte_free (mm, pgtable); }else {pmd_t entry; + +/* Deliver the page fault to userland */ +if (vma->vm_flags &VM_USERFAULT ) + goto do_sigbus; + entry =mk_huge_pmd (page, vma);page_add_new_anon_rmap (page, vma, haddr);set_pmd_at (mm, haddr, pmd, entry);
@@ -737,6 +742,9 @@ static int __do_huge_pmd_anonymous_page(struct mm_struct *mm, }return 0 ; +do_sigbus: +spin_unlock (&mm->page_table_lock); +return VM_FAULT_SIGBUS ; }static inline gfp_t alloc_hugepage_gfpmask (int defrag,gfp_t extra_gfp)
@@ -761,20 +769,17 @@ static inline struct page *alloc_hugepage(int defrag) }#endif -static bool set_huge_zero_page (pgtable_t pgtable,struct mm_struct *mm, +static void set_huge_zero_page (pgtable_t pgtable,struct mm_struct *mm,struct vm_area_struct *vma,unsigned long haddr,pmd_t *pmd,struct page *zero_page) {pmd_t entry; -if (!pmd_none (*pmd)) -return false; entry =mk_pmd (zero_page, vma->vm_page_prot); entry =pmd_wrprotect (entry); entry =pmd_mkhuge (entry);set_pmd_at (mm, haddr, pmd, entry);pgtable_trans_huge_deposit (mm, pgtable); mm->nr_ptes++; -return true; } int do_huge_pmd_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma,
@@ -794,6 +799,7 @@ int do_huge_pmd_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma, transparent_hugepage_use_zero_page()) {pgtable_t pgtable;struct page *zero_page; +int ret;bool set; pgtable =pte_alloc_one (mm, haddr);if (unlikely (!pgtable))
@@ -805,14 +811,24 @@ int do_huge_pmd_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma, goto out; }spin_lock (&mm->page_table_lock); - set =set_huge_zero_page (pgtable, mm, vma, haddr, pmd, - zero_page); + ret =0 ; + set = false; +if (pmd_none (*pmd)) { +if (vma->vm_flags &VM_USERFAULT ) + ret =VM_FAULT_SIGBUS ; +else { +set_huge_zero_page (pgtable, mm, vma, + haddr, pmd, + zero_page); + set = true; + } + }spin_unlock (&mm->page_table_lock);if (!set) {pte_free (mm, pgtable);put_huge_zero_page (); } -return 0 ; +return ret; } page =alloc_hugepage_vma (transparent_hugepage_defrag (vma), vma, haddr,numa_node_id (),0 );
@@ -886,16 +902,14 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,*/ if (is_huge_zero_pmd (pmd)) {struct page *zero_page; -bool set;/* * get_huge_zero_page() will never allocate a new page here, * since we already have a zero page to copy. It just takes a * reference. */ zero_page =get_huge_zero_page (); - set =set_huge_zero_page (pgtable, dst_mm, vma, addr, dst_pmd, +set_huge_zero_page (pgtable, dst_mm, vma, addr, dst_pmd, zero_page); -BUG_ON (!set);/* unexpected !pmd_none(dst_pmd) */ ret =0 ; goto out_unlock; }
mm/madvise.c +16 -0
@@ -93,6 +93,21 @@ static long madvise_behavior(struct vm_area_struct * vma,if (error) goto out;break ; +case MADV_USERFAULT : +if (vma->vm_ops) { + error = -EINVAL ; + goto out; + } + new_flags |=VM_USERFAULT ; +break ; +case MADV_NOUSERFAULT : +if (vma->vm_ops) { +WARN_ON (new_flags &VM_USERFAULT ); + error = -EINVAL ; + goto out; + } + new_flags &= ~VM_USERFAULT ; +break ; }if (new_flags == vma->vm_flags) {
@@ -405,6 +420,7 @@ madvise_behavior_valid(int behavior)case MADV_HUGEPAGE :case MADV_NOHUGEPAGE :#endif +case MADV_USERFAULT :case MADV_DONTDUMP :case MADV_DODUMP :return 1 ;
mm/memory.c +10 -0
@@ -3235,6 +3235,9 @@ static int do_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma, page_table =pte_offset_map_lock (mm, pmd, address, &ptl);if (!pte_none (*page_table)) goto unlock; +/* Deliver the page fault to userland, check inside PT lock */ +if (vma->vm_flags &VM_USERFAULT ) + goto sigbus; goto setpte; }
@@ -3262,6 +3265,10 @@ static int do_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma,if (!pte_none (*page_table)) goto release; +/* Deliver the page fault to userland, check inside PT lock */ +if (vma->vm_flags &VM_USERFAULT ) + goto sigbus; +inc_mm_counter_fast (mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, address); setpte:
@@ -3280,6 +3287,9 @@ oom_free_page:page_cache_release (page); oom:return VM_FAULT_OOM ; +sigbus: +pte_unmap_unlock (page_table, ptl); +return VM_FAULT_SIGBUS ; }/*
Re: [Qemu-devel] [PATCH 1/4] mm: madvise MADV_USERFAULT
Quoted 24 lines · Andrea Arcangeli
+++
b/mm/madvise.cmm/madvise.c +15
-0@@ -93,6 +93,21 @@ static long madvise_behavior(struct vm_area_struct * vma,if (error) goto out;break ; +case MADV_USERFAULT : +if (vma->vm_ops) { + error = -EINVAL ; + goto out; + } + new_flags |=VM_USERFAULT ; +break ; +case MADV_NOUSERFAULT : +if (vma->vm_ops) { +WARN_ON (new_flags &VM_USERFAULT ); + error = -EINVAL ; + goto out; + } + new_flags &= ~VM_USERFAULT ; +break ; }if (new_flags == vma->vm_flags) {
mm/madvise.c madvise_behavior_valid(int behavior) · line 423
Quoted 4 lines
mm/madvise.c +1
-0@@ -405,6 +420,7 @@ madvise_behavior_valid(int behavior)case MADV_HUGEPAGE :case MADV_NOHUGEPAGE :#endif +case MADV_USERFAULT :
Missing MADV_NOUSERFAULT: here
Quoted 3 lines
mm/madvise.c +0
-0case MADV_DONTDUMP :case MADV_DODUMP :return 1 ;
Re: [Qemu-devel] [PATCH 1/4] mm: madvise MADV_USERFAULT
Hi Andrew,
Quoted 8 lines · Andrew Jones
Review madvise_behavior_valid(int behavior) · line 423
Quoted 5 lines · Andrea Arcangeli
(quoted diff) +1
-0@@ -405,6 +420,7 @@ madvise_behavior_valid(int behavior) case MADV_HUGEPAGE: case MADV_NOHUGEPAGE: #endif + case MADV_USERFAULT:Missing
MADV_NOUSERFAULT: here
Right, thanks! Folded the below in patch 1/4.
mm/madvise.c +1 -0
@@ -421,6 +421,7 @@ madvise_behavior_valid(int behavior)case MADV_NOHUGEPAGE :#endif case MADV_USERFAULT : +case MADV_NOUSERFAULT :case MADV_DONTDUMP :case MADV_DODUMP :return 1 ;
PATCH 2/4 mm: rmap preparation for remap_anon_pages
In reply to Andrea Arcangeliremap_anon_pages (unlike remap_file_pages) tries to be non intrusive in the rmap code.
As far as the rmap code is concerned, rmap_anon_pages only alters the page->mapping and page->index. It does it while holding the page lock. However there are a few places that in presence of anon pages are allowed to do rmap walks without the page lock (split_huge_page and page_referenced_anon). Those places that are doing rmap walks without taking the page lock first, must be updated to re-check that the page->mapping didn't change after they obtained the anon_vma lock. remap_anon_pages takes the anon_vma lock for writing before altering the page->mapping, so if the page->mapping is still the same after obtaining the anon_vma lock (without the page lock), the rmap walks can go ahead safely (and remap_anon_pages will wait them to complete before proceeding).
remap_anon_pages serializes against itself with the page lock.
All other places taking the anon_vma lock while holding the mmap_sem for writing, don't need to check if the page->mapping has changed after taking the anon_vma lock, regardless of the page lock, because remap_anon_pages holds the mmap_sem for reading.
Overall this looks a fairly small change to the rmap code, notably less intrusive than the nonlinear vmas created by remap_file_pages.
There's one constraint enforced to allow this simplification: the source pages passed to remap_anon_pages must be mapped only in one vma, but this is not a limitation when used to handle userland page faults with MADV_USERFAULT. The source addresses passed to remap_anon_pages should be set as VM_DONTCOPY with MADV_DONTFORK to avoid any risk of the mapcount of the pages increasing, if fork runs in parallel in another thread, before or while remap_anon_pages runs.
--- mm/huge_memory.c | 24 ++++++++++++++++++++---- mm/rmap.c | 9 +++++++++ 2 files changed, 29 insertions(+), 4 deletions(-)
mm/huge_memory.c +20 -4
@@ -1824,6 +1824,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) {struct anon_vma *anon_vma;int ret =1 ; +struct address_space *mapping;BUG_ON (is_huge_zero_page (page));BUG_ON (!PageAnon (page));
@@ -1835,10 +1836,24 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)* page_lock_anon_vma_read except the write lock is taken to serialise * against parallel split or collapse operations. */ - anon_vma =page_get_anon_vma (page); -if (!anon_vma) - goto out; -anon_vma_lock_write (anon_vma); +for (;;) { + mapping =ACCESS_ONCE (page->mapping); + anon_vma =page_get_anon_vma (page); +if (!anon_vma) + goto out; +anon_vma_lock_write (anon_vma); +/* +* We don't hold the page lock here so +* remap_anon_pages_huge_pmd can change the anon_vma +* from under us until we obtain the anon_vma +* lock. Verify that we obtained the anon_vma lock +* before remap_anon_pages did. +*/ +if (likely (mapping ==ACCESS_ONCE (page->mapping))) +break ; +anon_vma_unlock_write (anon_vma); +put_anon_vma (anon_vma); + } ret =0 ;if (!PageCompound (page))
@@ -2294,6 +2309,7 @@ static void collapse_huge_page(struct mm_struct *mm,* Prevent all access to pagetables with the exception of * gup_fast later hanlded by the ptep_clear_flush and the VM * handled by the anon_vma lock + PG_lock. +* remap_anon_pages is prevented to race as well by the mmap_sem. */ down_write (&mm->mmap_sem);if (unlikely (khugepaged_test_exit (mm)))
mm/rmap.c +9 -0
@@ -448,6 +448,7 @@ struct anon_vma *page_lock_anon_vma_read(struct page *page)struct anon_vma *root_anon_vma;unsigned long anon_mapping; +repeat:rcu_read_lock (); anon_mapping = (unsigned long )ACCESS_ONCE (page->mapping);if ((anon_mapping &PAGE_MAPPING_FLAGS ) !=PAGE_MAPPING_ANON )
@@ -486,6 +487,14 @@ struct anon_vma *page_lock_anon_vma_read(struct page *page)rcu_read_unlock ();anon_vma_lock_read (anon_vma); +/* check if remap_anon_pages changed the anon_vma */ +if (unlikely ((unsigned long )ACCESS_ONCE (page->mapping) != anon_mapping)) { +anon_vma_unlock_read (anon_vma); +put_anon_vma (anon_vma); + anon_vma =NULL ; + goto repeat; + } +if (atomic_dec_and_test (&anon_vma->refcount)) {/* * Oops, we held the last refcount, release the lock
PATCH 3/4 mm: swp_entry_swapcount
In reply to Andrea ArcangeliProvide a new swapfile method for remap_anon_pages to verify the swap entry is mapped only in one vma before relocating the swap entry in a different virtual address. Otherwise if the swap entry is mapped in multiple vmas, when the page is swapped back in, it could get mapped in a non linear way in some anon_vma.
--- include/linux/swap.h | 6 ++++++ mm/swapfile.c | 13 +++++++++++++ 2 files changed, 19 insertions(+)
include/linux/swap.h +6 -0
@@ -389,6 +389,7 @@ extern unsigned int count_swap_pages(int, int);extern sector_t map_swap_page (struct page *,struct block_device **);extern sector_t swapdev_block (int ,pgoff_t );extern int page_swapcount (struct page *); +extern int swp_entry_swapcount (swp_entry_t entry);extern struct swap_info_struct *page_swap_info (struct page *);extern int reuse_swap_page (struct page *);extern int try_to_free_swap (struct page *);
@@ -489,6 +490,11 @@ static inline int page_swapcount(struct page *page)return 0 ; } +static inline int swp_entry_swapcount (swp_entry_t entry) +{ +return 0 ; +} +#define reuse_swap_page (page) (page_mapcount(page) == 1)static inline int try_to_free_swap (struct page *page)
mm/swapfile.c +13 -0
@@ -671,6 +671,19 @@ int page_swapcount(struct page *page)return count; } +int swp_entry_swapcount (swp_entry_t entry) +{ +int count =0 ; +struct swap_info_struct *p; + + p =swap_info_get (entry); +if (p) { + count =swap_count (p->swap_map[swp_offset (entry)]); +spin_unlock (&p->lock); + } +return count; +} +/* * We can write to an anon page without COW if there are no other references * to it. And as a side-effect, free up its swap: because the old content
PATCH 4/4 mm: sys_remap_anon_pages
In reply to Andrea ArcangeliThis new syscall will move anon pages across vmas, atomically and without touching the vmas.
It only works on non shared anonymous pages because those can be relocated without generating non linear anon_vmas in the rmap code.
It is the ideal mechanism to handle userspace page faults. Normally the destination vma will have VM_USERFAULT set with madvise(MADV_USERFAULT) while the source vma will normally have VM_DONTCOPY set with madvise(MADV_DONTFORK).
MADV_DONTFORK set in the source vma avoids remap_anon_pages to fail if the process forks during the userland page fault.
The thread triggering the sigbus signal handler by touching an unmapped hole in the MADV_USERFAULT region, should take care to receive the data belonging in the faulting virtual address in the source vma. The data can come from the network, storage or any other I/O device. After the data has been safely received in the private area in the source vma, it will call remap_anon_pages to map the page in the faulting address in the destination vma atomically. And finally it will return from the signal handler.
It is an alternative to mremap.
It only works if the vma protection bits are identical from the source and destination vma.
It can remap non shared anonymous pages within the same vma too.
If the source virtual memory range has any unmapped holes, or if the destination virtual memory range is not a whole unmapped hole, remap_anon_pages will fail with -EFAULT. This provides a very strict behavior to avoid any chance of memory corruption going unnoticed if there are userland race conditions. Only one thread should resolve the userland page fault at any given time for any given faulting address. This means that if two threads try to both call remap_anon_pages on the same destination address at the same time, the second thread will get an explicit -EFAULT retval from this syscall.
The destination range with VM_USERFAULT set should be completely empty or remap_anon_pages will fail with -EFAULT. It's recommended to call madvise(MADV_USERFAULT) immediately after the destination range has been allocated with malloc() or posix_memalign(), so that the VM_USERFAULT vma will be splitted before a tranparent hugepage fault could fill the VM_USERFAULT region if it doesn't start hugepage aligned. That will ensure the VM_USERFAULT area remains empty after allocation, regardless of its alignment.
The main difference with mremap is that if used to fill holes in unmapped anonymous memory vmas (if used in combination with MADV_USERFAULT) remap_anon_pages won't create lots of unmergeable vmas. mremap instead would create lots of vmas (because of non linear vma->vm_pgoff) leading to -ENOMEM failures (the number of vmas is limited).
MADV_USERFAULT and remap_anon_pages() can be tested with a program like below:
===
static unsigned char *c, *tmp;
void userfault_sighandler(int signum, siginfo_t *info, void *ctx)
{
unsigned char *addr = info->si_addr;
int len = 4096;
int ret;
addr = (unsigned char *) ((unsigned long) addr & ~((2*1024*1024)-1));
len = 2*1024*1024;
if (addr >= c && addr < c + SIZE) {
unsigned long offset = addr - c;
ret = syscall(SYS_remap_anon_pages, c+offset, tmp+offset, len);
if (ret)
perror("sigbus remap_anon_pages"), exit(1);
//printf("sigbus offset %lu\n", offset);
return;
}
printf("sigbus error addr %p c %p tmp %p\n", addr, c, tmp), exit(1);
}int main()
{
struct sigaction sa;
int ret;
unsigned long i;
/*
* Fails with THP due lack of alignment because of memset
* pre-filling the destination
*/
c = mmap(0, SIZE, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (c == MAP_FAILED)
perror("mmap"), exit(1);
tmp = mmap(0, SIZE, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (tmp == MAP_FAILED)
perror("mmap"), exit(1);
ret = posix_memalign((void **)&c, 2*1024*1024, SIZE);
if (ret)
perror("posix_memalign"), exit(1);
ret = posix_memalign((void **)&tmp, 2*1024*1024, SIZE);
if (ret)
perror("posix_memalign"), exit(1);
/*
* MADV_USERFAULT must run before memset, to avoid THP 2m
* faults to map memory into "tmp", if "tmp" isn't allocated
* with hugepage alignment.
*/
if (madvise(c, SIZE, MADV_USERFAULT))
perror("madvise"), exit(1);
memset(tmp, 0xaa, SIZE);
sa.sa_sigaction = userfault_sighandler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_SIGINFO;
sigaction(SIGBUS, &sa, NULL);
ret = syscall(SYS_remap_anon_pages, c, tmp, SIZE);
if (ret)
perror("remap_anon_pages"), exit(1);
for (i = 0; i < SIZE; i += 4096) {
if ((i/4096) % 2) {
/* exercise read and write MADV_USERFAULT */
c[i+1] = 0xbb;
}
if (c[i] != 0xaa)
printf("error %x offset %lu\n", c[i], i), exit(1);
}
return 0;
}===
With postcopy live migration to use a single network socket, the vcpu thread triggering the fault will normally use some interprocess communication with the thread doing the postcopy background transfer, to request a specific address, and it'll wait an ack before returning from the signal. The thread receiving the faulting address, will also solve the concurrency of the sigbus handler potentially happening simultaneously on the same address on different vcpu threads (so calling remap_anon_pages only once for each range received). So remap_anon_pages in the above testcase runs within the signal handler, but in production postcopy, it can run in a different thread.
--- arch/x86/syscalls/syscall_32.tbl | 1 + arch/x86/syscalls/syscall_64.tbl | 1 + include/linux/huge_mm.h | 6 + include/linux/syscalls.h | 3 + kernel/sys_ni.c | 1 + mm/fremap.c | 440 +++++++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 100 +++++++++ 7 files changed, 552 insertions(+)
arch/x86/syscalls/syscall_32.tbl +1 -0
@@ -357,3 +357,4 @@ 348 i386 process_vm_writev sys_process_vm_writev compat_sys_process_vm_writev 349 i386 kcmp sys_kcmp 350 i386 finit_module sys_finit_module +351 i386 remap_anon_pages sys_remap_anon_pages
arch/x86/syscalls/syscall_64.tbl +1 -0
@@ -320,6 +320,7 @@ 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp 313 common finit_module sys_finit_module +314 common remap_anon_pages sys_remap_anon_pages # # x32-specific system call numbers start at 512 to avoid cache impact
include/linux/huge_mm.h +6 -0
@@ -33,6 +33,12 @@ extern int move_huge_pmd(struct vm_area_struct *vma,extern int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot,int prot_numa); +extern int remap_anon_pages_huge_pmd (struct mm_struct *mm, +pmd_t *dst_pmd,pmd_t *src_pmd, +struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma, +unsigned long dst_addr, +unsigned long src_addr);enum transparent_hugepage_flag {TRANSPARENT_HUGEPAGE_FLAG ,
include/linux/syscalls.h +3 -0
@@ -486,6 +486,9 @@ asmlinkage long sys_mremap(unsigned long addr,asmlinkage long sys_remap_file_pages (unsigned long start,unsigned long size,unsigned long prot,unsigned long pgoff,unsigned long flags); +asmlinkage long sys_remap_anon_pages (unsigned long dst_start, +unsigned long src_start, +unsigned long len);asmlinkage long sys_msync (unsigned long start,size_t len,int flags);asmlinkage long sys_fadvise64 (int fd,loff_t offset,size_t len,int advice);asmlinkage long sys_fadvise64_64 (int fd,loff_t offset,loff_t len,int advice);
kernel/sys_ni.c +1 -0
@@ -172,6 +172,7 @@ cond_syscall(sys_mincore);cond_syscall (sys_madvise);cond_syscall (sys_mremap);cond_syscall (sys_remap_file_pages); +cond_syscall (sys_remap_anon_pages);cond_syscall (compat_sys_move_pages);cond_syscall (compat_sys_migrate_pages);
mm/fremap.c +440 -0
443 lines not on this page · raw message
mm/huge_memory.c +100 -0
106 lines not on this page · raw message
Re: [PATCH 4/4] mm: sys_remap_anon_pages
Quoted 5 lines · Andrea Arcangeli
===
static unsigned char *c, *tmp;
void
userfault_sighandler(int signum, siginfo_t *info, void *ctx)
oops, the hash of the test program got cut... so I append it below which is nicer without leading whitespaces.
===
#define _GNU_SOURCE
#include <sys/mman.h>
#include <pthread.h>
#include <strings.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <signal.h>
#include <sys/syscall.h>
#include <sys/types.h>
#define USE_USERFAULT
#define THP
#define MADV_USERFAULT 18
#define SIZE (1024*1024*1024)
#define SYS_remap_anon_pages 314
static unsigned char *c, *tmp;
void userfault_sighandler(int signum, siginfo_t *info, void *ctx)
{
unsigned char *addr = info->si_addr;
int len = 4096;
int ret;#ifdef THP
addr = (unsigned char *) ((unsigned long) addr & ~((2*1024*1024)-1));
len = 2*1024*1024;
#endif
if (addr >= c && addr < c + SIZE) {
unsigned long offset = addr - c;
ret = syscall(SYS_remap_anon_pages, c+offset, tmp+offset, len);
if (ret)
perror("sigbus remap_anon_pages"), exit(1);
//printf("sigbus offset %lu\n", offset);
return;
}
printf("sigbus error addr %p c %p tmp %p\n", addr, c, tmp), exit(1);
}int main()
{
struct sigaction sa;
int ret;
unsigned long i;
#ifndef THP
/*
* Fails with THP due lack of alignment because of memset
* pre-filling the destination
*/
c = mmap(0, SIZE, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (c == MAP_FAILED)
perror("mmap"), exit(1);
tmp = mmap(0, SIZE, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
if (tmp == MAP_FAILED)
perror("mmap"), exit(1);
#else
ret = posix_memalign((void **)&c, 2*1024*1024, SIZE);
if (ret)
perror("posix_memalign"), exit(1);
ret = posix_memalign((void **)&tmp, 2*1024*1024, SIZE);
if (ret)
perror("posix_memalign"), exit(1);
#endif
/*
* MADV_USERFAULT must run before memset, to avoid THP 2m
* faults to map memory into "tmp", if "tmp" isn't allocated
* with hugepage alignment.
*/
if (madvise(c, SIZE, MADV_USERFAULT))
perror("madvise"), exit(1);
memset(tmp, 0xaa, SIZE);
sa.sa_sigaction = userfault_sighandler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_SIGINFO;
sigaction(SIGBUS, &sa, NULL);#ifndef USE_USERFAULT
ret = syscall(SYS_remap_anon_pages, c, tmp, SIZE);
if (ret)
perror("remap_anon_pages"), exit(1);#endif
for (i = 0; i < SIZE; i += 4096) {
if ((i/4096) % 2) {
/* exercise read and write MADV_USERFAULT */
c[i+1] = 0xbb;
}
if (c[i] != 0xaa)
printf("error %x offset %lu\n", c[i], i), exit(1);
}
return 0;
}Quoted 10 lines · Andrea Arcangeli
Hello everyone,
this is a patchset to implement two new kernel features:
MADV_USERFAULTandremap_anon_pages.The combination of the two features are what I would propose to implement postcopy live migration, and in general demand paging of remote memory, hosted in different cloud nodes with KSM. It might also be used without virt to offload parts of memory to different nodes using some userland library and a network memory manager.
Interesting. The API you are proposing handles only user fault. How do you think about kernel case. I mean that KVM kernel module issues get_user_pages().
Exit to qemu with dedicated reason?
Quoted 22 lines
Postcopy live migration is currently implemented using a chardevice, which remains open for the whole VM lifetime and all virtual memory then becomes owned by the chardevice and it's not anonymous anymore.
http://lists.gnu.org/archive/html/qemu-devel/2012-10/msg05274.html
The main cons of the chardevice design is that all nice Linux MM features (like swapping/THP/KSM/automatic-NUMA-balancing) are disabled if the guest physical memory doesn't remain in anonymous memory. This is entirely solved by this alternative kernel solution. In fact
remap_anon_pageswill move THP pages natively by just updating two pmd pointers if alignment and length permits without any THP split.The other bonus is that
MADV_USERFAULTandremap_anon_pagesare implemented in the MM core andremap_anon_pagesfurthermore provides a functionality similar to what is already available for filebacked pages withremap_file_pages. That is usually more maintainable than having MM parts in a chardevice.In addition to asking review of the internals, this also need review the user APIs, as both those features are userland visible changes.
Quoted 8 lines
MADV_USERFAULTis only enabled for anonymous mappings so far but it could be extended. To be strict,-EINVALis returned if run on non anonymous mappings (where it would currently be a noop).The
remap_anon_pagessyscall API is not vectored, as I expect it used for demand paging only (where there can be just one faulting range per fault) or for large ranges where vectoring isn't going to provide performance advantages.
In case of precopy + postcopy optimization, dirty bitmap is sent after precopy phase and then clean pages are populated. In this population phase, vecotored API can be utilized. I'm not sure how much vectored API will contribute to shorten VM-switch time, though.
Quoted 4 lines
The current behavior of
remap_anon_pagesis very strict to avoid any chance of memory corruption going unnoticed, and it will return-EFAULTat the first sign of something unexpected (like a page already mapped in the destination pmd/pte, potentially signaling an userland
Quoted 8 lines
thread race condition with two threads userfaulting on the same destination address). mremap is not strict like that: it would drop the destination range silently and it would succeed in such a condition. So on the API side, I wonder if I should add a flag to
remap_anon_pagesto provide non-strict behavior more similar to mremap. OTOH not providing the permissive mremap behavior may actually be better to force userland to be strict and be sure it knows what it is doing (otherwise it should use mremap in the first place?).
It would be desirable to avoid complex thing in signal handler. Like sending page request to remote, receiving pages from remote. So signal handler would just queue requests to those dedicated threads and wait and requests would be serialized. Such strictness is not very critical, I guess. But others might find other use case...
thanks,
Quoted 30 lines
Comments welcome, thanks!
AndreaAndrea Arcangeli (4):
mm: madviseMADV_USERFAULT
mm: rmap preparation forremap_anon_pages
mm:swp_entry_swapcount
mm:sys_remap_anon_pagesarch/alpha/include/uapi/asm/mman.h | 3 + arch/mips/include/uapi/asm/mman.h | 3 + arch/parisc/include/uapi/asm/mman.h | 3 + arch/x86/syscalls/syscall_32.tbl | 1 + arch/x86/syscalls/syscall_64.tbl | 1 + arch/xtensa/include/uapi/asm/mman.h | 3 + include/linux/huge_mm.h | 6 + include/linux/mm.h | 1 + include/linux/mm_types.h | 2 +- include/linux/swap.h | 6 + include/linux/syscalls.h | 3 + include/uapi/asm-generic/mman-common.h | 3 + kernel/sys_ni.c | 1 + mm/fremap.c | 440 +++++++++++++++++++++++++++++++++ mm/huge_memory.c | 158 ++++++++++-- mm/madvise.c | 16 ++ mm/memory.c | 10 + mm/rmap.c | 9 + mm/swapfile.c | 13 + 19 files changed, 667 insertions(+), 15 deletions(-)
Signature · 1 line
yamahata
Hi Isaku,
Quoted 12 lines · Isaku Yamahata
Quoted 10 lines · Andrea Arcangeli
Hello everyone,
this is a patchset to implement two new kernel features:
MADV_USERFAULTandremap_anon_pages.The combination of the two features are what I would propose to implement postcopy live migration, and in general demand paging of remote memory, hosted in different cloud nodes with KSM. It might also be used without virt to offload parts of memory to different nodes using some userland library and a network memory manager.
Quoted 4 lines
Interesting. The API you are proposing handles only user fault. How do you think about kernel case. I mean that KVM kernel module issues
get_user_pages().
Exit to qemu with dedicated reason?
Correct. It's possible we want a more meaningful retval from get_user_pages too (right now sigbus would make gup return a too generic -EFAULT) by introducing a FOLL_USERFAULT in gup_flags.
So the KVM bits are still missing at this point.
Gleb also wants to enable the async page fault in the postcopy stage, so we immediately schedule a different guest process if the current guest process hits an userfault within KVM.
So the protocol with the postcopy thread will tell it "fill this pfn async" or "fill it synchronous". And Gleb likes kvm to talk to the postcopy thread (through a pipe?) directly to avoid exiting to userland.
But we could also return to userland, if we do, we don't need to teach the kernel about the postcopy thread protocol to require new pages synchronously (after running out of async page faults) or asynchronously (when async page faults are still availbale).
Clearly staying in the kernel is more efficient as it avoids an enter/exit cycle and kvm can be restarted immediately after a 9 byte write to the pipe with the postcopy thread.
Quoted 4 lines
In case of precopy + postcopy optimization, dirty bitmap is sent after precopy phase and then clean pages are populated. In this population phase, vecotored API can be utilized. I'm not sure how much vectored API will contribute to shorten VM-switch time, though.
But the network transfer won't be vectored, would it? If we pay an enter/exit kernel for the network transfer, I assume we'd run a remap_anon_pages after each chunk.
Also the postcopy thread won't transfer in the background too much data at once. It needs to react quick to a "urgent" userfault request coming from a vcpu thread.
Quoted 4 lines
It would be desirable to avoid complex thing in signal handler. Like sending page request to remote, receiving pages from remote. So signal handler would just queue requests to those dedicated threads and wait and requests would be serialized. Such strictness is not
Exactly, that's the idea, a separate thread will do the network transfer and then run remap_anon_pages. And if we immediately use async page faults it won't need to block until we run out of async page faults.
Quoted 1 line
very critical, I guess. But others might find other use case...
It's still somewhat useful to be strict in my view, as it will verify that we handle correctly the case of many vcpus userfaulting on the same address at the same time, everyone except the first shouldn't run remap_anon_pages.
Thanks!
Andrea
Quoted 14 lines · Andrea Arcangeli
The current behavior of
remap_anon_pagesis very strict to avoid any chance of memory corruption going unnoticed, and it will return-EFAULTat the first sign of something unexpected (like a page already mapped in the destination pmd/pte, potentially signaling an userland thread race condition with two threads userfaulting on the same destination address). mremap is not strict like that: it would drop the destination range silently and it would succeed in such a condition. So on the API side, I wonder if I should add a flag toremap_anon_pagesto provide non-strict behavior more similar to mremap. OTOH not providing the permissive mremap behavior may actually be better to force userland to be strict and be sure it knows what it is doing (otherwise it should use mremap in the first place?).
What about instead of adding a new syscall (remap_anon_pages) to instead extend mremap with new flags giving it a strict mode?
drew
Quoted 2 lines · Andrew Jones
What about instead of adding a new syscall (
remap_anon_pages) to instead extend mremap with new flags giving it a strict mode?
I actually thought about this and it's a very interesting argument.
When I thought about it, I felt the problem was that we won't know until we did some work (at least until reaching _each_ individual page structure of every page mapped in the mremapped range) if we can avoid the vma mangling or not. So then remap_anon_pages should be more efficient because it will just error out if strict mode is not possible, it won't need to search all page structures before it know if it has to do the vma mangling or not.
It likely is simpler too than to try to teach mremap to do both things after checking which one it can do.
Also, the same argument could be made about remap_file_pages too.
But it's still an interesting point, let's see what others thinks about it.