mm: add a new option MREMAP_DUP to mmrep syscall
Subsystems (7)
cover mm: add a new option MREMAP_DUP to mmrep syscall
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory region,
just like what fork() did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.
This patch is based on the commit a12183c62717ac4579319189a00f5883a18dff08 pulled from upstream (linux 3.9) on 2013-04-04, but I hope to sent it first to see if some case I missed to handle correctly, will try rebase to latest upstream code in next version.
simple test code:
#define _GNU_SOURCE
#include <sys/mman.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
int main(void)
{
int len = 4096 * 2 ;
void *old_addr = malloc(len);
old_addr = ((unsigned long)(old_addr + 4096) & (~0x3FF));
printf("mapping addr %p with len %d.\n", old_addr, len);
char *oldc = old_addr;
oldc[0] = 0;
oldc[1] = 1;
oldc[2] = 2;
oldc[3] = 3;
void *new_addr;
unsigned long new_addr_l;
new_addr = mremap(old_addr, len, 0, 4);
printf("result new addr %lx %p.\n", new_addr_l, new_addr);
char *newc = new_addr;
printf("old value is 0x%lx.\n", *((unsigned long *)oldc));
printf("new value is 0x%lx.\n", *((unsigned long *)newc));
newc[0] = 6;
printf("old value is 0x%lx.\n", *((unsigned long *)oldc));
printf("new value is 0x%lx.\n", *((unsigned long *)newc));
oldc[0] = 9;
printf("old value is 0x%lx.\n", *((unsigned long *)oldc));
printf("new value is 0x%lx.\n", *((unsigned long *)newc));
assert(0 == munmap(new_addr, len));}
Wenchao Xia (6):
mm: add parameter remove_old in move_huge_pmd()
mm : allow copy between different addresses for copy_one_pte() mm : export rss vec helper functions
mm : export is_cow_mapping()
mm : add parameter remove_old in move_page_tables
mm : add new option MREMAP_DUP to mremap() syscall
fs/exec.c | 2 +- include/linux/huge_mm.h | 2 +- include/linux/mm.h | 9 ++- include/uapi/linux/mman.h | 1 + mm/huge_memory.c | 6 +- mm/memory.c | 33 ++++---- mm/mremap.c | 200 +++++++++++++++++++++++++++++++++++++++++++-- 7 files changed, 224 insertions(+), 29 deletions(-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 1/6 mm: add parameter remove_old in move_huge_pmd()
From: Wenchao Xia <[email protected]>
--- include/linux/huge_mm.h | 2 +- mm/huge_memory.c | 6 ++++-- mm/mremap.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-)
include/linux/huge_mm.h +1 -1
@@ -29,7 +29,7 @@ extern int move_huge_pmd(struct vm_area_struct *vma,struct vm_area_struct *new_vma, unsigned long old_addr, unsigned long new_addr, unsigned long old_end, - pmd_t *old_pmd, pmd_t *new_pmd); + pmd_t *old_pmd, pmd_t *new_pmd, bool remove_old);extern int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot,int prot_numa);
mm/huge_memory.c +4 -2
@@ -1402,10 +1402,11 @@ int mincore_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,return ret; } +/* This function copy or moves pmd in same mm */ int move_huge_pmd (struct vm_area_struct *vma,struct vm_area_struct *new_vma,unsigned long old_addr,unsigned long new_addr,unsigned long old_end, -pmd_t *old_pmd,pmd_t *new_pmd) +pmd_t *old_pmd,pmd_t *new_pmd,bool remove_old) {int ret =0 ;pmd_t pmd;
@@ -1429,7 +1430,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma, ret =__pmd_trans_huge_lock (old_pmd, vma);if (ret ==1 ) { - pmd =pmdp_get_and_clear (mm, old_addr, old_pmd); + pmd = remove_old ? +pmdp_get_and_clear (mm, old_addr, old_pmd) : *old_pmd;VM_BUG_ON (!pmd_none (*new_pmd));set_pmd_at (mm, new_addr, new_pmd, pmd);spin_unlock (&mm->page_table_lock);
mm/mremap.c +1 -1
@@ -178,7 +178,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,if (extent ==HPAGE_PMD_SIZE ) err =move_huge_pmd (vma, new_vma, old_addr, new_addr, old_end, - old_pmd, new_pmd); + old_pmd, new_pmd, true);if (err >0 ) { need_flush = true;continue ;
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 2/6 mm : allow copy between different addresses for copy_one_pte()
In reply to [email protected]From: Wenchao Xia <[email protected]>
This function now can be used in pte copy in same process with different addresses. It is also exported.
--- include/linux/mm.h | 4 ++++ mm/memory.c | 27 ++++++++++++++------------- 2 files changed, 18 insertions(+), 13 deletions(-)
include/linux/mm.h +4 -0
@@ -963,6 +963,10 @@ int walk_page_range(unsigned long addr, unsigned long end, struct mm_walk *walk);void free_pgd_range (struct mmu_gather *tlb,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling); +unsigned long copy_one_pte (struct mm_struct *dst_mm,struct mm_struct *src_mm, +pte_t *dst_pte,pte_t *src_pte, +unsigned long dst_addr,unsigned long src_addr, +struct vm_area_struct *vma,int *rss);int copy_page_range (struct mm_struct *dst,struct mm_struct *src,struct vm_area_struct *vma); void unmap_mapping_range(struct address_space *mapping,
mm/memory.c +14 -13
@@ -824,15 +824,15 @@ out: }/* -* copy one vm_area from one task to the other. Assumes the page tables -* already present in the new task to be cleared in the whole range -* covered by this vma. +* copy one pte from @src_addr to @dst_addr. Assumes the page tables and vma +* already present in the @dst_addr, @src_addr and @src_pte is covered by +* @vma, @rss is a array of size NR_MM_COUNTERS used by caller to sync. dst_mm +* may be equal to src_mm. Return 0 or swap entry value. */ - -static inline unsigned long -copy_one_pte (struct mm_struct *dst_mm,struct mm_struct *src_mm, -pte_t *dst_pte,pte_t *src_pte,struct vm_area_struct *vma, -unsigned long addr,int *rss) +unsigned long copy_one_pte (struct mm_struct *dst_mm,struct mm_struct *src_mm, +pte_t *dst_pte,pte_t *src_pte, +unsigned long dst_addr,unsigned long src_addr, +struct vm_area_struct *vma,int *rss) {unsigned long vm_flags = vma->vm_flags;pte_t pte = *src_pte;
@@ -872,7 +872,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,*/ make_migration_entry_read (&entry); pte =swp_entry_to_pte (entry); -set_pte_at (src_mm, addr, src_pte, pte); +set_pte_at (src_mm, src_addr, + src_pte, pte); } } }
@@ -884,7 +885,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,* in the parent and the child */ if (is_cow_mapping (vm_flags)) { -ptep_set_wrprotect (src_mm, addr, src_pte); +ptep_set_wrprotect (src_mm, src_addr, src_pte); pte =pte_wrprotect (pte); }
@@ -896,7 +897,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte =pte_mkclean (pte); pte =pte_mkold (pte); - page =vm_normal_page (vma, addr, pte); + page =vm_normal_page (vma, src_addr, pte);if (page) {get_page (page);page_dup_rmap (page);
@@ -907,7 +908,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } out_set_pte: -set_pte_at (dst_mm, addr, dst_pte, pte); +set_pte_at (dst_mm, dst_addr, dst_pte, pte);return 0 ; }
@@ -951,7 +952,7 @@ again:continue ; } entry.val =copy_one_pte (dst_mm, src_mm, dst_pte, src_pte, - vma, addr, rss); + addr, addr, vma, rss);if (entry.val)break ; progress +=8 ;
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 3/6 mm : export rss vec helper functions
In reply to [email protected]From: Wenchao Xia <[email protected]>
--- include/linux/mm.h | 2 ++ mm/memory.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-)
include/linux/mm.h +2 -0
@@ -963,6 +963,8 @@ int walk_page_range(unsigned long addr, unsigned long end, struct mm_walk *walk);void free_pgd_range (struct mmu_gather *tlb,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling); +void init_rss_vec (int *rss); +void add_mm_rss_vec (struct mm_struct *mm,int *rss); unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte_t *dst_pte, pte_t *src_pte, unsigned long dst_addr, unsigned long src_addr,
mm/memory.c +2 -2
@@ -643,12 +643,12 @@ int __pte_alloc_kernel(pmd_t *pmd, unsigned long address)return 0 ; } -static inline void init_rss_vec (int *rss) +void init_rss_vec (int *rss) {memset (rss,0 ,sizeof (int ) *NR_MM_COUNTERS ); } -static inline void add_mm_rss_vec (struct mm_struct *mm,int *rss) +void add_mm_rss_vec (struct mm_struct *mm,int *rss) {int i;
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 4/6 mm : export is_cow_mapping()
In reply to [email protected]From: Wenchao Xia <[email protected]>
--- include/linux/mm.h | 1 + mm/memory.c | 2 +- 2 files changed, 2 insertions(+), 1 deletions(-)
include/linux/mm.h +1 -0
@@ -965,6 +965,7 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling);void init_rss_vec (int *rss);void add_mm_rss_vec (struct mm_struct *mm,int *rss); +bool is_cow_mapping (vm_flags_t flags); unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte_t *dst_pte, pte_t *src_pte, unsigned long dst_addr, unsigned long src_addr,
mm/memory.c +1 -1
@@ -723,7 +723,7 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE ); } -static inline bool is_cow_mapping (vm_flags_t flags) +bool is_cow_mapping (vm_flags_t flags) {return (flags & (VM_SHARED |VM_MAYWRITE )) ==VM_MAYWRITE ; }
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 5/6 mm : add parameter remove_old in move_page_tables
In reply to [email protected]From: Wenchao Xia <[email protected]>
--- fs/exec.c | 2 +- include/linux/mm.h | 2 +- mm/mremap.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 93 insertions(+), 8 deletions(-)
fs/exec.c +1 -1
@@ -603,7 +603,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)* process cleanup to remove whatever mess we made. */ if (length !=move_page_tables (vma, old_start, - vma, new_start, length, false)) + vma, new_start, length, false, true))return -ENOMEM ;lru_add_drain ();
include/linux/mm.h +1 -1
@@ -1085,7 +1085,7 @@ vm_is_stack(struct task_struct *task, struct vm_area_struct *vma, int in_group);extern unsigned long move_page_tables (struct vm_area_struct *vma,unsigned long old_addr,struct vm_area_struct *new_vma,unsigned long new_addr,unsigned long len, -bool need_rmap_locks); +bool need_rmap_locks,bool remove_old);extern unsigned long do_mremap (unsigned long addr,unsigned long old_len,unsigned long new_len,unsigned long flags,unsigned long new_addr);
mm/mremap.c +91 -6
138 lines not on this page · raw message
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC PATCH 6/6 mm : add new option MREMAP_DUP to mremap() syscall
In reply to [email protected]From: Wenchao Xia <[email protected]>
This option allow user space program getting a mirror for mem, that is two virtual mapping. The content is COW so it can be used for snapshot a region of mem.
Now shared memory is not COWED yet.
--- include/uapi/linux/mman.h | 1 + mm/mremap.c | 103 ++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 102 insertions(+), 2 deletions(-)
include/uapi/linux/mman.h +1 -0
@@ -5,6 +5,7 @@#define MREMAP_MAYMOVE 1#define MREMAP_FIXED 2 +#define MREMAP_DUP 4#define OVERCOMMIT_GUESS 0#define OVERCOMMIT_ALWAYS 1
mm/mremap.c +101 -2
133 lines not on this page · raw message
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory region,
just like whatfork()did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.
What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
ao? 2013-5-9 22:13, Mel Gorman a??e??:
Quoted 8 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory region,
just like whatfork()did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.
Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.
It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.
To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory
region,
just like whatfork()did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.
Quoted 5 lines
To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additionalcopy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.
A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the fork() approach is preferable.
If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.
Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.
Stefan
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
CCed KVM guys.
Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory
region,
just like whatfork()did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.Quoted 5 lines
To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additionalcopy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the
fork()approach is preferable.If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.
Quoted 3 lines
Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.
Yup, I agree with you that the new syscall sometimes is not a good solution.
Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.
In this case, we need to move the guest memory on old qemu instance to the new one, fork() can not help because we need to exec() a new instance, after that all memory mapping will be destroyed.
We tried to enable SPLICE_F_MOVE[2] for vmsplice() to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.
To coordinate with your fork() approach, maybe we can introduce a new flag for VMA, something like: VM_KEEP_ONEXEC, to tell exec() to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.
[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285
Quoted 68 lines · Xiao Guangrong
CCed KVM guys.
Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory
region,
just like whatfork()did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.Quoted 5 lines
To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additionalcopy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the
fork()approach is preferable.If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.Quoted 3 lines
Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.
Yup, I agree with you that the new syscall sometimes is not a good solution.
Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.
In this case, we need to move the guest memory on old qemu instance to the new one,
fork()can not help because we need toexec()a new instance, after that all memory mapping will be destroyed.We tried to enable SPLICE_F_MOVE[2] for
vmsplice()to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.
Quoted 7 lines
To coordinate with your
fork()approach, maybe we can introduce a new flag for VMA, something like:VM_KEEP_ONEXEC, to tellexec()to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285
Hi,
What is the purpose of snapshotting guest RAM here, in the context of local migration?
Quoted 74 lines · Marcelo Tosatti
Quoted 73 lines · Xiao Guangrong
CCed KVM guys.
Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory
region,
just like whatfork()did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.It have some troubles: parent - child communication, sometimes page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.Quoted 5 lines
To avoid complex communication for data control, and file content protecting, So let parent instead of child handling the data with a pipe, but this brings additional
copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the
fork()approach is preferable.If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.Quoted 3 lines
Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.
Yup, I agree with you that the new syscall sometimes is not a good solution.
Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.
In this case, we need to move the guest memory on old qemu instance to the new one,
fork()can not help because we need toexec()a new instance, after that all memory mapping will be destroyed.We tried to enable SPLICE_F_MOVE[2] for
vmsplice()to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.To coordinate with your
fork()approach, maybe we can introduce a new flag for VMA, something like:VM_KEEP_ONEXEC, to tellexec()to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.
Quoted 5 lines
Quoted 2 lines
[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285Hi,
Hi Marcelo,
Quoted 2 lines
What is the purpose of snapshotting guest RAM here, in the context of local migration?
RAM-shapshotting and local-migration are on the different ways. Why i asked for your guy’s suggestion here is beacuse i thought they need do a same thing that moves memory from one process to another in a efficient way. Your idea? :)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 86 lines · Xiao Guangrong
Quoted 74 lines · Marcelo Tosatti
Quoted 73 lines · Xiao Guangrong
CCed KVM guys.
Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory
region,
just like whatfork()did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.It have some troubles: parent - child communication, sometimes page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.Quoted 5 lines
To avoid complex communication for data control, and file content protecting, So let parent instead of child handling the data with a pipe, but this brings additional
copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the
fork()approach is preferable.If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.Quoted 3 lines
Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.
Yup, I agree with you that the new syscall sometimes is not a good solution.
Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.
In this case, we need to move the guest memory on old qemu instance to the new one,
fork()can not help because we need toexec()a new instance, after that all memory mapping will be destroyed.We tried to enable SPLICE_F_MOVE[2] for
vmsplice()to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.To coordinate with your
fork()approach, maybe we can introduce a new flag for VMA, something like:VM_KEEP_ONEXEC, to tellexec()to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.Quoted 5 lines
Quoted 2 lines
[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285Hi,
Hi Marcelo,
Quoted 7 lines
Quoted 2 lines
What is the purpose of snapshotting guest RAM here, in the context of local migration?
RAM-shapshotting and local-migration are on the different ways. Why i asked for your guy’s suggestion here is beacuse i thought they need do a same thing that moves memory from one process to another in a efficient way. Your idea? :)
Another possibility is to use memory that is not anonymous for guest RAM, such as hugetlbfs or tmpfs.
IIRC ksm and thp have limitations wrt tmpfs.
Still curious about RAM snapshotting.
Quoted 95 lines · Marcelo Tosatti
Quoted 86 lines · Xiao Guangrong
Quoted 74 lines · Marcelo Tosatti
Quoted 73 lines · Xiao Guangrong
CCed KVM guys.
Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory
region,
just like whatfork()did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.It have some troubles: parent - child communication, sometimes page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.Quoted 5 lines
To avoid complex communication for data control, and file content protecting, So let parent instead of child handling the data with a pipe, but this brings additional
copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the
fork()approach is preferable.If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.Quoted 3 lines
Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.
Yup, I agree with you that the new syscall sometimes is not a good solution.
Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.
In this case, we need to move the guest memory on old qemu instance to the new one,
fork()can not help because we need toexec()a new instance, after that all memory mapping will be destroyed.We tried to enable SPLICE_F_MOVE[2] for
vmsplice()to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.To coordinate with your
fork()approach, maybe we can introduce a new flag for VMA, something like:VM_KEEP_ONEXEC, to tellexec()to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.Quoted 5 lines
Quoted 2 lines
[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285Hi,
Hi Marcelo,
Quoted 7 lines
Quoted 2 lines
What is the purpose of snapshotting guest RAM here, in the context of local migration?
RAM-shapshotting and local-migration are on the different ways. Why i asked for your guy’s suggestion here is beacuse i thought they need do a same thing that moves memory from one process to another in a efficient way. Your idea? :)
Quoted 4 lines
Another possibility is to use memory that is not anonymous for guest RAM, such as hugetlbfs or tmpfs.
IIRC ksm and thp have limitations wrt tmpfs.
Yes, KSM and THP are what we're concerning about.
Quoted 2 lines
Still curious about RAM snapshotting.
Wen Chao, could you please tell it more?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · wenchao
ao? 2013-5-9 22:13, Mel Gorman a??e??:
Quoted 8 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory region,
just like whatfork()did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.
Quoted 6 lines
It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.
CC Pavel
I wounder if you can reuse the CRIU approach for memory snapshoting.
http://thread.gmane.org/gmane.linux.kernel/1483158/
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 15 lines · Kirill A. Shutemov
Quoted 14 lines · wenchao
ao? 2013-5-9 22:13, Mel Gorman a??e??:
Quoted 13 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory region,
just like whatfork()did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.
Quoted 8 lines
Quoted 6 lines
It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.CC Pavel
Thank you!
Quoted 1 line
I wounder if you can reuse the CRIU approach for memory snapshoting.
I doubt it. First of all, we need to have task's memory in existing external process which is not its child. With MREMAP_DUP we can't have this. And the most important thing is that we don't need pages duplication on modification. It's the waste of memory for our case. We just need to know the fact that the page has changed.
Wenchao, why can't you use existing KVM dirty-tracking for making mem snapshot? As per my understanding of how KVM MMU works you can
1 turn dirty track on
2 read pages from their original places
3 pick dirty bitmap and read changed pages several times 4 freeze guest
5 repeat step 3
6 release guest
Does it work for you?
This is very very similar to how we do mem snapshot with CRIU (dirty tracking is the soft-dirty patches from the link Kirill provided).
Quoted 2 lines
Thanks,
Pavel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
ao? 2013-5-11 22:16, Pavel Emelyanov a??e??:
Quoted 23 lines · Kirill A. Shutemov
Quoted 22 lines
Quoted 20 lines · wenchao
D'D?D? 2013-5-9 22:13, Mel Gorman DuD?D(C)D1D?DGBP:
Quoted 8 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines
From: Wenchao Xia <[email protected]>
This serial try to enable mremap syscall to cow some private memory region,
just like whatfork()did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.Quoted 5 lines
What not just
fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1)fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.
Quoted 4 lines
Quoted 1 line
CC Pavel
Thank you!
Sorry I forgot CC you, I have viewed the contents on CRIU website
before patching. :>
Quoted 17 lines
Quoted 1 line
I wounder if you can reuse the CRIU approach for memory snapshoting.
I doubt it. First of all, we need to have task's memory in existing external process which is not its child. With
MREMAP_DUPwe can't have this. And the most important thing is that we don't need pages duplication on modification. It's the waste of memory for our case. We just need to know the fact that the page has changed.Wenchao, why can't you use existing KVM dirty-tracking for making mem snapshot? As per my understanding of how KVM MMU works you can
1 turn dirty track on
2 read pages from their original places
3 pick dirty bitmap and read changed pages several times 4 freeze guest
5 repeat step 3
6 release guest
Quoted 5 lines
Does it work for you?
This is very very similar to how we do mem snapshot with CRIU (dirty tracking is the soft-dirty patches from the link Kirill provided).
It is different. Actually this approach is already used in qemu as
migration, also as a work around for snapshot. Dirty tracking actually
made a mirror of latest memory region, but for snapshot we need not
sync up the two mirror, and syncing up require frequently changed pages being saved several times. This brings extra trouble for following actions, and also require transfer speed > memory changing speed.
Looking at the block layer's function, for example, LVM2. Most of them
provided an API to do a snapshot by COW, so the idea comes: why not add
an same one for memory? Then the APIs are full for user. Later in
discuss Fork() seems one way to do it, with disadvantage that additional
process comes, so I hope to improve it.
Comparation:
Dirty tracking VS cow
CPU higher minimum Memory less higher I/O higher minimum
Since dirty tracking keeps sync with latest memory data, so it is
more ideal for migration than snapshot, but in principle migration is a bit different with snapshot. Further thinking, dirty tracking could work together with cow, to form an incremental snapshot chain, reducing the pages need to be written, but not related to this patch.
Base->delta->delta
Quoted 7 lines
Quoted 2 lines
Thanks,
Pavel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>