Extend the mincore() report bits
Subsystems (3)
cover Extend the mincore() report bits
Hi.
We're currently working on the "Checkpoint-Restart in the Userspace" project. The aim of the project is to dump a full (as full as possible) state of a running process set (e.g. an application or a container) and recreate them at exactly the same state later [1]
One of the parts of this state-dumping is dumping the memory of the processes. Since apps usually mmap() more memory, than they actually use, dumping the whole mapping contents is too expensive. To reduce this data we currently use the mincore() syscall to check which pages are currently in memory and should be dumped, and which are not and can be skipped. This simple trick reduces the dump size greatly, buy has three problems.
1. File pages, that were not mapped by a task, but that were brought to page cache somehow (readahead or shared lib usage by other task) are reported as present and we dump them, while we can skip them, since reading the page from file again at restore time gives the correct page;
2. File pages, that are mapped by private mapping but that are not yet cow-ed are also reported as present, but we can skip them as well -- they can be re-read from disk;
2. Pages, that are swapped out are not reported by existing mincore(), and we skip them, and this is a bug :(
That said, I propose to add 2 more bits to the mincore per-page report to address these problems. Plz, find details in the respective patches.
[1]
The project homepage is at http://criu.org
The project sources can be found here:
tools: https://github.com/cyrillos/crtools kernel: https://github.com/cyrillos/linux-2.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/3 mincore: Introduce named constant for existing bit
--- include/linux/mman.h | 2 ++ mm/huge_memory.c | 2 +- mm/mincore.c | 10 +++++----- 3 files changed, 8 insertions(+), 6 deletions(-)
include/linux/mman.h +2 -0
@@ -10,6 +10,8 @@#define OVERCOMMIT_ALWAYS 1#define OVERCOMMIT_NEVER 2 +#define MINCORE_RESIDENT 0x1 +#ifdef __KERNEL__#include <linux/mm.h> #include <linux/percpu_counter.h>
mm/huge_memory.c +1 -1
@@ -1045,7 +1045,7 @@ int mincore_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,* All logical pages in the range are present * if backed by a huge page. */ -memset (vec,1 , (end - addr) >>PAGE_SHIFT ); +memset (vec,MINCORE_RESIDENT , (end - addr) >>PAGE_SHIFT ); } }else spin_unlock (&vma->vm_mm->page_table_lock);
mm/mincore.c +5 -5
@@ -38,7 +38,7 @@ static void mincore_hugetlb_page_range(struct vm_area_struct *vma, addr & huge_page_mask(h)); present = ptep && !huge_pte_none (huge_ptep_get (ptep));while (1 ) { - *vec = present; + *vec = (present ?MINCORE_RESIDENT :0 ); vec++; addr +=PAGE_SIZE ;if (addr == end)
@@ -83,7 +83,7 @@ static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff)page_cache_release (page); } -return present; +return present ?MINCORE_RESIDENT :0 ; } static void mincore_unmapped_range(struct vm_area_struct *vma,
@@ -122,7 +122,7 @@ static void mincore_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (pte_none (pte))mincore_unmapped_range (vma, addr, next, vec);else if (pte_present (pte)) - *vec =1 ; + *vec =MINCORE_RESIDENT ;else if (pte_file (pte)) { pgoff =pte_to_pgoff (pte); *vec =mincore_page (vma->vm_file->f_mapping, pgoff);
@@ -131,14 +131,14 @@ static void mincore_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (is_migration_entry (entry)) {/* migration entries are always uptodate */ - *vec =1 ; + *vec =MINCORE_RESIDENT ; }else {#ifdef CONFIG_SWAP pgoff = entry.val; *vec =mincore_page (&swapper_space, pgoff);#else WARN_ON (1 ); - *vec =1 ; + *vec =MINCORE_RESIDENT ;#endif } }
Signature · 8 lines
1.5.5.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/3 mincore: Introduce the MINCORE_ANON bit
In reply to Pavel EmelyanovWhen creating a memory dump of a running application it is better to have the smallest possible set of pages. Using the existing mincore() bit helps, but not much -- it repotes pages from page cache, which have not necessarily being mapped by an application, and pages from private file mappings, that are not yet being cow-ed and thus their contents doesn't differ from file.
Introduce the 2nd bit of mincore, that reports whether a page is not backed by a file on disk. I.e. all pages from anonymous mappings and those pages from private file mappings, that have already being cow-ed by write.
--- include/linux/mman.h | 1 + mm/mincore.c | 15 +++++++++++++-- 2 files changed, 14 insertions(+), 2 deletions(-)
include/linux/mman.h +1 -0
@@ -11,6 +11,7 @@#define OVERCOMMIT_NEVER 2#define MINCORE_RESIDENT 0x1 +#define MINCORE_ANON 0x2#ifdef __KERNEL__#include <linux/mm.h>
mm/mincore.c +13 -2
@@ -38,7 +38,7 @@ static void mincore_hugetlb_page_range(struct vm_area_struct *vma, addr & huge_page_mask(h)); present = ptep && !huge_pte_none (huge_ptep_get (ptep));while (1 ) { - *vec = (present ?MINCORE_RESIDENT :0 ); + *vec = (present ?MINCORE_RESIDENT :0 ) |MINCORE_ANON ; vec++; addr +=PAGE_SIZE ;if (addr == end)
@@ -86,6 +86,17 @@ static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff)return present ?MINCORE_RESIDENT :0 ; } +static unsigned char mincore_pte (struct vm_area_struct *vma,unsigned long addr,pte_t pte) +{ +struct page *pg; + + pg =vm_normal_page (vma, addr, pte); +if (!pg) +return 0 ; +else +return PageAnon (pg) ?MINCORE_ANON :0 ; +} +static void mincore_unmapped_range (struct vm_area_struct *vma,unsigned long addr,unsigned long end,unsigned char *vec)
@@ -122,7 +133,7 @@ static void mincore_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (pte_none (pte))mincore_unmapped_range (vma, addr, next, vec);else if (pte_present (pte)) - *vec =MINCORE_RESIDENT ; + *vec =MINCORE_RESIDENT |mincore_pte (vma, addr, pte);else if (pte_file (pte)) { pgoff =pte_to_pgoff (pte); *vec =mincore_page (vma->vm_file->f_mapping, pgoff);
Signature · 8 lines
1.5.5.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/3] mincore: Introduce the MINCORE_ANON bit
mm/mincore.c static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff) · line 99
Quoted 11 lines
mm/mincore.c +11
-0+static unsigned char mincore_pte (struct vm_area_struct *vma,unsigned long addr,pte_t pte) +{ + struct page *pg; + + pg = vm_normal_page(vma, addr, pte); + if (!pg) + return 0; + else + return PageAnon(pg) ? MINCORE_ANON : 0; +} +
How do your program handle tmpfs pages (and/or ram device pages)?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/3] mincore: Introduce the MINCORE_ANON bit
Quoted 6 lines · KOSAKI Motohiro
Quoted 6 lines
mm/mincore.c +6
-0+static unsigned char mincore_pte (struct vm_area_struct *vma,unsigned long addr,pte_t pte) +{ +struct page *pg; + + pg =vm_normal_page (vma, addr, pte); +if (!pg)
Quoted 8 lines
mm/mincore.c static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff) · line 99
Quoted 5 lines
mm/mincore.c +5
-0+return 0 ; + else + return PageAnon(pg) ? MINCORE_ANON : 0; +} +How do your program handle tmpfs pages (and/or ram device pages)? .
Do you mean mapped files from tmpfs? Currently just any other file. Do you see problems with this patch wrt tmpfs?
Thanks,
Pavel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/3] mincore: Introduce the MINCORE_ANON bit
2011/12/26 Pavel Emelyanov <[email protected]>:
Quoted 15 lines · KOSAKI Motohiro
Quoted 5 lines
Quoted 5 lines
mm/mincore.c +5
-0+static unsigned char mincore_pte (struct vm_area_struct *vma,unsigned long addr,pte_t pte) +{ + struct page *pg; + + pg = vm_normal_page(vma, addr, pte);Quoted 8 lines
mm/mincore.c static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff) · line 99
Quoted 6 lines
+ if (!pg)
+ return 0;
+ else
+ return PageAnon(pg) ?MINCORE_ANON: 0;mm/mincore.c +2
-0+} +How do your program handle tmpfs pages (and/or ram device pages)?
Quoted 2 lines
Do you mean mapped files from tmpfs? Currently just any other file. Do you see problems with this patch wrt tmpfs?
If you don't save mapped file on tmpfs or other volatile devices, the process might not restored. The data might already destroyed. The common strategy are two,
1) save all opened file by different ways.
2) save all mapped file even though clean file cache.
In both case, we don't reduce freezed data size. So, I'm interesting you strategy.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/3] mincore: Introduce the MINCORE_ANON bit
Quoted 16 lines · KOSAKI Motohiro
2011/12/26 Pavel Emelyanov <[email protected]>:
Quoted 15 lines · KOSAKI Motohiro
Quoted 13 lines
mm/mincore.c static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff) · line 99
Quoted 11 lines
mm/mincore.c +11
-0+static unsigned char mincore_pte (struct vm_area_struct *vma,unsigned long addr,pte_t pte) +{ +struct page *pg; + + pg =vm_normal_page (vma, addr, pte); +if (!pg) +return 0 ; +else +return PageAnon (pg) ?MINCORE_ANON :0 ; +} +How do your program handle tmpfs pages (and/or ram device pages)?
Quoted 5 lines
Quoted 2 lines
Do you mean mapped files from tmpfs? Currently just any other file. Do you see problems with this patch wrt tmpfs?
If you don't save mapped file on tmpfs or other volatile devices, the process might not restored. The data might already destroyed.
Yes I know this, thanks :\
Quoted 7 lines
The common strategy are two,
1) save all opened file by different ways.
2) save all mapped file even though clean file cache.In both case, we don't reduce freezed data size. So, I'm interesting you strategy.
The tmpfs contents itself is supposed to be preserved, it's not a problem. The problem I'm trying to solve here is which page from task mappings (i.e. vm_area_struct-s) to save and which not to.
Do the proposed MINCORE_RESIDENT and MINCORE_ANON bits have problems with this from your POV?
Thanks,
Pavel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/3] mincore: Introduce the MINCORE_ANON bit
Quoted 5 lines
The tmpfs contents itself is supposed to be preserved, it's not a problem. The problem I'm trying to solve here is which page from task mappings (i.e. vm_area_struct-s) to save and which not to.
Do the proposed
MINCORE_RESIDENTandMINCORE_ANONbits have problems with this from your POV?
No problem.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/3 mincore: Introduce the MINCORE_SWAP bit
In reply to Pavel EmelyanovWhen a page is swapped out it should be included in the memory dump, but the existing mincore() doesn't report the set bit for such pages.
Thus add a bit for swapped out pages.
--- include/linux/mman.h | 1 + mm/mincore.c | 2 +- 2 files changed, 2 insertions(+), 1 deletions(-)
include/linux/mman.h +1 -0
@@ -12,6 +12,7 @@#define MINCORE_RESIDENT 0x1#define MINCORE_ANON 0x2 +#define MINCORE_SWAP 0x4#ifdef __KERNEL__#include <linux/mm.h>
mm/mincore.c +1 -1
@@ -146,7 +146,7 @@ static void mincore_pte_range(struct vm_area_struct *vma, pmd_t *pmd, }else {#ifdef CONFIG_SWAP pgoff = entry.val; - *vec =mincore_page (&swapper_space, pgoff); + *vec =mincore_page (&swapper_space, pgoff) |MINCORE_SWAP ;#else WARN_ON (1 ); *vec =MINCORE_RESIDENT ;
Signature · 8 lines
1.5.5.6 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 3/3] mincore: Introduce the MINCORE_SWAP bit
Quoted 4 lines · Pavel Emelyanov
When a page is swapped out it should be included in the memory dump, but the existing
mincore()doesn't report the set bit for such pages.Thus add a bit for swapped out pages.
We can reuse ANON bit there.
Plus you forget to handle migration-entries, they can be both file and anon too.
Quoted 25 lines
Signed-off-by Pavel Emelyanov<[email protected]>--- include/linux/mman.h | 1 + mm/mincore.c | 2 +- 2 files changed, 2 insertions(+), 1 deletions(-)include/linux/mman.h +1
-0@@ -12,6 +12,7 @@#define MINCORE_RESIDENT 0x1#define MINCORE_ANON 0x2 +#define MINCORE_SWAP 0x4#ifdef __KERNEL__#include <linux/mm.h> mm/mincore.c +0
-0
mm/mincore.c static void mincore_pte_range(struct vm_area_struct *vma, pmd_t *pmd, · line 152
Quoted 8 lines
mm/mincore.c +1
-1@@ -146,7 +146,7 @@ static void mincore_pte_range(struct vm_area_struct *vma, pmd_t *pmd, }else {#ifdef CONFIG_SWAP pgoff = entry.val; - *vec =mincore_page (&swapper_space, pgoff); + *vec =mincore_page (&swapper_space, pgoff) |MINCORE_SWAP ;#else WARN_ON (1 ); *vec =MINCORE_RESIDENT ;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>