mm: Add ability to monitor task's memory changes

mm/madvise.c static long madvise_behavior(struct vm_area_struct * vma, · line 77

Quoted 12 lines · Pavel Emelyanov

mm/madvise.c +11 -0

 	case MADV_DOTRACE:
+		/*
+		 * Protect pages to be read-only and force tasks to generate
+		 * #PFs on modification.
+		 *
+		 * It should be done before issuing trace-on event. Otherwise
+		 * we're leaving a short window after the 'on' event when tasks
+		 * can still modify pages.
+		 */
+		change_protection(vma, start, end,
+				vm_get_page_prot(vma->vm_flags & ~VM_READ),
+				vma_wants_writenotify(vma));

Should be VM_WRITE?

Hello,

This is an attempt to implement support for memory snapshot for the the checkpoint-restore project (http://criu.org).

To create a dump of an application(s) we save all the information about it to files. No surprise, the biggest part of such dump is the contents of tasks' memory. However, in some usage scenarios it's not required to get _all_ the task memory while creating a dump. For example, when doing periodical dumps it's only required to take full memory dump only at the first step and then take incremental changes of memory. Another example is live migration. In the simplest form it looks like -- create dump, copy it on the remote node then restore tasks from dump files. While all this dump-copy-restore thing goes all the process must be stopped. However, if we can monitor how tasks change their memory, we can dump and copy it in smaller chunks, periodically updating it and thus freezing tasks only at the very end for the very short time to pick up the recent changes.

That said, some help from kernel to watch how processes modify the contents of their memory is required. I'd like to propose one possible solution of this task -- with the help of page-faults and trace events.

Briefly the approach is -- remap some memory regions as read-only, get the #pf on task's attempt to modify the memory and issue a trace event of that. Since we're only interested in parts of memory of some tasks, make it possible to mark the vmas we're interested in and issue events for them only. Also, to be aware of tasks unmapping the vma-s being watched, also issue an event when the marked vma is removed (and for symmetry -- an event when a vma is marked).

What do you think about this approach? Is this way of supporting mem snapshot OK for you, or should we invent some better one?

Quoted 29 lines · Pavel Emelyanov

Hello,

This is an attempt to implement support for memory snapshot for the the checkpoint-restore project (http://criu.org).

To create a dump of an application(s) we save all the information about it to files. No surprise, the biggest part of such dump is the contents of tasks' memory. However, in some usage scenarios it's not required to get _all_ the task memory while creating a dump. For example, when doing periodical dumps it's only required to take full memory dump only at the first step and then take incremental changes of memory. Another example is live migration. In the simplest form it looks like -- create dump, copy it on the remote node then restore tasks from dump files. While all this dump-copy-restore thing goes all the process must be stopped. However, if we can monitor how tasks change their memory, we can dump and copy it in smaller chunks, periodically updating it and thus freezing tasks only at the very end for the very short time to pick up the recent changes.

That said, some help from kernel to watch how processes modify the contents of their memory is required. I'd like to propose one possible solution of this task -- with the help of page-faults and trace events.

Briefly the approach is -- remap some memory regions as read-only, get the #pf on task's attempt to modify the memory and issue a trace event of that. Since we're only interested in parts of memory of some tasks, make it possible to mark the vmas we're interested in and issue events for them only. Also, to be aware of tasks unmapping the vma-s being watched, also issue an event when the marked vma is removed (and for symmetry -- an event when a vma is marked).

Quoted 3 lines

What do you think about this approach? Is this way of supporting mem snapshot OK for you, or should we invent some better one?

The page fault mechanism is pretty obvious - anything that deals with dirty pages will end up having to do this. So there is nothing crazy about this.

What concerns me, however, is that should this go in, we'll have two dirty mem loggers in the kernel: one to support CRIU, one to support KVM. And the worst part: They have the exact the same purpose!!

So to begin with, I think one thing to consider, would be to generalize KVM's dirty memory notification so it can work on a normal process memory region. KVM api requires a "memory slot" to be passed, something we are unlikely to have. But KVM can easily keep its API and use an alternate mechanics, that's trivial...

Generally speaking, KVM will do polling with this ioctl. I prefer your tracing mechanism better. The only difference, is that KVM tends to transfer large chunks of memory in some loads - in the high gigs range. So the proposal tracing API should be able to optionally batch requests within a time frame.

It would also be good to hear what does the KVM guys think of it as well

Quoted 54 lines · Glauber Costa

Quoted 29 lines · Pavel Emelyanov

Hello,

This is an attempt to implement support for memory snapshot for the the checkpoint-restore project (http://criu.org).

To create a dump of an application(s) we save all the information about it to files. No surprise, the biggest part of such dump is the contents of tasks' memory. However, in some usage scenarios it's not required to get _all_ the task memory while creating a dump. For example, when doing periodical dumps it's only required to take full memory dump only at the first step and then take incremental changes of memory. Another example is live migration. In the simplest form it looks like -- create dump, copy it on the remote node then restore tasks from dump files. While all this dump-copy-restore thing goes all the process must be stopped. However, if we can monitor how tasks change their memory, we can dump and copy it in smaller chunks, periodically updating it and thus freezing tasks only at the very end for the very short time to pick up the recent changes.

That said, some help from kernel to watch how processes modify the contents of their memory is required. I'd like to propose one possible solution of this task -- with the help of page-faults and trace events.

Briefly the approach is -- remap some memory regions as read-only, get the #pf on task's attempt to modify the memory and issue a trace event of that. Since we're only interested in parts of memory of some tasks, make it possible to mark the vmas we're interested in and issue events for them only. Also, to be aware of tasks unmapping the vma-s being watched, also issue an event when the marked vma is removed (and for symmetry -- an event when a vma is marked).

Quoted 3 lines

What do you think about this approach? Is this way of supporting mem snapshot OK for you, or should we invent some better one?

The page fault mechanism is pretty obvious - anything that deals with dirty pages will end up having to do this. So there is nothing crazy about this.

What concerns me, however, is that should this go in, we'll have two dirty mem loggers in the kernel: one to support CRIU, one to support KVM. And the worst part: They have the exact the same purpose!!

So to begin with, I think one thing to consider, would be to generalize KVM's dirty memory notification so it can work on a normal process memory region. KVM api requires a "memory slot" to be passed, something we are unlikely to have. But KVM can easily keep its API and use an alternate mechanics, that's trivial...

Generally speaking, KVM will do polling with this ioctl. I prefer your tracing mechanism better. The only difference, is that KVM tends to transfer large chunks of memory in some loads - in the high gigs range. So the proposal tracing API should be able to optionally batch requests within a time frame.

Quoted 1 line

It would also be good to hear what does the KVM guys think of it as well

There are significant differences. KVM's dirty logging works for guest translations (NPT/shadow) and is optimized for specific use cases.

Above is about dirty logging of userspace memory areas.

This is an attempt to implement support for memory snapshot for the the checkpoint-restore project (http://criu.org).

To create a dump of an application(s) we save all the information about it to files. No surprise, the biggest part of such dump is the contents of tasks' memory. However, in some usage scenarios it's not required to get _all_ the task memory while creating a dump. For example, when doing periodical dumps it's only required to take full memory dump only at the first step and then take incremental changes of memory. Another example is live migration. In the simplest form it looks like -- create dump, copy it on the remote node then restore tasks from dump files. While all this dump-copy-restore thing goes all the process must be stopped. However, if we can monitor how tasks change their memory, we can dump and copy it in smaller chunks, periodically updating it and thus freezing tasks only at the very end for the very short time to pick up the recent changes.

That said, some help from kernel to watch how processes modify the contents of their memory is required. I'd like to propose one possible solution of this task -- with the help of page-faults and trace events.

Briefly the approach is -- remap some memory regions as read-only, get the #pf on task's attempt to modify the memory and issue a trace event of that. Since we're only interested in parts of memory of some tasks, make it possible to mark the vmas we're interested in and issue events for them only. Also, to be aware of tasks unmapping the vma-s being watched, also issue an event when the marked vma is removed (and for symmetry -- an event when a vma is marked).

What do you think about this approach? Is this way of supporting mem snapshot OK for you, or should we invent some better one?

Quoted 27 lines · On Fri, 30 Nov 2012 21:55:00 +0400 Pavel Emelyanov <[email protected]> wrote:

This is an attempt to implement support for memory snapshot for the the checkpoint-restore project (http://criu.org).

To create a dump of an application(s) we save all the information about it to files. No surprise, the biggest part of such dump is the contents of tasks' memory. However, in some usage scenarios it's not required to get _all_ the task memory while creating a dump. For example, when doing periodical dumps it's only required to take full memory dump only at the first step and then take incremental changes of memory. Another example is live migration. In the simplest form it looks like -- create dump, copy it on the remote node then restore tasks from dump files. While all this dump-copy-restore thing goes all the process must be stopped. However, if we can monitor how tasks change their memory, we can dump and copy it in smaller chunks, periodically updating it and thus freezing tasks only at the very end for the very short time to pick up the recent changes.

That said, some help from kernel to watch how processes modify the contents of their memory is required. I'd like to propose one possible solution of this task -- with the help of page-faults and trace events.

Briefly the approach is -- remap some memory regions as read-only, get the #pf on task's attempt to modify the memory and issue a trace event of that. Since we're only interested in parts of memory of some tasks, make it possible to mark the vmas we're interested in and issue events for them only. Also, to be aware of tasks unmapping the vma-s being watched, also issue an event when the marked vma is removed (and for symmetry -- an event when a vma is marked).

Quoted 2 lines

What do you think about this approach? Is this way of supporting mem snapshot OK for you, or should we invent some better one?

The patches look pretty simple.

Some performance numbers would be useful.

Is it reliable? Under what circumstances will the trace system drop events?

Please cc Steven Rostedt on tracing stuff - he is a diligent reviewer.

The proposed interface might be useful to things other than c/r. But it hasn't actually been described. Please include a full description of the proposed kernel/usersapce interface.

Two alternatives come to mind:

1) Use /proc/pid/pagemap (Documentation/vm/pagemap.txt) in some fashion to determine which pages have been touched.

2) At pagefault time, don't send an event: just mark the vma as "touched". Then add a userspace interface to sweep the vma tree testing, clearing and reporting the touched flags.

2a) Avoid the full linear search by propagating the "touched" flag
up the rbtree and do the sweep in a fashion similar to radix_tree_for_each_tagged().
.

Quoted 5 lines

Two alternatives come to mind:

1) Use /proc/pid/pagemap (Documentation/vm/pagemap.txt) in some fashion to determine which pages have been touched.

I thought about this. Unfortunately there's no free bits left in the pagemap entry. What can we do about it (other than introducing the pagemap2 file)?

Quoted 3 lines

2) At pagefault time, don't send an event: just mark the vma as "touched". Then add a userspace interface to sweep the vma tree testing, clearing and reporting the touched flags.

Per-vma granularity is not enough. In OpenVZ we've observed Oracle touching several pages in a hundred-megs anon mapping. Marking _part_ of the vma with the "node write-faults" bit would help, but there's currently no APIs that modifies vma and report some info back at the same time. Can you propose how it could look like?

Quoted 5 lines · On Tue, 04 Dec 2012 09:15:10 +0400 Pavel Emelyanov <[email protected]> wrote:

Quoted 5 lines

Two alternatives come to mind:

1) Use /proc/pid/pagemap (Documentation/vm/pagemap.txt) in some fashion to determine which pages have been touched.

Quoted 8 lines · Andrew Morton

Quoted 5 lines · On Tue, 04 Dec 2012 09:15:10 +0400 Pavel Emelyanov <[email protected]> wrote:

Quoted 5 lines

Two alternatives come to mind:

1) Use /proc/pid/pagemap (Documentation/vm/pagemap.txt) in some fashion to determine which pages have been touched.

[momentarily coming out of kernel retirement for old man rant]

This is a popular interface anti-pattern.

You shouldn't use an interface that gives you huge amount of STATE to detect small amounts of CHANGE via manual differentiation.

Quoted 14 lines · On Tue, 04 Dec 2012 18:17:08 -0600 Matt Mackall <[email protected]> wrote:

Quoted 8 lines · Andrew Morton

Quoted 5 lines · On Tue, 04 Dec 2012 09:15:10 +0400 Pavel Emelyanov <[email protected]> wrote:

Quoted 5 lines

Two alternatives come to mind:

1) Use /proc/pid/pagemap (Documentation/vm/pagemap.txt) in some fashion to determine which pages have been touched.

[momentarily coming out of kernel retirement for old man rant]

This is a popular interface anti-pattern.

Quoted 2 lines

You shouldn't use an interface that gives you huge amount of STATE to detect small amounts of CHANGE via manual differentiation.

I'm not sure that's what checkpoint-restart will be doing. If we want to determine "which pages have been touched since the last checkpoint ten minutes ago" then that set of touched pages *is* state. And it's not "small"!

Quoted 17 lines · Andrew Morton

Quoted 14 lines · On Tue, 04 Dec 2012 18:17:08 -0600 Matt Mackall <[email protected]> wrote:

Quoted 8 lines · Andrew Morton

Quoted 5 lines · On Tue, 04 Dec 2012 09:15:10 +0400 Pavel Emelyanov <[email protected]> wrote:

Quoted 5 lines

Two alternatives come to mind:

1) Use /proc/pid/pagemap (Documentation/vm/pagemap.txt) in some fashion to determine which pages have been touched.

[momentarily coming out of kernel retirement for old man rant]

This is a popular interface anti-pattern.

Quoted 7 lines

Quoted 2 lines

You shouldn't use an interface that gives you huge amount of STATE to detect small amounts of CHANGE via manual differentiation.

I'm not sure that's what checkpoint-restart will be doing. If we want to determine "which pages have been touched since the last checkpoint ten minutes ago" then that set of touched pages *is* state. And it's not "small"!

Yeah, there is definitely a middle-ground here between "I want high-frequency updates" and "I want to see the whole picture". The filesystem analogy is backups: we don't have any good way to say "find me all files changed since yesterday" short of "find all files". The closest thing is explicit snapshotting.

Quoted 26 lines · Matt Mackall

Quoted 17 lines · Andrew Morton

Quoted 14 lines · On Tue, 04 Dec 2012 18:17:08 -0600 Matt Mackall <[email protected]> wrote:

Quoted 8 lines · Andrew Morton

Quoted 5 lines · On Tue, 04 Dec 2012 09:15:10 +0400 Pavel Emelyanov <[email protected]> wrote:

Quoted 5 lines

Two alternatives come to mind:

1) Use /proc/pid/pagemap (Documentation/vm/pagemap.txt) in some fashion to determine which pages have been touched.

[momentarily coming out of kernel retirement for old man rant]

This is a popular interface anti-pattern.

Quoted 7 lines

Quoted 2 lines

You shouldn't use an interface that gives you huge amount of STATE to detect small amounts of CHANGE via manual differentiation.

I'm not sure that's what checkpoint-restart will be doing. If we want to determine "which pages have been touched since the last checkpoint ten minutes ago" then that set of touched pages *is* state. And it's not "small"!

Quoted 5 lines

Yeah, there is definitely a middle-ground here between "I want high-frequency updates" and "I want to see the whole picture". The filesystem analogy is backups: we don't have any good way to say "find me all files changed since yesterday" short of "find all files". The closest thing is explicit snapshotting.

For what is required for checkpoint-restore is -- we want to query the kernel for "what pages has been written to since moment X". But this "moment X" is a little bit more tricky than just "mark all pages r/o". Consider we're doing this periodically. So when defining the moment X for the 2nd time we should query the "changed" state and remap the respective page r/o atomically. Full snapshot is actually not required, since we don't need to keep the old copy of a page that is written to. Just a sign, that this page was modified is OK.

Quoted 7 lines

For what is required for checkpoint-restore is -- we want to query the kernel for "what pages has been written to since moment X". But this "moment X" is a little bit more tricky than just "mark all pages r/o". Consider we're doing this periodically. So when defining the moment X for the 2nd time we should query the "changed" state and remap the respective page r/o atomically. Full snapshot is actually not required, since we don't need to keep the old copy of a page that is written to. Just a sign, that this page was modified is OK.

How is all this going to work, btw? What is the interface to query page states and set them read-only? How will dirty pagecache and dirty swapcache be handled? And anonymous memory?