C/R minisummit notes

* What are the problems that the linux community can solve with the
checkpoint/restart ?

Eric Biederman reminds at the previous OLS nobody complained about the
checkpoint/restart

Hi,

I've placed a somewhat more detailed summary on the wiki:
http://wiki.openvz.org/Containers/Mini-summit_2008_notes

(also accessible from: http://wiki.openvz.org/Containers/Mini-summit_2008)

To further discuss technical details, let's schedule to meet while we are here for the OLS. I suggest the following for a start:

1) Dinner tonight at 7:30pm. Suggestions for a venue are welcome :)
2) Breakfast tomorrow before the OLS, at 8:30am at the congress center.

Please confirm your participation.

Hi,

I've placed a somewhat more detailed summary on the wiki:
http://wiki.openvz.org/Containers/Mini-summit_2008_notes

(also accessible from: http://wiki.openvz.org/Containers/Mini-summit_2008)

To further discuss technical details, let's schedule to meet while we are here for the OLS. I suggest the following for a start:

1) Dinner tonight at 7:30pm. Suggestions for a venue are welcome :)

2) Breakfast tomorrow before the OLS, at 8:30am at the congress center.

Hi,

I've placed a somewhat more detailed summary on the wiki:
http://wiki.openvz.org/Containers/Mini-summit_2008_notes

(also accessible from: http://wiki.openvz.org/Containers/Mini-summit_2008)

To further discuss technical details, let's schedule to meet while we are here for the OLS. I suggest the following for a start:

1) Dinner tonight at 7:30pm. Suggestions for a venue are welcome :)
2) Breakfast tomorrow before the OLS, at 8:30am at the congress center.

Please confirm your participation.

* What are the problems that the linux community can solve with the
checkpoint/restart ?

Eric Biederman reminds at the previous OLS nobody complained about the
checkpoint/restart

Pavel Emylianov : The startup of Oracle takes some minutes, if we
checkpoint just after the startup, Oracle can be restarted from this point later and provide fast startup

Oren Laaden : Time travel, we can do monotonic snapshot and go back on
one of this snaphost.

Eric Biedreman : Priority running, checkpoint/kill an application and
run another application with a bigger priority

Denis Lunev : Task migration, move application on one host to another host

Daniel Lezcano : SSI (task migration)

* Preparing the kernel internals

OL : Can we implement a kernel module and move CR functionality into
the kernel itself later ?

EB : Better to add a little CR functionnality into the kernel itself
and add more after.

DLu : Problem with kernel version

OL : Compatibility with intermediate kernel version should be possible
with userspace conversion tools

DLu : Non sequential file for checkpoint statefile is a challenge

OL : yes, but possible and useful for compression/encryption

We showed that there are five steps to realize a checkpoint:

1 - Pre-dump

2 - Freeze
3 - Dump
4 - Resume/kill
5 - Post-dump

At this point we state we want create a proof of concept and
checkpoint/restart the simplest application.

We will add iteratively more and more kernel resources.

Process hierarchy created from kernel or userspace ?

OL : Seems better to send a chunk of data to kernel and that restores
the processes hierarchy
PE : Agreed
OL : We should be able to checkpoint from inside the container, keep
that in mind for later.

=> we need a syscall or a ioctl

The first items to address before implementing the Checkpoint are:
1 - Make a container object (the context)
2 - Freeze the container (extend cgroup freezer ?)
3 - syscall | ioctl

First step:
* simplest application : A single process, without any file, no
checkpoint of text file (same file system for restart), no signals, no syscall in the application, no ipc/no msgq, no network

Second step:
* multiple processes + zombie state

Third step:
* files, pipe, signals, socketpair ?

This proof of concept must came with a documentation describing what is
supported, what is not supported and what we plan to do.

Quoted 33 lines · Daniel Lezcano

* What are the problems that the linux community can solve with the
checkpoint/restart ?

Eric Biederman reminds at the previous OLS nobody complained about the
checkpoint/restart

Pavel Emylianov : The startup of Oracle takes some minutes, if we
checkpoint just after the startup, Oracle can be restarted from this point later and provide fast startup

Oren Laaden : Time travel, we can do monotonic snapshot and go back on
one of this snaphost.

Eric Biedreman : Priority running, checkpoint/kill an application and
run another application with a bigger priority

Denis Lunev : Task migration, move application on one host to another host

Daniel Lezcano : SSI (task migration)

* Preparing the kernel internals

OL : Can we implement a kernel module and move CR functionality into
the kernel itself later ?

EB : Better to add a little CR functionnality into the kernel itself
and add more after.

DLu : Problem with kernel version

OL : Compatibility with intermediate kernel version should be possible
with userspace conversion tools

Quoted 7 lines

DLu : Non sequential file for checkpoint statefile is a challenge

OL : yes, but possible and useful for compression/encryption

We showed that there are five steps to realize a checkpoint:

1 - Pre-dump

I'd just add here that the pre-dump is where you might start writing memory to disk, trying to get disk and memory closer and closer to being the same until, at some point, you decide they are close enough that you can go on to step two, and attempt the freeze+dump+migrate/kill with minimal downtime.

Coming into the discussion my primary concern had been that doing a sys_checkpoint() system call would be tough to augment to provide this kind of incremental checkpoint, but this breakdown is great for that.

Quoted 5 lines

2 - Freeze
3 - Dump
4 - Resume/kill
5 - Post-dump

Quoted 2 lines

At this point we state we want create a proof of concept and
checkpoint/restart the simplest application.

By which we mean, start with a piece of step 3 (and maybe a bit of step 4).

Step 2 was pretty widely accepted to be the freezer subsystem, but noone seemed to be sure quite what the status of that was.

Matt, can you remind us how the freezer cgroup is doing?

Quoted 28 lines

We will add iteratively more and more kernel resources.

Process hierarchy created from kernel or userspace ?

OL : Seems better to send a chunk of data to kernel and that restores
the processes hierarchy
PE : Agreed
OL : We should be able to checkpoint from inside the container, keep
that in mind for later.

=> we need a syscall or a ioctl

The first items to address before implementing the Checkpoint are:
1 - Make a container object (the context)
2 - Freeze the container (extend cgroup freezer ?)
3 - syscall | ioctl

First step:
* simplest application : A single process, without any file, no
checkpoint of text file (same file system for restart), no signals, no syscall in the application, no ipc/no msgq, no network

Second step:
* multiple processes + zombie state

Third step:
* files, pipe, signals, socketpair ?

Quoted 2 lines

This proof of concept must came with a documentation describing what is
supported, what is not supported and what we plan to do.

And there was talk of making sure that if you attempt to checkpoint an app using unsupported resources, we return -EAGAIN. There had been murmurings about giving more meaningful feedback, but I have no idea what that would look like.

-serge
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Oren Laadan [[email protected]] wrote:

| 
| 
| Serge E. Hallyn wrote:
| > Quoting Daniel Lezcano (dlezcano-NmTC/[email protected]):
| >>   * What are the problems that the linux community can solve with the 
| >> checkpoint/restart ?
| >>
| >> 	Eric Biederman reminds at the previous OLS nobody complained about the 
| >> checkpoint/restart
| >>
| >> 	Pavel Emylianov : The startup of Oracle takes some minutes, if we 
| >> checkpoint just after the startup, Oracle can be restarted from this 
| >> point later and provide fast startup
| >>
| >> 	Oren Laaden : Time travel, we can do monotonic snapshot and go back on 
| >> one of this snaphost.
| >>
| >> 	Eric Biedreman : Priority running, checkpoint/kill an application and 
| >> run another application with a bigger priority
| >>
| >> 	Denis Lunev : Task migration, move application on one host to another host
| >>
| >> 	Daniel Lezcano : SSI (task migration)
| >>
| >>   * Preparing the kernel internals
| >>
| >> 	OL : Can we implement a kernel module and move CR functionality into 
| >> the kernel itself later ?
| >>
| >> 	EB : Better to add a little CR functionnality into the kernel itself 
| >> and add more after.
| >>
| >> 	DLu : Problem with kernel version
| >>
| >> 	OL : Compatibility with intermediate kernel version should be possible 
| >> with userspace conversion tools
| >>
| >> 	DLu : Non sequential file for checkpoint statefile is a challenge
| >>
| >> 	OL : yes, but possible and useful for compression/encryption
| >>
| >> 	We showed that there are five steps to realize a checkpoint:
| >>
| >> 	1 - Pre-dump
| > 
| > I'd just add here that the pre-dump is where you might start writing
| > memory to disk, trying to get disk and memory closer and closer to
| > being the same until, at some point, you decide they are close enough
| > that you can go on to step two, and attempt the freeze+dump+migrate/kill
| > with minimal downtime.
| > 
| > Coming into the discussion my primary concern had been that doing a
| > sys_checkpoint() system call would be tough to augment to provide this
| > kind of incremental checkpoint, but this breakdown is great for that.
| > 
| >> 	2 - Freeze
| >> 	3 - Dump
| >> 	4 - Resume/kill
| >> 	5 - Post-dump
| >>
| >> 	At this point we state we want create a proof of concept and 
| >> checkpoint/restart the simplest application.
| > 
| > By which we mean, start with a piece of step 3 (and maybe a bit of
| > step 4).
| 
| step 4 is also part of the freezer -- it's the unfreeze operation
| (or force a SIGKILL to all processes in the container).

Are steps 1-5 considered part of the sys_checkpoint() system call and if successful sys_checkpoint() returns after step 5 ?

If so, like Serge points out, it would be harder to optimize for incremental checkpoints (as each sys_checkpoint() would be independent) ?

But may not be something to worry about for POC.

No no, the idea (IIUC) is that if you want to do a very short-downtime migrate, you stay in step 1 for a long time, writing the container memory to disk, checking how different the disk img is from the memory image, updating the version on disk, checking again, etc. Then when you decide that the disk and memory are very close together, you quickly do steps 2-4, where 4 in this case is kill. In the meantime you would have been loading the disk data into memory ahead of time at the new machine, so you can also quickly complete the restart.

So 3, 'Dump', in this case really becomes "dump the metadata and any more changes that have happened." Presumably, if when you get to 3, you find that there was suddenly a lot of activity and there is too much data to write quickly, you bail on the migrate and step 4 is a resume rather than kill. Then you start again at step 1.

At least that was my understanding.

Currently we have three possibilities on how to name pid namespaces.
- indirect via processes
- pids
- names in the filesystem

We discussed this a bit in the hallway track and pids are look like the way to go. Pavel has a patch in progress to help sort this out.

The practical problem we have today is that we need a way to wait for the network namespace in particular and namespaces in general to exit.

At a first glance waitid(P_NS, <pid>,....) looks like a useful way to achieve this. After looking at wait a bit more it really is fundamentally just an exit status reaper of zombies, that has the option of blocking when the zombies do not yet exist. In any kind of event loop you would wait for SIGCHLD either as a signal or with signalfd.

So how shall we wait for a namespace to exit? My brainstorm tonight suggests inotify_add_watch(ifd, "/proc/ns/<pid>", IN_DELETE);

Eric

Quoted 17 lines · Eric W. Biederman

Currently we have three possibilities on how to name pid namespaces.
- indirect via processes
- pids
- names in the filesystem

We discussed this a bit in the hallway track and pids are look like the way to go. Pavel has a patch in progress to help sort this out.

The practical problem we have today is that we need a way to wait for the network namespace in particular and namespaces in general to exit.

At a first glance waitid(P_NS, <pid>,....) looks like a useful way to achieve this. After looking at wait a bit more it really is fundamentally just an exit status reaper of zombies, that has the option of blocking when the zombies do not yet exist. In any kind of event loop you would wait for SIGCHLD either as a signal or with signalfd.

Quoted 4 lines

So how shall we wait for a namespace to exit? My brainstorm tonight suggests inotify_add_watch(ifd, "/proc/ns/<pid>", IN_DELETE);

Eric

I'm sorry, I'm still not quite clear on...

Why?

You care about when the tasks exit, and you care about when network devices, for instance, need to be deleted (which you can presumably get uevents for, when they get moved back into init_net_ns).

Why do you care when the struct net actually gets deleted?

Quoted 25 lines · Serge E. Hallyn

Quoted 17 lines · Eric W. Biederman

Currently we have three possibilities on how to name pid namespaces.
- indirect via processes
- pids
- names in the filesystem

We discussed this a bit in the hallway track and pids are look like the way to go. Pavel has a patch in progress to help sort this out.

The practical problem we have today is that we need a way to wait for the network namespace in particular and namespaces in general to exit.

At a first glance waitid(P_NS, <pid>,....) looks like a useful way to achieve this. After looking at wait a bit more it really is fundamentally just an exit status reaper of zombies, that has the option of blocking when the zombies do not yet exist. In any kind of event loop you would wait for SIGCHLD either as a signal or with signalfd.

Quoted 4 lines

So how shall we wait for a namespace to exit? My brainstorm tonight suggests inotify_add_watch(ifd, "/proc/ns/<pid>", IN_DELETE);

Eric

I'm sorry, I'm still not quite clear on...

Quoted 7 lines

Why?

You care about when the tasks exit, and you care about when network devices, for instance, need to be deleted (which you can presumably get uevents for, when they get moved back into init_net_ns).

Why do you care when the struct net actually gets deleted?

IMO, if we consider a container being an aggregation of different namespaces, we should consider the container dies when all the namespaces are dead.

One good example is an application ran inside a container and doing a bulk data writing over the network. When the application finish its last
call to "send" it will exits. At this point, there is no more processes
running inside the container but we can not consider the container is dead because there are still some pending datas in the socket to be delivered to the peer.

Eric will post a patch to automatically destroy the virtual devices when the netns is destroyed, so there is no way to know if a network namespace is dead or not as the uevent socket will not deliver an event outside of the container.

Quoted 43 lines · Daniel Lezcano

Quoted 24 lines · Serge E. Hallyn

Quoted 17 lines · Eric W. Biederman

Currently we have three possibilities on how to name pid namespaces.
- indirect via processes
- pids
- names in the filesystem

We discussed this a bit in the hallway track and pids are look like the way to go. Pavel has a patch in progress to help sort this out.

The practical problem we have today is that we need a way to wait for the network namespace in particular and namespaces in general to exit.

At a first glance waitid(P_NS, <pid>,....) looks like a useful way to achieve this. After looking at wait a bit more it really is fundamentally just an exit status reaper of zombies, that has the option of blocking when the zombies do not yet exist. In any kind of event loop you would wait for SIGCHLD either as a signal or with signalfd.

Quoted 4 lines

So how shall we wait for a namespace to exit? My brainstorm tonight suggests inotify_add_watch(ifd, "/proc/ns/<pid>", IN_DELETE);

Eric

I'm sorry, I'm still not quite clear on...

Quoted 7 lines

Why?

You care about when the tasks exit, and you care about when network devices, for instance, need to be deleted (which you can presumably get uevents for, when they get moved back into init_net_ns).

Why do you care when the struct net actually gets deleted?

IMO, if we consider a container being an aggregation of different namespaces, we should consider the container dies when all the namespaces are dead.

One good example is an application ran inside a container and doing a bulk data writing over the network. When the application finish its last
call to "send" it will exits. At this point, there is no more processes
running inside the container but we can not consider the container is dead because there are still some pending datas in the socket to be delivered to the peer.

Quoted 4 lines

Eric will post a patch to automatically destroy the virtual devices when the netns is destroyed, so there is no way to know if a network namespace is dead or not as the uevent socket will not deliver an event outside of the container.

My question remains: who cares?

Quoted 43 lines · Daniel Lezcano

Quoted 24 lines · Serge E. Hallyn

Quoted 17 lines · Eric W. Biederman

Currently we have three possibilities on how to name pid namespaces.
- indirect via processes
- pids
- names in the filesystem

We discussed this a bit in the hallway track and pids are look like the way to go. Pavel has a patch in progress to help sort this out.

The practical problem we have today is that we need a way to wait for the network namespace in particular and namespaces in general to exit.

At a first glance waitid(P_NS, <pid>,....) looks like a useful way to achieve this. After looking at wait a bit more it really is fundamentally just an exit status reaper of zombies, that has the option of blocking when the zombies do not yet exist. In any kind of event loop you would wait for SIGCHLD either as a signal or with signalfd.

Quoted 4 lines

So how shall we wait for a namespace to exit? My brainstorm tonight suggests inotify_add_watch(ifd, "/proc/ns/<pid>", IN_DELETE);

Eric

I'm sorry, I'm still not quite clear on...

Quoted 7 lines

Why?

You care about when the tasks exit, and you care about when network devices, for instance, need to be deleted (which you can presumably get uevents for, when they get moved back into init_net_ns).

Why do you care when the struct net actually gets deleted?

IMO, if we consider a container being an aggregation of different namespaces, we should consider the container dies when all the namespaces are dead.

One good example is an application ran inside a container and doing a bulk data writing over the network. When the application finish its last
call to "send" it will exits. At this point, there is no more processes
running inside the container but we can not consider the container is dead because there are still some pending datas in the socket to be delivered to the peer.

Quoted 4 lines

Eric will post a patch to automatically destroy the virtual devices when the netns is destroyed, so there is no way to know if a network namespace is dead or not as the uevent socket will not deliver an event outside of the container.

My question remains: who cares?

Quoted 5 lines

Quoted 4 lines

Eric will post a patch to automatically destroy the virtual devices when the netns is destroyed, so there is no way to know if a network namespace is dead or not as the uevent socket will not deliver an event outside of the container.

Quoted 1 line

My question remains: who cares?

The container implementation in userspace. Let's imagine it sets some routes outside of the container to route the traffic to the container. It should remove these routes when the container dies. And the container should be considered as dead when the network has died and not when the last process of the container exits.

Let's have some more breakfast, tomorrow - Friday - morning. Same place, same time. If it doesn't rain we'll go outside ;)