Checkpoint/Restart mini-summit
Hi all,
Here is a proposition a more detailed agenda for the checkpoint/restart mini-summit. If everybody is ok with it, I will update the wiki.
Comments are welcome :)
Thanks
-- Daniel
========================================================================
The Checkpoint/restart is a very big topic and the time at the mini-summit is short, so I propose a list of document pointers to be read before the mini-summit, so we can address the checkpoint/restart topic directly and save precious time :)
* Documentation
* Zap : www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
* Metacluster : lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
* OpenVZ : http://wiki.openvz.org/Checkpointing_and_live_migration
* Checkpoint/Restart technology :
http://en.wikipedia.org/wiki/Application_checkpointing
* Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops
document
-------------------------------------------------------------------------------
This section is about how to prepare the kernel to implement the checkpoint/restart
* Preparing the kernel internals
* Identifying the kernel subsystems
* Identifying the process resources
* Identifying the frameworks for the CR
* Identifying the pieces to target first
Actually, one of the big interrogation is how we transmit the internal state to and from the kernel. There are some little patches doing the checkpoint/restart, taking into account a small part of the kernel resources. Some were made through netlink, others via /proc, others directly with a syscall. There were solutions proposed in the containers mailing list to use a core dump like file, or a CR filesystem. This section is to discuss about that.
* Passing the kernel internal state to/from userspace
* coredump like file
* swap per container
* netlinks
* CR filesystem
* army of different call for the CR (proc, existing syscalls, ...)
The following sections addresse the checkpoint/restart itself which can be split into three parts: the quescient point, the checkpoint and the restart.
* Checkpointing / Restarting
* Reaching a quescient point
* for the network
* for the processes
* for the asynchronous IO
* Checkpoint
* Preinstalled checkpoint signal handler ?
* syscall ?
* tar of a CR filesystem ?
* monolithic ?
* Dumping processes hierarchy
* Identifying the kernel resource dependencies
* Dumping system wide resources (per namespace ?)
* Dumping process wide resources (from process context ?) (Memory is in between system and process resource)
* Restarting
* New binary format handler ?
* Identifying the kernel resource dependencies
* Restoring the processes hierarchy
* Restoring system wide resources
* Restoring process wide resources
There is a posix draft, 1003.1m, which specify a CR semantic. This can be interesting to take it into account and provide an user API based on this specification so we can keep in mind this when we implement the CR in the kernel. I was not able to find the posix draft itself but the man of the CR IRIX implementation sticks to this specification.
* Determining the userspace API
* Posix 1003.1m (implementation in IRIX) ?
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi/0650/bks/SGI_Admin/CPR_OG/sgi_html/ch03.html
A message that is not in the archive (sent off-list or not yet received): 487C80C9.2040105-NmTC/[email protected]
Quoted 6 lines · Daniel Lezcano
Hi all,
Here is a proposition a more detailed agenda for the checkpoint/restart mini-summit. If everybody is ok with it, I will update the wiki.
Comments are welcome :)
A reading list is useful, even to help get some ideas circulating before we get there.
Ultimately the technical details will need to be resolve by people discussing things and sending patches back and forth on the mailing lists.
I don't think a detailed agenda is going to get us anywhere. Especially not one focused on the implementation details.
I think we need to start by seeing what we can agree on. Certainly we agree that checkpoint/restart needs to be part of the picture. What are the problems that the linux community can solve with checkpoint/restart.
Then we need to talk about what kind of implementation we want to merge into mainline. How do we sell it, and how do we implement it without affecting long term maintainability.
I think the granularity of our operations, and what state we save is important. I don't think how we save it is important unless it affects one of our requirements.
As for the posix draft and the historical Cray & SGI implementations. They were on the wrong track. The did not have namespace support so they could not in general restore their checkpoints.
There are also a lot of things you have failed to touch on, that I'm not going to go into now.
With any luck the mini-summit before OLS will be the start of a conversation that will go on all week, and continue on the mailing lists.
The real question is how do we coordinate our efforts to build a good linux checkpoint/restart implementation.
Quoted 8 lines
* Documentation
* Zap : www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
* Metacluster : lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
* OpenVZ : http://wiki.openvz.org/Checkpointing_and_live_migration
* Checkpoint/Restart technology :
http://en.wikipedia.org/wiki/Application_checkpointing
* Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops
document
There is also the classic emacs undump.
The very simple vmadump from bproc.
Eric
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Eric W. Biederman
Quoted 6 lines · Daniel Lezcano
Hi all,
Here is a proposition a more detailed agenda for the checkpoint/restart mini-summit. If everybody is ok with it, I will update the wiki.
Comments are welcome :)
A reading list is useful, even to help get some ideas circulating before we get there.
Quoted 6 lines
Ultimately the technical details will need to be resolve by people discussing things and sending patches back and forth on the mailing lists.
I don't think a detailed agenda is going to get us anywhere. Especially not one focused on the implementation details.
Right, the whole point of Daniel including a 'reading list' was just so that we can avoid wasting time discussing existing implementations. So he wasn't suggesting that we would be discussing those in detail, in fact quite the opposite.
Quoted 17 lines
I think we need to start by seeing what we can agree on. Certainly we agree that checkpoint/restart needs to be part of the picture. What are the problems that the linux community can solve with checkpoint/restart.
Then we need to talk about what kind of implementation we want to merge into mainline. How do we sell it, and how do we implement it without affecting long term maintainability.
I think the granularity of our operations, and what state we save is important. I don't think how we save it is important unless it affects one of our requirements.
As for the posix draft and the historical Cray & SGI implementations. They were on the wrong track. The did not have namespace support so they could not in general restore their checkpoints.
Quoted 6 lines
There are also a lot of things you have failed to touch on, that I'm not going to go into now.
With any luck the mini-summit before OLS will be the start of a conversation that will go on all week, and continue on the mailing lists.
Agreed.
This could be tough to pull off, but if we can walk out of there with a short focused list of coding todos with the intent of pumping out patches by the end of OLS, turning OLS into a bit of a hack-fest, that would imo be great.
(But then that's precisely what I like to do at conferences - sit by some wall and pick something completely new to code, while once in awhile getting up to chat or see a talk. Does that make me anti-social?)
Quoted 20 lines
The real question is how do we coordinate our efforts to build a good linux checkpoint/restart implementation.
Quoted 8 lines
* Documentation
* Zap : www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
* Metacluster : lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
* OpenVZ : http://wiki.openvz.org/Checkpointing_and_live_migration
* Checkpoint/Restart technology :
http://en.wikipedia.org/wiki/Application_checkpointing
* Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops
documentThere is also the classic emacs undump.
The very simple vmadump from bproc.Eric
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers
A message that is not in the archive (sent off-list or not yet received): 20080716151530.GA26496-r/[email protected]
Quoted 4 lines · Serge E. Hallyn
Right, the whole point of Daniel including a 'reading list' was just so that we can avoid wasting time discussing existing implementations. So he wasn't suggesting that we would be discussing those in detail, in fact quite the opposite.
Right however he was suggesting we discuss how to implement it without agreeing on basic principles. Maybe I'm wrong and all of the different development groups have a common idea of what needs to be done but I would be surprised if that were the case.
Quoted 10 lines
Quoted 3 lines
With any luck the mini-summit before OLS will be the start of a conversation that will go on all week, and continue on the mailing lists.
Agreed.
This could be tough to pull off, but if we can walk out of there with a short focused list of coding todos with the intent of pumping out patches by the end of OLS, turning OLS into a bit of a hack-fest, that would imo be great.
I totally agree.
Quoted 4 lines
(But then that's precisely what I like to do at conferences - sit by some wall and pick something completely new to code, while once in awhile getting up to chat or see a talk. Does that make me anti-social?)
It means you like to code!
Eric
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Eric W. Biederman
Quoted 4 lines · Serge E. Hallyn
Right, the whole point of Daniel including a 'reading list' was just so that we can avoid wasting time discussing existing implementations. So he wasn't suggesting that we would be discussing those in detail, in fact quite the opposite.
Right however he was suggesting we discuss how to implement it without agreeing on basic principles. Maybe I'm wrong and all of the different development groups have a common idea of what needs to be done but I would be surprised if that were the case.
IMO recent threads have clearly proven that you're right about basic principles. Question is what is the right level to start at first, and how do we go about reaching concensus? I.e. is the first question whether we should do a fully in-kernel checkpoint and restart vs entirely userspace vs a mix, or is there another place we should start?
It sounds like you have some good ideas in any case on where to start so I'm glad you'll be there :)
thanks,
-serge
Quoted 21 lines
Quoted 10 lines
Quoted 3 lines
With any luck the mini-summit before OLS will be the start of a conversation that will go on all week, and continue on the mailing lists.
Agreed.
This could be tough to pull off, but if we can walk out of there with a short focused list of coding todos with the intent of pumping out patches by the end of OLS, turning OLS into a bit of a hack-fest, that would imo be great.
I totally agree.
Quoted 4 lines
(But then that's precisely what I like to do at conferences - sit by some wall and pick something completely new to code, while once in awhile getting up to chat or see a talk. Does that make me anti-social?)
It means you like to code!
Eric
A message that is not in the archive (sent off-list or not yet received): 20080716193538.GA28393-r/[email protected]
Quoted 13 lines · Serge E. Hallyn
Quoted 11 lines · Eric W. Biederman
Quoted 4 lines · Serge E. Hallyn
Right, the whole point of Daniel including a 'reading list' was just so that we can avoid wasting time discussing existing implementations. So he wasn't suggesting that we would be discussing those in detail, in fact quite the opposite.
Right however he was suggesting we discuss how to implement it without agreeing on basic principles. Maybe I'm wrong and all of the different development groups have a common idea of what needs to be done but I would be surprised if that were the case.
Quoted 5 lines
IMO recent threads have clearly proven that you're right about basic principles. Question is what is the right level to start at first, and how do we go about reaching concensus? I.e. is the first question whether we should do a fully in-kernel checkpoint and restart vs entirely userspace vs a mix, or is there another place we should start?
Where all good kernel features start. With the necessary mechanisms in the kernel and the policy in user space.
If we want to replay all of the user space actions to create the environment at the time of the checkpoint we already have all of the kernel support we need as the applications go to their current state using current system calls.
I think we want something a bit more efficient.
Quoted 2 lines
It sounds like you have some good ideas in any case on where to start so I'm glad you'll be there :)
Will we have a white board or a large piece paper or something we can draw on and talk about?
Eric
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 31 lines · Eric W. Biederman
Quoted 13 lines · Serge E. Hallyn
Quoted 11 lines · Eric W. Biederman
Quoted 4 lines · Serge E. Hallyn
Right, the whole point of Daniel including a 'reading list' was just so that we can avoid wasting time discussing existing implementations. So he wasn't suggesting that we would be discussing those in detail, in fact quite the opposite.
Right however he was suggesting we discuss how to implement it without agreeing on basic principles. Maybe I'm wrong and all of the different development groups have a common idea of what needs to be done but I would be surprised if that were the case.
Quoted 5 lines
IMO recent threads have clearly proven that you're right about basic principles. Question is what is the right level to start at first, and how do we go about reaching concensus? I.e. is the first question whether we should do a fully in-kernel checkpoint and restart vs entirely userspace vs a mix, or is there another place we should start?
Where all good kernel features start. With the necessary mechanisms in the kernel and the policy in user space.
If we want to replay all of the user space actions to create the environment at the time of the checkpoint we already have all of the kernel support we need as the applications go to their current state using current system calls.
I think we want something a bit more efficient.
Quoted 8 lines
Quoted 2 lines
It sounds like you have some good ideas in any case on where to start so I'm glad you'll be there :)
Will we have a white board or a large piece paper or something we can draw on and talk about?
Eric
Hi C.,
I'm sorry after all the emails I'm not straight on what we're actually getting. We have a U-shaped room, but do we have a blackboard or whiteboard? Is there a speakerphone? Wireless or at least wired internet?
thanks,
-serge
A message that is not in the archive (sent off-list or not yet received): 20080717022729.GC21726-r/[email protected]
Hi Serge,
I can confirm the room and we can get you some type of a paper/white board. I should know tomorrow about the speakerphone and Internet access.
C.
Quoted 10 lines
Hi C.,
I'm sorry after all the emails I'm not straight on what we're actually getting. We have a U-shaped room, but do we have a blackboard or whiteboard? Is there a speakerphone? Wireless or at least wired internet?
thanks,
-serge
Quoted 55 lines · Eric W. Biederman
Quoted 6 lines · Daniel Lezcano
Hi all,
Here is a proposition a more detailed agenda for the checkpoint/restart mini-summit. If everybody is ok with it, I will update the wiki.
Comments are welcome :)
A reading list is useful, even to help get some ideas circulating before we get there.
Ultimately the technical details will need to be resolve by people discussing things and sending patches back and forth on the mailing lists.
I don't think a detailed agenda is going to get us anywhere. Especially not one focused on the implementation details.
I think we need to start by seeing what we can agree on. Certainly we agree that checkpoint/restart needs to be part of the picture. What are the problems that the linux community can solve with checkpoint/restart.
Then we need to talk about what kind of implementation we want to merge into mainline. How do we sell it, and how do we implement it without affecting long term maintainability.
I think the granularity of our operations, and what state we save is important. I don't think how we save it is important unless it affects one of our requirements.
As for the posix draft and the historical Cray & SGI implementations. They were on the wrong track. The did not have namespace support so they could not in general restore their checkpoints.
There are also a lot of things you have failed to touch on, that I'm not going to go into now.
With any luck the mini-summit before OLS will be the start of a conversation that will go on all week, and continue on the mailing lists.
The real question is how do we coordinate our efforts to build a good linux checkpoint/restart implementation.
Quoted 8 lines
* Documentation
* Zap : www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
* Metacluster : lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
* OpenVZ : http://wiki.openvz.org/Checkpointing_and_live_migration
* Checkpoint/Restart technology :
http://en.wikipedia.org/wiki/Application_checkpointing
* Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops
document
Quoted 2 lines
There is also the classic emacs undump.
The very simple vmadump from bproc.
Thanks Eric for all your comments. I agree the agenda is a little big, I will reduce it and I will add the points you raised. I have other points from by Oren I will add too, perhaps that will cover more aspect of the discussion.
-- Daniel
Quoted 8 lines · Daniel Lezcano
* Documentation
* Zap : www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
* Metacluster : lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
* OpenVZ : http://wiki.openvz.org/Checkpointing_and_live_migration
* Checkpoint/Restart technology :
http://en.wikipedia.org/wiki/Application_checkpointing
* Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops
document
And of course remote fork is a fun related concept.
Eric
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Eric W. Biederman
Quoted 8 lines · Daniel Lezcano
* Documentation
* Zap : www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
* Metacluster : lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
* OpenVZ : http://wiki.openvz.org/Checkpointing_and_live_migration
* Checkpoint/Restart technology :
http://en.wikipedia.org/wiki/Application_checkpointing
* Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops
document
Quoted 1 line
And of course remote fork is a fun related concept.
So Daniel, can you add the reading list to the wiki (including Eric's suggestions), maybe above the agenda?
thanks,
-serge
A message that is not in the archive (sent off-list or not yet received): 20080716151707.GB26496-r/[email protected]
Quoted 11 lines · Serge E. Hallyn
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Daniel Lezcano
* Documentation
* Zap : www.ncl.cs.columbia.edu/publications/usenix2007_fordist.pdf
* Metacluster : lxc.sourceforge.net/doc/ols2006/lxc-ols2006.pdf
* OpenVZ : http://wiki.openvz.org/Checkpointing_and_live_migration
* Checkpoint/Restart technology :
http://en.wikipedia.org/wiki/Application_checkpointing
* Virtual Servers and Checkpoint/Restart in Mainstream Linux : Sigops
document
Quoted 4 lines
Quoted 1 line
And of course remote fork is a fun related concept.
So Daniel, can you add the reading list to the wiki (including Eric's suggestions), maybe above the agenda?
Sure.