fill vdso with syscall32_setup_pages if TIF_IA32 on x86_64
Subsystems (2)
PATCH 1/1 fill vdso with syscall32_setup_pages if TIF_IA32 on x86_64
arch_setup_additional_pages() on x86_64 fills in a 64-bit vdso page.
With this patch, restart of 32-bit tasks (both self- and external checkpoints) on x86-64 succeeds.
--- mm/mmap.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
mm/mmap.c +4 -0
@@ -2324,6 +2324,10 @@ int special_mapping_restore(struct ckpt_ctx *ctx,* Even that, is very basic - call arch_setup_additional_pages * requiring the same mapping (start address) as before. */ +#ifdef CONFIG_X86_64 && CONFIG_COMPAT +if (test_thread_flag (TIF_IA32 )) +return syscall32_setup_pages (NULL , h->vm_start,0 ); +#endif return arch_setup_additional_pages(NULL, h->vm_start, 0); }#else /* !CONFIG_CHECKPOINT */
Signature · 1 line
1.6.0.6
A message that is not in the archive (sent off-list or not yet received): 20100127071636.GA16624-r/[email protected]
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> works 64bit 32bit -> ?????
Does it make sense to allow the opposite transition: 'restart' starts as a 32bit and becomes a 64bit after it restores the state from the image ?
And what about if you checkpoint on a 32 bit kernel and try to restart on a 64 bit kernel, and vice versa ? (in both cases, the program of course is 32bit, and we can assume same physical host for now).
Oren.
Quoted 27 lines · Serge E. Hallyn
arch_setup_additional_pages()onx86_64fills in a 64-bit vdso page.With this patch, restart of 32-bit tasks (both self- and external checkpoints) on x86-64 succeeds.
Signed-off-by Serge E. Hallyn <serue-r/[email protected]>--- mm/mmap.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)mm/mmap.c +4
-0@@ -2324,6 +2324,10 @@ int special_mapping_restore(struct ckpt_ctx *ctx,* Even that, is very basic - call arch_setup_additional_pages * requiring the same mapping (start address) as before. */ +#ifdef CONFIG_X86_64 && CONFIG_COMPAT +if (test_thread_flag (TIF_IA32 )) +return syscall32_setup_pages (NULL , h->vm_start,0 ); +#endif return arch_setup_additional_pages(NULL, h->vm_start, 0); }#else /* !CONFIG_CHECKPOINT */
A message that is not in the archive (sent off-list or not yet received): Pine.LNX.4.64.1001270954120.8974-CXF6herHY6ykSYb+qCZC/1i27PF6R63G9nwVQlTi/[email protected]
Quoted 6 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
Quoted 7 lines
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> works 64bit 32bit -> ?????
Actually the other way around works - /bin/restart_32 < 64bit.out works just fine. /bin/restart_64 < 32bit.out does not. The reason is that destroy_mm() ends up calling do_munmap on a 64-bit mapping after the switch to 32-bit had been made, and it refuses bc vma->vm_start > TASK_SIZE.
Perhaps getting it to work will be as simple as temporarily switching back to 64-bit during destroy_mm().
Quoted 8 lines
Does it make sense to allow the opposite transition: 'restart' starts as a 32bit and becomes a 64bit after it restores the state from the image ?
And what about if you checkpoint on a 32 bit kernel and try to restart on a 64 bit kernel, and vice versa ? (in both cases, the program of course is 32bit, and we can assume same physical host for now).
I have no hw right now where I could test such a thing. Do you?
A message that is not in the archive (sent off-list or not yet received): 20100127201037.GA23119-r/[email protected]
Quoted 14 lines · Serge E. Hallyn
Quoted 5 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
Quoted 7 lines
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> works 64bit 32bit -> ?????
Quoted 8 lines
Actually the other way around works -
/bin/restart_32< 64bit.out works just fine./bin/restart_64< 32bit.out does not. The reason is thatdestroy_mm()ends up callingdo_munmapon a 64-bit mapping after the switch to 32-bit had been made, and it refuses bcvma->vm_start>TASK_SIZE.Perhaps getting it to work will be as simple as temporarily switching back to 64-bit during
destroy_mm().
Interesting, I didn't think about it.
So yes, switching temporarily should work. An alternative we can do the call destroy_mm() earlier, as it may suit us.
Quoted 5 lines
Quoted 5 lines
Does it make sense to allow the opposite transition: 'restart' starts as a 32bit and becomes a 64bit after it restores the state from the image ?
Quoted 7 lines
Quoted 4 lines
And what about if you checkpoint on a 32 bit kernel and try to restart on a 64 bit kernel, and vice versa ? (in both cases, the program of course is 32bit, and we can assume same physical host for now).
I have no hw right now where I could test such a thing. Do you?
Couldn't you use the same machine you are using - just reboot it between checkpoint and restart with a 32bit kernel ... ?
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 41 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 5 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
Quoted 7 lines
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> works 64bit 32bit -> ?????Quoted 8 lines
Actually the other way around works -
/bin/restart_32< 64bit.out works just fine./bin/restart_64< 32bit.out does not. The reason is thatdestroy_mm()ends up callingdo_munmapon a 64-bit mapping after the switch to 32-bit had been made, and it refuses bcvma->vm_start>TASK_SIZE.Perhaps getting it to work will be as simple as temporarily switching back to 64-bit during
destroy_mm().Interesting, I didn't think about it.
So yes, switching temporarily should work. An alternative we can do the call
destroy_mm()earlier, as it may suit us.Quoted 10 lines
Quoted 9 lines
Does it make sense to allow the opposite transition: 'restart' starts as a 32bit and becomes a 64bit after it restores the state from the image ?
And what about if you checkpoint on a 32 bit kernel and try to restart on a 64 bit kernel, and vice versa ? (in both cases, the program of course is 32bit, and we can assume same physical host for now).
Quoted 5 lines
Quoted 2 lines
I have no hw right now where I could test such a thing. Do you?
Couldn't you use the same machine you are using - just reboot it between checkpoint and restart with a 32bit kernel ... ?
maybe - but i'm borrowing this machine (with no phys access) so don't want to get too risky :)
can give it a shot i guess
A message that is not in the archive (sent off-list or not yet received): 20100127211052.GA27579-r/[email protected]
Quoted 39 lines · Serge E. Hallyn
Quoted 38 lines · Oren Laadan
Quoted 13 lines · Serge E. Hallyn
Quoted 5 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
Quoted 7 lines
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> works 64bit 32bit -> ?????Quoted 8 lines
Actually the other way around works -
/bin/restart_32< 64bit.out works just fine./bin/restart_64< 32bit.out does not. The reason is thatdestroy_mm()ends up callingdo_munmapon a 64-bit mapping after the switch to 32-bit had been made, and it refuses bcvma->vm_start>TASK_SIZE.Perhaps getting it to work will be as simple as temporarily switching back to 64-bit during
destroy_mm().Interesting, I didn't think about it.
So yes, switching temporarily should work. An alternative we can do the call
destroy_mm()earlier, as it may suit us.Quoted 10 lines
Quoted 8 lines
Does it make sense to allow the opposite transition: 'restart' starts as a 32bit and becomes a 64bit after it restores the state from the image ?
And what about if you checkpoint on a 32 bit kernel and try to restart on a 64 bit kernel, and vice versa ? (in both cases, the program of course is 32bit, and we can assume same physical host for now).
I have no hw right now where I could test such a thing. Do you?
Quoted 8 lines
Quoted 2 lines
Couldn't you use the same machine you are using - just reboot it between checkpoint and restart with a 32bit kernel ... ?
maybe - but i'm borrowing this machine (with no phys access) so don't want to get too risky :)
can give it a shot i guess
Or you can run the 32bit kernel inside a VM on that machine...
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Oren Laadan
Quoted 9 lines · Serge E. Hallyn
Quoted 8 lines · Oren Laadan
Quoted 6 lines · Serge E. Hallyn
Quoted 5 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> works 64bit 32bit -> ?????
s/?????/Rejected/
CKPT_ARCH_ID is of course different for X86_32 than X86_64, so we refuse restart in restore_read_header().
-serge
A message that is not in the archive (sent off-list or not yet received): 20100205233800.GA17057-r/[email protected]
Quoted 17 lines · Serge E. Hallyn
Quoted 16 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> works
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works
32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works
32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]
32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2]
[0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
IIUC, your comment above means that they don't work because the CKPT_ARCH_ID is a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.
[2] The last two cases consider the case when the restart program itself has different bit-ness than the checkpointed program (and transition may occur in either direction). While lower priority, we would like this to work, too.
The question is whether the transition 64 -> 32 (or 32 ->64) from the 'restart' program to the restarting task should happen in the kernel as part of sys_restart(), or in user space using an execve() syscall before calling sys_restart().
Doing so in user space is not trivial when threads are involved, since the exec must then happen before the creation of threads (or it will kill them). This will complicate the implementation of the MakeForest() algorithm which relies on all all descendents seeing the same data structures.
Doing so in kernel should have been easy in theory, but in practice so far it isn't working; and it may be frowned upon by kernel people (allowing such transition not in exec).
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 56 lines · Oren Laadan
Quoted 17 lines · Serge E. Hallyn
Quoted 16 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> worksQuoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
Quoted 8 lines
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.[2] The last two cases consider the case when the restart program itself has different bit-ness than the checkpointed program (and transition may occur in either direction). While lower priority, we would like this to work, too.
Great table. Is it posted in the ckpt wiki too?
I could take care of that for you if not. Perhaps it belongs under the "Checklist"?
Quoted 4 lines
The question is whether the transition 64 -> 32 (or 32 ->64) from the 'restart' program to the restarting task should happen in the kernel as part of
sys_restart(), or in user space using anexecve()syscall before callingsys_restart().
The recent exec bug while switching personalities highlights the value, in my opinion, of keeping these transitions out of the restart syscall. There's great potential for nasty, long-term bugs in any code that deals with those kinds of switches. Keeping that code "in one place" is the best way to avoid adding similar bugs.
Quoted 5 lines
Doing so in user space is not trivial when threads are involved, since the exec must then happen before the creation of threads (or it will kill them). This will complicate the implementation of the
MakeForest()algorithm which relies on all all descendents seeing the same data structures.
True -- MakeForest is already rather complicated.
As for seeing the same data structures across exec, perhaps we should keep an fd open across exec and read/map the table from that. That means converting from struct task* to indices in the table for one thing. I have some RFC patches for that. It also means the table contents have to use the same layout between 32 and 64-bit -- also quite easy.
What I couldn't see was a good place to do the exec itself.
Cheers,
-Matt Helsley
A message that is not in the archive (sent off-list or not yet received): 20100206062650.GG3714-52DBMbEzqgQ/wnmkkaCWp/[email protected]
Quoted 66 lines · Matt Helsley
Quoted 56 lines · Oren Laadan
Quoted 17 lines · Serge E. Hallyn
Quoted 16 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> worksQuoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
Quoted 8 lines
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.[2] The last two cases consider the case when the restart program itself has different bit-ness than the checkpointed program (and transition may occur in either direction). While lower priority, we would like this to work, too.
Quoted 6 lines
Great table. Is it posted in the ckpt wiki too?
I could take care of that for you if not. Perhaps it belongs under the "Checklist"?
Yep, I'll add it.
Quoted 10 lines
Quoted 4 lines
The question is whether the transition 64 -> 32 (or 32 ->64) from the 'restart' program to the restarting task should happen in the kernel as part of
sys_restart(), or in user space using anexecve()syscall before callingsys_restart().The recent exec bug while switching personalities highlights the value, in my opinion, of keeping these transitions out of the restart syscall. There's great potential for nasty, long-term bugs in any code that deals with those kinds of switches. Keeping that code "in one place" is the best way to avoid adding similar bugs.
I agree with the concern.
There is even a stronger argument: doing it in user-space via exec will rid the need to repeat the kernel code for different archs.
The only caveat is that restarting mixes 32- and 64-bit programs will be slower because fo the exec calls. Then again, it can be optimized when someone complains (and provides kernel code for this).
Quoted 9 lines
Quoted 6 lines
Doing so in user space is not trivial when threads are involved, since the exec must then happen before the creation of threads (or it will kill them). This will complicate the implementation of the
MakeForest()algorithm which relies on all all descendents seeing the same data structures.True -- MakeForest is already rather complicated.
Quoted 7 lines
As for seeing the same data structures across exec, perhaps we should keep an fd open across exec and read/map the table from that. That means converting from struct task* to indices in the table for one thing. I have some RFC patches for that. It also means the table contents have to use the same layout between 32 and 64-bit -- also quite easy.
What I couldn't see was a good place to do the exec itself.
Can you send me the patches that you already have ?
There are a couple of more details:
* Checkpoint needs to also record the bit-ness of each process in the tasks table
* On restart, if all tasks have same bit-ness, then one exec at most is needed.
* Where to place the exec ? ideally for process (without threads)
it would happen right before sys_restart(), but for processes that have threads it should happen right before the fork().
* Passing an fd is a good idea - anonymous shared memory should do the trick.
* I'm a bit concerned about security, because 'restart' will likely be a setuid utility:
1) need to ensure that we are exec'ing the right program (e.g. use the /proc/self/exe link)
2) a new switch to 'restart' will say "use fd N for the data", but a user may provide arbitrary data - can they do harm ? we'll definitely need to be more cautious in handling the tasks array because in this case we don't construct it ourselves.
Oren.
[snip]
Quoted 25 lines
Quoted 25 lines
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.
Quoted 7 lines
Quoted 4 lines
[2] The last two cases consider the case when the restart program itself has different bit-ness than the checkpointed program (and transition may occur in either direction). While lower priority, we would like this to work, too.
Great table. Is it posted in the ckpt wiki too?
I updated the wiki:
http://ckpt.wiki.kernel.org/index.php/Architecture
For powerpc I just assumed it's like x86 ... :)
Can you please approve or modify for archs other than x86-{32,64} ?
Thanks,
Oren.
Quoted 52 lines · Oren Laadan
Quoted 17 lines · Serge E. Hallyn
Quoted 16 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> worksQuoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
Quoted 7 lines
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.
Well, you'd think so, but we also check for uts->machine, and want to eventually check for kernel config, both of which are obviously different.
After I comment out the obvious offending checks, it still fails to restart from x8632->x86-64. I can spend some time next week figuring out what we're not quite doing right as there shouldn't be a problem really. But do we definately want to go out of our way to try
and mask out the differences in this case, while trying to detect
cpu differences between two x86-32's for instance?
-serge
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 53 lines · Serge E. Hallyn
Quoted 52 lines · Oren Laadan
Quoted 16 lines · Serge E. Hallyn
Quoted 15 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> worksQuoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
Quoted 7 lines
Quoted 3 lines
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.Well, you'd think so, but we also check for
uts->machine, and want to eventually check for kernel config, both of which are obviously different.
Then we'll have to take that in account when we get to also check those other fields.
Quoted 7 lines
After I comment out the obvious offending checks, it still fails to restart from x8632->x86-64. I can spend some time next week figuring out what we're not quite doing right as there shouldn't be a problem really. But do we definately want to go out of our way to try
and mask out the differences in this case, while trying to detect
cpu differences between two x86-32's for instance?
I agree, there shouldn't be a problem really, and I expect this to be a very useful feature for migration/fault-tolerance.
Checking for differences between CPU's is a separate issue, and is orthogonal to migration (of 32bit programs) between 32 and 64 bit kernels.
I tend to answer "yes" - we should eventually refuse restart if we detect that the "configuration" at restart time differs from that at checkpoint time "sufficiently".
Now "configuration" is very basic - the architecture. I would like that to also include cpu features, kernel features, fpu capabilities... Also "sufficiently" is vaguely defined, because I don't know enough to describe it more precisely.
Ideally there will be some clever user-space logic that will detect and make a decision. And, yes, it will take a lot of details...
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 75 lines · Oren Laadan
Quoted 53 lines · Serge E. Hallyn
Quoted 52 lines · Oren Laadan
Quoted 16 lines · Serge E. Hallyn
Quoted 15 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> worksQuoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
Quoted 7 lines
Quoted 3 lines
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.Well, you'd think so, but we also check for
uts->machine, and want to eventually check for kernel config, both of which are obviously different.Then we'll have to take that in account when we get to also check those other fields.
Quoted 7 lines
After I comment out the obvious offending checks, it still fails to restart from x8632->x86-64. I can spend some time next week figuring out what we're not quite doing right as there shouldn't be a problem really. But do we definately want to go out of our way to try and mask out the differences in this case, while trying to detect cpu differences between two x86-32's for instance?
Quoted 2 lines
I agree, there shouldn't be a problem really, and I expect this to be a very useful feature for migration/fault-tolerance.
May be, but then perhaps this is the first case where we should be using a userspace checkpoing image rewriter to help us out. Otherwise we'll need to hardcode in the kernel that a task which was checkpointed on X86_32 should, on x86_64, have TIF_IA32 added to the thread_flags but may be restarted; etc. Should be doable, but kind of ugly...
-serge
A message that is not in the archive (sent off-list or not yet received): 20100208153145.GB9120-r/[email protected]
Quoted 51 lines · Serge E. Hallyn
Quoted 50 lines · Oren Laadan
Quoted 48 lines · Serge E. Hallyn
Quoted 47 lines · Oren Laadan
Quoted 16 lines · Serge E. Hallyn
Quoted 15 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> worksQuoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.
^^^^^^^^^^^^^^^^^^^^^^^^
---->
Quoted 15 lines
Quoted 6 lines
Quoted 3 lines
Well, you'd think so, but we also check for
uts->machine, and want to eventually check for kernel config, both of which are obviously different.Then we'll have to take that in account when we get to also check those other fields.
Quoted 8 lines
Quoted 6 lines
After I comment out the obvious offending checks, it still fails to restart from x8632->x86-64. I can spend some time next week figuring out what we're not quite doing right as there shouldn't be a problem really. But do we definately want to go out of our way to try and mask out the differences in this case, while trying to detect cpu differences between two x86-32's for instance?
I agree, there shouldn't be a problem really, and I expect this to be a very useful feature for migration/fault-tolerance.
Quoted 6 lines
May be, but then perhaps this is the first case where we should be using a userspace checkpoing image rewriter to help us out. Otherwise we'll need to hardcode in the kernel that a task which was checkpointed on
X86_32should, onx86_64, haveTIF_IA32added to thethread_flagsbut may be restarted; etc. Should be doable, but kind of ugly...
Indeed. I offered that path above :)
Since we are going to need the bit-ness of a task for the tree creation as well, how about:
1) Add the bit-ness property to the pids_arr[], e.g. as a flags field (we may need use it for other stuff later).
2) 'restart' already examines and possibly modifies pids_arr[], so in transition from 32->64 it will add that flag, and in the opposite transition it will check/remove that flag.
3) 'restart' will also change the header architecture as needed.
4) The kernel will verify that the bitness reported in pids_arr[] is the same as the actual process. (This is just a sanity check, of course).
Later we'll also make 'restart' use that bit-ness information to decide whether an exec() is needed to change own bit-ness.
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 98 lines · Oren Laadan
Quoted 51 lines · Serge E. Hallyn
Quoted 50 lines · Oren Laadan
Quoted 48 lines · Serge E. Hallyn
Quoted 47 lines · Oren Laadan
Quoted 16 lines · Serge E. Hallyn
Quoted 15 lines · Oren Laadan
Quoted 14 lines · Serge E. Hallyn
Quoted 13 lines · Oren Laadan
Quoted 12 lines · Serge E. Hallyn
Quoted 11 lines · Oren Laadan
Cool !
So what do we have working now for 64 bit kernel (for 32 bit kernel we know it works...):
'restart' checkpointed program program ---------------------------------------- 64bit 64bit -> works 32bit 32bit -> worksQuoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
64bit 32bit -> ?????s/?????/Rejected/
CKPT_ARCH_IDis of course different forX86_32thanX86_64, so we refuse restart inrestore_read_header().-serge
lol ... that's actually funny !
Anyway, in light of the IRC discussions, here are the cases again:
original original restart target
program kernel program kernel
-------- --------- -------- --------
64 bit 64 bit 64 bit 64 bit [0] works32 bit 32 bit 32 bit 32 bit [0] works
32 bit 64 bit 32 bit 64 bit [0] works32 bit 32 bit 32 bit 64 bit [1]
32 bit 64 bit 32 bit 32 bit [1]32 bit any 64 bit 64 bit [2]
64 bit 64 bit 32 bit 64 bit [2][0] The first 3 cases are "homogeneous", with conditions equal at checkpoint and restart. AFAIK, they work.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
[1] The next two cases consider 32 bit program, and vary only the environment - the kernel may change from 32 to 64 or back. We want them to work.
IIUC, your comment above means that they don't work because the
CKPT_ARCH_IDis a mismatch. The fix should be trivial - either make 'restart' modify it, or make the kernel tolerate it.^^^^^^^^^^^^^^^^^^^^^^^^---->
Quoted 15 lines
Quoted 6 lines
Quoted 3 lines
Well, you'd think so, but we also check for
uts->machine, and want to eventually check for kernel config, both of which are obviously different.Then we'll have to take that in account when we get to also check those other fields.
Quoted 8 lines
Quoted 6 lines
After I comment out the obvious offending checks, it still fails to restart from x8632->x86-64. I can spend some time next week figuring out what we're not quite doing right as there shouldn't be a problem really. But do we definately want to go out of our way to try and mask out the differences in this case, while trying to detect cpu differences between two x86-32's for instance?
I agree, there shouldn't be a problem really, and I expect this to be a very useful feature for migration/fault-tolerance.
Quoted 6 lines
May be, but then perhaps this is the first case where we should be using a userspace checkpoing image rewriter to help us out. Otherwise we'll need to hardcode in the kernel that a task which was checkpointed on
X86_32should, onx86_64, haveTIF_IA32added to thethread_flagsbut may be restarted; etc. Should be doable, but kind of ugly...Indeed. I offered that path above :)
Since we are going to need the bit-ness of a task for the tree creation as well, how about:
1) Add the bit-ness property to the pids_arr[], e.g. as a flags field (we may need use it for other stuff later).
2) 'restart' already examines and possibly modifies pids_arr[], so in transition from 32->64 it will add that flag, and in the opposite transition it will check/remove that flag.
Quoted 8 lines
3) 'restart' will also change the header architecture as needed.
4) The kernel will verify that the bitness reported in pids_arr[] is the same as the actual process. (This is just a sanity check, of course).
Later we'll also make 'restart' use that bit-ness information to decide whether an
exec()is needed to change own bit-ness.
It'll mean yet another arch-dependent hook used early in the checkpoint path, but if we want to restarted mixed-bit containers i guess it's what we'll need.
Still I really don't think it's all that mean to just say we don't support it: at checkpoint we refuse with a meaningful log message including pids of task which are COMPAT, and the end-user can use that info to checkpoint those applications separately as subtrees, kill them, then checkpoint the container, then restart the applications.
If to my surprise there turn out to be people who care, then we can make the necessary changes to accomodate them. But IMO we have enough to worry about right now.
-serge