C/R and stdio redirection

Suppose we create a container and redirect its stdout/stderr as follows:

	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1

If we attempt to checkpoint the container 'foo', we fail bc one of the fds in the application refers to /tmp/xyz.out, which is also in use outside the container (specifically sys_checkpoint() fails due to the "alien mount ns" check in ckpt_fill_fname()).

It can be argued, 'foo' is not a strict container (since it shares the fd with another container). For this reason, we currently need the CHECKPOINT_SUBTREE flag in lxc-checkpoint.

We initially thought that solving mount-namespaces will solve this, but realized that they are both separate problems. Mount-namespace C/R addresses preserving the mounts within the container and /tmp/xyz.out is outside the container.

So if an application container needs to redirect stdio as above, we should either
a) disable/ignore the alien-mount-ns check or

b) try and start the application something like:

$ cat /tmp/wrapper
/path/to/app > /tmp/xyz.out 2>&1

$ lxc-execute --name foo -- /tmp/wrapper

with the difference being /tmp/xyz.out is now inside the container's /tmp filesystem rather than in the parent container.

Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also set - we had discussed this before and considered it hacky.

Or are there other solutions to this stdio redirection issue ?

I think that this should be solved by a file substitution facility combined with an ability to ignore "unsupported files" at checkpoint.
File substitution is the ability to tell sys_restart() to substitute some of caller's file descriptors to checkpointed file descriptors. Ignoring "unsupported files" means that the checkpoint could record that a given file descriptor was shared by some files_struct, but the file descriptor itself is not checkpointed because of various reasons, like file descriptor outside the container, unsupported file systems, etc.

Both of those mechanisms have been developed in Kerrighed, and Matthieu (in Cc) should start a discussion about this in the next weeks.

Thanks,

Louis

Quoted 3 lines · Louis Rilling

Both of those mechanisms have been developed in Kerrighed, and Matthieu (in Cc) should start a discussion about this in the next weeks.

Matthieu doesn't appear on the Cc line...

Suppose we create a container and redirect its stdout/stderr as follows:

	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1

If we attempt to checkpoint the container 'foo', we fail bc one of the fds in the application refers to /tmp/xyz.out, which is also in use outside the container (specifically sys_checkpoint() fails due to the "alien mount ns" check in ckpt_fill_fname()).

It can be argued, 'foo' is not a strict container (since it shares the fd with another container). For this reason, we currently need the CHECKPOINT_SUBTREE flag in lxc-checkpoint.

We initially thought that solving mount-namespaces will solve this, but realized that they are both separate problems. Mount-namespace C/R addresses preserving the mounts within the container and /tmp/xyz.out is outside the container.

So if an application container needs to redirect stdio as above, we should either
a) disable/ignore the alien-mount-ns check or

b) try and start the application something like:

$ cat /tmp/wrapper
/path/to/app > /tmp/xyz.out 2>&1

$ lxc-execute --name foo -- /tmp/wrapper

with the difference being /tmp/xyz.out is now inside the container's /tmp filesystem rather than in the parent container.

Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also set - we had discussed this before and considered it hacky.

Or are there other solutions to this stdio redirection issue ?

Greg Kurz [gkurz-NmTC/[email protected]] wrote:

| On Tue, 2010-09-07 at 13:03 -0700, Sukadev Bhattiprolu wrote:
| > Suppose we create a container and redirect its stdout/stderr as follows:
| > 
| > 	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1
| > 
| > If we attempt to checkpoint the container 'foo', we fail bc one of the
| > fds in the application refers to /tmp/xyz.out, which is also in use
| > outside the container (specifically sys_checkpoint() fails due to the
| > "alien mount ns" check in ckpt_fill_fname()).
| > 
| > It can be argued, 'foo' is not a strict container (since it shares the
| > fd with another container).  For this reason, we currently need the
| > CHECKPOINT_SUBTREE flag in lxc-checkpoint.
| > 
| > We initially thought that solving mount-namespaces will solve this, but
| > realized that they are both separate problems. Mount-namespace C/R addresses
| > preserving the mounts within the container and /tmp/xyz.out is outside
| > the container.
| > 
| > So if an application container needs to redirect stdio as above, we should
| > either 
| > 	a) disable/ignore the alien-mount-ns check or 
| > 
| > 	b) try and start the application something like:
| > 
| > 		$ cat /tmp/wrapper
| > 		/path/to/app > /tmp/xyz.out 2>&1
| > 
| > 		$ lxc-execute --name foo --  /tmp/wrapper
| > 
| > with the difference being /tmp/xyz.out is now inside the container's /tmp
| > filesystem rather than in the parent container.
| > 
| > Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
| > set - we had discussed this before and considered it hacky.
| > 
| > Or are there other solutions to this stdio redirection issue ?
| > 
| 
| To be more accurate, this issue is about fd leaking from a parent
| container to its descendants. The fd numbers may be anything else than
| 0,1 or 2 and the underlying files may be regular files, pipes,
| sockets... For example, in the HPC world, stdio are often sockets
| inheritated from a rshd like daemon.

I agree that fd substitution is the right way to go.

However, Matt Helsley and I were discussing this and wondered if we should ignore the redirection and expect to user to specify it during restart.

i.e if container was created like this:

	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1

and checkpointed, can we expect the user to restart it like this ?

	lxc-restart --name foo --statefile ckpt.img >> /tmp/xyz.out 

i.e user has to redo the redirection or the output would go to stdout.

Doing this would somehow seem to match a (bogus container) like:

	lxc-execute --name foo -- /path/to/app | sort

If this container is checkpointed/restarted, we can't really redirect the output of the app to 'sort' right ? So expecting the user to redo the redirection on restart would treat both redirections ('>' and '|') in a consistent way ?

Quoted 65 lines · Sukadev Bhattiprolu

Greg Kurz [gkurz-NmTC/[email protected]] wrote:

| On Tue, 2010-09-07 at 13:03 -0700, Sukadev Bhattiprolu wrote:
|>  Suppose we create a container and redirect its stdout/stderr as follows:
|>
|>  	lxc-execute -name foo -- /path/to/app>  /tmp/xyz.out 2>&1
|>
|>  If we attempt to checkpoint the container 'foo', we fail bc one of the
|>  fds in the application refers to /tmp/xyz.out, which is also in use
|>  outside the container (specifically sys_checkpoint() fails due to the
|>  "alien mount ns" check in ckpt_fill_fname()).
|>
|>  It can be argued, 'foo' is not a strict container (since it shares the
|>  fd with another container).  For this reason, we currently need the
|>  CHECKPOINT_SUBTREE flag in lxc-checkpoint.
|>
|>  We initially thought that solving mount-namespaces will solve this, but
|>  realized that they are both separate problems. Mount-namespace C/R addresses
|>  preserving the mounts within the container and /tmp/xyz.out is outside
|>  the container.
|>
|>  So if an application container needs to redirect stdio as above, we should
|>  either
|>  	a) disable/ignore the alien-mount-ns check or
|>
|>  	b) try and start the application something like:
|>
|>  		$ cat /tmp/wrapper
|>  		/path/to/app>  /tmp/xyz.out 2>&1
|>
|>  		$ lxc-execute --name foo --  /tmp/wrapper
|>
|>  with the difference being /tmp/xyz.out is now inside the container's /tmp
|>  filesystem rather than in the parent container.
|>
|>  Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
|>  set - we had discussed this before and considered it hacky.
|>
|>  Or are there other solutions to this stdio redirection issue ?
|>
|
| To be more accurate, this issue is about fd leaking from a parent
| container to its descendants. The fd numbers may be anything else than
| 0,1 or 2 and the underlying files may be regular files, pipes,
| sockets... For example, in the HPC world, stdio are often sockets
| inheritated from a rshd like daemon.

I agree that fd substitution is the right way to go.

However, Matt Helsley and I were discussing this and wondered if we should ignore the redirection and expect to user to specify it during restart.

i.e if container was created like this:

	lxc-execute -name foo -- /path/to/app>  /tmp/xyz.out 2>&1

and checkpointed, can we expect the user to restart it like this ?

	lxc-restart --name foo --statefile ckpt.img>>  /tmp/xyz.out

i.e user has to redo the redirection or the output would go to stdout.

Doing this would somehow seem to match a (bogus container) like:

	lxc-execute --name foo -- /path/to/app | sort

Quoted 4 lines

If this container is checkpointed/restarted, we can't really redirect the output of the app to 'sort' right ? So expecting the user to redo the redirection on restart would treat both redirections ('>' and '|') in a consistent way ?

From the fd substitution point of view, this means that lxc-restart would
automatically request the substitution of its stdout to the checkpointed container init's stdout?

This sounds reasonable to me at least. Especially since the container is usually not supposed to know where the host is redirecting its stdout.

Thanks,

Louis