Clone PTS namespace

Devpts namespace patchset

In continuation of the implementation of containers in mainline, we need to support multiple PTY namespaces so that the PTY index (ie the tty names) in one container is independent of the PTY indices of other containers. For instance this would allow each container to have a '/dev/pts/0' PTY and refer to different terminals.

Quoted 8 lines · sukadev-r/[email protected] wrote:

Devpts namespace patchset
In continuation of the implementation of containers in mainline, we need to
support multiple PTY namespaces so that the PTY index (ie the tty names) in
one container is independent of the PTY indices of other containers. For instance this would allow each container to have a '/dev/pts/0' PTY and refer to different terminals.

Why do we "need" this? There isn't a fundamental need for this to be a dense numberspace (in fact, there are substantial reasons why it's a bad idea; the only reason the namespace is dense at the moment is because of the hideously bad handing of utmp in glibc.) Other than indicies, this seems to be a more special case of device isolation across namespaces, would that be a more useful problem to solve across the board?

We want to provide isolation between containers, meaning PTYs in container C1 should not be accessible to processes in C2 (unless C2 is an ancestor).

The other reason for this in the longer term is for checkpoint/restart. When restarting an application we want to make sure that the PTY indices it was using is available and isolated.

A complete device-namespace could solve this, but IIUC, is being planned in the longer term. We are hoping this would provide the isolation in the near-term without being too intrusive or impeding the implementation of the device namespace.

Quoted 2 lines · [email protected] wrote:

We want to provide isolation between containers, meaning PTYs in container C1 should not be accessible to processes in C2 (unless C2 is an ancestor).

Yes, I certainly can understand the desire for isolation. That wasn't what my question was about.

Quoted 3 lines

The other reason for this in the longer term is for checkpoint/restart. When restarting an application we want to make sure that the PTY indices it was using is available and isolated.

OK, this would be the motivation for index isolation.

Quoted 4 lines

A complete device-namespace could solve this, but IIUC, is being planned in the longer term. We are hoping this would provide the isolation in the near-term without being too intrusive or impeding the implementation of the device namespace.

I'm just worried about the accumulation of what feels like ad hoc namespaces, causing a very large combination matrix, a lot of which don't make sense.

Quoted 1 line

I'm just worried about the accumulation of what feels like ad hoc namespaces, causing a very large combination matrix, a lot of which don't make sense.

Hmm, if we were to just call this CLONE_NEWDEV, would that (a) make sense and (b) suitably address your (certainly valid) concern?

Basically for now CLONE_NEWDEV wouldn't yet be fully implemented, only unsharing unix98 ptys...

Quoted 8 lines · sukadev-r/[email protected] wrote:

Devpts namespace patchset

In continuation of the implementation of containers in mainline, we need to support multiple PTY namespaces so that the PTY index (ie the tty names) in one container is independent of the PTY indices of other containers. For instance this would allow each container to have a '/dev/pts/0' PTY and refer to different terminals.

Why do we "need" this? There isn't a fundamental need for this to be a dense numberspace (in fact, there are substantial reasons why it's a bad idea; the only reason the namespace is dense at the moment is because of the hideously bad handing of utmp in glibc.) Other than indicies, this seems to be a more special case of device isolation across namespaces, would that be a more useful problem to solve across the board?

Quoted 10 lines · H. Peter Anvin

Quoted 8 lines · [email protected] wrote:

Devpts namespace patchset

In continuation of the implementation of containers in mainline, we need to support multiple PTY namespaces so that the PTY index (ie the tty names) in one container is independent of the PTY indices of other containers. For instance this would allow each container to have a '/dev/pts/0' PTY and refer to different terminals.

Quoted 6 lines

Why do we "need" this? There isn't a fundamental need for this to be a dense numberspace (in fact, there are substantial reasons why it's a bad idea; the only reason the namespace is dense at the moment is because of the hideously bad handing of utmp in glibc.) Other than indicies, this seems to be a more special case of device isolation across namespaces, would that be a more useful problem to solve across the board?

In short application migration. When you move a running applicaiton from one machine to another you want to be able to keep the same pseudo devices.

The isolation that you have noticed is also an important application and like the rest of the namespaces if we can solve the duplicate identifier problem needed to restore checkpoints we also largely solve the isolation problem.

This problem is much larger then ptys. ptys are the really in your face aspect of it. There are a more pseudo devices in the kernel and it is the device number to device mapping that we are abstracting. So this really should be done as a device namespace not a pty namespace.

I would be happy if the first version of the device namespace could not map anything but pty's (assuming an incremental implementation path). I really don't think we should do a special case for each kind of device.

Oh and just skimming the patch summary I'm pretty certain this implementation breaks /sys/class/tty/ptyXX/uevent. Which is another reason why it would be good to have a single device namespace. So we only to capture one more namespace and figure out how to deal with it when mounting sysfs.

Feh, so of course sysfs would have the most interactions for a device namespace, but now we have pty, network, and user namespace all needing some sort of sysfs solution. For a quickfix for CONFIG_USER_SCHED+CONFIG_USER_NS, I just moved /sys/kernel/uids/<uid> to /sys/kernel/uids/<userns_address>/<uid>. But what would be a *good* general solution?

ln -s /sys /proc/self/sys?

Quoted 8 lines · Serge E. Hallyn

Feh, so of course sysfs would have the most interactions for a device namespace, but now we have pty, network, and user namespace all needing some sort of sysfs solution. For a quickfix for CONFIG_USER_SCHED+CONFIG_USER_NS, I just moved /sys/kernel/uids/<uid> to /sys/kernel/uids/<userns_address>/<uid>. But what would be a *good* general solution?

ln -s /sys /proc/self/sys?

LOL

In /proc I prefer the /proc/self approach because we can do it and it is just much easier to setup and use. (Plus we have weird problems if we try and capture more then the pid namespace).

For other filesystems the only really viable option is to capture namespaces at mount time, as we are doing for devpts and proc with respect to the pid namespace.

For the network namespace where it is very much more then a single directory with symlinks from physical devices pointing at logical network interfaces.

My last effort in that area was ok'd by Tejun lightly tested by a few others and misplaced by gregkh so I don't think we have a real problem with resurrecting those patches cleaning them up a bit and merging them.

The biggest gotcha with sysfs is that the VFS locking for the dcache is in the wrong order for distributed filesystems, where we would like to make the change atomically on the server and them make the change in the local cache. Or in this case the sysfs internal data structures. The truly nasty case is supporting rename (as sysfs does) as the VFS is not at all happy if you just punt and shoot down the dentries and instantiate new ones.

I'm hoping to be able to get back at this in the week or so as things settle down from my move. My last patches should be in my proof of concept network namespace tree, if they don't show up elsewhere.

It isn't my perception that we have a design problem rather, we just need to move an important piece of code in a subtle and moderately uninteresting direction for it's primary maintainer.

Further what I did for the network namespace should easily handle the uid/gid namespace and should be a good starting place for a general device namespace.

Quoted 8 lines · Serge E. Hallyn

Feh, so of course sysfs would have the most interactions for a device namespace, but now we have pty, network, and user namespace all needing some sort of sysfs solution. For a quickfix for CONFIG_USER_SCHED+CONFIG_USER_NS, I just moved /sys/kernel/uids/<uid> to /sys/kernel/uids/<userns_address>/<uid>. But what would be a *good* general solution?

ln -s /sys /proc/self/sys?

LOL

In /proc I prefer the /proc/self approach because we can do it and it is just much easier to setup and use. (Plus we have weird problems if we try and capture more then the pid namespace).

For other filesystems the only really viable option is to capture namespaces at mount time, as we are doing for devpts and proc with respect to the pid namespace.

For the network namespace where it is very much more then a single directory with symlinks from physical devices pointing at logical network interfaces.

My last effort in that area was ok'd by Tejun lightly tested by a few others and misplaced by gregkh so I don't think we have a real problem with resurrecting those patches cleaning them up a bit and merging them.

The biggest gotcha with sysfs is that the VFS locking for the dcache is in the wrong order for distributed filesystems, where we would like to make the change atomically on the server and them make the change in the local cache. Or in this case the sysfs internal data structures. The truly nasty case is supporting rename (as sysfs does) as the VFS is not at all happy if you just punt and shoot down the dentries and instantiate new ones.

I'm hoping to be able to get back at this in the week or so as things settle down from my move. My last patches should be in my proof of concept network namespace tree, if they don't show up elsewhere.

It isn't my perception that we have a design problem rather, we just need to move an important piece of code in a subtle and moderately uninteresting direction for it's primary maintainer.

Further what I did for the network namespace should easily handle the uid/gid namespace and should be a good starting place for a general device namespace.

Eric

Quoted 4 lines

I'm hoping to be able to get back at this in the week or so as things settle down from my move. My last patches should be in my proof of concept network namespace tree, if they don't show up elsewhere.

Is that the tree I'd get from

git-fetch
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/linux-2.6-netns.git
master:ebieder.master

? So I'd add a user_ns to the struct sysfs_tag_info?

If so I'll give it a whirl.

Quoted 10 lines · Serge E. Hallyn

Quoted 4 lines

I'm hoping to be able to get back at this in the week or so as things settle down from my move. My last patches should be in my proof of concept network namespace tree, if they don't show up elsewhere.

Is that the tree I'd get from

git-fetch
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/linux-2.6-netns.git
master:ebieder.master

Yes.

Quoted 3 lines

? So I'd add a user_ns to the struct sysfs_tag_info?

If so I'll give it a whirl.

Sounds good. My apologies I keep being almost on the verge of getting someplace.

Quoted 18 lines · Eric W. Biederman

Quoted 10 lines · Serge E. Hallyn

Quoted 4 lines

I'm hoping to be able to get back at this in the week or so as things settle down from my move. My last patches should be in my proof of concept network namespace tree, if they don't show up elsewhere.

Is that the tree I'd get from

git-fetch
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/linux-2.6-netns.git
master:ebieder.master

Yes.

Quoted 3 lines

? So I'd add a user_ns to the struct sysfs_tag_info?

If so I'll give it a whirl.

Quoted 2 lines

Sounds good. My apologies I keep being almost on the verge of getting someplace.

Ok I've got the sysfs relevant patches ported to 2.6.25, and am looking at how to extend it to handle /sys/kernel/uids. You have tagging tied intimately to struct class. So the question is should I generalize the taggint to deal with kobjects instead, or create a struct class user and make /sys/kernel/uids a symlink to /sys/class/user/uids?

Opinions?

thanks,
-serge

PS: If you want me to post the patchset before handling userns I can do that, otherwise I was just going to wait until I'm done with userns.

Quoted 22 lines · Serge E. Hallyn

Quoted 20 lines · Eric W. Biederman

Quoted 10 lines · Serge E. Hallyn

Quoted 4 lines

I'm hoping to be able to get back at this in the week or so as things settle down from my move. My last patches should be in my proof of concept network namespace tree, if they don't show up elsewhere.

Is that the tree I'd get from

git-fetch
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/linux-2.6-netns.git
master:ebieder.master

Yes.

Quoted 3 lines

? So I'd add a user_ns to the struct sysfs_tag_info?

If so I'll give it a whirl.

Sounds good. My apologies I keep being almost on the verge of getting someplace.

Quoted 5 lines

Ok I've got the sysfs relevant patches ported to 2.6.25, and am looking at how to extend it to handle /sys/kernel/uids. You have tagging tied intimately to struct class. So the question is should I generalize the taggint to deal with kobjects instead, or create a struct class user and make /sys/kernel/uids a symlink to /sys/class/user/uids?

Heh, never mind, I was thinking class was a kobject class, not a device class :) So I'll just have to generalize tagging.

Quoted 29 lines · Serge E. Hallyn

Quoted 27 lines · Serge E. Hallyn

Quoted 12 lines · Eric W. Biederman

Quoted 10 lines · Serge E. Hallyn

Quoted 4 lines

I'm hoping to be able to get back at this in the week or so as things settle down from my move. My last patches should be in my proof of concept network namespace tree, if they don't show up elsewhere.

Is that the tree I'd get from

git-fetch
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/linux-2.6-netns.git
master:ebieder.master

Quoted 8 lines

Yes.

Quoted 3 lines

? So I'd add a user_ns to the struct sysfs_tag_info?

If so I'll give it a whirl.

Sounds good. My apologies I keep being almost on the verge of getting someplace.

Ok I've got the sysfs relevant patches ported to 2.6.25, and am looking at how to extend it to handle /sys/kernel/uids. You have tagging tied intimately to struct class. So the question is should I generalize the taggint to deal with kobjects instead, or create a struct class user and make /sys/kernel/uids a symlink to /sys/class/user/uids?

Quoted 2 lines

Heh, never mind, I was thinking class was a kobject class, not a device class :) So I'll just have to generalize tagging.

Yes. You just need a way to get the tags there.

At the level of sysfs it is fairly general. Getting through the kobject layer is a different story.

I suspect since you are working on this and I seem to be stuck in molasses at the moment it makes sense to figure out what it will take to handle the uid namespace before pushing these patches again.

Taking a quick look and having a clue what we will need to do for a theoretical device namespace is also a possibility.

Heh, well I tried several approaches - adding tag_ops to kset, to ktype, etc. Finally ended up just calling sysfs_enable_tagging on /sys/kernel/uids when that is created. It's now working perfectly.

Quoted 4 lines

I suspect since you are working on this and I seem to be stuck in molasses at the moment it makes sense to figure out what it will take to handle the uid namespace before pushing these patches again.

I had ported your patches to 2.6.25, but Benjamin in the meantime ported them to 2.6.25-mm1. Since that's closer to the -net tree it's a more useful port, so I'll let him post his patchset. Then I'll send the userns patch on top of that. While I'm not actually able to send network traffic over a veth dev (I probably am still not setting it up right), I am able to pass veth devices into network namespaces, and the user namespaces are properly handled.

I believe Benjamin did notice a problem with some symlinks not existing, and I think we want one more patch on top of yours removing the hold_net() from sysfs_mount, which I don't think was what you really wanted to do. By simply removing that, if all tasks in a netns go away, the netns actually goes away and a lookup under a bind-mounted copy of its /sys/class/net is empty.

Anyway the patches should be hitting the list next week.

Quoted 2 lines

Taking a quick look and having a clue what we will need to do for a theoretical device namespace is also a possibility.

I'm not sure I'm familiar enough with the kobject/class/sysfs/device relationships yet to comment on that. It doesn't look like it should really be a problem, though simply adding tags to every directory under /sys/class (/sys/class/tty, /sys/class/usb_device, etc) doesn't seem like necessarily the nicest way to go...

Quoted 3 lines · Serge E. Hallyn

Heh, well I tried several approaches - adding tag_ops to kset, to ktype, etc. Finally ended up just calling sysfs_enable_tagging on /sys/kernel/uids when that is created. It's now working perfectly.

Sounds good.

Quoted 13 lines

Quoted 4 lines

I suspect since you are working on this and I seem to be stuck in molasses at the moment it makes sense to figure out what it will take to handle the uid namespace before pushing these patches again.

I had ported your patches to 2.6.25, but Benjamin in the meantime ported them to 2.6.25-mm1. Since that's closer to the -net tree it's a more useful port, so I'll let him post his patchset. Then I'll send the userns patch on top of that. While I'm not actually able to send network traffic over a veth dev (I probably am still not setting it up right), I am able to pass veth devices into network namespaces, and the user namespaces are properly handled.

Quoted 6 lines

I believe Benjamin did notice a problem with some symlinks not existing, and I think we want one more patch on top of yours removing the hold_net() from sysfs_mount, which I don't think was what you really wanted to do. By simply removing that, if all tasks in a netns go away, the netns actually goes away and a lookup under a bind-mounted copy of its /sys/class/net is empty.

I will have to look, I need to refresh myself on where all of this code is. I think hold_net was what I wanted. A record that there is a user but not something that will keep the network namespace from going away.

Essentially hold_net should be a debugging check rather then a real limitation.

Quoted 1 line

Anyway the patches should be hitting the list next week.

Cool. We can figure out what we need to do to merge them from there.

Quoted 8 lines

Quoted 2 lines

Taking a quick look and having a clue what we will need to do for a theoretical device namespace is also a possibility.

I'm not sure I'm familiar enough with the kobject/class/sysfs/device relationships yet to comment on that. It doesn't look like it should really be a problem, though simply adding tags to every directory under /sys/class (/sys/class/tty, /sys/class/usb_device, etc) doesn't seem like necessarily the nicest way to go...

True. And the goal is something maintainable. There are still a lot of implications of a device namespace left unexamined so we shall see.

Eric