RFC: netfilter: nf_conntrack: add support for "conntrack zones"

The attached largish patch adds support for "conntrack zones", which are virtual conntrack tables that can be used to seperate connections from different zones, allowing to handle multiple connections with equal identities in conntrack and NAT.

A zone is simply a numerical identifier associated with a network device that is incorporated into the various hashes and used to distinguish entries in addition to the connection tuples. Additionally it is used to seperate conntrack defragmentation queues. An iptables target for the raw table could be used alternatively to the network device for assigning conntrack entries to zones.

This is mainly useful when connecting multiple private networks using the same addresses (which unfortunately happens occasionally)

to pass
the packets through a set of veth devices and SNAT each network to a unique address, after which they can pass through the "main" zone and be handled like regular non-clashing packets and/or have NAT applied a second time based f.i. on the outgoing interface.

Something like this, with multiple tunl and veth devices, each pair using a unique zone:

<tunl0 / zone 1>

     |
  PREROUTING
     |
  FORWARD
     |
  POSTROUTING: SNAT to unique network
     |
  <veth1 / zone 1>
  <veth0 / zone 0>
     |
  PREROUTING
     |
  FORWARD
     |
  POSTROUTING: SNAT to eth0 address
     |

<eth0>

As probably everyone has noticed, this is quite similar to what you can do using network namespaces. The main reason for not using network namespaces is that its an all-or-nothing approach, you can't virtualize just connection tracking.

Beside the difficulties in
managing different namespaces from f.i. an IKE or PPP daemon running in the initial namespace,

network namespaces have a quite large
overhead, especially when used with a large conntrack table.

I'm not too fond of this partial feature duplication myself, but I couldn't think of a better way to do this without the downsides of using namespaces. Having partially shared network namespaces would be great, but it doesn't seem to fit in the design very well. I'm open for any better suggestion :)

Ive had an equivalent discussion with B Greear (CCed) at one point on something similar, curious if you solve things differently - couldnt tell from the patch if you address it.

Comments inline:

Quoted 13 lines · Patrick McHardy

The attached largish patch adds support for "conntrack zones", which are virtual conntrack tables that can be used to seperate connections from different zones, allowing to handle multiple connections with equal identities in conntrack and NAT.

A zone is simply a numerical identifier associated with a network device that is incorporated into the various hashes and used to distinguish entries in addition to the connection tuples. Additionally it is used to seperate conntrack defragmentation queues. An iptables target for the raw table could be used alternatively to the network device for assigning conntrack entries to zones.

Quoted 2 lines

This is mainly useful when connecting multiple private networks using the same addresses (which unfortunately happens occasionally)

Agreed that this would be a main driver of such a feature. Which means that you need zones (or whatever noun other people use) to work on not just netfilter, but also routing, ipsec etc.

As a digression: this is trivial to solve with network namespaces.

Quoted 6 lines

to pass
the packets through a set of veth devices and SNAT each network to a unique address, after which they can pass through the "main" zone and be handled like regular non-clashing packets and/or have NAT applied a second time based f.i. on the outgoing interface.

The fundamental question i have is:
how you deal with overlapping addresses?
i.e zone1 uses 10.0.0.1 and zone2 uses 10.0.0.1 but they are for different NAT users/endpoints.

Quoted 4 lines

As probably everyone has noticed, this is quite similar to what you can do using network namespaces. The main reason for not using network namespaces is that its an all-or-nothing approach, you can't virtualize just connection tracking.

Unless there is a clever approach for overlapping IP addresses (my question above), i dont see a way around essentially virtualizing the whole stack which clone(CLONE_NEWNET) provides..

Quoted 3 lines

Beside the difficulties in
managing different namespaces from f.i. an IKE or PPP daemon running in the initial namespace,

This is a valid concern against the namespace approach. Existing tools of course could be taught to know about namespaces - and one could argue that if you can resolve the overlap IP address issue, then you _have to_ modify user space anyways.

Quoted 2 lines

network namespaces have a quite large
overhead, especially when used with a large conntrack table.

Elaboration needed.
You said the size in 64 bit increases to 152B per conntrack i think?

Do you have a hand-wave figure we can use as a metric to elaborate this point? What would a typical user of this feature have in number of "zones" and how many contracks per zone? Actually we could also look at extremes (huge number vs low numbers)...

You may also wanna look as a metric at code complexity/maintainability of this scheme vs namespace (which adds zero changes to the kernel).

I am pretty sure you will soon be "zoning" on other pieces of the net stack ;->

Quoted 5 lines

I'm not too fond of this partial feature duplication myself, but I couldn't think of a better way to do this without the downsides of using namespaces. Having partially shared network namespaces would be great, but it doesn't seem to fit in the design very well. I'm open for any better suggestion :)

My opinions above.

BTW, why not use skb->mark instead of creating a new semantic construct?

Quoted 3 lines · jamal

Agreed that this would be a main driver of such a feature. Which means that you need zones (or whatever noun other people use) to work on not just netfilter, but also routing, ipsec etc.

Routing already works fine. I believe IPsec should also work already, but I haven't tried it.

The zone is set based on some other criteria (in this case the incoming device).

The packets make one pass through the stack
to a veth device and are SNATed in POSTROUTING to non-clashing addresses.

Quoted 4 lines

This is a valid concern against the namespace approach. Existing tools of course could be taught to know about namespaces - and one could argue that if you can resolve the overlap IP address issue, then you _have to_ modify user space anyways.

I don't think thats true.

In any case its completely impractical
to modify every userspace tool that does something with networking and potentially make complex configuration changes to have all those namespaces interact nicely.

Currently they are simply not
very well suited for virtualizing selected parts of networking.

I'm not sure whether there is a typical user for overlapping networks :) I know of setups with ~150 overlapping networks.

The number of conntracks per zone doesn't matter since the table is shared between all zones. network namespaces would allocate 150 tables, each of the same size, which might be quite large.

Quoted 2 lines

You may also wanna look as a metric at code complexity/maintainability of this scheme vs namespace (which adds zero changes to the kernel).

There's not a lot of complexity, its basically passing a numeric identifier around in a few spots and comparing it. Something like TOS handling in the routing code.

Quoted 1 line

BTW, why not use skb->mark instead of creating a new semantic construct?

Because people are already using it for different purposes.

Quoted 7 lines · Patrick McHardy

Quoted 3 lines · jamal

Agreed that this would be a main driver of such a feature. Which means that you need zones (or whatever noun other people use) to work on not just netfilter, but also routing, ipsec etc.

Routing already works fine. I believe IPsec should also work already, but I haven't tried it.

maybe further discussion would clarify this point..

Quoted 2 lines

The zone is set based on some other criteria (in this case the incoming device).

If you are using a netdev as a reference point, then I take it if you add vlans should be possible to do multiple zones on a single physical netdev? Or is there some other way to satisfy that?

Quoted 3 lines

The packets make one pass through the stack
to a veth device and are SNATed in POSTROUTING to non-clashing addresses.

Ok - makes sense.
i.e NAT would work; and policy routing as well as arp would be fine. Also it looks to be sufficiently useful to fit a specific use case you are interested in.
But back to my question on routing, ipsec etc (and you may not be interested in solving this problem, but it is what i was getting to earlier). Lets take for example:
a) network tables like SAD/SPD tables: how you would separate those on a per-zone basis? i.e 10.0.0.1/zone1 could use different policy/association than 10.0.0.1/zone2

b) dynamic protocols (routing, IKE etc): how do you do that without making both sides understand what is going on?

Quoted 5 lines

Quoted 4 lines

This is a valid concern against the namespace approach. Existing tools of course could be taught to know about namespaces - and one could argue that if you can resolve the overlap IP address issue, then you _have to_ modify user space anyways.

I don't think thats true.

Refer to my statements above for an example.

Quoted 4 lines

In any case its completely impractical
to modify every userspace tool that does something with networking and potentially make complex configuration changes to have all those namespaces interact nicely.

Agreed. But the major ones like iproute2 etc could be taught. We have namespaces in the kernel already, over a period of time I think changing the user space tools would a sensible evolution.

Quoted 2 lines

Currently they are simply not
very well suited for virtualizing selected parts of networking.

My contention is that it is a lot less headache to just virtualize all the network stack and then use what you want than it is to go and selectively changing the network objects.
Note: if i wanted today i could run racoon on every namespace unchanged and it would work or i could modify racoon to understand namespaces...

Quoted 7 lines

I'm not sure whether there is a typical user for overlapping networks :) I know of setups with ~150 overlapping networks.

The number of conntracks per zone doesn't matter since the table is shared between all zones. network namespaces would allocate 150 tables, each of the same size, which might be quite large.

Thats what i was looking for ..
So the difference, to pick the 150 zones example so as to put a number around it, is namespaces will consume 150.X bytes (where X is the overhead of a conntrack table) and you approach will be (X + 152) bytes, correct?
What is the typical sizeof X?

Quoted 5 lines

Quoted 2 lines

You may also wanna look as a metric at code complexity/maintainability of this scheme vs namespace (which adds zero changes to the kernel).

There's not a lot of complexity, its basically passing a numeric identifier around in a few spots and comparing it. Something like TOS handling in the routing code.

I think the challenge is whether zones will have to encroach on other net stack objects or not. You are already touching structure netdev...

Quoted 2 lines · jamal

b) dynamic protocols (routing, IKE etc): how do you do that without making both sides understand what is going on?

In case of IPsec the outer addresses are different, its only the selectors which will have similar addresses. A keying deamon should have no trouble with this. The ifindex would be needed in the selectors though to make sure each policy is used for the correct traffic.

A routing daemon is unrealistic to be used in this scenario, at least a single one for all the overlapping networks.

Quoted 3 lines

Agreed. But the major ones like iproute2 etc could be taught. We have namespaces in the kernel already, over a period of time I think changing the user space tools would a sensible evolution.

Yes, that might be useful in any case. But I don't think it would even work for iproute or other standalone programs, a process can't associate to an existing namespace except through clone(). So it needs to run as child of a process already associated with the namespace.

(X + 152) bytes,

Quoted 2 lines

correct?
What is the typical sizeof X?

No, to give some correct number. Assuming a conntrack table of 10MB (large, but reasonable depending on the number of connections) we get an overhead of:

namespaces: 150 * 10MB memory use
"zones": 152 bytes increased code size

Both approaches additionally need one extra connection tracking entry of ~300 bytes per connection that is actually handled twice.

That will go away once I add a target for classification.

Quoted 9 lines

Quoted 3 lines

Agreed. But the major ones like iproute2 etc could be taught. We have namespaces in the kernel already, over a period of time I think changing the user space tools would a sensible evolution.

Yes, that might be useful in any case. But I don't think it would even work for iproute or other standalone programs, a process can't associate to an existing namespace except through clone(). So it needs to run as child of a process already associated with the namespace.

The mechanics are not there, yet. But if i had sufficient permission, and was able to find the namespaces when i ask and/or get events when it is created it should be an issue of sending it a message. The current approach to say migrate a veth via iproute2 requires we know the pid of the target namespace. Thats a usability issue. I tried to muck with namespaces and if you use a library like lxc you can do it - but it is a hack as it stands today (and merging iproute2 with lxc is questionable).

jamal <[email protected]> writes:

This is one of the long standing issues that we have always known we needed to solve, but have not taken the time to do it. Now that the need is more real it looks about time to solve this one.

There are currently two problems.
1) A process is needed to hold a reference to the network namespace.
2) We use pids which are an awkward way of talking about network namespaces.

The solution I have been playing with involves.
- Using a file descriptor to refer to a network namespace.
- Using a trivial virtual filesystem to persistently hold onto a namespace without the need of a process.
- Have a convention of mounting the fs at something like /var/run/netns/<name>

Quoted 12 lines · Eric W. Biederman

jamal <[email protected]> writes:

This is one of the long standing issues that we have always known we needed to solve, but have not taken the time to do it. Now that the need is more real it looks about time to solve this one.

There are currently two problems.
1) A process is needed to hold a reference to the network namespace.
2) We use pids which are an awkward way of talking about network namespaces.

Quoted 7 lines

The solution I have been playing with involves.
- Using a file descriptor to refer to a network namespace.
- Using a trivial virtual filesystem to persistently hold onto a namespace without the need of a process.
- Have a convention of mounting the fs at something like /var/run/netns/<name>

I didnt quiet follow how i could use the above to do:
"ip ns <name/id> route add blah" from namespace0.

I tend to think in packets and wires instead of files;
How about just allowing a "control" channel from which
i could discover the namespace?
Example, assuming i have the right permissions:
1) listen to async events example on a multicast bus when
a namespace is created or destroyed. Provide me a little more info on the created namespace such as its pid, name(?), types of namespace, etc
2) send a query to dump existing namespace or query by name, id etc. I get the same details as above.

using genetlink should provide you with sufficient ability to do this.

What I am thinking is:

"ip ns <name> route add blah" is:
fd = open("/var/run/netns/<name>");
sys_setns(fd); /* Like unshare but takes an existing namespace */ /* Then the rest of the existing ip command */

"ip ns list" is:
dfd = open("/var/run/netns", O_DIRECTORY);
getdents(dfd, buf, count);

"ip ns new <name>" is:
unshare(CLONE_NEWNS);
fd = nsfd(NETNS); mkdir("/var/run/netns/<name>"); mount("none", "/var/run/netns/<name>", "ns", 0, fd);

Using unix domain names means that which namespaces you see is under control of userspace. Which allows for nested containers (something I use today), and ultimately container migration.

Using genetlink userspace doesn't result in a nestable implementation unless I introduce yet another namespace, ugh.

Quoted 2 lines · Eric W. Biederman

What I am thinking is:

Quoted 4 lines

"ip ns <name> route add blah" is:
fd = open("/var/run/netns/<name>");
sys_setns(fd); /* Like unshare but takes an existing namespace */ /* Then the rest of the existing ip command */

The other two below make some sense; For the above:
Does the point after sys_setns(fd) allow me to do io inside ns <name>? Can i do open() and get a fd from ns <name>?

Quoted 10 lines

"ip ns list" is:
dfd = open("/var/run/netns", O_DIRECTORY);
getdents(dfd, buf, count);

"ip ns new <name>" is:
unshare(CLONE_NEWNS);
fd = nsfd(NETNS); mkdir("/var/run/netns/<name>"); mount("none", "/var/run/netns/<name>", "ns", 0, fd);

Quoted 3 lines

Using unix domain names means that which namespaces you see is under control of userspace. Which allows for nested containers (something I use today), and ultimately container migration.

The only problem that i see is events are not as nice. I take it i am going to get something like an inotify when a new namespace is created?

Quoted 2 lines

Using genetlink userspace doesn't result in a nestable implementation unless I introduce yet another namespace, ugh.

Is it not just a naming convention that you are dealing with? Example in your scheme above a nested namespace shows up as:
/var/run/netns/<name>/<nestedname>, no?

Quoted 2 lines · jamal

Does the point after sys_setns(fd) allow me to do io inside ns <name>? Can i do open() and get a fd from ns <name>?

Yes. My intention is that current->nsproxy->net_ns be changed. We can already change it in unshare so this is feasible.

Quoted 2 lines

The only problem that i see is events are not as nice. I take it i am going to get something like an inotify when a new namespace is created?

Yes. Inotify would at the very least see that mkdir. You could also use poll on /proc/mounts to see the set of mounts change.

Quoted 3 lines

Is it not just a naming convention that you are dealing with? Example in your scheme above a nested namespace shows up as:
/var/run/netns/<name>/<nestedname>, no?

No. More like:

For the outer namespace:
/var/run/netns/<name>

For the inner namespace:
/some/random/fs/path/to/a/chroot/var/run/netns/<name>

For a doubly nested scenario:
/some/random/fs/path/to/a/chroot/some/other/random/fs/path/to/another/chroot/var/run/netns/<name>

Since I would be using mount namespaces instead of chroot it is not strictly required that the fs paths nest at all.

Added Daniel to the discussion..

Quoted 5 lines · Eric W. Biederman

Quoted 2 lines · jamal

Does the point after sys_setns(fd) allow me to do io inside ns <name>? Can i do open() and get a fd from ns <name>?

Quoted 2 lines

Yes. My intention is that current->nsproxy->net_ns be changed. We can already change it in unshare so this is feasible.

I like it if it makes it as easy as it sounds;-> With lxc, i essentially have to create a proxy process inside the namespace that i use unix domain to open fds inside the ns. Do i still need that?

Quoted 5 lines

Quoted 2 lines

The only problem that i see is events are not as nice. I take it i am going to get something like an inotify when a new namespace is created?

Yes. Inotify would at the very least see that mkdir. You could also use poll on /proc/mounts to see the set of mounts change.

It is not as nice but livable. I suppose attributes of the specific namespace are retrieved somewhere there as well..

That point of the mount to hold a persistent reference to the namespace without using a process.

The point of the of the to be written set_ns call is to change the default network namespace of the process such that all future open/bind/socket calls happen in the referenced network namespace.

The are a few stray places like sysfs where it is the mount point not current->nsproxy->net_ns that will determine what we see.

Attributes of the specific namespace?

Quoted 6 lines · Eric W. Biederman

That point of the mount to hold a persistent reference to the namespace without using a process.

The point of the of the to be written set_ns call is to change the default network namespace of the process such that all future open/bind/socket calls happen in the referenced network namespace.

Ok, i like it ;-> Patches RSN? Let me if you want someone to test..

Quoted 2 lines

The are a few stray places like sysfs where it is the mount point not current->nsproxy->net_ns that will determine what we see.

Is sysfs considered "usable enough" for namespaces?

Quoted 1 line

Attributes of the specific namespace?

Well, example what is being un/shared etc.

Quoted 6 lines · Eric W. Biederman

That point of the mount to hold a persistent reference to the namespace without using a process.

The point of the of the to be written set_ns call is to change the default network namespace of the process such that all future open/bind/socket calls happen in the referenced network namespace.

Ok, i like it ;-> Patches RSN? Let me if you want someone to test..

Quoted 2 lines

The are a few stray places like sysfs where it is the mount point not current->nsproxy->net_ns that will determine what we see.

Is sysfs considered "usable enough" for namespaces?

Quoted 1 line

Attributes of the specific namespace?

Well, example what is being un/shared etc.

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <[email protected]>

---

Quoted 24 lines · Eric W. Biederman

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Quoted 9 lines

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <[email protected]>

---

Is it planned to support all the namespaces for 'nsfd' ?
I mean will it be possible to specify an Or'ed combination of nstype to grab a reference for several namespaces at a time of the targeted process ?

for example : nsfd( 1234, NSTYPE_NET | NSTYPE_IPC, NSTYPE_MNT)

Quoted 40 lines · Daniel Lezcano

Quoted 24 lines · Eric W. Biederman

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Quoted 10 lines

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <[email protected]>

---

Is it planned to support all the namespaces for 'nsfd' ?
I mean will it be possible to specify an Or'ed combination of nstype to grab a reference for several namespaces at a time of the targeted process ?

Quoted 2 lines

for example : nsfd( 1234, NSTYPE_NET | NSTYPE_IPC, NSTYPE_MNT)

No, the plan is only one namespace at a time.

It would not be much of a change to support multiple namespaces, but I don't think I want to go there. Bitmaps filling up are ugly and I don't see what would be gained.

I does make sense to support all of the namespaces we can support with unshare, but with nstype as an enumeration not as a bitmap.

This is slightly better than the earlier version that used a netlink socket as the reference as I can give it the semantics of a deleted file and only when that file goes away drop the reference on the namespace. It is also better in that this interface can support all of the namespaces, without adding yet another syscall.

Quoted 6 lines

No, the plan is only one namespace at a time.

It would not be much of a change to support multiple namespaces, but I don't think I want to go there. Bitmaps filling up are ugly and I don't see what would be gained.

The idea I had in mind when I asked this question was if we can "move" a process inside a container, aka a set of namespaces :)

Quoted 3 lines

I does make sense to support all of the namespaces we can support with unshare, but with nstype as an enumeration not as a bitmap.

I suppose when you say "to support all of the namespaces we can support with *unshare*", you exclude the pid namespace which is created only with clone, right ? Do you think we can extend the concept to all the namespaces including the pid_namespace ?

Quoted 6 lines

No, the plan is only one namespace at a time.

It would not be much of a change to support multiple namespaces, but I don't think I want to go there. Bitmaps filling up are ugly and I don't see what would be gained.

The idea I had in mind when I asked this question was if we can "move" a process inside a container, aka a set of namespaces :)

Quoted 3 lines

I does make sense to support all of the namespaces we can support with unshare, but with nstype as an enumeration not as a bitmap.

I suppose when you say "to support all of the namespaces we can support with *unshare*", you exclude the pid namespace which is created only with clone, right ? Do you think we can extend the concept to all the namespaces including the pid_namespace ?

Quoted 40 lines · Daniel Lezcano

Quoted 24 lines · Eric W. Biederman

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Quoted 10 lines

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]>

---

Is it planned to support all the namespaces for 'nsfd' ?
I mean will it be possible to specify an Or'ed combination of nstype to grab a reference for several namespaces at a time of the targeted process ?

Quoted 2 lines

for example : nsfd( 1234, NSTYPE_NET | NSTYPE_IPC, NSTYPE_MNT)

No, the plan is only one namespace at a time.

It would not be much of a change to support multiple namespaces, but I don't think I want to go there. Bitmaps filling up are ugly and I don't see what would be gained.

I does make sense to support all of the namespaces we can support with unshare, but with nstype as an enumeration not as a bitmap.

This is slightly better than the earlier version that used a netlink socket as the reference as I can give it the semantics of a deleted file and only when that file goes away drop the reference on the namespace. It is also better in that this interface can support all of the namespaces, without adding yet another syscall.

No, the plan is only one namespace at a time.

No, the plan is only one namespace at a time.

Quoted 24 lines · Eric W. Biederman

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Quoted 9 lines

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]>

---

Is it planned to support all the namespaces for 'nsfd' ?
I mean will it be possible to specify an Or'ed combination of nstype to grab a reference for several namespaces at a time of the targeted process ?

for example : nsfd( 1234, NSTYPE_NET | NSTYPE_IPC, NSTYPE_MNT)

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]>

---

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]>

Introduce two new system calls:
int nsfd(pid_t pid, unsigned long nstype);
int setns(unsigned long nstype, int fd);

These two new system calls address three specific problems that can make namespaces hard to work with.
- Namespaces require a dedicated process to pin them in memory.
- It is not possible to use a namespace unless you are the child of the original creator.
- Namespaces don't have names that userspace can use to talk about them.

The nsfd() system call returns a file descriptor that can be used to talk about a specific namespace, and to keep the specified namespace alive.

The fd returned by nsfd() can be bind mounted as:
mount --bind /proc/self/fd/N /some/filesystem/path to keep the namespace alive indefinitely as long as it is mounted.

open works on the fd returned by nsfd() so another process can get a hold of it and do interesting things.

Overall that allows for persistent naming of namespaces according to userspace policy.

setns() allows changing the namespace of the current process to a namespace that originates with nsfd().

Signed-off-by Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]>

---

This is just my first pass at this, and not yet compiled tested. I was pleasantly surprised at how easy all of this was to implement.

fs/nsfd.c new file +12 -0

+static struct inode *nsfd_mkinode(void)
+{
+	struct inode *inode;
+	inode = new_inode(nsfd_mnt->mnt_sb);
+	if (!inode)
+		return ERR_PTR(-ENOMEM);
+
+	inode->i_fop = &nsfd_file_operations;
+
+	/*
+	 * Mark the inode dirty from the very beginning,
+	 * that way it will never be moved to the dirty

Quoted 16 lines

fs/nsfd.c new file +16 -0

+static struct inode *nsfd_mkinode(void)
+{
+	struct inode *inode;
+	inode = new_inode(nsfd_mnt->mnt_sb);
+	if (!inode)
+		return ERR_PTR(-ENOMEM);
+
+	inode->i_fop = &nsfd_file_operations;
+
+	/*
+	 * Mark the inode dirty from the very beginning,
+	 * that way it will never be moved to the dirty
+	 * list because mark_inode_dirty() will think that
+	 * it already _is_ on the dirty list.
+	 */
+	inode->i_state = I_DIRTY;

fs/nsfd.c line 121

Quoted 6 lines

fs/nsfd.c new file +6 -0

+	inode->i_mode = S_IRUSR | S_IWUSR;
+	inode->i_uid = current_fsuid();
+	inode->i_gid = current_fsgid();
+	inode->i_atime = inode->i_mtime = inode->i_ctime = CURRENT_TIME;
+	return inode;
+}

Why not use anon inodes?

Quoted 8 lines

include/linux/nstype.h new file +2 -0

new file mode 100644
index 0000000..3bdf856
@@ -0,0 +1,6 @@
+#ifndef _LINUX_NSTYPE_H
+#define _LINUX_NSTYPE_H

include/linux/nstype.h line 6

Quoted 4 lines

include/linux/nstype.h new file +4 -0

+
+#define NSTYPE_NET 0
+
+#endif /* _LINUX_NSTYPE_H */

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

Eric

Quoted 6 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

Quoted 5 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

Quoted 2 lines

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

Why don't we have a syscall that allows us to enter every namespace?

Eric

_______________________________________________
Containers mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/containers

Quoted 7 lines · Eric W. Biederman

Quoted 5 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

Quoted 7 lines

Quoted 2 lines

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

That's how I understood the question, and I, too, wonder why not ?

By the way, an alternative to using bitmap is to change the prototype of setns() to accept an array of FD's:

	int setns(int *fds, int nfds);

So the process will atomically enter all the namespaces as specified by the FDs.

Quoted 16 lines · Oren Laadan

Quoted 5 lines · Eric W. Biederman

Quoted 3 lines · Pavel Emelyanov

Quoted 3 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

That's how I understood the question, and I, too, wonder why not ?

Quoted 7 lines

By the way, an alternative to using bitmap is to change the prototype of setns() to accept an array of FD's:

	int setns(int *fds, int nfds);

So the process will atomically enter all the namespaces as specified by the FDs.

We could. Mostly I implemented things in the simplest way possible.

Semantically I know of no reason why need to enter more than one namespace at once, and I don't expect entering a namespace to be on anyone's fast path so every last drop of performance was not crucial.

The only justification I can think of for more than one namespace at a time is that because we have a synchronize_rcu() in the kernel we can loop in the kernel much more quickly than we can loop in userspace.

Can't think of a specific scenario, but I wonder if there would be a problem (security or otherwise) with a process that only partly belongs to a container, even if for a short time ?

Can't think of a specific scenario, but I wonder if there would be a problem (security or otherwise) with a process that only partly belongs to a container, even if for a short time ?

Quoted 16 lines · Oren Laadan

Quoted 5 lines · Eric W. Biederman

Quoted 3 lines · Pavel Emelyanov

Quoted 3 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

That's how I understood the question, and I, too, wonder why not ?

Quoted 7 lines

By the way, an alternative to using bitmap is to change the prototype of setns() to accept an array of FD's:

	int setns(int *fds, int nfds);

So the process will atomically enter all the namespaces as specified by the FDs.

We could. Mostly I implemented things in the simplest way possible.

Semantically I know of no reason why need to enter more than one namespace at once, and I don't expect entering a namespace to be on anyone's fast path so every last drop of performance was not crucial.

The only justification I can think of for more than one namespace at a time is that because we have a synchronize_rcu() in the kernel we can loop in the kernel much more quickly than we can loop in userspace.

Quoted 7 lines · Eric W. Biederman

Quoted 5 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

Quoted 7 lines

Quoted 2 lines

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

That's how I understood the question, and I, too, wonder why not ?

By the way, an alternative to using bitmap is to change the prototype of setns() to accept an array of FD's:

	int setns(int *fds, int nfds);

So the process will atomically enter all the namespaces as specified by the FDs.

Quoted 5 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

Quoted 2 lines

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

Why don't we have a syscall that allows us to enter every namespace?

Eric

_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Quoted 5 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

Quoted 2 lines

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

Why don't we have a syscall that allows us to enter every namespace?

Eric

Quoted 5 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

Quoted 2 lines

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

Why don't we have a syscall that allows us to enter every namespace?

Eric

Quoted 10 lines · Eric W. Biederman

Quoted 7 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

Quoted 4 lines

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

Exactly. Please add at least the NSTYPE_NSPROXY or whatever, that will pin all namespaces of a given pid from the very beginning.

Quoted 1 line

Why don't we have a syscall that allows us to enter every namespace?

This one is done in the patch, no?

Although the approach is OK for me, there's one design issue, that came up to my mind recently: can we use this fd to wail for a namespace to stop? I currently don't see this ability, but this is something I require badly.

Quoted 10 lines · Pavel Emelyanov

Quoted 9 lines · Eric W. Biederman

Quoted 7 lines · Pavel Emelyanov

Quoted 5 lines

Quoted 3 lines

Yet another set of per-namespace IDs along with CLONE_NEWXXX ones? I currently have a way to create all namespaces we have with one syscall. Why don't we have an ability to enter them all with one syscall?

The CLONE_NEWXXX series of bits has been an royal pain to work with, and it appears to be unnecessary complications for no gain.

That's the answer for the "Yet another set..." question. How about the "Why don't we have..." one?

Quoted 6 lines

Quoted 4 lines

I am not certain which question you are asking:

Why don't we have an ability to enter all namespaces with one syscall invocation?

Exactly. Please add at least the NSTYPE_NSPROXY or whatever, that will pin all namespaces of a given pid from the very beginning.

For nsfd(2) that is doable. At least for now setns can't restore it.

Quoted 7 lines

Quoted 1 line

Why don't we have a syscall that allows us to enter every namespace?

This one is done in the patch, no?

Although the approach is OK for me, there's one design issue, that came up to my mind recently: can we use this fd to wail for a namespace to stop? I currently don't see this ability, but this is something I require badly.

I have designed these file descriptors to pin the namespaces, so waiting for them to exit isn't something they can do now. It makes a lot of sense to have similar ones that take weak references to the namespaces that we can use to wait for a namespace to exit.

Eric

Thanks. What's the problem with setns?

Quoted 4 lines

I have designed these file descriptors to pin the namespaces, so waiting for them to exit isn't something they can do now. It makes a lot of sense to have similar ones that take weak references to the namespaces that we can use to wait for a namespace to exit.

Yes, I saw this from patches. Eric, I'd very much appreciate if we workout a solution that will allow us to kill two birds with one stone. I do not want to invent yet another bunch of system calls for "taking weak reference".

As a "brain storm" start up. Can we use inotify/dnotify for this? Or maybe we should better equip the nsfd call with flags argument and add a flag for weak reference? In that case - how shall we get a notification about namespace is dead? With poll? Maybe worth making the sys_close return only when the namespace is dead (by providing a proper ->release callback of a file)?

Quoted 1 line · Pavel Emelyanov

Thanks. What's the problem with setns?

joining a preexisting namespace is roughly the same problem as unsharing a namespace. We simply haven't figure out how to do it safely for the pid and the uid namespaces.

Quoted 8 lines

Quoted 4 lines

I have designed these file descriptors to pin the namespaces, so waiting for them to exit isn't something they can do now. It makes a lot of sense to have similar ones that take weak references to the namespaces that we can use to wait for a namespace to exit.

Yes, I saw this from patches. Eric, I'd very much appreciate if we workout a solution that will allow us to kill two birds with one stone. I do not want to invent yet another bunch of system calls for "taking weak reference".

Definitely. I only consider the current interface to be a mushy not set in stone.

Quoted 6 lines

As a "brain storm" start up. Can we use inotify/dnotify for this? Or maybe we should better equip the nsfd call with flags argument and add a flag for weak reference? In that case - how shall we get a notification about namespace is dead? With poll? Maybe worth making the sys_close return only when the namespace is dead (by providing a proper ->release callback of a file)?

We would want poll to work, anything else is a weird work-around. The challenging part is that we don't have any infrastructure for notifying when a namespace goes away. So that has to be built before we can wire it up to userspace. I don't expect it is too difficult but there is work to be done.

Eric

Quoted 4 lines · Eric W. Biederman

Quoted 1 line · Pavel Emelyanov

Thanks. What's the problem with setns?

Quoted 3 lines

joining a preexisting namespace is roughly the same problem as unsharing a namespace. We simply haven't figure out how to do it safely for the pid and the uid namespaces.

The pid may change after this for sure. What problems do you know about it? What if we try to allocate the same PID in a new space or return -EBUSY? This will be a good starting point. If we manage to fix it later this will not break the API at all.

Quoted 2 lines

Definitely. I only consider the current interface to be a mushy not set in stone.

OK. The interface is good. I just don't want you to send it for an inclusion until we decide what to do with waiting.

Poll is OK with me. As far as the notification is concerned - that's also done in OpenVZ. If you are OK to wait for a week or two I can do it for net namespaces.

Quoted 7 lines · Pavel Emelyanov

Quoted 6 lines · Eric W. Biederman

Quoted 1 line · Pavel Emelyanov

Thanks. What's the problem with setns?

joining a preexisting namespace is roughly the same problem as unsharing a namespace. We simply haven't figure out how to do it safely for the pid and the uid namespaces.

Quoted 4 lines

The pid may change after this for sure. What problems do you know about it? What if we try to allocate the same PID in a new space or return -EBUSY? This will be a good starting point. If we manage to fix it later this will not break the API at all.

Parentage. The pid is the identity of a process and all kinds of things make assumptions in all kinds of strange places. I don't see how waitpid can work if you change the pid.

glibc doesn't cope if you change someones pid.

Quoted 4 lines

Quoted 2 lines

Definitely. I only consider the current interface to be a mushy not set in stone.

OK. The interface is good. I just don't want you to send it for an inclusion until we decide what to do with waiting.

Sure. I am get a jump on 2.6.35 not aiming for inclusion this merge window. There is plenty of time.

Quoted 3 lines

Poll is OK with me. As far as the notification is concerned - that's also done in OpenVZ. If you are OK to wait for a week or two I can do it for net namespaces.

Seems reasonable.

Eric

Quoted 14 lines · Eric W. Biederman

Quoted 11 lines · Pavel Emelyanov

Quoted 6 lines · Eric W. Biederman

Quoted 1 line · Pavel Emelyanov

Thanks. What's the problem with setns?

joining a preexisting namespace is roughly the same problem as unsharing a namespace. We simply haven't figure out how to do it safely for the pid and the uid namespaces.

The pid may change after this for sure. What problems do you know about it? What if we try to allocate the same PID in a new space or return -EBUSY? This will be a good starting point. If we manage to fix it later this will not break the API at all.

Quoted 3 lines

Parentage. The pid is the identity of a process and all kinds of things make assumptions in all kinds of strange places. I don't see how waitpid can work if you change the pid.

Agree. But what if we enter a pid space, which is a subnamespace of a current one? In that case parent will still see the task by its old pid. We can restrict first version of entering with this rule as well and this restriction will not block us in typical usecase (I mean enter a container from a host).

Quoted 1 line

glibc doesn't cope if you change someones pid.

OK, but what if we try to allocate the same pid returning -EBUSY on failure?

My aim is to provide even a restricted enter. For most of the cases this should work and make our lives easier. So two restrictions currently:
a) enter a sub namespace
b) allocate the same pid as we have now

Hm? :)

Quoted 14 lines · Pavel Emelyanov

Quoted 13 lines · Eric W. Biederman

Quoted 11 lines · Pavel Emelyanov

Quoted 6 lines · Eric W. Biederman

Quoted 1 line · Pavel Emelyanov

Thanks. What's the problem with setns?

joining a preexisting namespace is roughly the same problem as unsharing a namespace. We simply haven't figure out how to do it safely for the pid and the uid namespaces.

The pid may change after this for sure. What problems do you know about it? What if we try to allocate the same PID in a new space or return -EBUSY? This will be a good starting point. If we manage to fix it later this will not break the API at all.

Quoted 7 lines

Quoted 3 lines

Parentage. The pid is the identity of a process and all kinds of things make assumptions in all kinds of strange places. I don't see how waitpid can work if you change the pid.

Agree. But what if we enter a pid space, which is a subnamespace of a current one? In that case parent will still see the task by its old pid. We can restrict first version of entering with this rule as well and this restriction will not block us in typical usecase (I mean enter a container from a host).

When I was thinking about pid namespaces and unshare last time. The idea I came to was we unshare of the pid namespace should only affect which pid namespace your children are in.

I remember that do that there were a few cases where you would have to access task->pid->pid_ns instead of task->nsproxy->pid_ns, but essentially it was pretty simple.

Quoted 8 lines

Quoted 1 line

glibc doesn't cope if you change someones pid.

OK, but what if we try to allocate the same pid returning -EBUSY on failure?

My aim is to provide even a restricted enter. For most of the cases this should work and make our lives easier. So two restrictions currently:
a) enter a sub namespace
b) allocate the same pid as we have now

Quoted 1 line

Hm? :)

Replacing struct pid is guaranteed to do all kinds of nasty things with signal handling and the like, de_thread is nasty enough and you are talking something worse. So if we can change pid namespaces without changing the pid I am for it.

Quoted 4 lines

Replacing struct pid is guaranteed to do all kinds of nasty things with signal handling and the like, de_thread is nasty enough and you are talking something worse. So if we can change pid namespaces without changing the pid I am for it.

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example). What is the difference with the sys_bindns or the sys_hijack, proposed a couple of years ago ?

I did a suggestion some weeks ago about a new syscall 'cloneat' where the child process becomes the child of the targeted process specified in the syscall. Maybe it would be interesting to replace the 'setns' by, or add, a cloneat' syscall with the file descriptor passed as parameter. The copy_process function shall not use the nsproxy of the caller but the one provided in the fd argument.

The newly created process becomes the child of the process where we retrieve the namespace with nsfd and this one have to 'waitpid' it, (the caller of 'cloneat' can not wait it). It's a bit similar with the CLONE_PARENT flag, except the creation order is inverted (the father creates for the child).

So when entering the container, we specify the pid 1 of the container which is usually a child reaper.

Does it make sense ?

Quoted 4 lines

Replacing struct pid is guaranteed to do all kinds of nasty things with signal handling and the like, de_thread is nasty enough and you are talking something worse. So if we can change pid namespaces without changing the pid I am for it.

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example). What is the difference with the sys_bindns or the sys_hijack, proposed a couple of years ago ?

I did a suggestion some weeks ago about a new syscall 'cloneat' where the child process becomes the child of the targeted process specified in the syscall. Maybe it would be interesting to replace the 'setns' by, or add, a cloneat' syscall with the file descriptor passed as parameter. The copy_process function shall not use the nsproxy of the caller but the one provided in the fd argument.

The newly created process becomes the child of the process where we retrieve the namespace with nsfd and this one have to 'waitpid' it, (the caller of 'cloneat' can not wait it). It's a bit similar with the CLONE_PARENT flag, except the creation order is inverted (the father creates for the child).

So when entering the container, we specify the pid 1 of the container which is usually a child reaper.

Does it make sense ?

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example). What is the difference with the sys_bindns or the sys_hijack, proposed a couple of years ago ?

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example).

Quoted 39 lines · Eric W. Biederman

Quoted 14 lines · Pavel Emelyanov

Quoted 13 lines · Eric W. Biederman

Quoted 11 lines · Pavel Emelyanov

Quoted 6 lines · Eric W. Biederman

Quoted 1 line · Pavel Emelyanov

Thanks. What's the problem with setns?

joining a preexisting namespace is roughly the same problem as unsharing a namespace. We simply haven't figure out how to do it safely for the pid and the uid namespaces.

The pid may change after this for sure. What problems do you know about it? What if we try to allocate the same PID in a new space or return -EBUSY? This will be a good starting point. If we manage to fix it later this will not break the API at all.

Quoted 7 lines

Quoted 3 lines

Parentage. The pid is the identity of a process and all kinds of things make assumptions in all kinds of strange places. I don't see how waitpid can work if you change the pid.

Agree. But what if we enter a pid space, which is a subnamespace of a current one? In that case parent will still see the task by its old pid. We can restrict first version of entering with this rule as well and this restriction will not block us in typical usecase (I mean enter a container from a host).

When I was thinking about pid namespaces and unshare last time. The idea I came to was we unshare of the pid namespace should only affect which pid namespace your children are in.

I remember that do that there were a few cases where you would have to access task->pid->pid_ns instead of task->nsproxy->pid_ns, but essentially it was pretty simple.

Quoted 8 lines

Quoted 1 line

glibc doesn't cope if you change someones pid.

OK, but what if we try to allocate the same pid returning -EBUSY on failure?

My aim is to provide even a restricted enter. For most of the cases this should work and make our lives easier. So two restrictions currently:
a) enter a sub namespace
b) allocate the same pid as we have now

Quoted 5 lines

Quoted 1 line

Hm? :)

Replacing struct pid is guaranteed to do all kinds of nasty things with signal handling and the like, de_thread is nasty enough and you are talking something worse. So if we can change pid namespaces without changing the pid I am for it.

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example). What is the difference with the sys_bindns or the sys_hijack, proposed a couple of years ago ?

I did a suggestion some weeks ago about a new syscall 'cloneat' where the child process becomes the child of the targeted process specified in the syscall. Maybe it would be interesting to replace the 'setns' by, or add, a 'cloneat' syscall with the file descriptor passed as parameter. The copy_process function shall not use the nsproxy of the caller but the one provided in the fd argument.

The newly created process becomes the child of the process where we retrieve the namespace with nsfd and this one have to 'waitpid' it, (the caller of 'cloneat' can not wait it). It's a bit similar with the CLONE_PARENT flag, except the creation order is inverted (the father creates for the child).

So when entering the container, we specify the pid 1 of the container which is usually a child reaper.

Does it make sense ?

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example). What is the difference with the sys_bindns or the sys_hijack, proposed a couple of years ago ?

I am of two mind about my patches. Right now they are a brilliant proof of concept that we can name namespaces without needing a namespace for the names of namespaces, and start to be a practical
solution to the join problem. At the same time, I'm not certain
I like a solution that requires yet more syscalls so I ask myself is there not yet a simpler way.

I am of two mind about my patches. Right now they are a brilliant proof of concept that we can name namespaces without needing a namespace for the names of namespaces, and start to be a practical
solution to the join problem. At the same time, I'm not certain
I like a solution that requires yet more syscalls so I ask myself is there not yet a simpler way.

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example).

Quoted 4 lines

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example).

The answer is - the one, that used to be. I see no problems with it. Do you?

Quoted 5 lines · Pavel Emelyanov

Quoted 4 lines

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example).

Quoted 2 lines

The answer is - the one, that used to be. I see no problems with it. Do you?

But perhaps it could be named "namespacefd" instead of nsfd, to reduce potential clashes (because glibc will usually just use the same name when making the syscall available as a C function).

Quoted 5 lines · Pavel Emelyanov

Quoted 4 lines

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example).

Quoted 2 lines

The answer is - the one, that used to be. I see no problems with it. Do you?

But perhaps it could be named "namespacefd" instead of nsfd, to reduce potential clashes (because glibc will usually just use the same name when making the syscall available as a C function).

Quoted 4 lines

I agree with all the points you and Pavel you talked about but I don't feel comfortable to have the current process to switch the pid namespace because of the process tree hierarchy (what will be the parent of the process when you enter the pid namespace for example).

The answer is - the one, that used to be. I see no problems with it. Do you?

Pavel Emelyanov [[email protected]] wrote:

| > I agree with all the points you and Pavel you talked about but I don't 
| > feel comfortable to have the current process to switch the pid namespace 
| > because of the process tree hierarchy (what will be the parent of the 
| > process when you enter the pid namespace for example).
| 
| The answer is - the one, that used to be. I see no problems with it.
| Do you?

Just to be clear, when a process unshares its pid namespace, it takes on additional pid nr (== 1) in the new namespace but retains its original pid nr(s) in the parent (ancestor) namespaces right ?

i.e the process becomes the container-init of the new namespace. When it exits, all its children belonging to the new namespace are killed too, but any children in the parent namespace (i.e children created before unshare()) are not killed.

After the unshare() the process will not be able to signal any children it created before the unshare() (bc their active pid namespaces are different)

From an implementation point of view who gets pid 1 when the child_reaper is

Eric W. Biederman [[email protected]] wrote:

| 
| I think replacing a struct pid for another struct pid allocated in
| descendant pid_namespace (but has all of the same struct upid values
| as the first struct pid) is a disastrous idea.  It destroys the

True. Sorry, I did not mean we would need a new 'struct pid' for an existing process. I think we talked earlier of finding a way of attaching additional pid numbers to the same struct pid.

Quoted 6 lines · Sukadev Bhattiprolu

Eric W. Biederman [[email protected]] wrote:

| 
| I think replacing a struct pid for another struct pid allocated in
| descendant pid_namespace (but has all of the same struct upid values
| as the first struct pid) is a disastrous idea.  It destroys the

Quoted 3 lines

True. Sorry, I did not mean we would need a new 'struct pid' for an existing process. I think we talked earlier of finding a way of attaching additional pid numbers to the same struct pid.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

Quoted 8 lines · Eric W. Biederman

Quoted 6 lines · Sukadev Bhattiprolu

Eric W. Biederman [[email protected]] wrote:

| 
| I think replacing a struct pid for another struct pid allocated in
| descendant pid_namespace (but has all of the same struct upid values
| as the first struct pid) is a disastrous idea.  It destroys the

Quoted 7 lines

Quoted 3 lines

True. Sorry, I did not mean we would need a new 'struct pid' for an existing process. I think we talked earlier of finding a way of attaching additional pid numbers to the same struct pid.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

Heh, and then (browsing through your copy_process() patch hunks) the next forked task becomes the child reaper for the new pidns? <shrug> why not I guess.

Now if that child reaper then gets killed, will the idle task get killed too?

And if not, then idle task can just re-populating the new pidns with new idle tasks...

If this brought us a step closer to entering an existing pidns that would be one thing, but is there actually any advantage to being able to unshare a new pidns? Oh, I guess there is - PAM can then use it at login, which might be neat.

Quoted 17 lines · Serge E. Hallyn

Quoted 8 lines · Eric W. Biederman

Quoted 6 lines · Sukadev Bhattiprolu

Eric W. Biederman [[email protected]] wrote:

| 
| I think replacing a struct pid for another struct pid allocated in
| descendant pid_namespace (but has all of the same struct upid values
| as the first struct pid) is a disastrous idea.  It destroys the

Quoted 7 lines

Quoted 3 lines

True. Sorry, I did not mean we would need a new 'struct pid' for an existing process. I think we talked earlier of finding a way of attaching additional pid numbers to the same struct pid.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

Quoted 5 lines

Heh, and then (browsing through your copy_process() patch hunks) the next forked task becomes the child reaper for the new pidns? <shrug> why not I guess.

Now if that child reaper then gets killed, will the idle task get killed too?

No.

Quoted 2 lines

And if not, then idle task can just re-populating the new pidns with new idle tasks...

After zap_pid_namespace interesting...

Quoted 4 lines

If this brought us a step closer to entering an existing pidns that would be one thing, but is there actually any advantage to being able to unshare a new pidns? Oh, I guess there is - PAM can then use it at login, which might be neat.

I have to say that the semantics of my patch are unworkable for unshare. Unless I am mistaken for PAM to use it requires that the current process fully change and become what it needs to be. Requiring an extra fork to fully complete the process is a problem.

Scratch one bright idea.

So an unshare of the pid namespace that doesn't really take effect until we fork may actually be usable from pam, and in fact is probably the preferred implementation. It looks like neither openssh nor login from util-linux-ng will cope properly with getting any pid back from wait() except the pid of their child.

So an unshare of the pid namespace that doesn't really take effect until we fork may actually be usable from pam, and in fact is probably the preferred implementation. It looks like neither openssh nor login from util-linux-ng will cope properly with getting any pid back from wait() except the pid of their child.

Quoted 17 lines · Serge E. Hallyn

Quoted 8 lines · Eric W. Biederman

Quoted 6 lines · Sukadev Bhattiprolu

Eric W. Biederman [ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]] wrote:

| 
| I think replacing a struct pid for another struct pid allocated in
| descendant pid_namespace (but has all of the same struct upid values
| as the first struct pid) is a disastrous idea.  It destroys the

Quoted 7 lines

Quoted 3 lines

True. Sorry, I did not mean we would need a new 'struct pid' for an existing process. I think we talked earlier of finding a way of attaching additional pid numbers to the same struct pid.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

Quoted 5 lines

Heh, and then (browsing through your copy_process() patch hunks) the next forked task becomes the child reaper for the new pidns? <shrug> why not I guess.

Now if that child reaper then gets killed, will the idle task get killed too?

No.

Quoted 2 lines

And if not, then idle task can just re-populating the new pidns with new idle tasks...

After zap_pid_namespace interesting...

Quoted 4 lines

If this brought us a step closer to entering an existing pidns that would be one thing, but is there actually any advantage to being able to unshare a new pidns? Oh, I guess there is - PAM can then use it at login, which might be neat.

I have to say that the semantics of my patch are unworkable for unshare. Unless I am mistaken for PAM to use it requires that the current process fully change and become what it needs to be. Requiring an extra fork to fully complete the process is a problem.

Scratch one bright idea.

Quoted 8 lines · Eric W. Biederman

Quoted 6 lines · Sukadev Bhattiprolu

Eric W. Biederman [ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]] wrote:

| 
| I think replacing a struct pid for another struct pid allocated in
| descendant pid_namespace (but has all of the same struct upid values
| as the first struct pid) is a disastrous idea.  It destroys the

Quoted 7 lines

Quoted 3 lines

True. Sorry, I did not mean we would need a new 'struct pid' for an existing process. I think we talked earlier of finding a way of attaching additional pid numbers to the same struct pid.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

Heh, and then (browsing through your copy_process() patch hunks) the next forked task becomes the child reaper for the new pidns? <shrug> why not I guess.

Now if that child reaper then gets killed, will the idle task get killed too?

And if not, then idle task can just re-populating the new pidns with new idle tasks...

If this brought us a step closer to entering an existing pidns that would be one thing, but is there actually any advantage to being able to unshare a new pidns? Oh, I guess there is - PAM can then use it at login, which might be neat.

Quoted 6 lines · Sukadev Bhattiprolu

Eric W. Biederman [ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]] wrote:

| 
| I think replacing a struct pid for another struct pid allocated in
| descendant pid_namespace (but has all of the same struct upid values
| as the first struct pid) is a disastrous idea.  It destroys the

Quoted 3 lines

True. Sorry, I did not mean we would need a new 'struct pid' for an existing process. I think we talked earlier of finding a way of attaching additional pid numbers to the same struct pid.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

Quoted 3 lines

I just played with this and if you make the semantics of unshare(CLONE_NEWPID) to be that you become the idle task aka pid 0, and not the init task pid 1 the implementation is trivial.

This is not ... handy - if after enter you have pid 0 you obviously can't perform 2 parallel enters.

The way I see it:

As far as the numbers reported to the userspace are concerned:
1. task, that enters is still visible by its old parent by old pid
2. task, that enters gets some pid within the entering namespace and reports its parent pid to have pid 1 (init obviously doesn't care)
3. we _can_ try to allocate new pid equal to the old one so that glibc stays happy

As far as the pointers are concerned:
1. parent pointer doesn't change
2. task_pid(tsk) one (i.e. struct pid * one) _can_ change if
a) we don't allow threads enter (de_thread problem is handeled)
b) we don't allow leave the group/session, i.e. check, that there is the only one task that enters lives in its pgid/sid
c) we wait for the quiescent state to pass by before destroying the old pid to handle race with sys_kill()

2 parallel enters? I meant you have pid 0 in the entered pid namespace. You have pid 0 because your pid simply does not map.

There is nothing that makes to parallel enters impossible in that. Even today we have one thread per cpu that has task->pid == &init_struct_pid which is pid 0.

For the case of unshare where we are designed to be used with PAM I don't think my proposed semantics work. For a join needed an extra fork before you are really in the pid namespace should be minor.

That doesn't handle the case of cached struct pids. A good example is waitpid, where it waits for a specific struct pid. Which means that allocating a new struct pid and changing task->pid will cause waitpid(pid) to wait forever...

To change struct pid would require the refcount on struct pid to show no references from anywhere except the task_struct.

At the cost of a little memory we can solve that problem for unshare if we have a an extra upid in struct pid, how we verify there is space in struct pid I'm not certain.

I do think that at least until someone calls exec the namespace pids are reported to the process itself should not change. That is kill and

waitpid etc. Which suggests an implementation the opposite of what I proposed. With ns_of_pid(task_pid(current)) being used as the pid namespace of children, and current->nsproxy->pid_ns not changing in the case of unshare.

Shrug.

Or perhaps this is a case where we use we can implement join with an extra process but we can't implement unshare, because the effect cannot be immediate.

Eric

Quoted 2 lines

2 parallel enters? I meant you have pid 0 in the entered pid namespace. You have pid 0 because your pid simply does not map.

Oh, I see.

Quoted 3 lines

There is nothing that makes to parallel enters impossible in that. Even today we have one thread per cpu that has task->pid == &init_struct_pid which is pid 0.

How about the forked processes then? Who will be their parent?

Quoted 3 lines

For the case of unshare where we are designed to be used with PAM I don't think my proposed semantics work. For a join needed an extra fork before you are really in the pid namespace should be minor.

Hm... One more proposal - can we adopt the planned new fork_with_pids system call to fork the process right into a new pid namespace?

Quoted 4 lines

That doesn't handle the case of cached struct pids. A good example is waitpid, where it waits for a specific struct pid. Which means that allocating a new struct pid and changing task->pid will cause waitpid(pid) to wait forever...

OK. Good example. Thanks.

Quoted 2 lines

To change struct pid would require the refcount on struct pid to show no references from anywhere except the task_struct.

I think this is OK to return -EBUSY for this. And fix the waitpid respectively not to block this common case. All the others I think can be stayed as is.

Quoted 6 lines

At the cost of a little memory we can solve that problem for unshare if we have a an extra upid in struct pid, how we verify there is space in struct pid I'm not certain.

I do think that at least until someone calls exec the namespace pids are reported to the process itself should not change. That is kill and

Wait a second - in that case the wait will be blocked too! No?

Quoted 5 lines

waitpid etc. Which suggests an implementation the opposite of what I proposed. With ns_of_pid(task_pid(current)) being used as the pid namespace of children, and current->nsproxy->pid_ns not changing in the case of unshare.

Quoted 5 lines

Shrug.

Or perhaps this is a case where we use we can implement join with an extra process but we can't implement unshare, because the effect cannot be immediate.

Well, I'm talking only about the join now.