Per-user namespace process accounting
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
Finally if we use different UID/GID maps we can not do live migration to another node because the UIDs may be already in use.
So I'm proposing one hack modifying unshare_userns() to allocate new user_struct for the cred that is created for the first task creating the user_ns and free it in exit_creds().
Can you please comment on that?
Or suggest a better solution?
Best regards,
Marian
- --
Marian Marinov
Founder & CEO of 1H Ltd.
Jabber/GTalk: [email protected]
ICQ: 7556201
Mobile: +359 886 660 270
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iEYEARECAAYFAlOG1Y0ACgkQ4mt9JeIbjJQREACbBrax+ztBj2Y0P2jY3qYEUY9T JJ0AnAqLj3pqFFjXCkczEydV1V0LdzQ3
=8M+P
-----END PGP SIGNATURE-----
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
Quoted 11 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet
as much as possible. That has been true ever since NFS was invented
and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.
Quoted 5 lines
Finally if we use different UID/GID maps we can not do live migration to another node because the UIDs may be already in use.
So I'm proposing one hack modifying
unshare_userns()to allocate newuser_structfor the cred that is created for the first task creating theuser_nsand free it inexit_creds().
I do not like the idea of having user_structs be per user namespace, and deliberately made the code not work that way.
Quoted 1 line
Can you please comment on that?
I have been pondering having some recursive resources limits that are per user namespace and if all you are worried about are process counts that might work. I don't honestly know what makes sense at the moment.
Eric
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 10 lines · Eric W. Biederman
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
And I understand that very well ;)
Quoted 10 lines
Quoted 10 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
Quoted 8 lines
Quoted 2 lines
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet as much as possible. That has been true
ever since NFS was invented and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Quoted 7 lines
Quoted 6 lines
Finally if we use different UID/GID maps we can not do live migration to another node because the UIDs may be already in use.
So I'm proposing one hack modifying
unshare_userns()to allocate newuser_structfor the cred that is created for the first task creating theuser_nsand free it inexit_creds().
Quoted 7 lines
I do not like the idea of having
user_structsbe per user namespace, and deliberately made the code not work that way.Quoted 1 line
Can you please comment on that?
I have been pondering having some recursive resources limits that are per user namespace and if all you are worried about are process counts that might work. I don't honestly know what makes sense at the moment.
It seams to me that the only limit(from RLIMIT) that are generally a problem for the namespaces is number of processes and pending signals.
This is why I proposed the above modification. However I'm not sure if the places I have chosen are right and also I'm not really convinced that having per-namespace user_struct is the right approach for the process counter.
Quoted 3 lines
Eric
Marian
- --
Marian Marinov
Founder & CEO of 1H Ltd.
Jabber/GTalk: [email protected]
ICQ: 7556201
Mobile: +359 886 660 270
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iEYEARECAAYFAlOHDqoACgkQ4mt9JeIbjJRLPACZARH6agr856HeoB3Ub+e6U1PI ICgAoLbQTRM2SqcYOLep7WPIeuoiw4aB
=/Ii4
-----END PGP SIGNATURE-----
Quoted 36 lines · Marian Marinov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 10 lines · Eric W. Biederman
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
And I understand that very well ;)
Quoted 10 lines
Quoted 10 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
Quoted 8 lines
Quoted 2 lines
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet as much as possible. That has been true
ever since NFS was invented and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.
Quoted 5 lines
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
Quoted 41 lines · Serge Hallyn
Quoted 40 lines · Marian Marinov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 10 lines · Eric W. Biederman
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
And I understand that very well ;)
Quoted 10 lines
Quoted 10 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
Quoted 8 lines
Quoted 2 lines
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet as much as possible. That has been true
ever since NFS was invented and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
Quoted 7 lines
Quoted 1 line
We do not share filesystems between containers, we offer them block devices.
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Thanks,
Pavel
Quoted 44 lines · Pavel Emelyanov
Quoted 43 lines · Serge Hallyn
Quoted 41 lines · Marian Marinov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 10 lines · Eric W. Biederman
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
And I understand that very well ;)
Quoted 10 lines
Quoted 10 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
Quoted 8 lines
Quoted 2 lines
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet as much as possible. That has been true
ever since NFS was invented and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Quoted 8 lines
Quoted 5 lines
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
Quoted 51 lines · Serge Hallyn
Quoted 50 lines · Pavel Emelyanov
Quoted 48 lines · Serge Hallyn
Quoted 36 lines · Marian Marinov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 10 lines · Eric W. Biederman
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
And I understand that very well ;)
Quoted 10 lines
Quoted 10 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
Quoted 8 lines
Quoted 2 lines
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet as much as possible. That has been true
ever since NFS was invented and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.Quoted 5 lines
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
Quoted 6 lines
Quoted 2 lines
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
I was thinking about "lightweight mapping" which is simple shifting. Since we're trying to make this co-work with user-ns mappings, simple uid/gid shift should be enough. Please, correct me if I'm wrong.
If I'm not, then it looks to be enough to have two per-sb or per-mnt values for uid and gid shift. Per-mnt for now looks more promising, since container's FS may be just a bind-mount from shared disk.
Quoted 6 lines
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
User-space crossing? From my point of view it would be enough if we just turn uid/gid read from disk (well, from whenever FS gets them) into uids, that would match the user-ns's ones, this sould cover the VFS layer and related syscalls only, which is, IIRC stat-s family and chown.
Ouch, and the whole quota engine :\
Thanks,
Pavel
Quoted 59 lines · Pavel Emelyanov
Quoted 51 lines · Serge Hallyn
Quoted 50 lines · Pavel Emelyanov
Quoted 48 lines · Serge Hallyn
Quoted 36 lines · Marian Marinov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 10 lines · Eric W. Biederman
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
And I understand that very well ;)
Quoted 10 lines
Quoted 10 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
Quoted 8 lines
Quoted 2 lines
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet as much as possible. That has been true
ever since NFS was invented and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.Quoted 5 lines
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
Quoted 6 lines
Quoted 2 lines
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
Quoted 7 lines
I was thinking about "lightweight mapping" which is simple shifting. Since we're trying to make this co-work with user-ns mappings, simple uid/gid shift should be enough. Please, correct me if I'm wrong.
If I'm not, then it looks to be enough to have two per-sb or per-mnt values for uid and gid shift. Per-mnt for now looks more promising, since container's FS may be just a bind-mount from shared disk.
per-sb would work. per-mnt would as you say be nicer, but I don't see how it can be done since parts of the vfs which get inodes but no mnt information would not be able to figure out the shifts.
Quoted 20 lines
Quoted 6 lines
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
User-space crossing? From my point of view it would be enough if we just turn uid/gid read from disk (well, from whenever FS gets them) into uids, that would match the user-ns's ones, this sould cover the VFS layer and related syscalls only, which is, IIRC stat-s family and chown.
Ouch, and the whole quota engine :\
Thanks,
Pavel
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
Quoted 74 lines · Pavel Emelyanov
Quoted 51 lines · Serge Hallyn
Quoted 50 lines · Pavel Emelyanov
Quoted 48 lines · Serge Hallyn
Quoted 36 lines · Marian Marinov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 10 lines · Eric W. Biederman
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
That is deliberate.
And I understand that very well ;)
Quoted 10 lines
Quoted 10 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
We are provisioning the containers from a template. The template has a lot of files 500k and more. And chowning these causes a lot of I/O and also slows down provisioning considerably.
Quoted 8 lines
Quoted 2 lines
The other problem is that when we migrate one container from one host machine to another the IDs may be already in use on the new machine and we need to chown all the files again.
You should have the same uid allocations for all machines in your fleet as much as possible. That has been true
ever since NFS was invented and is not new here. You can avoid the cost of chowning if you untar your files inside of your user namespace. You can have different maps per machine if you are crazy enough to do that. You can even have shared uids that you use to share files between containers as long as none of those files is setuid. And map those shared files to some kind of nobody user in your user namespace.Quoted 5 lines
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
Quoted 6 lines
Quoted 2 lines
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
I was thinking about "lightweight mapping" which is simple shifting. Since we're trying to make this co-work with user-ns mappings, simple uid/gid shift should be enough. Please, correct me if I'm wrong.
If I'm not, then it looks to be enough to have two per-sb or per-mnt values for uid and gid shift. Per-mnt for now looks more promising, since container's FS may be just a bind-mount from shared disk.
Quoted 6 lines
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
Quoted 6 lines
User-space crossing? From my point of view it would be enough if we just turn uid/gid read from disk (well, from whenever FS gets them) into uids, that would match the user-ns's ones, this sould cover the VFS layer and related syscalls only, which is, IIRC stat-s family and chown.
Ouch, and the whole quota engine :\
And posix acls.
But all of this is 90% done already. I think today we just have conversions to the initial user namespace. We just need a few tweaks to allow it and a per superblock user namespace setting.
Eric
Quoted 22 lines · Serge Hallyn
Quoted 8 lines · Pavel Emelyanov
Quoted 7 lines · Serge Hallyn
Quoted 5 lines · Marian Marinov
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Quoted 8 lines
Quoted 5 lines
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
Quoted 6 lines
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
There is a simple solution.
We pick the filesystems we choose to support.
We add privileged mounting in a user namespace.
We create the user and mount namespace.
Global root goes into the target mount namespace with setns and performs the mounts.
90% of that work is already done.
As long as we don't plan to support XFS (as it XFS likes to expose it's implementation details to userspace) it should be quite straight forward.
The permission check change would probably only need to be:
@@ -2180,6 +2245,10 @@ static int do_new_mount(struct path *path, const char *fstype, int flags,
return -ENODEV;
if (user_ns != &init_user_ns) {+ if (!(type->fs_flags & FS_UNPRIV_MOUNT) && !capable(CAP_SYS_ADMIN)) {
+ put_filesystem(type);
+ return -EPERM;
+ }
if (!(type->fs_flags & FS_USERNS_MOUNT)) {
put_filesystem(type);
return -EPERM;There are also a few funnies with capturing the user namespace of the filesystem when we perform the mount (in the superblock?), and not allowing a mount of that same filesystem in a different user namespace.
But as long as the kuid conversions don't measurably slow down the filesystem when mounted in the initial mount and user namespaces I don't see how this would be a problem for anyone, and is very little code.
Eric
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 47 lines · Eric W. Biederman
Quoted 17 lines · Serge Hallyn
Quoted 8 lines · Pavel Emelyanov
Quoted 7 lines · Serge Hallyn
Quoted 5 lines · Marian Marinov
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Quoted 7 lines
Quoted 4 lines
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Quoted 7 lines
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
There is a simple solution.
We pick the filesystems we choose to support. We add privileged mounting in a user namespace. We create the user and mount namespace. Global root goes into the target mount namespace with setns and performs the mounts.
90% of that work is already done.
As long as we don't plan to support XFS (as it XFS likes to expose it's implementation details to userspace) it should be quite straight forward.
The permission check change would probably only need to be:
(quoted diff) +0
-0@@ -2180,6 +2245,10 @@ static int do_new_mount(struct path *path, const char *fstype, int flags, return -ENODEV; if (user_ns != &init_user_ns) { + if (!(type->fs_flags & FS_UNPRIV_MOUNT) && !capable(CAP_SYS_ADMIN)){ +
put_filesystem(type); + return-EPERM; + } if
(!(type->fs_flags &FS_USERNS_MOUNT)) {put_filesystem(type); return-EPERM;
Quoted 6 lines
There are also a few funnies with capturing the user namespace of the filesystem when we perform the mount (in the superblock?), and not allowing a mount of that same filesystem in a different user namespace.
But as long as the kuid conversions don't measurably slow down the filesystem when mounted in the initial mount and user namespaces I don't see how this would be a problem for anyone, and is very little code.
This may solve one of the problems, but it does not solve the issue with UID/GID maps that overlap in different user namespaces.
In our cases, this means breaking container migration mechanisms.
Will this at all be addressed or I'm the only one here that has this sort of requirement?
Marian
Quoted 3 lines
Eric
- --
Marian Marinov
Founder & CEO of 1H Ltd.
Jabber/GTalk: hackman-/[email protected]
ICQ: 7556201
Mobile: +359 886 660 270
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iEYEARECAAYFAlOOQIgACgkQ4mt9JeIbjJRUWQCgsp/dN0WBy9iLJmsjO8KB+Bin HiIAoIkm8TlcJr4UnbJOAHoYgPVHhg4P
=B9xA
-----END PGP SIGNATURE-----
Quoted 62 lines · Marian Marinov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 47 lines · Eric W. Biederman
Quoted 17 lines · Serge Hallyn
Quoted 8 lines · Pavel Emelyanov
Quoted 7 lines · Serge Hallyn
Quoted 5 lines · Marian Marinov
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Quoted 7 lines
Quoted 4 lines
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Quoted 7 lines
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
There is a simple solution.
We pick the filesystems we choose to support. We add privileged mounting in a user namespace. We create the user and mount namespace. Global root goes into the target mount namespace with setns and performs the mounts.
90% of that work is already done.
As long as we don't plan to support XFS (as it XFS likes to expose it's implementation details to userspace) it should be quite straight forward.
The permission check change would probably only need to be:
(quoted diff) +0
-0@@ -2180,6 +2245,10 @@ static int do_new_mount(struct path *path, const char *fstype, int flags, return -ENODEV; if (user_ns != &init_user_ns) { + if (!(type->fs_flags & FS_UNPRIV_MOUNT) && !capable(CAP_SYS_ADMIN)){ +
put_filesystem(type); + return-EPERM; + } if
(!(type->fs_flags &FS_USERNS_MOUNT)) {put_filesystem(type); return-EPERM;Quoted 6 lines
There are also a few funnies with capturing the user namespace of the filesystem when we perform the mount (in the superblock?), and not allowing a mount of that same filesystem in a different user namespace.
But as long as the kuid conversions don't measurably slow down the filesystem when mounted in the initial mount and user namespaces I don't see how this would be a problem for anyone, and is very little code.
This may solve one of the problems, but it does not solve the issue with UID/GID maps that overlap in different user namespaces.
In our cases, this means breaking container migration mechanisms.
Quoted 1 line
Will this at all be addressed or I'm the only one here that has this sort of requirement?
You're not. The openstack scenario has the same problem. So we have a single base rootfs in a qcow2 or raw file which we want to mount into multiple containers, each of which has a distinct set of uid mappings.
We'd like some way to identify uid mappings at mount time, without having to walk the whole rootfs to chown every file.
(Of course safety would demand that the shared qcow2 use a set of high subuids, NOT host uids - i.e. if we end up allowing a container to own files owned by 0 on the host - even in a usually unmapped qcow2 - there's danger we'd rather avoid, see again Andy's suggestions of accidentidally auto-mounted filesystem images which happen to share a UUID with host's / or /etc. So we'd want to map uids 100000-106536 in the qcow2 to uids 0-65536 in the container, which in turn map to uids 200000-206536 on the host)
-serge
Quoted 39 lines · Eric W. Biederman
Quoted 22 lines · Serge Hallyn
Quoted 8 lines · Pavel Emelyanov
Quoted 7 lines · Serge Hallyn
Quoted 5 lines · Marian Marinov
We are not using NFS. We are using a shared block storage that offers us snapshots. So provisioning new containers is extremely cheep and fast. Comparing that with untar is comparing a race car with Smart. Yes it can be done and no, I do not believe we should go backwards.
We do not share filesystems between containers, we offer them block devices.
Quoted 8 lines
Quoted 5 lines
Yes, this is a real nuisance for openstack style deployments.
One nice solution to this imo would be a very thin stackable filesystem which does uid shifting, or, better yet, a non-stackable way of shifting uids at mount.
I vote for non-stackable way too. Maybe on generic VFS level so that filesystems don't bother with it. From what I've seen, even simple stacking is quite a challenge.
Do you have any ideas for how to go about it? It seems like we'd have to have separate inodes per mapping for each file, which is why of course stacking seems "natural" here.
Quoted 6 lines
Trying to catch the uid/gid at every kernel-userspace crossing seems like a design regression from the current userns approach. I suppose we could continue in the kuid theme and introduce a iiud/igid for the in-kernel inode uid/gid owners. Then allow a user privileged in some ns to create a new mount associated with a different mapping for any ids over which he is privileged.
There is a simple solution.
We pick the filesystems we choose to support.
We add privileged mounting in a user namespace.
We create the user and mount namespace.
Global root goes into the target mount namespace with setns and performs the mounts.
Quoted 5 lines
90% of that work is already done.
As long as we don't plan to support XFS (as it XFS likes to expose it's implementation details to userspace) it should be quite straight forward.
Any implementation which doesn't support XFS is unviable from a distro point of view. The whole reason we're fighting to get USER_NS enabled in distros goes back to lack of XFS support (they basically refused to turn it on until it wasn't a choice between XFS and USER_NS). If we put them in a position where they choose a namespace feature or XFS, they'll choose XFS.
XFS developers aren't unreasonable ... they'll help if we ask. I mean it was them who eventually helped us get USER_NS turned on in the first place.
James
Quoted 8 lines · James Bottomley
Quoted 6 lines · Eric W. Biederman
90% of that work is already done.
As long as we don't plan to support XFS (as it XFS likes to expose it's implementation details to userspace) it should be quite straight forward.
Quoted 6 lines
Any implementation which doesn't support XFS is unviable from a distro point of view. The whole reason we're fighting to get
USER_NSenabled in distros goes back to lack of XFS support (they basically refused to turn it on until it wasn't a choice between XFS andUSER_NS). If we put them in a position where they choose a namespace feature or XFS, they'll choose XFS.
This isn't the same dicotomy. This is a simple case of not being able to use XFS mounted inside of a user namespace. Which does not cause any regression from the current use cases. The previous case was that XFS would not build at all.
There were valid technical reasons but part of the reason the XFS conversion took so long was my social engineering the distro's to not enable the latest bling until there was a chance for the initial crop of bugs to be fixed.
Quoted 3 lines
XFS developers aren't unreasonable ... they'll help if we ask. I mean it was them who eventually helped us get
USER_NSturned on in the first place.
Fair enough. But XFS is not the place to start.
For most filesystems the only really hard part is finding the handful of places where we actually need some form of error handling when on disk uids don't map to in core kuids. Which ultimately should wind up with maybe a 20 line patch for most filesystems.
For XFS there are two large obstacles to overcome.
- XFS journal replay does not work when the XFS filesystem is moved from a host with one combination of wordsize and endianness to a host with a different combination of wordsize and edianness. This makes XFS a bad choice of a filesystem to move between hosts in a sparse file. Every other filesystem in the kernel handles this better.
- The XFS code base has a large the largest number of any ioctls of any filesystem in the linux kernel. This increases the amount of code that has to be converted. Combine that with the fact that the XFS developers chose to convert from kuids and kgids at the VFS<->FS layer instead of at the FS<->disk layer it becomes quite easy to miss changing code in an ioctl or a quota check by accident. Which all adds up to the fact that converting XFS to be mountable with a non 1-1 mapping of filesystem uids and system kuids is going to be a lot more than a simple 20 line patch.
All of that said what becomes attractive about this approach is that it gets us to the point where people can ask questions about mounting normal filesystems unprivileged and the entire reason it won't be allowed are (no block devices to mount from) and concern that the filesystem error handling code is not sufficient to ward off evil users that create bad filesystem images.
Eric
Quoted 4 lines · Marian Marinov
Hello,
I have the following proposition.
Quoted 8 lines
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
If this matters, we also suffer from the same problem here. So we support any implementation that would address it.
Cheers,
Alin.
_______________________________________________
lxc-devel mailing list
[email protected]
http://lists.linuxcontainers.org/listinfo/lxc-devel
Quoted 7 lines · Alin Dobre
Quoted 6 lines · Marian Marinov
Hello,
I have the following proposition.
Number of currently running processes is accounted at the root user namespace. The problem I'm facing is that multiple containers in different user namespaces share the process counters.
Most ppl here probably are aware of this, but the previous, never-completed user namespace implementation provided this and only this. We (mostly Eric and I) spent years looking for clean ways to make that implementation, which had some advantages (including this one), complete. We did have a few POCs which worked but were unsatisfying. The two things which were never convincing were (a) conversion of all uid checks to be namespace-safe, and (b) storing namespace identifiers on disk. (As I say we did have solutions to these, but not satisfying ones). These are the two things which the new implementation address *beautifully*.
Quoted 8 lines
Quoted 5 lines
So if containerX runs 100 with UID 99, containerY should have NPROC limit of above 100 in order to execute any processes with ist own UID 99.
I know that some of you will tell me that I should not provision all of my containers with the same UID/GID maps, but this brings another problem.
If this matters, we also suffer from the same problem here. So we support any implementation that would address it.
ISTM the only reasonable answer here (at least for now) is to make it more convenient to isolate uid ranges, by providing a way to shift uids at mount time as has been discussed a bit.
If we go down the route of talking about uid 99 in ns 1 vs uid 99 in ns 2, then people will also expect isolation at file access time, and we're back to all the disadvantages of the first userns implementation.
(If someone proves me wrong by suggesting a clean solution, then awesome)
-serge
_______________________________________________
lxc-devel mailing list
[email protected]
http://lists.linuxcontainers.org/listinfo/lxc-devel