Use cases for multiple uid mapping?
We had a discussion in the hackroom at LPC talking about use cases for a shiftfs style setup where there are different mappings of uids to disk.
In the discussion we had a couple of ideas of kernel developments we should look at that address some of these.
- Fix rlimits in user namespaces (This potentially allows multiple containers to run with the same userids simplifying the mapping problem).
- Look at extending kuid_t to 64bits and using the highbits to implement uids that are private to user namespaces and don't map out.
- Look at ways for allowing setgroups unprivileged.
Together this has the potential that the existing uid & gid mappings will be able to function the same as the proposed fusid mappings. Fingers crossed.
I had some problems with audio and a lot of people were talking
quickly. So I did not manage to capture everyone's use cases. And I
definitely was not able to see how everyone's use cases interacted with the changes we are looking at.
I know for certain I missed Serge's usecase (apologies).
Can people follow up to this and report their use cases?
There are some real challenges and I would like to see if we can solve them, while avoiding scary problems like changing uids on write.
Eric
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
Quoted 31 lines · Eric W. Biederman
We had a discussion in the hackroom at LPC talking about use cases for a shiftfs style setup where there are different mappings of uids to disk.
In the discussion we had a couple of ideas of kernel developments we should look at that address some of these.
- Fix rlimits in user namespaces (This potentially allows multiple containers to run with the same userids simplifying the mapping problem).
- Look at extending
kuid_tto 64bits and using the highbits to implement uids that are private to user namespaces and don't map out.- Look at ways for allowing setgroups unprivileged.
Together this has the potential that the existing uid & gid mappings will be able to function the same as the proposed fusid mappings. Fingers crossed.
I had some problems with audio and a lot of people were talking
quickly. So I did not manage to capture everyone's use cases. And I
definitely was not able to see how everyone's use cases interacted with the changes we are looking at.I know for certain I missed Serge's usecase (apologies).
Quoted 7 lines
Can people follow up to this and report their use cases?
There are some real challenges and I would like to see if we can solve them, while avoiding scary problems like changing uids on write.
Eric
Hi Eric,
It was great to have you and everyone else participate in that session. There was indeed a lot covered in there and a lot of use cases we'll have to keep in mind!
The main outcomes we captured I believe were:
- Fixing rlimits in user namespaces such that one namespace cannot
affect another
- Consider switching the kernel uid/git types to 64bit, allowing for
a hidden 32bit identifier and fully usable 32bit uid/gid range for the container
- Find a way to allow setgroups() in a user namespace while keeping
in mind the case of groups used for negative access control
- Add optional enforcement that overflow uid/gid as set through
sysctl cannot be used as regular uid/gid on the system
Christian is working on a comprehensive summary of the session based on the shared notes document we circulated at the beginning of the session which will try to separate the different topics and cover use cases and potential solutions for each as we discussed in the BoF.
We'll make sure that everyone who attended the BoF gets CCed on that, on top of the usual mailing-lists so that should any of the use raised use cases have been missed in the notes, those folks can raise them again so we don't end up missing anything :)
Stéphane
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
Quoted 48 lines · Stéphane Graber via Containers
Quoted 31 lines · Eric W. Biederman
We had a discussion in the hackroom at LPC talking about use cases for a shiftfs style setup where there are different mappings of uids to disk.
In the discussion we had a couple of ideas of kernel developments we should look at that address some of these.
- Fix rlimits in user namespaces (This potentially allows multiple containers to run with the same userids simplifying the mapping problem).
- Look at extending
kuid_tto 64bits and using the highbits to implement uids that are private to user namespaces and don't map out.- Look at ways for allowing setgroups unprivileged.
Together this has the potential that the existing uid & gid mappings will be able to function the same as the proposed fusid mappings. Fingers crossed.
I had some problems with audio and a lot of people were talking
quickly. So I did not manage to capture everyone's use cases. And I
definitely was not able to see how everyone's use cases interacted with the changes we are looking at.I know for certain I missed Serge's usecase (apologies).
Quoted 7 lines
Can people follow up to this and report their use cases?
There are some real challenges and I would like to see if we can solve them, while avoiding scary problems like changing uids on write.
Eric
Hi Eric,
It was great to have you and everyone else participate in that session. There was indeed a lot covered in there and a lot of use cases we'll have to keep in mind!
Quoted 6 lines
The main outcomes we captured I believe were:
- Fixing rlimits in user namespaces such that one namespace cannot
affect another
- Consider switching the kernel uid/git types to 64bit, allowing for
a hidden 32bit identifier and fully usable 32bit uid/gid range for the container
Seconding this. We run into a lot of problems with high UIDs. Some systems think that uid 2**31-1 (I think it's 2**31-1) is their version of 'nobody'. Being able to have 64-bit UIDs would solve this.
Also making sure this works with NFSv4 (both ID mapped, and non- ID mapped) is very important to us. It's fine if the UID / GID is just "passthrough", and we don't need different mappings between different filesystems in NFS (for now). We would like to be able to be able to run idmapper in the user / pid / net namespace of the container.
Quoted 15 lines
- Find a way to allow
setgroups()in a user namespace while keeping
in mind the case of groups used for negative access control
- Add optional enforcement that overflow uid/gid as set through
sysctl cannot be used as regular uid/gid on the systemChristian is working on a comprehensive summary of the session based on the shared notes document we circulated at the beginning of the session which will try to separate the different topics and cover use cases and potential solutions for each as we discussed in the BoF.
We'll make sure that everyone who attended the BoF gets CCed on that, on top of the usual mailing-lists so that should any of the use raised use cases have been missed in the notes, those folks can raise them again so we don't end up missing anything :)
Quoted 5 lines
Stéphane
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
On Fri, 2020-08-28 at 10:17 -0500, Eric W. Biederman wrote:
[...]
Quoted 5 lines
Can people follow up to this and report their use cases?
There are some real challenges and I would like to see if we can solve them, while avoiding scary problems like changing uids on write.
The main problem shiftfs has that fsuid doesn't is write at privileged id, so mainly safely writing to kuid 0.
shiftfs tried to solve this by exposing the shifted mount point safely, but maybe there's another way to do it.
James
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
Quoted 23 lines · Eric W. Biederman
We had a discussion in the hackroom at LPC talking about use cases for a shiftfs style setup where there are different mappings of uids to disk.
In the discussion we had a couple of ideas of kernel developments we should look at that address some of these.
- Fix rlimits in user namespaces (This potentially allows multiple containers to run with the same userids simplifying the mapping problem).
- Look at extending
kuid_tto 64bits and using the highbits to implement uids that are private to user namespaces and don't map out.- Look at ways for allowing setgroups unprivileged.
Together this has the potential that the existing uid & gid mappings will be able to function the same as the proposed fusid mappings. Fingers crossed.
Quoted 8 lines
I had some problems with audio and a lot of people were talking
quickly. So I did not manage to capture everyone's use cases. And I
definitely was not able to see how everyone's use cases interacted with the changes we are looking at.I know for certain I missed Serge's usecase (apologies).
Can people follow up to this and report their use cases?
Sorry - I'll do so later this week.
Quoted 4 lines
There are some real challenges and I would like to see if we can solve them, while avoiding scary problems like changing uids on write.
Quoted 10 lines
Eric
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
Quoted 29 lines · Serge E. Hallyn
Quoted 28 lines · Eric W. Biederman
We had a discussion in the hackroom at LPC talking about use cases for a shiftfs style setup where there are different mappings of uids to disk.
In the discussion we had a couple of ideas of kernel developments we should look at that address some of these.
- Fix rlimits in user namespaces (This potentially allows multiple containers to run with the same userids simplifying the mapping problem).
- Look at extending
kuid_tto 64bits and using the highbits to implement uids that are private to user namespaces and don't map out.- Look at ways for allowing setgroups unprivileged.
Together this has the potential that the existing uid & gid mappings will be able to function the same as the proposed fusid mappings. Fingers crossed.
I had some problems with audio and a lot of people were talking
quickly. So I did not manage to capture everyone's use cases. And I
definitely was not able to see how everyone's use cases interacted with the changes we are looking at.
Quoted 5 lines
Quoted 3 lines
I know for certain I missed Serge's usecase (apologies).
Can people follow up to this and report their use cases?
Sorry - I'll do so later this week.
Thank you.
I know we have the OCI use case of overlayfs and sharing storage between containers.
I know we have the lxc case of not wanting to be strangled by ulimits. So not using the same uid between containers even when it is logically the same users.
I know the brainstorming was going a lot of different directions and I piped up and said that we should probably focus on handling the stranger cases with fuse mounts, and the other capabilities we have now.
It really will be valuable to understand the other cases so we don't code ourselves into a corner that only works for the most vocal of the developers.
Eric
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
Quoted 34 lines · Eric W. Biederman
Quoted 31 lines · Serge E. Hallyn
Quoted 30 lines · Eric W. Biederman
We had a discussion in the hackroom at LPC talking about use cases for a shiftfs style setup where there are different mappings of uids to disk.
In the discussion we had a couple of ideas of kernel developments we should look at that address some of these.
- Fix rlimits in user namespaces (This potentially allows multiple containers to run with the same userids simplifying the mapping problem).
- Look at extending
kuid_tto 64bits and using the highbits to implement uids that are private to user namespaces and don't map out.- Look at ways for allowing setgroups unprivileged.
Together this has the potential that the existing uid & gid mappings will be able to function the same as the proposed fusid mappings. Fingers crossed.
I had some problems with audio and a lot of people were talking
quickly. So I did not manage to capture everyone's use cases. And I
definitely was not able to see how everyone's use cases interacted with the changes we are looking at.I know for certain I missed Serge's usecase (apologies).
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Can people follow up to this and report their use cases?
Sorry - I'll do so later this week.
Thank you.
I know we have the OCI use case of overlayfs and sharing storage between containers.
Is there a description of this use case, and ideas of possible solutions?
There was a use case that was brought to a recent Kubernetes sig-node meeting, but I am not sure it is the same one, so I'll describe this one.
We're working to enable user namespaces in Kubernetes with two possible setups:
1. Each Kubernetes pod has its own userns but with the same user id mapping (to make it easy to manage shared volumes)
2. Each Kubernetes pod has its own userns with non-overlapping user id mapping (providing additional isolation between pods)
Pods could be executed with a Kubernetes sidecar such as Envoy, meaning that if there are 'n' pods running on a node, there will be 'n' containers running the Envoy container image. In the second setup, each Envoy container will have a different user id mapping.
The Envoy container image will be snapshotted in as many copies, and each copy will be chown'ed appropriately for each non-overlapping user id mapping. Each copy will then be used as the lowerdir for the overlayfs rootfs of the container.
This results in a waste of disk space. Without user namespaces, we don't have this problem, since the same directory can be used as lowerdir for each container's overlayfs rootfs.
This is not specific to sidecars, but they exacerbate the problem.
I was not present in the Plumbers discussions, so I don't know if possible solutions were discussed. For example: could overlayfs get a mount option to specify an id mapping to apply on the lowerdir? In the case of containerd + runc, the mount option would probably need to be an id mapping rather than a reference to an existing user namespace, because the overlayfs mount is set up by containerd before runc is executed, so before the user namespace is unshared.
Alban
Quoted 8 lines
I know we have the lxc case of not wanting to be strangled by ulimits. So not using the same uid between containers even when it is logically the same users.
I know the brainstorming was going a lot of different directions and I piped up and said that we should probably focus on handling the stranger cases with fuse mounts, and the other capabilities we have now.
Quoted 5 lines
It really will be valuable to understand the other cases so we don't code ourselves into a corner that only works for the most vocal of the developers.
Eric
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
Quoted 66 lines · Alban Crequy
Quoted 34 lines · Eric W. Biederman
Quoted 31 lines · Serge E. Hallyn
Quoted 30 lines · Eric W. Biederman
We had a discussion in the hackroom at LPC talking about use cases for a shiftfs style setup where there are different mappings of uids to disk.
In the discussion we had a couple of ideas of kernel developments we should look at that address some of these.
- Fix rlimits in user namespaces (This potentially allows multiple containers to run with the same userids simplifying the mapping problem).
- Look at extending
kuid_tto 64bits and using the highbits to implement uids that are private to user namespaces and don't map out.- Look at ways for allowing setgroups unprivileged.
Together this has the potential that the existing uid & gid mappings will be able to function the same as the proposed fusid mappings. Fingers crossed.
I had some problems with audio and a lot of people were talking
quickly. So I did not manage to capture everyone's use cases. And I
definitely was not able to see how everyone's use cases interacted with the changes we are looking at.I know for certain I missed Serge's usecase (apologies).
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Can people follow up to this and report their use cases?
Sorry - I'll do so later this week.
Thank you.
I know we have the OCI use case of overlayfs and sharing storage between containers.
Is there a description of this use case, and ideas of possible solutions?
There was a use case that was brought to a recent Kubernetes sig-node meeting, but I am not sure it is the same one, so I'll describe this one.
We're working to enable user namespaces in Kubernetes with two possible setups:
1. Each Kubernetes pod has its own userns but with the same user id mapping (to make it easy to manage shared volumes)
2. Each Kubernetes pod has its own userns with non-overlapping user id mapping (providing additional isolation between pods)Pods could be executed with a Kubernetes sidecar such as Envoy, meaning that if there are 'n' pods running on a node, there will be 'n' containers running the Envoy container image. In the second setup, each Envoy container will have a different user id mapping.
The Envoy container image will be snapshotted in as many copies, and each copy will be chown'ed appropriately for each non-overlapping user id mapping. Each copy will then be used as the lowerdir for the overlayfs rootfs of the container.
Quoted 8 lines
This results in a waste of disk space. Without user namespaces, we don't have this problem, since the same directory can be used as lowerdir for each container's overlayfs rootfs.
This is not specific to sidecars, but they exacerbate the problem.
I was not present in the Plumbers discussions, so I don't know if possible solutions were discussed. For example: could overlayfs get a
Hey Alban,
Tycho, Aleksa, Serge and I are collaborating on a patchset that intends to cover all of these use-cases and that we hope to have ready for review early after the next merge window.
Christian
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers