Building a BSD-jail clone out of namespaces
Prompted by the new userns support merged in the 3.8/3.9 kernels, I've been playing with namespaces and trying to understand how I could use them to build containers to replace some of my uses of qemu-kvm virtual machines.
I've successfully created a fakeroot-type container running as an unprivileged user by unsharing everything including CLONE_NEWUSER, and can map a block of host UIDs for that environment by writing to /proc/PID/[ug]id_map from a helper process running as root.
However, what I'm hoping for in practice is to be able to create containers whose access to its filesystem subtree is untranslated, i.e. uid/gid N in the container maps to uid/gid N in a subdirectory of the filesystem, but which is still isolated from the rest of the host filesystem and can't do externally privileged things. This is pretty much what a BSD jail provides, for example.
Is this possible to achieve securely using the mechanisms now available? (I'm assuming that parent directory permissions prevent unprivileged host users from getting at these container filesystems, exactly as is necessary to make BSD jails safe.)
As a first step, I naively tried running as root and unsharing everything with
unshare(CLONE_NEWIPC | CLONE_NEWNS | CLONE_NEWNET | CLONE_NEWPID
| CLONE_NEWUTS | CLONE_NEWUSER);
before execing a shell[1]. From another root process in the host namespace, I then wrote a pass-through mapping 0 0 4294967295 to /proc/PID/[ug]id_map.
The result initially looks plausible, with the PID namespace preventing signals being sent from one container to another, despite those processes sharing the same user ID in the top-level user namespace.
However, unfortunately I still have too many privileges with respect to the host. Whilst (for example) I can't mknod, I can mount a sysfs or procfs and apparently write to them with host root privileges to reconfigure the host kernel. I suspect there will be other things I haven't secured by this recipe too.
I also tried tightening things up by dropping capabilities from my root user and preventing capability grant on exec by setting and locking SECBIT_NOROOT on before starting the container. However, I'm not sure this really makes any difference---does CLONE_NEWUSER drop all capabilities with respect to the parent namespace?
[1] In this description, I'm ignoring the part where I lock into a new root filesystem, but presumably the way to do this is by pivot_root into a bind mount?
Best wishes,
Chris.
Quoted 22 lines · Chris Webb
Prompted by the new userns support merged in the 3.8/3.9 kernels, I've been playing with namespaces and trying to understand how I could use them to build containers to replace some of my uses of qemu-kvm virtual machines.
I've successfully created a fakeroot-type container running as an unprivileged user by unsharing everything including
CLONE_NEWUSER, and can map a block of host UIDs for that environment by writing to /proc/PID/[ug]id_map from a helper process running as root.However, what I'm hoping for in practice is to be able to create containers whose access to its filesystem subtree is untranslated, i.e. uid/gid N in the container maps to uid/gid N in a subdirectory of the filesystem, but which is still isolated from the rest of the host filesystem and can't do externally privileged things. This is pretty much what a BSD jail provides, for example.
Is this possible to achieve securely using the mechanisms now available? (I'm assuming that parent directory permissions prevent unprivileged host users from getting at these container filesystems, exactly as is necessary to make BSD jails safe.)
Quoted 8 lines
As a first step, I naively tried running as root and unsharing everything with
unshare(CLONE_NEWIPC |
CLONE_NEWNS|CLONE_NEWNET|CLONE_NEWPID| CLONE_NEWUTS | CLONE_NEWUSER);before execing a shell[1]. From another root process in the host namespace, I then wrote a pass-through mapping 0 0 4294967295 to /proc/PID/[ug]id_map.
That will work, but you really don't want to run with uid == 0 mapped to uid == 0. There are too many things in /proc and /sys and similar that grant access to uid == 0.
Quoted 9 lines
The result initially looks plausible, with the PID namespace preventing signals being sent from one container to another, despite those processes sharing the same user ID in the top-level user namespace.
However, unfortunately I still have too many privileges with respect to the host. Whilst (for example) I can't mknod, I can mount a sysfs or procfs and apparently write to them with host root privileges to reconfigure the host kernel. I suspect there will be other things I haven't secured by this recipe too.
Yes. I recommend having a dedicated range of uids for your container to prevent this kind of silliness. Or at the very least a separate mapping of uid == 0.
Quoted 5 lines
I also tried tightening things up by dropping capabilities from my root user and preventing capability grant on exec by setting and locking
SECBIT_NOROOTon before starting the container. However, I'm not sure this really makes any difference---doesCLONE_NEWUSERdrop all capabilities with respect to the parent namespace?
Yes. CLONE_NEWUSER drops all capabilities with respect to the parent namespace.
Quoted 3 lines
[1] In this description, I'm ignoring the part where I lock into a new root filesystem, but presumably the way to do this is by
pivot_rootinto a bind mount?
Yes pivot_root and bind mount work.
ERic
Quoted 3 lines · Eric W. Biederman
That will work, but you really don't want to run with uid == 0 mapped to uid == 0. There are too many things in /proc and /sys and similar that grant access to uid == 0.
Many thanks for the swift reply. If I map UID zero in the userns to a non-zero UID outside (say -1), is there any way to use the userns UIDs instead of host UIDs when accessing the container's root filesystem so I don't end up with strange file ownerships on disk? This would prevent me from using the same filesystem on physical hosts or in VMs.
I don't think there's any kernel mechanism that lets me apply a UID translation layer as part of a bind mount is there?
Cheers,
Chris.
Quoted 11 lines · Chris Webb
Quoted 3 lines · Eric W. Biederman
That will work, but you really don't want to run with uid == 0 mapped to uid == 0. There are too many things in /proc and /sys and similar that grant access to uid == 0.
Many thanks for the swift reply. If I map UID zero in the userns to a non-zero UID outside (say -1), is there any way to use the userns UIDs instead of host UIDs when accessing the container's root filesystem so I don't end up with strange file ownerships on disk? This would prevent me from using the same filesystem on physical hosts or in VMs.
Hmm. I guess it depends on how your VM is reading them. If it is
blocked based access to the filesystem you have a problem. If the VM
is effectively NFS mounting the filesystem you can do all kinds of things.
It is possible to just change the user namespace and setup your mapping, effectively running your VM in the user namespace, and that would allow the VM to see your mapped uids.
Quoted 2 lines
I don't think there's any kernel mechanism that lets me apply a UID translation layer as part of a bind mount is there?
No. In principle you could mount the filesystem inside of the user namespace but in practice no filesystems support that yet.
Eric
Quoted 8 lines · Eric W. Biederman
Hmm. I guess it depends on how your VM is reading them. If it is
blocked based access to the filesystem you have a problem. If the VM
is effectively NFS mounting the filesystem you can do all kinds of things.It is possible to just change the user namespace and setup your mapping, effectively running your VM in the user namespace, and that would allow the VM to see your mapped uids.
In some cases I was thinking of mounting a filesystem directly from a block device, but more often it would be directories in a local host filesystem. I use qemu's built in virtio 9p-over-pci to pass these in at present.
So in principle, that does mean I could store UIDs translated and wrap everything else I do at host level in a userns translation layer as well, but it's quite an intrusive thing to do and I imagine it would preclude lightweight throwaway containers where I share the host filesystem read-only into a container.
This is why I was quite keen to avoid mangled ownerships in the host filesystems at all, but from what you say, that goal sounds like this might be rather tricky to achieve.
Quoted 2 lines
There are too many things in /proc and /sys and similar that grant access to uid == 0.
Ah yes, I can see why this is a thorny one. Is it just the synthetic filesystems like /proc and /sys that are the problem, or are there loads of other places in the kernel that assume uid == 0 implies privilege? I.e. is it 'just' a matter of somehow securing access to procfs and sysfs, or a much wider issue?
Best wishes,
Chris.
Quoted 7 lines · Chris Webb
Quoted 5 lines · Eric W. Biederman
Hmm. I guess it depends on how your VM is reading them. If it is
blocked based access to the filesystem you have a problem. If the VM
is effectively NFS mounting the filesystem you can do all kinds of things.
Quoted 7 lines
Quoted 3 lines
It is possible to just change the user namespace and setup your mapping, effectively running your VM in the user namespace, and that would allow the VM to see your mapped uids.
In some cases I was thinking of mounting a filesystem directly from a block device, but more often it would be directories in a local host filesystem. I use qemu's built in virtio 9p-over-pci to pass these in at present.
Interesting. I hadn't seen that feature. That makes 9p much more interesting that I thought it was.
Quoted 5 lines
So in principle, that does mean I could store UIDs translated and wrap everything else I do at host level in a userns translation layer as well, but it's quite an intrusive thing to do and I imagine it would preclude lightweight throwaway containers where I share the host filesystem read-only into a container.
Not being able to share the host filesystem into a container is a downside of the current implementation. In principle you can have an overlay style filesystem that munges the uids and removes this limitation, but that doesn't currently exist.
Quoted 3 lines
This is why I was quite keen to avoid mangled ownerships in the host filesystems at all, but from what you say, that goal sounds like this might be rather tricky to achieve.
If you don't try to share the host root filesystem you can achieve the sharing pretty easily by just running qemu in a user namespace. So that qemu or whatever else serves the 9p protocol sees the filesystem with all of the uids and gids translated.
Quoted 8 lines
Quoted 2 lines
There are too many things in /proc and /sys and similar that grant access to uid == 0.
Ah yes, I can see why this is a thorny one. Is it just the synthetic filesystems like /proc and /sys that are the problem, or are there loads of other places in the kernel that assume uid == 0 implies privilege? I.e. is it 'just' a matter of somehow securing access to procfs and sysfs, or a much wider issue?
It is a wider issue. Capabilities cover most of places in the kernel where the kernel tests if you have privilege but there are other filesystems like devtmpsfs, and the occasional silly piece of kernel code that should be using capabilities but is not. Beyond the kernel there are files like /etc/shadow that only root is allowed to read.
Which all boils down to the fact that for the inconvience of using a separate range of uids a lot of other problems just go away.
Eric
Quoted 8 lines · Eric W. Biederman
It is a wider issue. Capabilities cover most of places in the kernel where the kernel tests if you have privilege but there are other filesystems like devtmpsfs, and the occasional silly piece of kernel code that should be using capabilities but is not. Beyond the kernel there are files like
/etc/shadowthat only root is allowed to read.Which all boils down to the fact that for the inconvience of using a separate range of uids a lot of other problems just go away.
Hi. Thanks for the clarifications here, which make a lot of sense.
Quoted 4 lines
Not being able to share the host filesystem into a container is a downside of the current implementation. In principle you can have an overlay style filesystem that munges the uids and removes this limitation, but that doesn't currently exist.
Yes, given the design means I can't just have an identity UID/GID mapping, this seems like the building block I'm missing to get namespace IDs instead of host IDs stored on disk. I imagine it might be fairly straightforward for me to take a simple 'example' stacked filesystem like wrapfs and teach it to map UIDs and GIDs. I'll have to take a look.
Cheers,
Chris.
Quoted 3 lines · Chris Webb
Prompted by the new userns support merged in the 3.8/3.9 kernels, I've been playing with namespaces and trying to understand how I could use them to build containers to replace some of my uses of qemu-kvm virtual machines.
I now have most things working as I'd want and am just polishing my userspace container tool before release to make sure it fits well with common conventions such as those mentioned at
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
and parses /etc/subuid and /etc/subgid files in the format you've defined them in your shadow patches. I was delighted by how it all nests nicely, provided I bind mount my /dev nodes from the level above rather than try to mknod them in the outer container.
I'd like to arrange for slightly different behaviour when the tool is run at the top-level 'host' user namespace, for example warning about attempts to map the dangerous UID 0.
Is there a canonical way to detect when I'm in the top-level user namespace? I can clearly try doing something which should be impossible for a non-top-level root user like opening /proc/kpageflags for reading or /proc/sys/ctrl-alt-del for writing, but I wondered if there was something more idiomatic as a test? (Some sort of 'get parent namespace' that might return null at top-level maybe?)
Cheers,
Chris.