shiftfs: uid/gid shifting filesystem

fs/shiftfs.c new file +26 -0

+static struct dentry *shiftfs_lookup(struct inode *dir, struct dentry *dentry,
+				     unsigned int flags)
+{
+	struct dentry *real = dir->i_private, *new;
+	struct inode *reali = real->d_inode, *newi;
+	const struct cred *oldcred, *newcred;
+
+	/* note: violation of usual fs rules here: dentries are never
+	 * added with d_add.  This is because we want no dentry cache
+	 * for shiftfs.  All lookups proceed through the dentry cache
+	 * of the underlying filesystem, meaning we always see any
+	 * changes in the underlying */
+
+	inode_lock(reali);
+	oldcred = shiftfs_new_creds(&newcred, dentry->d_sb);
+	new = lookup_one_len(dentry->d_name.name, real, dentry->d_name.len);
+	shiftfs_old_creds(oldcred, &newcred);
+	inode_unlock(reali);
+
+	if (IS_ERR(new))
+		return new;
+
+	dentry->d_fsdata = new;
+
+	if (!new->d_inode)
+		return NULL;

Quoted 32 lines · James Bottomley

+static struct dentry *shiftfs_lookup(struct inode *dir, struct dentry *dentry,

fs/shiftfs.c new file +7 -0

+				     unsigned int flags)
+{
+	struct dentry *real = dir->i_private, *new;
+	struct inode *reali = real->d_inode, *newi;
+	const struct cred *oldcred, *newcred;
+
+	/* note: violation of usual fs rules here: dentries are

never

(quoted diff) +1 -0

+	 * added with d_add.  This is because we want no dentry

cache

(quoted diff) +1 -0

+	 * for shiftfs.  All lookups proceed through the dentry

cache

fs/shiftfs.c new file +17 -1

+	 * of the underlying filesystem, meaning we always see any
+	 * changes in the underlying */
+
+	inode_lock(reali);
+	oldcred = shiftfs_new_creds(&newcred, dentry->d_sb);
+	new = lookup_one_len(dentry->d_name.name, real, dentry
->d_name.len);
+	shiftfs_old_creds(oldcred, &newcred);
+	inode_unlock(reali);
+
+	if (IS_ERR(new))
+		return new;
+
+	dentry->d_fsdata = new;
+
+	if (!new->d_inode)
+		return NULL;
+

fs/shiftfs.c line 526

Quoted 10 lines

fs/shiftfs.c new file +9 -1

+	newi = shiftfs_new_inode(dentry->d_sb, new->d_inode
->i_mode, new);
+	if (!newi) {
+		dput(new);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	d_splice_alias(newi, dentry);
+
+	return NULL;

This is utter crap. First of all, d_splice_alias() *WILL* hash them, so you get all the coherency problems, in spades. Moreover, if you

did manage to avoid hashing, you would get something absolutely unusable.
* no mounting of anything on top of that thing
* performance shot to hell
and that's just for starters. I hadn't looked into the locking and semantics issues - those would really depends upon how you would achieve that "no dentry cache" thing; again, right now that's *not* what your code is doing.

PS: and then there's the choice of name. I mean, just try to say it over the phone several times in a row...

[This patch is updated for the new VFS APIs in 4.7-rc1; it's also been updated as Serge has been hammering on it]

My use case for this is that I run a lot of unprivileged architectural emulation containers on my system using user namespaces. Details here:

http://blog.hansenpartnership.com/unprivileged-build-containers/

They're mostly for building non-x86 stuff (like aarch64 and arm secure boot and mips images). For builds, I have all the environments in my home directory with downshifted uids; however, sometimes I need to use them to administer real images that run on systems, meaning the uids are the usual privileged ones not the downshifted ones. The only current choice I have is to start the emulation as root so the uid/gids match. The reason for this filesystem is to use my standard unprivileged containers to maintain these images. The way I do this is crack the image with a loop and then shift the uids before bringing up the container. I usually loop mount into /var/tmp/images/, so it's owned by real root there:

jarvis:~ # ls -l /var/tmp/images/mips|head -4
total 0
drwxr-xr-x 1 root root 8192 May 12 08:33 bin
drwxr-xr-x 1 root root 6 May 12 08:33 boot
drwxr-xr-x 1 root root 167 May 12 08:33 dev

And I usually run my build containers with a uid_map of

         0     100000       1000
      1000       1000          1
     65534     101000          1

(maps 0-999 shifted, then shifts nobody to 1000 and keeps my uid [1000] fixed so I can mount my home directory into the namespace) and something similar with gid_map. So I shift mount the mips image with

mount -t shiftfs -o uidmap=0:100000:1000,uidmap=65534:101000:1,gidmap=0:100000:100,gidmap=1 01:100101:899,gidmap=65533:101000:2 /var/tmp/images/mips /home/jejb/containers/mips

and I now see it as

jejb@jarvis:~> ls -l containers/mips|head -4
total 0
drwxr-xr-x 1 100000 100000 8192 May 12 08:33 bin/
drwxr-xr-x 1 100000 100000 6 May 12 08:33 boot/
drwxr-xr-x 1 100000 100000 167 May 12 08:33 dev/

Like my usual unprivileged build roots and I can now use an unprivileged container to enter and administer the image.

It seems like a lot of container systems need to do something similar when they try and provide unprivileged access to standard images.
Right at the moment, the security mechanism only allows root in the
host to use this, but it's not impossible to come up with a scheme for marking trees that can safely be shift mounted by unprivileged user namespaces.

James

---

 fs/Kconfig                 |   8 +
 fs/Makefile                |   1 +
 fs/shiftfs.c               | 877 +++++++++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/magic.h |   2 +
 4 files changed, 888 insertions(+)

[This patch is updated for the new VFS APIs in 4.7-rc1; it's also been updated as Serge has been hammering on it]

My use case for this is that I run a lot of unprivileged architectural emulation containers on my system using user namespaces. Details here:

http://blog.hansenpartnership.com/unprivileged-build-containers/

They're mostly for building non-x86 stuff (like aarch64 and arm secure boot and mips images). For builds, I have all the environments in my home directory with downshifted uids; however, sometimes I need to use them to administer real images that run on systems, meaning the uids are the usual privileged ones not the downshifted ones. The only current choice I have is to start the emulation as root so the uid/gids match. The reason for this filesystem is to use my standard unprivileged containers to maintain these images. The way I do this is crack the image with a loop and then shift the uids before bringing up the container. I usually loop mount into /var/tmp/images/, so it's owned by real root there:

jarvis:~ # ls -l /var/tmp/images/mips|head -4
total 0
drwxr-xr-x 1 root root 8192 May 12 08:33 bin
drwxr-xr-x 1 root root 6 May 12 08:33 boot
drwxr-xr-x 1 root root 167 May 12 08:33 dev

And I usually run my build containers with a uid_map of

         0     100000       1000
      1000       1000          1
     65534     101000          1

(maps 0-999 shifted, then shifts nobody to 1000 and keeps my uid [1000] fixed so I can mount my home directory into the namespace) and something similar with gid_map. So I shift mount the mips image with

mount -t shiftfs -o uidmap=0:100000:1000,uidmap=65534:101000:1,gidmap=0:100000:100,gidmap=1 01:100101:899,gidmap=65533:101000:2 /var/tmp/images/mips /home/jejb/containers/mips

and I now see it as

jejb@jarvis:~> ls -l containers/mips|head -4
total 0
drwxr-xr-x 1 100000 100000 8192 May 12 08:33 bin/
drwxr-xr-x 1 100000 100000 6 May 12 08:33 boot/
drwxr-xr-x 1 100000 100000 167 May 12 08:33 dev/

Like my usual unprivileged build roots and I can now use an unprivileged container to enter and administer the image.

It seems like a lot of container systems need to do something similar when they try and provide unprivileged access to standard images.
Right at the moment, the security mechanism only allows root in the
host to use this, but it's not impossible to come up with a scheme for marking trees that can safely be shift mounted by unprivileged user namespaces.

James

---

 fs/Kconfig                 |   8 +
 fs/Makefile                |   1 +
 fs/shiftfs.c               | 877 +++++++++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/magic.h |   2 +
 4 files changed, 888 insertions(+)

As I sent a reply in a ... wrong way, I do it again. my question was:
Why isn't it done at the vfs layer when you mount the fs in different userns, instead of using a separate filesystem for it?

I believe it could be useful to be able to mount all filesystems in
userns with autoshifted uids, although I do not know security implications for that usage.

W dniu 01.06.2016 o 02:29, James Bottomley pisze:

Quoted 86 lines

[This patch is updated for the new VFS APIs in 4.7-rc1; it's also been
updated as Serge has been hammering on it]

My use case for this is that I run a lot of unprivileged architectural
emulation containers on my system using user namespaces. Details here:

http://blog.hansenpartnership.com/unprivileged-build-containers/

They're mostly for building non-x86 stuff (like aarch64 and arm secure
boot and mips images). For builds, I have all the environments in my
home directory with downshifted uids; however, sometimes I need to use
them to administer real images that run on systems, meaning the uids
are the usual privileged ones not the downshifted ones. The only current choice I have is to start the emulation as root so the uid/gids
match. The reason for this filesystem is to use my standard unprivileged containers to maintain these images. The way I do this is
crack the image with a loop and then shift the uids before bringing up
the container. I usually loop mount into /var/tmp/images/, so it's owned by real root there:

jarvis:~ # ls -l /var/tmp/images/mips|head -4
total 0
drwxr-xr-x 1 root root 8192 May 12 08:33 bin
drwxr-xr-x 1 root root 6 May 12 08:33 boot
drwxr-xr-x 1 root root 167 May 12 08:33 dev

And I usually run my build containers with a uid_map of

         0     100000       1000
      1000       1000          1
     65534     101000          1

(maps 0-999 shifted, then shifts nobody to 1000 and keeps my uid [1000]
fixed so I can mount my home directory into the namespace) and something similar with gid_map. So I shift mount the mips image with

mount -t shiftfs -o uidmap=0:100000:1000,uidmap=65534:101000:1,gidmap=0:100000:100,gidm ap=1
01:100101:899,gidmap=65533:101000:2 /var/tmp/images/mips /home/jejb/containers/mips

and I now see it as

jejb@jarvis:~> ls -l containers/mips|head -4
total 0
drwxr-xr-x 1 100000 100000 8192 May 12 08:33 bin/
drwxr-xr-x 1 100000 100000 6 May 12 08:33 boot/
drwxr-xr-x 1 100000 100000 167 May 12 08:33 dev/

Like my usual unprivileged build roots and I can now use an unprivileged container to enter and administer the image.

It seems like a lot of container systems need to do something similar
when they try and provide unprivileged access to standard images.
Right at the moment, the security mechanism only allows root in
the
host to use this, but it's not impossible to come up with a scheme for
marking trees that can safely be shift mounted by unprivileged user namespaces.

James

---

 fs/Kconfig                 |   8 +
 fs/Makefile                |   1 +
 fs/shiftfs.c               | 877
+++++++++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/magic.h |   2 +
 4 files changed, 888 insertions(+)

Quoted 3 lines · Michał Zegan

As I sent a reply in a ... wrong way, I do it again. my question was:
Why isn't it done at the vfs layer when you mount the fs in different userns, instead of using a separate filesystem for it?

Well, that is what this patch does:

http://thread.gmane.org/gmane.linux.kernel/2214882

However, the reason it doesn't work for me is that I want to be able to unpack the image into a subdirectory (so I'm not dedicating a whole filesystem for this). This is primarily for a docker hack IBM is working on to allow each container instance to use a separate uid/gid range, so I need something that behaves much more like a bind mount.

Quoted 3 lines

I believe it could be useful to be able to mount all filesystems in
userns with autoshifted uids, although I do not know security implications for that usage.

As long as you don't need to subdivide the volume, it works nicely.
However, from a security point of view, that entire volume is now
effectively freely writeable by anyone who can set up a userns. If you follow the shiftfs route, you can break off writeable subdirectories for each namespace shift, but they can't cross over into writing subdirectories that belong to other user namespaces (assuming the uids are fully segregated).

Quoted 11 lines · James Bottomley

Quoted 5 lines · Michał Zegan

As I sent a reply in a ... wrong way, I do it again. my question was:
Why isn't it done at the vfs layer when you mount the fs in different
userns, instead of using a separate filesystem for it?

Well, that is what this patch does:

http://thread.gmane.org/gmane.linux.kernel/2214882

Quoted 8 lines

However, the reason it doesn't work for me is that I want to be able to
unpack the image into a subdirectory (so I'm not dedicating a whole filesystem for this). This is primarily for a docker hack IBM is working on to allow each container instance to use a separate uid/gid
range, so I need something that behaves much more like a bind mount.

I thought that you were using a loop device ?

that's precisely one of the main case that's solved with that
solution... mount the portable fs image into a loop device, set the shift which will be only active into that subdirectory...

Quoted 11 lines

Quoted 3 lines

I believe it could be useful to be able to mount all filesystems
in userns with autoshifted uids, although I do not know security implications for that usage.

As long as you don't need to subdivide the volume, it works nicely.
However, from a security point of view, that entire volume is now
effectively freely writeable by anyone who can set up a userns. If you follow the shiftfs route, you can break off writeable subdirectories for each namespace shift, but they can't cross over into writing subdirectories that belong to other user namespaces (assuming the uids are fully segregated).

As said in the other email, I'm not really sure about the use case at all... but I give you this quick test with:
https://gist.githubusercontent.com/tixxdz/6b84c2c3bd6cb987c82255602ec
70f23/raw/97c9ab76878f9d7415583c00b22ca0e4a948847b/userns_test.c

$ mkdir shifted-fedora-tree && sudo mount -t shiftfs 
-ouidmap=0:1000000:65536,gidmap=0:1000000:65536 ~/fedora-tree/
shifted-fedora-tree

[tixxdz@fedora-kvm bin]$ sudo ./userns-test -m -U -M "0 1000000 1" /bin/bash
uid=0(root) gid=65534(nfsnobody) groups=65534(nfsnobody) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [root@fedora-kvm bin]# cat /proc/self/uid_map

         0    1000000          1

[root@fedora-kvm bin]# echo "$(id -u)_not_a_sandboxed_app" >> shifted -fedora-tree/etc/fedora-release
[root@fedora-kvm bin]# exit
exit
[tixxdz@fedora-kvm bin]$ sudo ./userns-test -m -U -M "48 1000000 1" /bin/bash
[apache@fedora-kvm bin]$ id
uid=48(apache) gid=65534(nfsnobody) groups=65534(nfsnobody) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [apache@fedora-kvm bin]$ echo "$(id -u)_not_a_sandboxed_app" >> shifted-fedora-tree/etc/fedora-release
[apache@fedora-kvm bin]$ exit
exit
[tixxdz@fedora-kvm bin]$ sudo ./userns-test -m -U -M "70 1000000 1" /bin/bash
[avahi@fedora-kvm bin]$ id
uid=70(avahi) gid=65534(nfsnobody) groups=65534(nfsnobody) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [avahi@fedora-kvm bin]$ echo "$(id -u)_not_a_sandboxed_app" >> shifted-fedora-tree/etc/fedora-release
[avahi@fedora-kvm bin]$ exit
exit
[tixxdz@fedora-kvm bin]$ sudo ./userns-test -m -U -M "1000 1000000 1" /bin/bash
[tixxdz@fedora-kvm bin]$ id
uid=1000(tixxdz) gid=65534(nfsnobody) groups=65534(nfsnobody) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [tixxdz@fedora-kvm bin]$ echo "$(id -u)_not_a_sandboxed_app" >> shifted-fedora-tree/etc/fedora-release

[tixxdz@fedora-kvm bin]$ exit
exit
[tixxdz@fedora-kvm bin]$ cat ~/fedora-tree/etc/fedora-release Fedora release 23 (Twenty Three) 0_not_a_sandboxed_app
48_not_a_sandboxed_app
70_not_a_sandboxed_app
1000_not_a_sandboxed_app

Quoted 20 lines · Djalal Harouni

Quoted 19 lines · James Bottomley

Quoted 5 lines · Michał Zegan

As I sent a reply in a ... wrong way, I do it again. my question was:
Why isn't it done at the vfs layer when you mount the fs in different
userns, instead of using a separate filesystem for it?

Well, that is what this patch does:

http://thread.gmane.org/gmane.linux.kernel/2214882

However, the reason it doesn't work for me is that I want to be able to
unpack the image into a subdirectory (so I'm not dedicating a whole filesystem for this). This is primarily for a docker hack IBM is working on to allow each container instance to use a separate uid/gid
range, so I need something that behaves much more like a bind mount.

Quoted 1 line

I thought that you were using a loop device ?

No, for Architectural emulation containers, I use file roots, so they're subdirectories of my home directory. The interesting issues Serge discovered are on ext4, which I needed a loop device to reproduce (my home directory is xfs) if that's where the confusion arises?

Thinking about containers in general, a significant amount use bind mounted file roots because that's a nice use case that hypervisors can't match without clusterable filesystems. However, I do know some containers that are block image based, so whatever solution is chosen has to support both.

Quoted 22 lines

that's precisely one of the main case that's solved with that
solution... mount the portable fs image into a loop device, set the shift which will be only active into that subdirectory...

Quoted 11 lines

Quoted 3 lines

I believe it could be useful to be able to mount all filesystems
in userns with autoshifted uids, although I do not know security implications for that usage.

As long as you don't need to subdivide the volume, it works nicely.
However, from a security point of view, that entire volume is now
effectively freely writeable by anyone who can set up a userns. If you follow the shiftfs route, you can break off writeable subdirectories for each namespace shift, but they can't cross over into writing subdirectories that belong to other user namespaces (assuming the uids are fully segregated).

As said in the other email, I'm not really sure about the use case at all... but I give you this quick test with:
https://gist.githubusercontent.com/tixxdz/6b84c2c3bd6cb987c82255602ec
70f23/raw/97c9ab76878f9d7415583c00b22ca0e4a948847b/userns_test.c

Quoted 3 lines

$ mkdir shifted-fedora-tree && sudo mount -t shiftfs 
-ouidmap=0:1000000:65536,gidmap=0:1000000:65536 ~/fedora-tree/
shifted-fedora-tree

This is basically what I do for my container roots. However, after that I tend to set them up with scripts. I've attached my latest build -container script at the bottom. As you can see from my script, all my build containers are in /home/jejb/containers.