Skip to contentSkip to conversation

mnt: add support for non-rootfs initramfs

PATCH In review Ignat Korchagin 5 participants 8 messages Mar 5, 2020 – Feb 25 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mnt: add support for non-rootfs initramfs

The main need for this is to support container runtimes on stateless Linux system (pivot_root system call from initramfs).

Normally, the task of initramfs is to mount and switch to a "real" root filesystem. However, on stateless systems (booting over the network) it is just convenient to have your "real" filesystem as initramfs from the start.

This, however, breaks different container runtimes, because they usually use pivot_root system call after creating their mount namespace. But pivot_root does not work from initramfs, because initramfs runs form rootfs, which is the root of the mount tree and can't be unmounted.

One can solve this problem from userspace, but it is much more cumbersome. We either have to create a multilayered archive for initramfs, where the outer layer creates a tmpfs filesystem and unpacks the inner layer, switches root and does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd kernel cmdline option, unpack initramfs to rootfs, run a script to create our target tmpfs root, unpack the same initramfs there, switch root to it and again properly cleanup the old root, thus unpacking the same archive twice and also wasting memory, because kernel stores compressed initramfs image indefinitely.

With this change we can ask the kernel (by specifying nonroot_initramfs kernel cmdline option) to create a "leaf" tmpfs mount for us and switch root to it before the initramfs handling code, so initramfs gets unpacked directly into the "leaf" tmpfs with rootfs being empty and no need to clean up anything.

---
 fs/namespace.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
fs/namespace.c +47 -0
index 85b5f7bea82e..a1ec862e8146 100644
@@ -3701,6 +3701,49 @@ static void __init init_mount_tree(void)
 	set_fs_root(current->fs, &root);
 }
 
+#if IS_ENABLED(CONFIG_TMPFS)
+static int __initdata nonroot_initramfs;
+
+static int __init nonroot_initramfs_param(char *str)
+{
+	if (*str)
+		return 0;
+	nonroot_initramfs = 1;
+	return 1;
+}
+__setup("nonroot_initramfs", nonroot_initramfs_param);
+
+static void __init init_nonroot_initramfs(void)
+{
+	int err;
+
+	if (!nonroot_initramfs)
+		return;
+
+	err = ksys_mkdir("/root", 0700);
+	if (err < 0)
+		goto out;
+
+	err = do_mount("tmpfs", "/root", "tmpfs", 0, NULL);
+	if (err)
+		goto out;
+
+	err = ksys_chdir("/root");
+	if (err)
+		goto out;
+
+	err = do_mount(".", "/", NULL, MS_MOVE, NULL);
+	if (err)
+		goto out;
+
+	err = ksys_chroot(".");
+	if (!err)
+		return;
+out:
+	printk(KERN_WARNING "Failed to create a non-root filesystem for initramfs\n");
+}
+#endif /* IS_ENABLED(CONFIG_TMPFS) */
+
 void __init mnt_init(void)
 {
 	int err;
@@ -3734,6 +3777,10 @@ void __init mnt_init(void)
 	shmem_init();
 	init_rootfs();
 	init_mount_tree();
+
+#if IS_ENABLED(CONFIG_TMPFS)
+	init_nonroot_initramfs();
+#endif
 }
 
 void put_mnt_ns(struct mnt_namespace *ns)
Signature · 1 line
2.20.1
maintainer
#1 lore raw
Quoted 21 lines · Ignat Korchagin

The main need for this is to support container runtimes on stateless Linux system (pivot_root system call from initramfs).

Normally, the task of initramfs is to mount and switch to a "real" root filesystem. However, on stateless systems (booting over the network) it is just convenient to have your "real" filesystem as initramfs from the start.

This, however, breaks different container runtimes, because they usually use pivot_root system call after creating their mount namespace. But pivot_root does not work from initramfs, because initramfs runs form rootfs, which is the root of the mount tree and can't be unmounted.

One can solve this problem from userspace, but it is much more cumbersome. We either have to create a multilayered archive for initramfs, where the outer layer creates a tmpfs filesystem and unpacks the inner layer, switches root and does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd kernel cmdline option, unpack initramfs to rootfs, run a script to create our target tmpfs root, unpack the same initramfs there, switch root to it and again properly cleanup the old root, thus unpacking the same archive twice and also wasting memory, because kernel stores compressed initramfs image indefinitely.

Quoted 4 lines

With this change we can ask the kernel (by specifying nonroot_initramfs kernel cmdline option) to create a "leaf" tmpfs mount for us and switch root to it before the initramfs handling code, so initramfs gets unpacked directly into the "leaf" tmpfs with rootfs being empty and no need to clean up anything.

IDGI. Why not simply this as the first thing from your userland:

	mount("/", "/", NULL, MS_BIND | MS_REC, NULL);
	chdir("/..");
	chroot(".");

3 syscalls and you should be all set...

#2 lore raw
Quoted 28 lines · Al Viro
Quoted 21 lines · Ignat Korchagin

The main need for this is to support container runtimes on stateless Linux system (pivot_root system call from initramfs).

Normally, the task of initramfs is to mount and switch to a "real" root filesystem. However, on stateless systems (booting over the network) it is just convenient to have your "real" filesystem as initramfs from the start.

This, however, breaks different container runtimes, because they usually use pivot_root system call after creating their mount namespace. But pivot_root does not work from initramfs, because initramfs runs form rootfs, which is the root of the mount tree and can't be unmounted.

One can solve this problem from userspace, but it is much more cumbersome. We either have to create a multilayered archive for initramfs, where the outer layer creates a tmpfs filesystem and unpacks the inner layer, switches root and does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd kernel cmdline option, unpack initramfs to rootfs, run a script to create our target tmpfs root, unpack the same initramfs there, switch root to it and again properly cleanup the old root, thus unpacking the same archive twice and also wasting memory, because kernel stores compressed initramfs image indefinitely.

Quoted 4 lines

With this change we can ask the kernel (by specifying nonroot_initramfs kernel cmdline option) to create a "leaf" tmpfs mount for us and switch root to it before the initramfs handling code, so initramfs gets unpacked directly into the "leaf" tmpfs with rootfs being empty and no need to clean up anything.

Quoted 5 lines

IDGI. Why not simply this as the first thing from your userland:

        mount("/", "/", NULL, MS_BIND | MS_REC, NULL);
        chdir("/..");
        chroot(".");

3 syscalls and you should be all set...

(sorry for duplicate - didn't press "reply all" the first time) Container people really prefer pivot_root over chroot due to some security concerns around chroot.
As far as my (probably limited) understanding goes, while the above approach will make it work,
it will have the same security implications as just using chroot: we trick the system to perform
pivot_root, however we don't get rid of the actual host root filesystem in the cloned namespace.

#3 lore raw
In reply to Ignat Korchagin
Quoted 8 lines · Ignat Korchagin

The main need for this is to support container runtimes on stateless Linux system (pivot_root system call from initramfs).

Normally, the task of initramfs is to mount and switch to a "real" root filesystem. However, on stateless systems (booting over the network) it is just convenient to have your "real" filesystem as initramfs from the start.

Quoted 5 lines

This, however, breaks different container runtimes, because they usually use pivot_root system call after creating their mount namespace. But pivot_root does not work from initramfs, because initramfs runs form rootfs, which is the root of the mount tree and can't be unmounted.

Can you say more about why this is a problem? We use pivot_root to pivot from the initramfs rootfs to the newly discovered and mounted real root ... the same mechanism should work for a container (mount namespace) running from initramfs ... why doesn't it?

The sequence usually looks like: create and enter a mount namespace, build a tmpfs for the container in some $root directory then do

cd $root
mkdir old-root
pivot_root . old-root
mount --
make-rprivate /old-root
umount -l /old-root
rmdir /old-root

Once that's done you're disconnected from the initramfs root. The sequence is really no accident because it's what the initramfs would have done to pivot to the new root anyway (that's where container people got it from).

James

#4 lore raw
Quoted 15 lines · James Bottomley
Quoted 8 lines · Ignat Korchagin

The main need for this is to support container runtimes on stateless Linux system (pivot_root system call from initramfs).

Normally, the task of initramfs is to mount and switch to a "real" root filesystem. However, on stateless systems (booting over the network) it is just convenient to have your "real" filesystem as initramfs from the start.

Quoted 5 lines

This, however, breaks different container runtimes, because they usually use pivot_root system call after creating their mount namespace. But pivot_root does not work from initramfs, because initramfs runs form rootfs, which is the root of the mount tree and can't be unmounted.

Quoted 4 lines

Can you say more about why this is a problem? We use pivot_root to pivot from the initramfs rootfs to the newly discovered and mounted real root ... the same mechanism should work for a container (mount namespace) running from initramfs ... why doesn't it?

Not sure how it interacts with mount namespaces, but we don't use pivot_root to go from rootfs to the real root. We use switch_root, which moves the new root onto the old / using mount with MS_MOVE and then chroot to it.

https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt

Quoted 21 lines

The sequence usually looks like: create and enter a mount namespace, build a tmpfs for the container in some $root directory then do

cd $root
mkdir old-root
pivot_root . old-root
mount --
make-rprivate /old-root
umount -l /old-root
rmdir /old-root

Once that's done you're disconnected from the initramfs root. The sequence is really no accident because it's what the initramfs would have done to pivot to the new root anyway (that's where container people got it from).

James

#5 lore raw
Quoted 22 lines · Arvind Sankar
Quoted 15 lines · James Bottomley
Quoted 8 lines · Ignat Korchagin

The main need for this is to support container runtimes on stateless Linux system (pivot_root system call from initramfs).

Normally, the task of initramfs is to mount and switch to a "real" root filesystem. However, on stateless systems (booting over the network) it is just convenient to have your "real" filesystem as initramfs from the start.

Quoted 5 lines

This, however, breaks different container runtimes, because they usually use pivot_root system call after creating their mount namespace. But pivot_root does not work from initramfs, because initramfs runs form rootfs, which is the root of the mount tree and can't be unmounted.

Quoted 4 lines

Can you say more about why this is a problem? We use pivot_root to pivot from the initramfs rootfs to the newly discovered and mounted real root ... the same mechanism should work for a container (mount namespace) running from initramfs ... why doesn't it?

Quoted 28 lines

Not sure how it interacts with mount namespaces, but we don't use pivot_root to go from rootfs to the real root. We use switch_root, which moves the new root onto the old / using mount with MS_MOVE and then chroot to it.

https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt

Quoted 21 lines

The sequence usually looks like: create and enter a mount namespace, build a tmpfs for the container in some $root directory then do

cd $root
mkdir old-root
pivot_root . old-root
mount --
make-rprivate /old-root
umount -l /old-root
rmdir /old-root

Once that's done you're disconnected from the initramfs root. The sequence is really no accident because it's what the initramfs would have done to pivot to the new root anyway (that's where container people got it from).

James

Yes, to add to Arvind's point the above sequence will only work for "old style" initrd (block ramdisk with some filesystem image on top), but will not work for the "new style" initramfs (just a disguised tmpfs). The sequence will fail on "pivot_root" with EINVAL (see pivot_root(2)). In fact this patch conceptually tries to have the same behaviour as with "old style" initrd. As currently, if you use initrd:
1. The kernel will create an empty "dummy" initramfs
2. Create a ramdisk
3. Unpack the FS image into the ramdisk
4. Mount the the disk
5. Do switch_root/move etc

So we have initial mount tree as: rootfs->some_initrd_fs (and pivot_root works here and you get empty rootfs by default)

With this option we have similar in the end: rootfs->tmpfs and rootfs is empty, because the kernel never unpacked anything there.

#6 lore raw
In reply to Ignat Korchagin

Following up on this: is there a way to move this forward somehow or we're stuck with complex userspace code to achieve the same, like
https://git.kernel.org/pub/scm/libs/klibc/klibc.git/tree/usr/kinit/run-init/runinitlib.c,
but for tmpfs?

Just FYI, here is an example of a fix for a security issue, which is caused by using chroot vs pivot_root in containers:
https://github.com/opencontainers/runc/commit/28a697cce3e4f905dca700eda81d681a30eef9cd

Alternatively, if the use-case is not generic enough, we could keep the patch to ourselves - just would appreciate some advice/potential concerns with this approach which we might have overlooked.

Thanks,
Ignat

Quoted 100 lines · Ignat Korchagin

The main need for this is to support container runtimes on stateless Linux system (pivot_root system call from initramfs).

Normally, the task of initramfs is to mount and switch to a "real" root filesystem. However, on stateless systems (booting over the network) it is just convenient to have your "real" filesystem as initramfs from the start.

This, however, breaks different container runtimes, because they usually use pivot_root system call after creating their mount namespace. But pivot_root does not work from initramfs, because initramfs runs form rootfs, which is the root of the mount tree and can't be unmounted.

One can solve this problem from userspace, but it is much more cumbersome. We either have to create a multilayered archive for initramfs, where the outer layer creates a tmpfs filesystem and unpacks the inner layer, switches root and does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd kernel cmdline option, unpack initramfs to rootfs, run a script to create our target tmpfs root, unpack the same initramfs there, switch root to it and again properly cleanup the old root, thus unpacking the same archive twice and also wasting memory, because kernel stores compressed initramfs image indefinitely.

With this change we can ask the kernel (by specifying nonroot_initramfs kernel cmdline option) to create a "leaf" tmpfs mount for us and switch root to it before the initramfs handling code, so initramfs gets unpacked directly into the "leaf" tmpfs with rootfs being empty and no need to clean up anything.

---
 fs/namespace.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
fs/namespace.c +47 -1
index 85b5f7bea82e..a1ec862e8146 100644
@@ -3701,6 +3701,49 @@ static void __init init_mount_tree(void)
        set_fs_root(current->fs, &root);
 }
 
+#if IS_ENABLED(CONFIG_TMPFS)
+static int __initdata nonroot_initramfs;
+
+static int __init nonroot_initramfs_param(char *str)
+{
+       if (*str)
+               return 0;
+       nonroot_initramfs = 1;
+       return 1;
+}
+__setup("nonroot_initramfs", nonroot_initramfs_param);
+
+static void __init init_nonroot_initramfs(void)
+{
+       int err;
+
+       if (!nonroot_initramfs)
+               return;
+
+       err = ksys_mkdir("/root", 0700);
+       if (err < 0)
+               goto out;
+
+       err = do_mount("tmpfs", "/root", "tmpfs", 0, NULL);
+       if (err)
+               goto out;
+
+       err = ksys_chdir("/root");
+       if (err)
+               goto out;
+
+       err = do_mount(".", "/", NULL, MS_MOVE, NULL);
+       if (err)
+               goto out;
+
+       err = ksys_chroot(".");
+       if (!err)
+               return;
+out:
+       printk(KERN_WARNING "Failed to create a non-root filesystem for initramfs\n");
+}
+#endif /* IS_ENABLED(CONFIG_TMPFS) */
+
 void __init mnt_init(void)
 {
        int err;
@@ -3734,6 +3777,10 @@ void __init mnt_init(void)
        shmem_init();
        init_rootfs();
        init_mount_tree();
+
+#if IS_ENABLED(CONFIG_TMPFS)
+       init_nonroot_initramfs();
+#endif
 }
 
 void put_mnt_ns(struct mnt_namespace *ns)
--

2.20.1

#7 lore raw
In reply to Ignat Korchagin

Ignat Korchagin <[email protected]>:

Quoted 4 lines

With this change we can ask the kernel (by specifying nonroot_initramfs kernel cmdline option) to create a "leaf" tmpfs mount for us and switch root to it before the initramfs handling code, so initramfs gets unpacked directly into the "leaf" tmpfs with rootfs being empty and no need to clean up anything.

This problem is solved by nullfs patches:

https://lore.kernel.org/all/[email protected]/

They are already in mainline.

Signature · 1 line
Askar Safin