Composefs: an opportunistically sharing verified image filesystem

Giuseppe Scrivano and I have recently been working on a new project we call composefs. This is the first time we propose this publically and we would like some feedback on it.

At its core, composefs is a way to construct and use read only images that are used similar to how you would use e.g. loop-back mounted squashfs images. On top of this composefs has two fundamental features. First it allows sharing of file data (both on disk and in page cache) between images, and secondly it has dm-verity like validation on read.

Let me first start with a minimal example of how this can be used, before going into the details:

Suppose we have this source for an image:

rootfs/

├── dir
│   └── another_a
├── file_a
└── file_b

We can then use this to generate an image file and a set of content-addressed backing files:

# mkcomposefs --digest-store=objects rootfs/ rootfs.img
# ls -l rootfs.img objects/*/*

(quoted diff) +0 -3

-rw-------. 1 root root   10 Nov 18 13:20 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
-rw-------. 1 root root   10 Nov 18 13:20 objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
-rw-r--r--. 1 root root 4228 Nov 18 13:20 rootfs.img

The rootfs.img file contains all information about directory and file metadata plus references to the backing files by name. We can now mount this and look at the result:

# mount -t composefs rootfs.img -o basedir=objects /mnt
# ls  /mnt/
dir  file_a  file_b
# cat /mnt/file_a
content_a

When reading this file the kernel is actually reading the backing file, in a fashion similar to overlayfs. Since the backing file is content-addressed, the objects directory can be shared for multiple images, and any files that happen to have the same content are shared. I refer to this as opportunistic sharing, as it is different than the more course-grained explicit sharing used by e.g. container base images.

The next step is the validation. Note how the object files have fs-verity enabled. In fact, they are named by their fs-verity digest:

# fsverity digest objects/*/*
sha256:02927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
sha256:cc3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f

The generated filesystm image may contain the expected digest for the backing files. When the backing file digest is incorrect, the open will fail, and if the open succeeds, any other on-disk file-changes will be detected by fs-verity:

# cat objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
content_a
# rm -f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# echo modified > objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# cat /mnt/file_a
WARNING: composefs backing file '3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f' unexpectedly had no fs-verity digest
cat: /mnt/file_a: Input/output error

This re-uses the existing fs-verity functionallity to protect against changes in file contents, while adding on top of it protection against changes in filesystem metadata and structure. I.e. protecting against replacing a fs-verity enabled file or modifying file permissions or xattrs.

To be fully verified we need another step: we use fs-verity on the image itself. Then we pass the expected digest on the mount command line (which will be verified at mount time):

# fsverity enable rootfs.img
# fsverity digest rootfs.img
sha256:da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 rootfs.img
# mount -t composefs rootfs.img -o basedir=objects,digest=da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 /mnt

So, given a trusted set of mount options (say unlocked from TPM), we have a fully verified filesystem tree mounted, with opportunistic finegrained sharing of identical files.

So, why do we want this? There are two initial users. First of all we want to use the opportunistic sharing for the podman container image baselayer. The idea is to use a composefs mount as the lower directory in an overlay mount, with the upper directory being the container work dir. This will allow automatical file-level disk and page-cache sharning between any two images, independent of details like the permissions and timestamps of the files.

Secondly we are interested in using the verification aspects of composefs in the ostree project. Ostree already supports a content-addressed object store, but it is currently referenced by hardlink farms. The object store and the trees that reference it are signed and verified at download time, but there is no runtime verification. If we replace the hardlink farm with a composefs image that points into the existing object store we can use the verification to implement runtime verification.

In fact, the tooling to create composefs images is 100% reproducible, so all we need is to add the composefs image fs-verity digest into the ostree commit. Then the image can be reconstructed from the ostree commit info, generating a file with the same fs-verity digest.

These are the usecases we're currently interested in, but there seems to be a breadth of other possible uses. For example, many systems use loopback mounts for images (like lxc or snap), and these could take advantage of the opportunistic sharing. We've also talked about using fuse to implement a local cache for the backing files. I.e. you would have the second basedir be a fuse filesystem. On lookup failure in the first basedir it downloads the file and saves it in the first basedir for later lookups. There are many interesting possibilities here.

The patch series contains some documentation on the file format and how to use the filesystem.

The userspace tools (and a standalone kernel module) is available here:
https://github.com/containers/composefs

Initial work on ostree integration is here:
https://github.com/ostreedev/ostree/pull/2640

Changes since v2:
- Simplified filesystem format to use fixed size inodes. This resulted in simpler (now < 2k lines) code as well as higher performance at the cost of slightly (~40%) larger images.
- We now use multi-page mappings from the page cache, which removes limits on sizes of xattrs and makes the dirent handling code simpler.
- Added more documentation about the on-disk file format.
- General cleanups based on review comments.

Quoted 130 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we call composefs. This is the first time we propose this publically and we would like some feedback on it.

At its core, composefs is a way to construct and use read only images that are used similar to how you would use e.g. loop-back mounted squashfs images. On top of this composefs has two fundamental features. First it allows sharing of file data (both on disk and in page cache) between images, and secondly it has dm-verity like validation on read.

Let me first start with a minimal example of how this can be used, before going into the details:

Suppose we have this source for an image:

rootfs/

├── dir
│   └── another_a
├── file_a
└── file_b

We can then use this to generate an image file and a set of content-addressed backing files:

# mkcomposefs --digest-store=objects rootfs/ rootfs.img
# ls -l rootfs.img objects/*/*

(quoted diff) +0 -3

-rw-------. 1 root root   10 Nov 18 13:20 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
-rw-------. 1 root root   10 Nov 18 13:20 objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
-rw-r--r--. 1 root root 4228 Nov 18 13:20 rootfs.img

The rootfs.img file contains all information about directory and file metadata plus references to the backing files by name. We can now mount this and look at the result:

# mount -t composefs rootfs.img -o basedir=objects /mnt
# ls  /mnt/
dir  file_a  file_b
# cat /mnt/file_a
content_a

When reading this file the kernel is actually reading the backing file, in a fashion similar to overlayfs. Since the backing file is content-addressed, the objects directory can be shared for multiple images, and any files that happen to have the same content are shared. I refer to this as opportunistic sharing, as it is different than the more course-grained explicit sharing used by e.g. container base images.

The next step is the validation. Note how the object files have fs-verity enabled. In fact, they are named by their fs-verity digest:

# fsverity digest objects/*/*
sha256:02927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
sha256:cc3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f

The generated filesystm image may contain the expected digest for the backing files. When the backing file digest is incorrect, the open will fail, and if the open succeeds, any other on-disk file-changes will be detected by fs-verity:

# cat objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
content_a
# rm -f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# echo modified > objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# cat /mnt/file_a
WARNING: composefs backing file '3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f' unexpectedly had no fs-verity digest
cat: /mnt/file_a: Input/output error

This re-uses the existing fs-verity functionallity to protect against changes in file contents, while adding on top of it protection against changes in filesystem metadata and structure. I.e. protecting against replacing a fs-verity enabled file or modifying file permissions or xattrs.

To be fully verified we need another step: we use fs-verity on the image itself. Then we pass the expected digest on the mount command line (which will be verified at mount time):

# fsverity enable rootfs.img
# fsverity digest rootfs.img
sha256:da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 rootfs.img
# mount -t composefs rootfs.img -o basedir=objects,digest=da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 /mnt

So, given a trusted set of mount options (say unlocked from TPM), we have a fully verified filesystem tree mounted, with opportunistic finegrained sharing of identical files.

So, why do we want this? There are two initial users. First of all we want to use the opportunistic sharing for the podman container image baselayer. The idea is to use a composefs mount as the lower directory in an overlay mount, with the upper directory being the container work dir. This will allow automatical file-level disk and page-cache sharning between any two images, independent of details like the permissions and timestamps of the files.

Secondly we are interested in using the verification aspects of composefs in the ostree project. Ostree already supports a content-addressed object store, but it is currently referenced by hardlink farms. The object store and the trees that reference it are signed and verified at download time, but there is no runtime verification. If we replace the hardlink farm with a composefs image that points into the existing object store we can use the verification to implement runtime verification.

In fact, the tooling to create composefs images is 100% reproducible, so all we need is to add the composefs image fs-verity digest into the ostree commit. Then the image can be reconstructed from the ostree commit info, generating a file with the same fs-verity digest.

These are the usecases we're currently interested in, but there seems to be a breadth of other possible uses. For example, many systems use loopback mounts for images (like lxc or snap), and these could take advantage of the opportunistic sharing. We've also talked about using fuse to implement a local cache for the backing files. I.e. you would have the second basedir be a fuse filesystem. On lookup failure in the first basedir it downloads the file and saves it in the first basedir for later lookups. There are many interesting possibilities here.

The patch series contains some documentation on the file format and how to use the filesystem.

The userspace tools (and a standalone kernel module) is available here:
https://github.com/containers/composefs

Initial work on ostree integration is here:
https://github.com/ostreedev/ostree/pull/2640

Quoted 9 lines

Changes since v2:
- Simplified filesystem format to use fixed size inodes. This resulted in simpler (now < 2k lines) code as well as higher performance at the cost of slightly (~40%) larger images.
- We now use multi-page mappings from the page cache, which removes limits on sizes of xattrs and makes the dirent handling code simpler.
- Added more documentation about the on-disk file format.
- General cleanups based on review comments.

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

Any current functionality gap in erofs and/or in overlayfs cannot be considered as a reason to maintain a new filesystem driver unless you come up with an explanation why closing that functionality gap is not possible or why the erofs+overlayfs alternative would be inferior to maintaining a new filesystem driver.

From the conversations so far, it does not seem like Gao thinks that the functionality gap in erofs cannot be closed and I don't see why the functionality gap in overlayfs cannot be closed.

Are we missing something?

Thanks,
Amir.

Hi Amir,

Quoted 1 line · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <[email protected]> wrote:

Quoted 8 lines

Quoted 1 line

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to what composefs does.

One big difference is that overlay still requires at least a syscall for each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.

I also don't see how overlay would solve the verified image problem. We would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.

There are at the moment two ways to handle container images, both somehow guided by the available file systems in the kernel.

- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.

What we are proposing with composefs is a way to have "dm-verity" style validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

The composefs manifest blob could be generated remotely and signed. A client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Regards,
Giuseppe

Quoted 14 lines

Any current functionality gap in erofs and/or in overlayfs cannot be considered as a reason to maintain a new filesystem driver unless you come up with an explanation why closing that functionality gap is not possible or why the erofs+overlayfs alternative would be inferior to maintaining a new filesystem driver.

From the conversations so far, it does not seem like Gao thinks
that the functionality gap in erofs cannot be closed and I don't see why the functionality gap in overlayfs cannot be closed.

Are we missing something?

Thanks,
Amir.

Quoted 4 lines · Giuseppe Scrivano

Hi Amir,

Quoted 1 line · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <[email protected]> wrote:

...

Quoted 37 lines

Quoted 8 lines

Quoted 1 line

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to
what composefs does.
One big difference is that overlay still requires at least a syscall for
each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.
I also don't see how overlay would solve the verified image problem. We
would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.
There are at the moment two ways to handle container images, both somehow
guided by the available file systems in the kernel.
- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

Quoted 8 lines

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.
What we are proposing with composefs is a way to have "dm-verity" style
validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

That is not a new idea honestly, including chain of trust. Even laterly out-of-tree incremental fs using fs-verity for this as well, except that it's in a real self-contained way.

Quoted 5 lines

The composefs manifest blob could be generated remotely and signed. A
client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Back to the topic, after thinking something I have to make a compliment for reference.

First, EROFS had the same internal dissussion and decision at that time almost _two years ago_ (June 2021), it means:

a) Some internal people really suggested EROFS could develop an entire new file-based in-kernel local cache subsystem (as you called local CAS, whatever) with stackable file interface so that the exist Nydus image service [1] (as ostree, and maybe ostree can use it as well) don't need to modify anything to use exist blobs;

b) Reuse exist fscache/cachefiles;

The reason why we (especially me) finally selected b) because:

- see the people discussion of Google's original Incremental FS topic [2] [3] in 2019, as Amir already mentioned. At that time all fs folks really like to reuse exist subsystem for in-kernel caching rather than reinvent another new in-kernel wheel for local cache.

[ Reinventing a new wheel is not hard (fs or caching), just
makes Linux more fragmented. Especially a new filesystem is just proposed to generate images full of massive massive new magical symlinks with *overriden* uid/gid/permissions to replace regular files. ]

- in-kernel cache implementation usually met several common potential security issues; reusing exist subsystem can make all fses addressed them and benefited from it.

- Usually an exist widely-used userspace implementation is never an excuse for a new in-kernel feature.

Although David Howells is always quite busy these months to develop new netfs interface, otherwise (we think) we should already support failover, multiple daemon/dirs, daemonless and more.

I know that you guys repeatedly say it's a self-contained stackable fs and has few code (the same words as Incfs folks [3] said four years ago already), four reasons make it weak IMHO:

- I think core EROFS is about 2~3 kLOC as well if compression, sysfs and fscache are all code-truncated.

Also, it's always welcome that all people could submit patches for cleaning up. I always do such cleanups from time to time and makes it better.

- "Few code lines" is somewhat weak because people do develop new features, layout after upstream.

Such claim is usually _NOT_ true in the future if you guys do more to optimize performance, new layout or even do your own lazy pulling with your local CAS codebase in the future unless
you *promise* you once dump the code, and do bugfix only like Christian said [4].

From LWN.net comments, I do see the opposite possibility that you'd like to develop new features later.

- In the past, all in-tree kernel filesystems were designed and implemented without some user-space specific indication, including Nydus and ostree (I did see a lot of discussion between folks before in ociv2 brainstorm [5]).

That is why EROFS selected exist in-kernel fscache and made userspace Nydus adapt it:

even (here called) manifest on-disk format ---
EROFS call primary device --- they call Nydus bootstrap;

I'm not sure why it becomes impossible for ... ($$$$).

In addition, if fscache is used, it can also use fsverity_get_digest() to enable fsverity for non-on-demand files.

But again I think even Google's folks think that is (somewhat) broken so that they added fs-verity to its incFS in a self-contained way in Feb 2021 [6].

Finally, again, I do hope a LSF/MM discussion for this new overlay model (full of massive magical symlinks to override permission.)

[1] https://github.com/dragonflyoss/image-service
[2] https://lore.kernel.org/r/CAK8JDrFZW1jwOmhq+YVDPJi9jWWrCRkwpqQ085EouVSyzw-1cg@mail.gmail.com/
[3] https://lore.kernel.org/r/CAK8JDrGRzA+yphpuX+GQ0syRwF_p2Fora+roGCnYqB5E1eOmXA@mail.gmail.com/
[4] https://lore.kernel.org/r/20230117101202.4v4zxuj2tbljogbx@wittgenstein/
[5] https://hackmd.io/@cyphar/ociv2-brainstorm
[6] https://android-review.googlesource.com/c/kernel/common/+/1444521

Thanks,
Gao Xiang

Quoted 18 lines

Regards,
Giuseppe

Quoted 14 lines

Any current functionality gap in erofs and/or in overlayfs cannot be considered as a reason to maintain a new filesystem driver unless you come up with an explanation why closing that functionality gap is not possible or why the erofs+overlayfs alternative would be inferior to maintaining a new filesystem driver.

From the conversations so far, it does not seem like Gao thinks
that the functionality gap in erofs cannot be closed and I don't see why the functionality gap in overlayfs cannot be closed.

Are we missing something?

Thanks,
Amir.

Quoted 98 lines · Gao Xiang

Quoted 4 lines · Giuseppe Scrivano

Hi Amir,

Quoted 1 line · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <[email protected]> wrote:

...

Quoted 37 lines

Quoted 8 lines

Quoted 1 line

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to
what composefs does.
One big difference is that overlay still requires at least a syscall for
each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.
I also don't see how overlay would solve the verified image problem. We
would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.
There are at the moment two ways to handle container images, both somehow
guided by the available file systems in the kernel.
- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

Quoted 8 lines

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.
What we are proposing with composefs is a way to have "dm-verity" style
validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

That is not a new idea honestly, including chain of trust. Even laterly out-of-tree incremental fs using fs-verity for this as well, except that it's in a real self-contained way.

Quoted 5 lines

The composefs manifest blob could be generated remotely and signed. A
client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Back to the topic, after thinking something I have to make a compliment for reference.

First, EROFS had the same internal dissussion and decision at that time almost _two years ago_ (June 2021), it means:

a) Some internal people really suggested EROFS could develop an entire new file-based in-kernel local cache subsystem (as you called local CAS, whatever) with stackable file interface so that the exist Nydus image service [1] (as ostree, and maybe ostree can use it as well) don't need to modify anything to use exist blobs;

   b) Reuse exist fscache/cachefiles;

The reason why we (especially me) finally selected b) because:

- see the people discussion of Google's original Incremental FS topic [2] [3] in 2019, as Amir already mentioned. At that time all fs folks really like to reuse exist subsystem for in-kernel caching rather than reinvent another new in-kernel wheel for local cache.

[ Reinventing a new wheel is not hard (fs or caching), just
makes Linux more fragmented. Especially a new filesystem is just proposed to generate images full of massive massive new magical symlinks with *overriden* uid/gid/permissions to replace regular files. ]

- in-kernel cache implementation usually met several common potential security issues; reusing exist subsystem can make all fses addressed them and benefited from it.

Quoted 7 lines

- Usually an exist widely-used userspace implementation is never an excuse for a new in-kernel feature.

Although David Howells is always quite busy these months to develop new netfs interface, otherwise (we think) we should already support failover, multiple daemon/dirs, daemonless and more.

we have not added any new cache system. overlay does "layer deduplication" and in similar way composefs does "file deduplication". That is not a built-in feature, it is just a side effect of how things are packed together.

Using fscache seems like a good idea and it has many advantages but it is a centralized cache mechanism and it looks like a potential problem when you think about allowing mounts from a user namespace.

As you know as I've contacted you, I've looked at EROFS in the past and tried to get our use cases to work with it before thinking about submitting composefs upstream.

From what I could see EROFS and composefs use two different approaches
to solve a similar problem, but it is not possible to do exactly with EROFS what we are trying to do. To oversimplify it: I see EROFS as a block device that uses fscache, and composefs as an overlay for files instead of directories.

Sure composefs is quite simple and you could embed the composefs features in EROFS and let EROFS behave as composefs when provided a similar manifest file. But how is that any better than having a

separate implementation that does just one thing well instead of merging different paradigms together?

Quoted 27 lines

I know that you guys repeatedly say it's a self-contained stackable fs and has few code (the same words as Incfs folks [3] said four years ago already), four reasons make it weak IMHO:

- I think core EROFS is about 2~3 kLOC as well if compression, sysfs and fscache are all code-truncated.

Also, it's always welcome that all people could submit patches for cleaning up. I always do such cleanups from time to time and makes it better.

- "Few code lines" is somewhat weak because people do develop new features, layout after upstream.

Such claim is usually _NOT_ true in the future if you guys do more to optimize performance, new layout or even do your own lazy pulling with your local CAS codebase in the future unless
you *promise* you once dump the code, and do bugfix only like Christian said [4].

From LWN.net comments, I do see the opposite possibility that you'd like to develop new features later.

Quoted 5 lines

- In the past, all in-tree kernel filesystems were designed and implemented without some user-space specific indication, including Nydus and ostree (I did see a lot of discussion between folks before in ociv2 brainstorm [5]).

Since you are mentioning OCI:

Potentially composefs can be the file system that enables something very close to "ociv2", but it won't need to be called v2 since it is completely compatible with the current OCI image format.

It won't require a different image format, just a seekable tarball that is compatible with old "v1" clients and we need to provide the composefs manifest file.

The seekable tarball allows individual files to be retrieved. OCI clients will not need to pull the entire tarball, but only the individual files that are not already present in the local CAS. They won't also need to create the overlay layout at all, as we do today, since it is already described with the composefs manifest file.

The manifest is portable on different machines with different configurations, as you can use multiple CAS when mounting composefs.

Some users might have a local CAS, some others could have a secondary CAS on a network file system and composefs support all these configurations with the same signed manifest file.

Quoted 3 lines

That is why EROFS selected exist in-kernel fscache and made userspace Nydus adapt it:

Quoted 5 lines

even (here called) manifest on-disk format ---
EROFS call primary device --- they call Nydus bootstrap;

I'm not sure why it becomes impossible for ... ($$$$).

I am not sure what you mean, care to elaborate?

Quoted 5 lines

In addition, if fscache is used, it can also use fsverity_get_digest() to enable fsverity for non-on-demand files.

Quoted 7 lines

But again I think even Google's folks think that is (somewhat) broken so that they added fs-verity to its incFS in a self-contained way in Feb 2021 [6].

Finally, again, I do hope a LSF/MM discussion for this new overlay model (full of massive magical symlinks to override permission.)

you keep pointing it out but nobody is overriding any permission. The "symlinks" as you call them are just a way to refer to the payload files so they can be shared among different mounts. It is the same idea used by "overlay metacopy" and nobody is complaining about it being a security issue (because it is not).

The files in the CAS are owned by the user that creates the mount, so there is no need to circumvent any permission check to access them. We use fs-verity for these files to make sure they are not modified by a malicious user that could get access to them (e.g. a container breakout).

Regards,
Giuseppe

Quoted 1 line

Quoted 107 lines · Giuseppe Scrivano

Quoted 98 lines · Gao Xiang

Quoted 4 lines · Giuseppe Scrivano

Hi Amir,

Quoted 1 line · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <[email protected]> wrote:

...

Quoted 37 lines

Quoted 8 lines

Quoted 1 line

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to
what composefs does.
One big difference is that overlay still requires at least a syscall for
each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.
I also don't see how overlay would solve the verified image problem. We
would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.
There are at the moment two ways to handle container images, both somehow
guided by the available file systems in the kernel.
- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

Quoted 8 lines

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.
What we are proposing with composefs is a way to have "dm-verity" style
validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

That is not a new idea honestly, including chain of trust. Even laterly out-of-tree incremental fs using fs-verity for this as well, except that it's in a real self-contained way.

Quoted 5 lines

The composefs manifest blob could be generated remotely and signed. A
client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Back to the topic, after thinking something I have to make a compliment for reference.

First, EROFS had the same internal dissussion and decision at that time almost _two years ago_ (June 2021), it means:

a) Some internal people really suggested EROFS could develop an entire new file-based in-kernel local cache subsystem (as you called local CAS, whatever) with stackable file interface so that the exist Nydus image service [1] (as ostree, and maybe ostree can use it as well) don't need to modify anything to use exist blobs;

   b) Reuse exist fscache/cachefiles;

The reason why we (especially me) finally selected b) because:

- see the people discussion of Google's original Incremental FS topic [2] [3] in 2019, as Amir already mentioned. At that time all fs folks really like to reuse exist subsystem for in-kernel caching rather than reinvent another new in-kernel wheel for local cache.

[ Reinventing a new wheel is not hard (fs or caching), just
makes Linux more fragmented. Especially a new filesystem is just proposed to generate images full of massive massive new magical symlinks with *overriden* uid/gid/permissions to replace regular files. ]

- in-kernel cache implementation usually met several common potential security issues; reusing exist subsystem can make all fses addressed them and benefited from it.

Quoted 7 lines

- Usually an exist widely-used userspace implementation is never an excuse for a new in-kernel feature.

Although David Howells is always quite busy these months to develop new netfs interface, otherwise (we think) we should already support failover, multiple daemon/dirs, daemonless and more.

Quoted 8 lines

we have not added any new cache system. overlay does "layer deduplication" and in similar way composefs does "file deduplication". That is not a built-in feature, it is just a side effect of how things are packed together.
Using fscache seems like a good idea and it has many advantages but it
is a centralized cache mechanism and it looks like a potential problem when you think about allowing mounts from a user namespace.

I think Christian [1] had the same feeling of my own at that time:

"I'm pretty skeptical of this plan whether we should add more filesystems
that are mountable by unprivileged users. FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format. The track record of bugs exploitable due to userns isn't making this very attractive."

Yes, you could add fs-verity, but EROFS could add fs-verity (or just use dm-verity) as well, but it doesn't change _anything_ about concerns of "allowing mounts from a user namespace".

Quoted 9 lines

As you know as I've contacted you, I've looked at EROFS in the past and tried to get our use cases to work with it before thinking about submitting composefs upstream.
From what I could see EROFS and composefs use two different
approaches
to solve a similar problem, but it is not possible to do exactly with EROFS what we are trying to do. To oversimplify it: I see EROFS as a block device that uses fscache, and composefs as an overlay for files instead of directories.

I don't think so honestly. EROFS "Multiple device" feature is actually "multiple blobs" feature if you really think "device" is block device.

Primary device -- primary blob -- "composefs manifest blob" Blob device -- data blobs -- "composefs backing files"

any difference?

Quoted 3 lines

Sure composefs is quite simple and you could embed the composefs features in EROFS and let EROFS behave as composefs when provided a similar manifest file. But how is that any better than having a

EROFS always has such feature since v5.16, we called primary device, or Nydus concept --- "bootstrap file".

Quoted 2 lines

separate implementation that does just one thing well instead of merging different paradigms together?

It's exist fs on-disk compatible (people can deploy the same image to wider scenarios), or you could modify/enhacnce any in-kernel local fs to do so like I already suggested, such as enhancing "fs/romfs" and make it maintained again due to this magic symlink feature

(because composefs don't have other on-disk requirements other than
a symlink path and a SHA256 verity digest from its original requirement. Any local fs can be enhanced like this.)

Quoted 32 lines

Quoted 27 lines

I know that you guys repeatedly say it's a self-contained stackable fs and has few code (the same words as Incfs folks [3] said four years ago already), four reasons make it weak IMHO:

- I think core EROFS is about 2~3 kLOC as well if compression, sysfs and fscache are all code-truncated.

Also, it's always welcome that all people could submit patches for cleaning up. I always do such cleanups from time to time and makes it better.

- "Few code lines" is somewhat weak because people do develop new features, layout after upstream.

Such claim is usually _NOT_ true in the future if you guys do more to optimize performance, new layout or even do your own lazy pulling with your local CAS codebase in the future unless
you *promise* you once dump the code, and do bugfix only like Christian said [4].

From LWN.net comments, I do see the opposite possibility that you'd like to develop new features later.

Quoted 5 lines

- In the past, all in-tree kernel filesystems were designed and implemented without some user-space specific indication, including Nydus and ostree (I did see a lot of discussion between folks before in ociv2 brainstorm [5]).

Quoted 9 lines

Since you are mentioning OCI:
Potentially composefs can be the file system that enables something very
close to "ociv2", but it won't need to be called v2 since it is completely compatible with the current OCI image format.
It won't require a different image format, just a seekable tarball that
is compatible with old "v1" clients and we need to provide the composefs manifest file.

May I ask did you really look into what Nydus + EROFS already did (as you mentioned we discussed before)?

Your "composefs manifest file" is exactly "Nydus bootstrap file", see:
https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-design.md

"Rafs is a filesystem image containing a separated metadata blob and
several data-deduplicated content-addressable data blobs. In a typical rafs filesystem, the metadata is stored in bootstrap while the data is stored in blobfile.
...

bootstrap: The metadata is a merkle tree (I think that is typo, should be filesystem tree) whose nodes represents a regular filesystem's directory/file a leaf node refers to a file and contains hash value of its file data.
Root node and internal nodes refer to directories and contain the
hash value
of their children nodes."

Nydus is already supported "It won't require a different image format, just a seekable tarball that is compatible with old "v1" clients and we need to provide the composefs manifest file." feature in v2.2 and will be released later.

Quoted 19 lines

The seekable tarball allows individual files to be retrieved. OCI clients will not need to pull the entire tarball, but only the individual files that are not already present in the local CAS. They won't also need to create the overlay layout at all, as we do today, since it is already described with the composefs manifest file.
The manifest is portable on different machines with different configurations, as you can use multiple CAS when mounting composefs. Some users might have a local CAS, some others could have a secondary
CAS on a network file system and composefs support all these configurations with the same signed manifest file.

Quoted 7 lines

That is why EROFS selected exist in-kernel fscache and made userspace Nydus adapt it:

even (here called) manifest on-disk format ---
EROFS call primary device --- they call Nydus bootstrap;

Quoted 2 lines

Quoted 1 line

I'm not sure why it becomes impossible for ... ($$$$).

I am not sure what you mean, care to elaborate?

I just meant these concepts are actually the same concept with different names and:
Nydus is a 2020 stuff;

EROFS + primary device is a 2021-mid stuff.

Quoted 11 lines

Quoted 11 lines

In addition, if fscache is used, it can also use fsverity_get_digest() to enable fsverity for non-on-demand files.

But again I think even Google's folks think that is (somewhat) broken so that they added fs-verity to its incFS in a self-contained way in Feb 2021 [6].

Finally, again, I do hope a LSF/MM discussion for this new overlay model (full of massive magical symlinks to override permission.)

Quoted 6 lines

you keep pointing it out but nobody is overriding any permission. The
"symlinks" as you call them are just a way to refer to the payload files so they can be shared among different mounts. It is the same idea used by "overlay metacopy" and nobody is complaining about it being a security issue (because it is not).

See overlay documentation clearly wrote such metacopy behavior:
https://docs.kernel.org/filesystems/overlayfs.html

"
Do not use metacopy=on with untrusted upper/lower directories. Otherwise it is possible that an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower pointed by REDIRECT. This should not be possible on local system as setting “trusted.” xattrs will require CAP_SYS_ADMIN. But it should be possible for untrusted layers like from a pen drive. "

Do we really need such behavior working on another fs especially with on-disk format? At least Christian said,
"FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format."

Quoted 5 lines

The files in the CAS are owned by the user that creates the mount, so
there is no need to circumvent any permission check to access them. We use fs-verity for these files to make sure they are not modified by a malicious user that could get access to them (e.g. a container breakout).

fs-verity is not always enforcing and it's broken here if fsverity is not supported in underlay fses, that is another my arguable point.

Thanks,
Gao Xiang

[1] https://lore.kernel.org/linux-fsdevel/20230117152756.jbwmeq724potyzju@wittgenstein/

Quoted 2 lines

Regards,
Giuseppe

Quoted 125 lines · Gao Xiang

Quoted 107 lines · Giuseppe Scrivano

Quoted 98 lines · Gao Xiang

Quoted 4 lines · Giuseppe Scrivano

Hi Amir,

Quoted 1 line · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <[email protected]> wrote:

...

Quoted 37 lines

Quoted 8 lines

Quoted 1 line

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to
what composefs does.
One big difference is that overlay still requires at least a syscall for
each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.
I also don't see how overlay would solve the verified image problem. We
would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.
There are at the moment two ways to handle container images, both somehow
guided by the available file systems in the kernel.
- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

Quoted 8 lines

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.
What we are proposing with composefs is a way to have "dm-verity" style
validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

That is not a new idea honestly, including chain of trust. Even laterly out-of-tree incremental fs using fs-verity for this as well, except that it's in a real self-contained way.

Quoted 5 lines

The composefs manifest blob could be generated remotely and signed. A
client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Back to the topic, after thinking something I have to make a compliment for reference.

First, EROFS had the same internal dissussion and decision at that time almost _two years ago_ (June 2021), it means:

a) Some internal people really suggested EROFS could develop an entire new file-based in-kernel local cache subsystem (as you called local CAS, whatever) with stackable file interface so that the exist Nydus image service [1] (as ostree, and maybe ostree can use it as well) don't need to modify anything to use exist blobs;

b) Reuse exist fscache/cachefiles;

The reason why we (especially me) finally selected b) because:

- see the people discussion of Google's original Incremental FS topic [2] [3] in 2019, as Amir already mentioned. At that time all fs folks really like to reuse exist subsystem for in-kernel caching rather than reinvent another new in-kernel wheel for local cache.

[ Reinventing a new wheel is not hard (fs or caching), just
makes Linux more fragmented. Especially a new filesystem is just proposed to generate images full of massive massive new magical symlinks with *overriden* uid/gid/permissions to replace regular files. ]

- in-kernel cache implementation usually met several common potential security issues; reusing exist subsystem can make all fses addressed them and benefited from it.

Quoted 7 lines

- Usually an exist widely-used userspace implementation is never an excuse for a new in-kernel feature.

Although David Howells is always quite busy these months to develop new netfs interface, otherwise (we think) we should already support failover, multiple daemon/dirs, daemonless and more.

Quoted 8 lines

we have not added any new cache system. overlay does "layer deduplication" and in similar way composefs does "file deduplication". That is not a built-in feature, it is just a side effect of how things are packed together.
Using fscache seems like a good idea and it has many advantages but it
is a centralized cache mechanism and it looks like a potential problem when you think about allowing mounts from a user namespace.

I think Christian [1] had the same feeling of my own at that time:

"I'm pretty skeptical of this plan whether we should add more filesystems
that are mountable by unprivileged users. FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format. The track record of bugs exploitable due to userns isn't making this very attractive."

Quoted 3 lines

Yes, you could add fs-verity, but EROFS could add fs-verity (or just use dm-verity) as well, but it doesn't change _anything_ about concerns of "allowing mounts from a user namespace".

I've mentioned that as a potential feature we could add in future, given the simplicity of the format and that it uses a CAS for its data instead of fscache. Each user can have and use their own store to mount the images.

At this point it is just a wish from userspace, as it would improve a few real use cases we have.

Having the possibility to run containers without root privileges is a big deal for many users, look at Flatpak apps for example, or rootless Podman. Mounting and validating images would be a a big security improvement. It is something that is not possible at the moment as fs-verity doesn't cover the directory structure and dm-verity seems out of reach from a user namespace.

Composefs delegates the entire logic of dealing with files to the underlying file system in a similar way to overlay.

Forging the inode metadata from a user namespace mount doesn't look like an insurmountable problem as well since it is already possible with a FUSE filesystem.

So the proposal/wish here is to have a very simple format, that at some point could be considered safe to mount from a user namespace, in addition to overlay and FUSE.

Quoted 19 lines

Quoted 11 lines

As you know as I've contacted you, I've looked at EROFS in the past and tried to get our use cases to work with it before thinking about submitting composefs upstream.
From what I could see EROFS and composefs use two different
approaches
to solve a similar problem, but it is not possible to do exactly with EROFS what we are trying to do. To oversimplify it: I see EROFS as a block device that uses fscache, and composefs as an overlay for files instead of directories.

I don't think so honestly. EROFS "Multiple device" feature is actually "multiple blobs" feature if you really think "device" is block device.

Primary device -- primary blob -- "composefs manifest blob" Blob device -- data blobs -- "composefs backing files"

Quoted 1 line

any difference?

I wouldn't expect any substancial difference between two RO file systems.

Please correct me if I am wrong: EROFS uses 16 bits for the blob device ID, so if we map each file to a single blob device we are kind of limited on how many files we can have.

Sure this is just an artificial limit and can be bumped in a future version but the major difference remains: EROFS uses the blob device through fscache while the composefs files are looked up in the specified repositories.

Quoted 82 lines

Quoted 4 lines

Sure composefs is quite simple and you could embed the composefs features in EROFS and let EROFS behave as composefs when provided a similar manifest file. But how is that any better than having a

EROFS always has such feature since v5.16, we called primary device, or Nydus concept --- "bootstrap file".

Quoted 2 lines

separate implementation that does just one thing well instead of merging different paradigms together?

It's exist fs on-disk compatible (people can deploy the same image to wider scenarios), or you could modify/enhacnce any in-kernel local fs to do so like I already suggested, such as enhancing "fs/romfs" and make it maintained again due to this magic symlink feature

(because composefs don't have other on-disk requirements other than
a symlink path and a SHA256 verity digest from its original requirement. Any local fs can be enhanced like this.)

Quoted 32 lines

Quoted 27 lines

I know that you guys repeatedly say it's a self-contained stackable fs and has few code (the same words as Incfs folks [3] said four years ago already), four reasons make it weak IMHO:

- I think core EROFS is about 2~3 kLOC as well if compression, sysfs and fscache are all code-truncated.

Also, it's always welcome that all people could submit patches for cleaning up. I always do such cleanups from time to time and makes it better.

- "Few code lines" is somewhat weak because people do develop new features, layout after upstream.

Such claim is usually _NOT_ true in the future if you guys do more to optimize performance, new layout or even do your own lazy pulling with your local CAS codebase in the future unless
you *promise* you once dump the code, and do bugfix only like Christian said [4].

From LWN.net comments, I do see the opposite possibility that you'd like to develop new features later.

Quoted 5 lines

- In the past, all in-tree kernel filesystems were designed and implemented without some user-space specific indication, including Nydus and ostree (I did see a lot of discussion between folks before in ociv2 brainstorm [5]).

Quoted 9 lines

Since you are mentioning OCI:
Potentially composefs can be the file system that enables something very
close to "ociv2", but it won't need to be called v2 since it is completely compatible with the current OCI image format.
It won't require a different image format, just a seekable tarball that
is compatible with old "v1" clients and we need to provide the composefs manifest file.

May I ask did you really look into what Nydus + EROFS already did (as you mentioned we discussed before)?

Your "composefs manifest file" is exactly "Nydus bootstrap file", see:
https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-design.md

"Rafs is a filesystem image containing a separated metadata blob and
several data-deduplicated content-addressable data blobs. In a typical rafs filesystem, the metadata is stored in bootstrap while the data is stored in blobfile.
...

bootstrap: The metadata is a merkle tree (I think that is typo, should be filesystem tree) whose nodes represents a regular filesystem's directory/file a leaf node refers to a file and contains hash value of its file data.
Root node and internal nodes refer to directories and contain the
hash value
of their children nodes."

Quoted 4 lines

Nydus is already supported "It won't require a different image format, just a seekable tarball that is compatible with old "v1" clients and we need to provide the composefs manifest file." feature in v2.2 and will be released later.

Nydus is not using a tarball compatible with OCI v1.

It defines a media type "application/vnd.oci.image.layer.nydus.blob.v1", that means it is not compatible with existing clients that don't know about it and you need special handling for that.

Anyway, let's not bother LKML folks with these userspace details. It has no relevance to the kernel and what file systems do.

Quoted 21 lines

Quoted 21 lines

The seekable tarball allows individual files to be retrieved. OCI clients will not need to pull the entire tarball, but only the individual files that are not already present in the local CAS. They won't also need to create the overlay layout at all, as we do today, since it is already described with the composefs manifest file.
The manifest is portable on different machines with different configurations, as you can use multiple CAS when mounting composefs. Some users might have a local CAS, some others could have a secondary
CAS on a network file system and composefs support all these configurations with the same signed manifest file.

Quoted 7 lines

That is why EROFS selected exist in-kernel fscache and made userspace Nydus adapt it:

even (here called) manifest on-disk format ---
EROFS call primary device --- they call Nydus bootstrap;

Quoted 6 lines

Quoted 2 lines

Quoted 1 line

I'm not sure why it becomes impossible for ... ($$$$).

I am not sure what you mean, care to elaborate?

I just meant these concepts are actually the same concept with different names and:

   Nydus is a 2020 stuff;

CRFS[1] is 2019 stuff.

Quoted 33 lines

EROFS + primary device is a 2021-mid stuff.

Quoted 11 lines

Quoted 11 lines

In addition, if fscache is used, it can also use fsverity_get_digest() to enable fsverity for non-on-demand files.

But again I think even Google's folks think that is (somewhat) broken so that they added fs-verity to its incFS in a self-contained way in Feb 2021 [6].

Finally, again, I do hope a LSF/MM discussion for this new overlay model (full of massive magical symlinks to override permission.)

Quoted 6 lines

you keep pointing it out but nobody is overriding any permission. The
"symlinks" as you call them are just a way to refer to the payload files so they can be shared among different mounts. It is the same idea used by "overlay metacopy" and nobody is complaining about it being a security issue (because it is not).

See overlay documentation clearly wrote such metacopy behavior:
https://docs.kernel.org/filesystems/overlayfs.html

"
Do not use metacopy=on with untrusted upper/lower directories. Otherwise it is possible that an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower pointed by REDIRECT. This should not be possible on local system as setting “trusted.” xattrs will require CAP_SYS_ADMIN. But it should be possible for untrusted layers like from a pen drive. "

Quoted 4 lines

Do we really need such behavior working on another fs especially with on-disk format? At least Christian said,
"FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format."

If users want to do something really weird then they can always find a way but the composefs lookup is limited under the directories specified at mount time, so it is not possible to access any file outside the repository.

Quoted 6 lines

Quoted 5 lines

The files in the CAS are owned by the user that creates the mount, so
there is no need to circumvent any permission check to access them. We use fs-verity for these files to make sure they are not modified by a malicious user that could get access to them (e.g. a container breakout).

Quoted 2 lines

fs-verity is not always enforcing and it's broken here if fsverity is not supported in underlay fses, that is another my arguable point.

It is a trade-off. It is up to the user to pick a configuration that allows using fs-verity if they care about this feature.

Regards,
Giuseppe

[1] https://github.com/google/crfs

Quoted 146 lines · Giuseppe Scrivano

Quoted 125 lines · Gao Xiang

Quoted 107 lines · Giuseppe Scrivano

Quoted 98 lines · Gao Xiang

Quoted 4 lines · Giuseppe Scrivano

Hi Amir,

Quoted 1 line · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <[email protected]> wrote:

...

Quoted 37 lines

Quoted 8 lines

Quoted 1 line

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to
what composefs does.
One big difference is that overlay still requires at least a syscall for
each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.
I also don't see how overlay would solve the verified image problem. We
would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.
There are at the moment two ways to handle container images, both somehow
guided by the available file systems in the kernel.
- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

Quoted 8 lines

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.
What we are proposing with composefs is a way to have "dm-verity" style
validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

That is not a new idea honestly, including chain of trust. Even laterly out-of-tree incremental fs using fs-verity for this as well, except that it's in a real self-contained way.

Quoted 5 lines

The composefs manifest blob could be generated remotely and signed. A
client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Back to the topic, after thinking something I have to make a compliment for reference.

First, EROFS had the same internal dissussion and decision at that time almost _two years ago_ (June 2021), it means:

a) Some internal people really suggested EROFS could develop an entire new file-based in-kernel local cache subsystem (as you called local CAS, whatever) with stackable file interface so that the exist Nydus image service [1] (as ostree, and maybe ostree can use it as well) don't need to modify anything to use exist blobs;

b) Reuse exist fscache/cachefiles;

The reason why we (especially me) finally selected b) because:

- see the people discussion of Google's original Incremental FS topic [2] [3] in 2019, as Amir already mentioned. At that time all fs folks really like to reuse exist subsystem for in-kernel caching rather than reinvent another new in-kernel wheel for local cache.

[ Reinventing a new wheel is not hard (fs or caching), just
makes Linux more fragmented. Especially a new filesystem is just proposed to generate images full of massive massive new magical symlinks with *overriden* uid/gid/permissions to replace regular files. ]

- in-kernel cache implementation usually met several common potential security issues; reusing exist subsystem can make all fses addressed them and benefited from it.

Quoted 7 lines

- Usually an exist widely-used userspace implementation is never an excuse for a new in-kernel feature.

Although David Howells is always quite busy these months to develop new netfs interface, otherwise (we think) we should already support failover, multiple daemon/dirs, daemonless and more.

Quoted 8 lines

we have not added any new cache system. overlay does "layer deduplication" and in similar way composefs does "file deduplication". That is not a built-in feature, it is just a side effect of how things are packed together.
Using fscache seems like a good idea and it has many advantages but it
is a centralized cache mechanism and it looks like a potential problem when you think about allowing mounts from a user namespace.

I think Christian [1] had the same feeling of my own at that time:

"I'm pretty skeptical of this plan whether we should add more filesystems
that are mountable by unprivileged users. FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format. The track record of bugs exploitable due to userns isn't making this very attractive."

Quoted 3 lines

Yes, you could add fs-verity, but EROFS could add fs-verity (or just use dm-verity) as well, but it doesn't change _anything_ about concerns of "allowing mounts from a user namespace".

I've mentioned that as a potential feature we could add in future, given
the simplicity of the format and that it uses a CAS for its data instead of fscache. Each user can have and use their own store to mount the images.
At this point it is just a wish from userspace, as it would improve a
few real use cases we have.
Having the possibility to run containers without root privileges is a
big deal for many users, look at Flatpak apps for example, or rootless Podman. Mounting and validating images would be a a big security improvement. It is something that is not possible at the moment as fs-verity doesn't cover the directory structure and dm-verity seems out of reach from a user namespace.
Composefs delegates the entire logic of dealing with files to the

Quoted 8 lines

underlying file system in a similar way to overlay. Forging the inode metadata from a user namespace mount doesn't look like an insurmountable problem as well since it is already possible with a FUSE filesystem.
So the proposal/wish here is to have a very simple format, that at some
point could be considered safe to mount from a user namespace, in addition to overlay and FUSE.

My response is quite similar to
https://lore.kernel.org/r/CAJfpeguyajzHwhae=4PWLF4CUBorwFWeybO-xX6UBD2Ekg81fg@mail.gmail.com/

Quoted 18 lines

Quoted 18 lines

Quoted 10 lines

As you know as I've contacted you, I've looked at EROFS in the past and tried to get our use cases to work with it before thinking about submitting composefs upstream.
From what I could see EROFS and composefs use two different
approaches
to solve a similar problem, but it is not possible to do exactly with EROFS what we are trying to do. To oversimplify it: I see EROFS as a block device that uses fscache, and composefs as an overlay for files instead of directories.

I don't think so honestly. EROFS "Multiple device" feature is actually "multiple blobs" feature if you really think "device" is block device.

Primary device -- primary blob -- "composefs manifest blob" Blob device -- data blobs -- "composefs backing files"

Quoted 7 lines

Quoted 1 line

any difference?

I wouldn't expect any substancial difference between two RO file systems.
Please correct me if I am wrong: EROFS uses 16 bits for the blob device
ID, so if we map each file to a single blob device we are kind of limited on how many files we can have.

I was here just to represent "composefs manifest file" concept rather than device ID.

Quoted 4 lines

Sure this is just an artificial limit and can be bumped in a future version but the major difference remains: EROFS uses the blob device through fscache while the composefs files are looked up in the specified repositories.

No, fscache can also open any cookie when opening file. Again, even with fscache, EROFS doesn't need to modify _any_ on-disk format to:

- record a "cookie id" for such special "magical symlink" with a similar symlink on-disk format (or whatever on-disk format with data, just with a new on-disk flag);

- open such "cookie id" on demand when opening such EROFS file just as any other network fses. I don't think blob device is limited here.

some difference now?

Quoted 86 lines

Quoted 82 lines

Quoted 4 lines

Sure composefs is quite simple and you could embed the composefs features in EROFS and let EROFS behave as composefs when provided a similar manifest file. But how is that any better than having a

EROFS always has such feature since v5.16, we called primary device, or Nydus concept --- "bootstrap file".

Quoted 2 lines

separate implementation that does just one thing well instead of merging different paradigms together?

It's exist fs on-disk compatible (people can deploy the same image to wider scenarios), or you could modify/enhacnce any in-kernel local fs to do so like I already suggested, such as enhancing "fs/romfs" and make it maintained again due to this magic symlink feature

(because composefs don't have other on-disk requirements other than
a symlink path and a SHA256 verity digest from its original requirement. Any local fs can be enhanced like this.)

Quoted 32 lines

Quoted 27 lines

I know that you guys repeatedly say it's a self-contained stackable fs and has few code (the same words as Incfs folks [3] said four years ago already), four reasons make it weak IMHO:

- I think core EROFS is about 2~3 kLOC as well if compression, sysfs and fscache are all code-truncated.

Also, it's always welcome that all people could submit patches for cleaning up. I always do such cleanups from time to time and makes it better.

- "Few code lines" is somewhat weak because people do develop new features, layout after upstream.

Such claim is usually _NOT_ true in the future if you guys do more to optimize performance, new layout or even do your own lazy pulling with your local CAS codebase in the future unless
you *promise* you once dump the code, and do bugfix only like Christian said [4].

From LWN.net comments, I do see the opposite possibility that you'd like to develop new features later.

Quoted 5 lines

- In the past, all in-tree kernel filesystems were designed and implemented without some user-space specific indication, including Nydus and ostree (I did see a lot of discussion between folks before in ociv2 brainstorm [5]).

Quoted 9 lines

Since you are mentioning OCI:
Potentially composefs can be the file system that enables something very
close to "ociv2", but it won't need to be called v2 since it is completely compatible with the current OCI image format.
It won't require a different image format, just a seekable tarball that
is compatible with old "v1" clients and we need to provide the composefs manifest file.

May I ask did you really look into what Nydus + EROFS already did (as you mentioned we discussed before)?

Your "composefs manifest file" is exactly "Nydus bootstrap file", see:
https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-design.md

"Rafs is a filesystem image containing a separated metadata blob and
several data-deduplicated content-addressable data blobs. In a typical rafs filesystem, the metadata is stored in bootstrap while the data is stored in blobfile.
...

bootstrap: The metadata is a merkle tree (I think that is typo, should be filesystem tree) whose nodes represents a regular filesystem's directory/file a leaf node refers to a file and contains hash value of its file data.
Root node and internal nodes refer to directories and contain the
hash value
of their children nodes."

Quoted 4 lines

Nydus is already supported "It won't require a different image format, just a seekable tarball that is compatible with old "v1" clients and we need to provide the composefs manifest file." feature in v2.2 and will be released later.

Quoted 5 lines

Nydus is not using a tarball compatible with OCI v1.
It defines a media type "application/vnd.oci.image.layer.nydus.blob.v1", that
means it is not compatible with existing clients that don't know about it and you need special handling for that.

I am not sure what you're saying: "media type" is quite out of topic here.

If you said "mkcomposefs" is done in the server side, what is the media type of such manifest files?

And why not Nydus cannot do in the same way?
https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-zran.md

Quoted 3 lines

Anyway, let's not bother LKML folks with these userspace details. It
has no relevance to the kernel and what file systems do.

I'd like to avoid, I did't say anything about userspace details, I just would like to say
"merged filesystem tree is also _not_ a new idea of composefs" not "media type", etc.

Quoted 26 lines

Quoted 26 lines

Quoted 16 lines

The seekable tarball allows individual files to be retrieved. OCI clients will not need to pull the entire tarball, but only the individual files that are not already present in the local CAS. They won't also need to create the overlay layout at all, as we do today, since it is already described with the composefs manifest file.
The manifest is portable on different machines with different configurations, as you can use multiple CAS when mounting composefs. Some users might have a local CAS, some others could have a secondary
CAS on a network file system and composefs support all these configurations with the same signed manifest file.

Quoted 3 lines

That is why EROFS selected exist in-kernel fscache and made userspace Nydus adapt it:

Quoted 6 lines

Quoted 5 lines

even (here called) manifest on-disk format ---
EROFS call primary device --- they call Nydus bootstrap;

I'm not sure why it becomes impossible for ... ($$$$).

I am not sure what you mean, care to elaborate?

I just meant these concepts are actually the same concept with different names and:

   Nydus is a 2020 stuff;

Quoted 1 line

CRFS[1] is 2019 stuff.

Does CRFS have anything similiar to a merged filesystem tree?

Here we talked about local CAS:
I have no idea CRFS has anything similar to it.

Quoted 49 lines

Quoted 33 lines

EROFS + primary device is a 2021-mid stuff.

Quoted 11 lines

Quoted 11 lines

In addition, if fscache is used, it can also use fsverity_get_digest() to enable fsverity for non-on-demand files.

But again I think even Google's folks think that is (somewhat) broken so that they added fs-verity to its incFS in a self-contained way in Feb 2021 [6].

Finally, again, I do hope a LSF/MM discussion for this new overlay model (full of massive magical symlinks to override permission.)

Quoted 6 lines

you keep pointing it out but nobody is overriding any permission. The
"symlinks" as you call them are just a way to refer to the payload files so they can be shared among different mounts. It is the same idea used by "overlay metacopy" and nobody is complaining about it being a security issue (because it is not).

See overlay documentation clearly wrote such metacopy behavior:
https://docs.kernel.org/filesystems/overlayfs.html

"
Do not use metacopy=on with untrusted upper/lower directories. Otherwise it is possible that an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower pointed by REDIRECT. This should not be possible on local system as setting “trusted.” xattrs will require CAP_SYS_ADMIN. But it should be possible for untrusted layers like from a pen drive. "

Quoted 4 lines

Do we really need such behavior working on another fs especially with on-disk format? At least Christian said,
"FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format."

If users want to do something really weird then they can always find a
way but the composefs lookup is limited under the directories specified at mount time, so it is not possible to access any file outside the repository.

Quoted 6 lines

Quoted 5 lines

The files in the CAS are owned by the user that creates the mount, so
there is no need to circumvent any permission check to access them. We use fs-verity for these files to make sure they are not modified by a malicious user that could get access to them (e.g. a container breakout).

Quoted 5 lines

Quoted 2 lines

fs-verity is not always enforcing and it's broken here if fsverity is not supported in underlay fses, that is another my arguable point.

It is a trade-off. It is up to the user to pick a configuration that
allows using fs-verity if they care about this feature.

I don't think fsverity is optional with your plan.

I wrote this all because it seems I didn't mention the original motivation to use fscache in v2: kernel already has such in-kernel local cache, and people liked to use it in 2019 rather than another stackable way (as mentioned in incremental fs thread.)

Thanks,
Gao Xiang

Quoted 3 lines

Regards,
Giuseppe
[1] https://github.com/google/crfs

Quoted 156 lines · Gao Xiang

Quoted 146 lines · Giuseppe Scrivano

Quoted 125 lines · Gao Xiang

Quoted 107 lines · Giuseppe Scrivano

Quoted 98 lines · Gao Xiang

Quoted 4 lines · Giuseppe Scrivano

Hi Amir,

Quoted 1 line · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <[email protected]> wrote:

...

Quoted 37 lines

Quoted 8 lines

Quoted 1 line

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to
what composefs does.
One big difference is that overlay still requires at least a syscall for
each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.
I also don't see how overlay would solve the verified image problem. We
would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.
There are at the moment two ways to handle container images, both somehow
guided by the available file systems in the kernel.
- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

Quoted 8 lines

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.
What we are proposing with composefs is a way to have "dm-verity" style
validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

That is not a new idea honestly, including chain of trust. Even laterly out-of-tree incremental fs using fs-verity for this as well, except that it's in a real self-contained way.

Quoted 5 lines

The composefs manifest blob could be generated remotely and signed. A
client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Back to the topic, after thinking something I have to make a compliment for reference.

First, EROFS had the same internal dissussion and decision at that time almost _two years ago_ (June 2021), it means:

a) Some internal people really suggested EROFS could develop an entire new file-based in-kernel local cache subsystem (as you called local CAS, whatever) with stackable file interface so that the exist Nydus image service [1] (as ostree, and maybe ostree can use it as well) don't need to modify anything to use exist blobs;

b) Reuse exist fscache/cachefiles;

The reason why we (especially me) finally selected b) because:

- see the people discussion of Google's original Incremental FS topic [2] [3] in 2019, as Amir already mentioned. At that time all fs folks really like to reuse exist subsystem for in-kernel caching rather than reinvent another new in-kernel wheel for local cache.

[ Reinventing a new wheel is not hard (fs or caching), just
makes Linux more fragmented. Especially a new filesystem is just proposed to generate images full of massive massive new magical symlinks with *overriden* uid/gid/permissions to replace regular files. ]

- in-kernel cache implementation usually met several common potential security issues; reusing exist subsystem can make all fses addressed them and benefited from it.

Quoted 7 lines

- Usually an exist widely-used userspace implementation is never an excuse for a new in-kernel feature.

Although David Howells is always quite busy these months to develop new netfs interface, otherwise (we think) we should already support failover, multiple daemon/dirs, daemonless and more.

Quoted 8 lines

we have not added any new cache system. overlay does "layer deduplication" and in similar way composefs does "file deduplication". That is not a built-in feature, it is just a side effect of how things are packed together.
Using fscache seems like a good idea and it has many advantages but it
is a centralized cache mechanism and it looks like a potential problem when you think about allowing mounts from a user namespace.

I think Christian [1] had the same feeling of my own at that time:

"I'm pretty skeptical of this plan whether we should add more filesystems
that are mountable by unprivileged users. FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format. The track record of bugs exploitable due to userns isn't making this very attractive."

Quoted 3 lines

Yes, you could add fs-verity, but EROFS could add fs-verity (or just use dm-verity) as well, but it doesn't change _anything_ about concerns of "allowing mounts from a user namespace".

I've mentioned that as a potential feature we could add in future, given
the simplicity of the format and that it uses a CAS for its data instead of fscache. Each user can have and use their own store to mount the images.
At this point it is just a wish from userspace, as it would improve a
few real use cases we have.
Having the possibility to run containers without root privileges is a
big deal for many users, look at Flatpak apps for example, or rootless Podman. Mounting and validating images would be a a big security improvement. It is something that is not possible at the moment as fs-verity doesn't cover the directory structure and dm-verity seems out of reach from a user namespace.
Composefs delegates the entire logic of dealing with files to the

Quoted 8 lines

underlying file system in a similar way to overlay. Forging the inode metadata from a user namespace mount doesn't look like an insurmountable problem as well since it is already possible with a FUSE filesystem.
So the proposal/wish here is to have a very simple format, that at some
point could be considered safe to mount from a user namespace, in addition to overlay and FUSE.

Quoted 2 lines

My response is quite similar to
https://lore.kernel.org/r/CAJfpeguyajzHwhae=4PWLF4CUBorwFWeybO-xX6UBD2Ekg81fg@mail.gmail.com/

I don't see how that applies to what I said about unprivileged mounts, except the part about lazy download where I agree with Miklos that should be handled through FUSE and that is something possible with composefs:

mount -t composefs composefs -obasedir=/path/to/store:/mnt/fuse /mnt/cfs

where /mnt/fuse is handled by a FUSE file system that takes care of loading the files from the remote server, and possibly write them to /path/to/store once they are completed.

So each user could have their "lazy download" without interfering with other users or the centralized cache.

Quoted 37 lines

Quoted 18 lines

Quoted 18 lines

Quoted 10 lines

As you know as I've contacted you, I've looked at EROFS in the past and tried to get our use cases to work with it before thinking about submitting composefs upstream.
From what I could see EROFS and composefs use two different
approaches
to solve a similar problem, but it is not possible to do exactly with EROFS what we are trying to do. To oversimplify it: I see EROFS as a block device that uses fscache, and composefs as an overlay for files instead of directories.

I don't think so honestly. EROFS "Multiple device" feature is actually "multiple blobs" feature if you really think "device" is block device.

Primary device -- primary blob -- "composefs manifest blob" Blob device -- data blobs -- "composefs backing files"

Quoted 7 lines

Quoted 1 line

any difference?

I wouldn't expect any substancial difference between two RO file systems.
Please correct me if I am wrong: EROFS uses 16 bits for the blob device
ID, so if we map each file to a single blob device we are kind of limited on how many files we can have.

I was here just to represent "composefs manifest file" concept rather than device ID.

Quoted 4 lines

Sure this is just an artificial limit and can be bumped in a future version but the major difference remains: EROFS uses the blob device through fscache while the composefs files are looked up in the specified repositories.

No, fscache can also open any cookie when opening file. Again, even with fscache, EROFS doesn't need to modify _any_ on-disk format to:

Quoted 8 lines

- record a "cookie id" for such special "magical symlink" with a similar symlink on-disk format (or whatever on-disk format with data, just with a new on-disk flag);

- open such "cookie id" on demand when opening such EROFS file just as any other network fses. I don't think blob device is limited here.

some difference now?

recording the "cookie id" is done by a singleton userspace daemon that controls the cachefiles device and requires one operation for each file before the image can be mounted.

Is that the case or I misunderstood something?

Quoted 92 lines

Quoted 86 lines

Quoted 82 lines

Quoted 4 lines

Sure composefs is quite simple and you could embed the composefs features in EROFS and let EROFS behave as composefs when provided a similar manifest file. But how is that any better than having a

EROFS always has such feature since v5.16, we called primary device, or Nydus concept --- "bootstrap file".

Quoted 2 lines

separate implementation that does just one thing well instead of merging different paradigms together?

It's exist fs on-disk compatible (people can deploy the same image to wider scenarios), or you could modify/enhacnce any in-kernel local fs to do so like I already suggested, such as enhancing "fs/romfs" and make it maintained again due to this magic symlink feature

(because composefs don't have other on-disk requirements other than
a symlink path and a SHA256 verity digest from its original requirement. Any local fs can be enhanced like this.)

Quoted 32 lines

Quoted 27 lines

I know that you guys repeatedly say it's a self-contained stackable fs and has few code (the same words as Incfs folks [3] said four years ago already), four reasons make it weak IMHO:

- I think core EROFS is about 2~3 kLOC as well if compression, sysfs and fscache are all code-truncated.

Also, it's always welcome that all people could submit patches for cleaning up. I always do such cleanups from time to time and makes it better.

- "Few code lines" is somewhat weak because people do develop new features, layout after upstream.

Such claim is usually _NOT_ true in the future if you guys do more to optimize performance, new layout or even do your own lazy pulling with your local CAS codebase in the future unless
you *promise* you once dump the code, and do bugfix only like Christian said [4].

From LWN.net comments, I do see the opposite possibility that you'd like to develop new features later.

Quoted 5 lines

- In the past, all in-tree kernel filesystems were designed and implemented without some user-space specific indication, including Nydus and ostree (I did see a lot of discussion between folks before in ociv2 brainstorm [5]).

Quoted 9 lines

Since you are mentioning OCI:
Potentially composefs can be the file system that enables something very
close to "ociv2", but it won't need to be called v2 since it is completely compatible with the current OCI image format.
It won't require a different image format, just a seekable tarball that
is compatible with old "v1" clients and we need to provide the composefs manifest file.

May I ask did you really look into what Nydus + EROFS already did (as you mentioned we discussed before)?

Your "composefs manifest file" is exactly "Nydus bootstrap file", see:
https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-design.md

"Rafs is a filesystem image containing a separated metadata blob and
several data-deduplicated content-addressable data blobs. In a typical rafs filesystem, the metadata is stored in bootstrap while the data is stored in blobfile.
...

bootstrap: The metadata is a merkle tree (I think that is typo, should be filesystem tree) whose nodes represents a regular filesystem's directory/file a leaf node refers to a file and contains hash value of its file data.
Root node and internal nodes refer to directories and contain the
hash value
of their children nodes."

Quoted 4 lines

Nydus is already supported "It won't require a different image format, just a seekable tarball that is compatible with old "v1" clients and we need to provide the composefs manifest file." feature in v2.2 and will be released later.

Quoted 5 lines

Nydus is not using a tarball compatible with OCI v1.
It defines a media type "application/vnd.oci.image.layer.nydus.blob.v1", that
means it is not compatible with existing clients that don't know about it and you need special handling for that.

Quoted 8 lines

I am not sure what you're saying: "media type" is quite out of topic here.

If you said "mkcomposefs" is done in the server side, what is the media type of such manifest files?

And why not Nydus cannot do in the same way?
https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-zran.md

I am not talking about the manifest or the bootstrap file, I am talking about the data blobs.

Quoted 35 lines

Quoted 3 lines

Anyway, let's not bother LKML folks with these userspace details. It
has no relevance to the kernel and what file systems do.

I'd like to avoid, I did't say anything about userspace details, I just would like to say
"merged filesystem tree is also _not_ a new idea of composefs" not "media type", etc.

Quoted 26 lines

Quoted 26 lines

Quoted 16 lines

The seekable tarball allows individual files to be retrieved. OCI clients will not need to pull the entire tarball, but only the individual files that are not already present in the local CAS. They won't also need to create the overlay layout at all, as we do today, since it is already described with the composefs manifest file.
The manifest is portable on different machines with different configurations, as you can use multiple CAS when mounting composefs. Some users might have a local CAS, some others could have a secondary
CAS on a network file system and composefs support all these configurations with the same signed manifest file.

Quoted 3 lines

That is why EROFS selected exist in-kernel fscache and made userspace Nydus adapt it:

Quoted 6 lines

Quoted 5 lines

even (here called) manifest on-disk format ---
EROFS call primary device --- they call Nydus bootstrap;

I'm not sure why it becomes impossible for ... ($$$$).

I am not sure what you mean, care to elaborate?

I just meant these concepts are actually the same concept with different names and:

   Nydus is a 2020 stuff;

Quoted 6 lines

Quoted 1 line

CRFS[1] is 2019 stuff.

Does CRFS have anything similiar to a merged filesystem tree?

Here we talked about local CAS:
I have no idea CRFS has anything similar to it.

yes it does and it uses it with a FUSE file system. So neither composefs nor EROFS have invented anything here.

Anyway, does it really matter who made what first? I don't see how it helps to understand if there are relevant differences in composefs to justify its presence in the kernel.

Quoted 55 lines

Quoted 54 lines

Quoted 33 lines

EROFS + primary device is a 2021-mid stuff.

Quoted 11 lines

Quoted 11 lines

In addition, if fscache is used, it can also use fsverity_get_digest() to enable fsverity for non-on-demand files.

But again I think even Google's folks think that is (somewhat) broken so that they added fs-verity to its incFS in a self-contained way in Feb 2021 [6].

Finally, again, I do hope a LSF/MM discussion for this new overlay model (full of massive magical symlinks to override permission.)

Quoted 6 lines

you keep pointing it out but nobody is overriding any permission. The
"symlinks" as you call them are just a way to refer to the payload files so they can be shared among different mounts. It is the same idea used by "overlay metacopy" and nobody is complaining about it being a security issue (because it is not).

See overlay documentation clearly wrote such metacopy behavior:
https://docs.kernel.org/filesystems/overlayfs.html

"
Do not use metacopy=on with untrusted upper/lower directories. Otherwise it is possible that an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower pointed by REDIRECT. This should not be possible on local system as setting “trusted.” xattrs will require CAP_SYS_ADMIN. But it should be possible for untrusted layers like from a pen drive. "

Quoted 4 lines

Do we really need such behavior working on another fs especially with on-disk format? At least Christian said,
"FUSE and Overlayfs are adventurous enough and they don't have their own on-disk format."

If users want to do something really weird then they can always find a
way but the composefs lookup is limited under the directories specified at mount time, so it is not possible to access any file outside the repository.

Quoted 8 lines

Quoted 5 lines

The files in the CAS are owned by the user that creates the mount, so
there is no need to circumvent any permission check to access them. We use fs-verity for these files to make sure they are not modified by a malicious user that could get access to them (e.g. a container breakout).

fs-verity is not always enforcing and it's broken here if fsverity is not supported in underlay fses, that is another my arguable point.

It is a trade-off. It is up to the user to pick a configuration that
allows using fs-verity if they care about this feature.

Quoted 1 line

I don't think fsverity is optional with your plan.

yes it is optional. without fs-verity it would behave the same as today with overlay mounts without any fs-verity.

How does validation work in EROFS for files served from fscache and that are on a remote file system?

Quoted 5 lines

I wrote this all because it seems I didn't mention the original motivation to use fscache in v2: kernel already has such in-kernel local cache, and people liked to use it in 2019 rather than another stackable way (as mentioned in incremental fs thread.)

still for us the stackable way works better.

Quoted 6 lines

Thanks,
Gao Xiang

Quoted 3 lines

Regards,
Giuseppe
[1] https://github.com/google/crfs

Giuseppe Scrivano <[email protected]> writes:

Quoted 3 lines

How does validation work in EROFS for files served from fscache and that are on a remote file system?

nevermind my last question, I guess it would still go through the block device in EROFS.
This is clearly a point in favor of a block device approach that a stacking file system like overlay or composefs cannot achieve without support from the underlying file system.

Hi Amir,

Quoted 147 lines · Amir Goldstein

Quoted 130 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we call composefs. This is the first time we propose this publically and we would like some feedback on it.

At its core, composefs is a way to construct and use read only images that are used similar to how you would use e.g. loop-back mounted squashfs images. On top of this composefs has two fundamental features. First it allows sharing of file data (both on disk and in page cache) between images, and secondly it has dm-verity like validation on read.

Let me first start with a minimal example of how this can be used, before going into the details:

Suppose we have this source for an image:

rootfs/

├── dir
│   └── another_a
├── file_a
└── file_b

We can then use this to generate an image file and a set of content-addressed backing files:

# mkcomposefs --digest-store=objects rootfs/ rootfs.img
# ls -l rootfs.img objects/*/*

(quoted diff) +0 -3

-rw-------. 1 root root   10 Nov 18 13:20 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
-rw-------. 1 root root   10 Nov 18 13:20 objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
-rw-r--r--. 1 root root 4228 Nov 18 13:20 rootfs.img

The rootfs.img file contains all information about directory and file metadata plus references to the backing files by name. We can now mount this and look at the result:

# mount -t composefs rootfs.img -o basedir=objects /mnt
# ls  /mnt/
dir  file_a  file_b
# cat /mnt/file_a
content_a

When reading this file the kernel is actually reading the backing file, in a fashion similar to overlayfs. Since the backing file is content-addressed, the objects directory can be shared for multiple images, and any files that happen to have the same content are shared. I refer to this as opportunistic sharing, as it is different than the more course-grained explicit sharing used by e.g. container base images.

The next step is the validation. Note how the object files have fs-verity enabled. In fact, they are named by their fs-verity digest:

# fsverity digest objects/*/*
sha256:02927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
sha256:cc3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f

The generated filesystm image may contain the expected digest for the backing files. When the backing file digest is incorrect, the open will fail, and if the open succeeds, any other on-disk file-changes will be detected by fs-verity:

# cat objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
content_a
# rm -f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# echo modified > objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# cat /mnt/file_a
WARNING: composefs backing file '3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f' unexpectedly had no fs-verity digest
cat: /mnt/file_a: Input/output error

This re-uses the existing fs-verity functionallity to protect against changes in file contents, while adding on top of it protection against changes in filesystem metadata and structure. I.e. protecting against replacing a fs-verity enabled file or modifying file permissions or xattrs.

To be fully verified we need another step: we use fs-verity on the image itself. Then we pass the expected digest on the mount command line (which will be verified at mount time):

# fsverity enable rootfs.img
# fsverity digest rootfs.img
sha256:da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 rootfs.img
# mount -t composefs rootfs.img -o basedir=objects,digest=da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 /mnt

So, given a trusted set of mount options (say unlocked from TPM), we have a fully verified filesystem tree mounted, with opportunistic finegrained sharing of identical files.

So, why do we want this? There are two initial users. First of all we want to use the opportunistic sharing for the podman container image baselayer. The idea is to use a composefs mount as the lower directory in an overlay mount, with the upper directory being the container work dir. This will allow automatical file-level disk and page-cache sharning between any two images, independent of details like the permissions and timestamps of the files.

Secondly we are interested in using the verification aspects of composefs in the ostree project. Ostree already supports a content-addressed object store, but it is currently referenced by hardlink farms. The object store and the trees that reference it are signed and verified at download time, but there is no runtime verification. If we replace the hardlink farm with a composefs image that points into the existing object store we can use the verification to implement runtime verification.

In fact, the tooling to create composefs images is 100% reproducible, so all we need is to add the composefs image fs-verity digest into the ostree commit. Then the image can be reconstructed from the ostree commit info, generating a file with the same fs-verity digest.

These are the usecases we're currently interested in, but there seems to be a breadth of other possible uses. For example, many systems use loopback mounts for images (like lxc or snap), and these could take advantage of the opportunistic sharing. We've also talked about using fuse to implement a local cache for the backing files. I.e. you would have the second basedir be a fuse filesystem. On lookup failure in the first basedir it downloads the file and saves it in the first basedir for later lookups. There are many interesting possibilities here.

The patch series contains some documentation on the file format and how to use the filesystem.

The userspace tools (and a standalone kernel module) is available here:
https://github.com/containers/composefs

Initial work on ostree integration is here:
https://github.com/ostreedev/ostree/pull/2640

Quoted 9 lines

Changes since v2:
- Simplified filesystem format to use fixed size inodes. This resulted in simpler (now < 2k lines) code as well as higher performance at the cost of slightly (~40%) larger images.
- We now use multi-page mappings from the page cache, which removes limits on sizes of xattrs and makes the dirent handling code simpler.
- Added more documentation about the on-disk file format.
- General cleanups based on review comments.

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I think Dave explained quite well why using overlay is not comparable to what composefs does.

One big difference is that overlay still requires at least a syscall for each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.

I also don't see how overlay would solve the verified image problem. We would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.

There are at the moment two ways to handle container images, both somehow guided by the available file systems in the kernel.

- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.

What we are proposing with composefs is a way to have "dm-verity" style validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

The composefs manifest blob could be generated remotely and signed. A client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Quoted 159 lines · Giuseppe Scrivano

Hi Amir,

Quoted 153 lines · Amir Goldstein

Quoted 130 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we call composefs. This is the first time we propose this publically and we would like some feedback on it.

At its core, composefs is a way to construct and use read only images that are used similar to how you would use e.g. loop-back mounted squashfs images. On top of this composefs has two fundamental features. First it allows sharing of file data (both on disk and in page cache) between images, and secondly it has dm-verity like validation on read.

Let me first start with a minimal example of how this can be used, before going into the details:

Suppose we have this source for an image:

rootfs/

├── dir
│   └── another_a
├── file_a
└── file_b

We can then use this to generate an image file and a set of content-addressed backing files:

# mkcomposefs --digest-store=objects rootfs/ rootfs.img
# ls -l rootfs.img objects/*/*

(quoted diff) +0 -3

-rw-------. 1 root root   10 Nov 18 13:20 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
-rw-------. 1 root root   10 Nov 18 13:20 objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
-rw-r--r--. 1 root root 4228 Nov 18 13:20 rootfs.img

The rootfs.img file contains all information about directory and file metadata plus references to the backing files by name. We can now mount this and look at the result:

# mount -t composefs rootfs.img -o basedir=objects /mnt
# ls  /mnt/
dir  file_a  file_b
# cat /mnt/file_a
content_a

When reading this file the kernel is actually reading the backing file, in a fashion similar to overlayfs. Since the backing file is content-addressed, the objects directory can be shared for multiple images, and any files that happen to have the same content are shared. I refer to this as opportunistic sharing, as it is different than the more course-grained explicit sharing used by e.g. container base images.

The next step is the validation. Note how the object files have fs-verity enabled. In fact, they are named by their fs-verity digest:

# fsverity digest objects/*/*
sha256:02927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
sha256:cc3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f

The generated filesystm image may contain the expected digest for the backing files. When the backing file digest is incorrect, the open will fail, and if the open succeeds, any other on-disk file-changes will be detected by fs-verity:

# cat objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
content_a
# rm -f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# echo modified > objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# cat /mnt/file_a
WARNING: composefs backing file '3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f' unexpectedly had no fs-verity digest
cat: /mnt/file_a: Input/output error

This re-uses the existing fs-verity functionallity to protect against changes in file contents, while adding on top of it protection against changes in filesystem metadata and structure. I.e. protecting against replacing a fs-verity enabled file or modifying file permissions or xattrs.

To be fully verified we need another step: we use fs-verity on the image itself. Then we pass the expected digest on the mount command line (which will be verified at mount time):

# fsverity enable rootfs.img
# fsverity digest rootfs.img
sha256:da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 rootfs.img
# mount -t composefs rootfs.img -o basedir=objects,digest=da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 /mnt

So, given a trusted set of mount options (say unlocked from TPM), we have a fully verified filesystem tree mounted, with opportunistic finegrained sharing of identical files.

So, why do we want this? There are two initial users. First of all we want to use the opportunistic sharing for the podman container image baselayer. The idea is to use a composefs mount as the lower directory in an overlay mount, with the upper directory being the container work dir. This will allow automatical file-level disk and page-cache sharning between any two images, independent of details like the permissions and timestamps of the files.

Secondly we are interested in using the verification aspects of composefs in the ostree project. Ostree already supports a content-addressed object store, but it is currently referenced by hardlink farms. The object store and the trees that reference it are signed and verified at download time, but there is no runtime verification. If we replace the hardlink farm with a composefs image that points into the existing object store we can use the verification to implement runtime verification.

In fact, the tooling to create composefs images is 100% reproducible, so all we need is to add the composefs image fs-verity digest into the ostree commit. Then the image can be reconstructed from the ostree commit info, generating a file with the same fs-verity digest.

These are the usecases we're currently interested in, but there seems to be a breadth of other possible uses. For example, many systems use loopback mounts for images (like lxc or snap), and these could take advantage of the opportunistic sharing. We've also talked about using fuse to implement a local cache for the backing files. I.e. you would have the second basedir be a fuse filesystem. On lookup failure in the first basedir it downloads the file and saves it in the first basedir for later lookups. There are many interesting possibilities here.

The patch series contains some documentation on the file format and how to use the filesystem.

The userspace tools (and a standalone kernel module) is available here:
https://github.com/containers/composefs

Initial work on ostree integration is here:
https://github.com/ostreedev/ostree/pull/2640

Quoted 9 lines

Changes since v2:
- Simplified filesystem format to use fixed size inodes. This resulted in simpler (now < 2k lines) code as well as higher performance at the cost of slightly (~40%) larger images.
- We now use multi-page mappings from the page cache, which removes limits on sizes of xattrs and makes the dirent handling code simpler.
- Added more documentation about the on-disk file format.
- General cleanups based on review comments.

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

Quoted 3 lines

I think Dave explained quite well why using overlay is not comparable to what composefs does.

Where? Can I get a link please?

If there are good reasons why composefs is superior to erofs+overlayfs Please include them in the submission, since several developers keep raising the same questions - that is all I ask.

Quoted 8 lines

One big difference is that overlay still requires at least a syscall for each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.

I think you did not understand my suggestion:

overlay read-only mount:
layer 1: erofs mount of a precomposed image (same as mkcomposefs) layer 2: any pre-existing fs path with /blocks repository layer 3: any per-existing fs path with /blocks repository
...

The mkcomposefs flow is exactly the same in this suggestion the upper layer image is created without any syscalls and removed without any syscalls.

Overlayfs already has the feature of redirecting from upper layer to relative paths in lower layers.

Quoted 6 lines

I also don't see how overlay would solve the verified image problem. We would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.

The entire erofs image would be fsverified including the overlayfs xattrs. That is exactly the same model as composefs.
I am not even saying that your model is wrong, only that you are within reach of implementing it with existing subsystems.

Quoted 16 lines

There are at the moment two ways to handle container images, both somehow guided by the available file systems in the kernel.

- A single image mounted as a block device.
- A list of tarballs (OCI image) that are unpacked and mounted as overlay layers.

One big advantage of the block devices model is that you can use dm-verity, this is something we miss today with OCI container images that use overlay.

What we are proposing with composefs is a way to have "dm-verity" style validation based on fs-verity and the possibility to share individual files instead of layers. These files can also be on different file systems, which is something not possible with the block device model.

Quoted 5 lines

The composefs manifest blob could be generated remotely and signed. A client would need just to validate the signature for the manifest blob and from there retrieve the files that are not in the local CAS (even from an insecure source) and mount directly the manifest file.

Excellent description of the problem.
I agree that we need a hybrid solution between the block and tarball image model.

All I am saying is that this solution can use existing kernel components and existing established on-disk formats (erofs+overlayfs).

What was missing all along was the userspace component (i.e. composefs) and I am very happy that you guys are working on this project.

These userspace tools could be useful for other use cases. For example, overlayfs is able to describe a large directory rename with redirect xattr since v4.9, but image composing tools do not make use of that, so an OCI image describing a large dir rename will currently contain all the files within.

Once again, you may or may not be able to use erofs and overlayfs out of the box for your needs, but so far I did not see any functionality gap that is not possible to close.

Please let me know if you know of such gaps or if my proposal does not meet the goals of composefs.

Quoted 160 lines · Amir Goldstein

Quoted 159 lines · Giuseppe Scrivano

Hi Amir,

Quoted 153 lines · Amir Goldstein

Quoted 130 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we call composefs. This is the first time we propose this publically and we would like some feedback on it.

At its core, composefs is a way to construct and use read only images that are used similar to how you would use e.g. loop-back mounted squashfs images. On top of this composefs has two fundamental features. First it allows sharing of file data (both on disk and in page cache) between images, and secondly it has dm-verity like validation on read.

Let me first start with a minimal example of how this can be used, before going into the details:

Suppose we have this source for an image:

rootfs/

├── dir
│   └── another_a
├── file_a
└── file_b

We can then use this to generate an image file and a set of content-addressed backing files:

# mkcomposefs --digest-store=objects rootfs/ rootfs.img
# ls -l rootfs.img objects/*/*

(quoted diff) +0 -3

-rw-------. 1 root root   10 Nov 18 13:20 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
-rw-------. 1 root root   10 Nov 18 13:20 objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
-rw-r--r--. 1 root root 4228 Nov 18 13:20 rootfs.img

The rootfs.img file contains all information about directory and file metadata plus references to the backing files by name. We can now mount this and look at the result:

# mount -t composefs rootfs.img -o basedir=objects /mnt
# ls  /mnt/
dir  file_a  file_b
# cat /mnt/file_a
content_a

When reading this file the kernel is actually reading the backing file, in a fashion similar to overlayfs. Since the backing file is content-addressed, the objects directory can be shared for multiple images, and any files that happen to have the same content are shared. I refer to this as opportunistic sharing, as it is different than the more course-grained explicit sharing used by e.g. container base images.

The next step is the validation. Note how the object files have fs-verity enabled. In fact, they are named by their fs-verity digest:

# fsverity digest objects/*/*
sha256:02927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
sha256:cc3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f

The generated filesystm image may contain the expected digest for the backing files. When the backing file digest is incorrect, the open will fail, and if the open succeeds, any other on-disk file-changes will be detected by fs-verity:

# cat objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
content_a
# rm -f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# echo modified > objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# cat /mnt/file_a
WARNING: composefs backing file '3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f' unexpectedly had no fs-verity digest
cat: /mnt/file_a: Input/output error

This re-uses the existing fs-verity functionallity to protect against changes in file contents, while adding on top of it protection against changes in filesystem metadata and structure. I.e. protecting against replacing a fs-verity enabled file or modifying file permissions or xattrs.

To be fully verified we need another step: we use fs-verity on the image itself. Then we pass the expected digest on the mount command line (which will be verified at mount time):

# fsverity enable rootfs.img
# fsverity digest rootfs.img
sha256:da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 rootfs.img
# mount -t composefs rootfs.img -o basedir=objects,digest=da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 /mnt

So, given a trusted set of mount options (say unlocked from TPM), we have a fully verified filesystem tree mounted, with opportunistic finegrained sharing of identical files.

So, why do we want this? There are two initial users. First of all we want to use the opportunistic sharing for the podman container image baselayer. The idea is to use a composefs mount as the lower directory in an overlay mount, with the upper directory being the container work dir. This will allow automatical file-level disk and page-cache sharning between any two images, independent of details like the permissions and timestamps of the files.

Secondly we are interested in using the verification aspects of composefs in the ostree project. Ostree already supports a content-addressed object store, but it is currently referenced by hardlink farms. The object store and the trees that reference it are signed and verified at download time, but there is no runtime verification. If we replace the hardlink farm with a composefs image that points into the existing object store we can use the verification to implement runtime verification.

In fact, the tooling to create composefs images is 100% reproducible, so all we need is to add the composefs image fs-verity digest into the ostree commit. Then the image can be reconstructed from the ostree commit info, generating a file with the same fs-verity digest.

These are the usecases we're currently interested in, but there seems to be a breadth of other possible uses. For example, many systems use loopback mounts for images (like lxc or snap), and these could take advantage of the opportunistic sharing. We've also talked about using fuse to implement a local cache for the backing files. I.e. you would have the second basedir be a fuse filesystem. On lookup failure in the first basedir it downloads the file and saves it in the first basedir for later lookups. There are many interesting possibilities here.

The patch series contains some documentation on the file format and how to use the filesystem.

The userspace tools (and a standalone kernel module) is available here:
https://github.com/containers/composefs

Initial work on ostree integration is here:
https://github.com/ostreedev/ostree/pull/2640

Quoted 9 lines

Changes since v2:
- Simplified filesystem format to use fixed size inodes. This resulted in simpler (now < 2k lines) code as well as higher performance at the cost of slightly (~40%) larger images.
- We now use multi-page mappings from the page cache, which removes limits on sizes of xattrs and makes the dirent handling code simpler.
- Added more documentation about the on-disk file format.
- General cleanups based on review comments.

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

Quoted 5 lines

Quoted 3 lines

I think Dave explained quite well why using overlay is not comparable to what composefs does.

Where? Can I get a link please?

I am referring to this message: https://lore.kernel.org/lkml/[email protected]/

Quoted 21 lines

If there are good reasons why composefs is superior to erofs+overlayfs Please include them in the submission, since several developers keep raising the same questions - that is all I ask.

Quoted 8 lines

One big difference is that overlay still requires at least a syscall for each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.

I think you did not understand my suggestion:

overlay read-only mount:
layer 1: erofs mount of a precomposed image (same as mkcomposefs) layer 2: any pre-existing fs path with /blocks repository layer 3: any per-existing fs path with /blocks repository
...

Quoted 3 lines

The mkcomposefs flow is exactly the same in this suggestion the upper layer image is created without any syscalls and removed without any syscalls.

mkcomposefs is supposed to be used server side, when the image is built. The clients that will mount the image don't have to create it (at least for images that will provide the manifest).

So this is quite different as in the overlay model we must create the layout, that is the equivalent of the composefs manifest, on any node the image is pulled to.

Quoted 2 lines

Overlayfs already has the feature of redirecting from upper layer to relative paths in lower layers.

Could you please provide more information on how you would compose the overlay image first?

From what I can see, it still requires at least one syscall for each file in the image to be created and these images are not portable to a different machine.

Should we always make "/blocks" a whiteout to prevent it is leaked in the container?

And what prevents files under "/blocks" to be replaced with a different version? I think fs-verity on the EROFS image itself won't cover it.

Quoted 11 lines

Quoted 6 lines

I also don't see how overlay would solve the verified image problem. We would have the same problem we have today with fs-verity as it can only validate a single file but not the entire directory structure. Changes that affect the layer containing the trusted.overlay.{metacopy,redirect} xattrs won't be noticed.

The entire erofs image would be fsverified including the overlayfs xattrs. That is exactly the same model as composefs.
I am not even saying that your model is wrong, only that you are within reach of implementing it with existing subsystems.

now we can do:

mount -t composefs rootfs.img -o basedir=objects,digest=da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 /mnt

that is quite useful for mounting the OS image, as is the OSTree case.

How would that be possible with the setup you are proposing? Would overlay gain a new "digest=" kind of option to validate its first layer?

Quoted 163 lines · Giuseppe Scrivano

Quoted 160 lines · Amir Goldstein

Quoted 159 lines · Giuseppe Scrivano

Hi Amir,

Quoted 153 lines · Amir Goldstein

Quoted 130 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we call composefs. This is the first time we propose this publically and we would like some feedback on it.

At its core, composefs is a way to construct and use read only images that are used similar to how you would use e.g. loop-back mounted squashfs images. On top of this composefs has two fundamental features. First it allows sharing of file data (both on disk and in page cache) between images, and secondly it has dm-verity like validation on read.

Let me first start with a minimal example of how this can be used, before going into the details:

Suppose we have this source for an image:

rootfs/

├── dir
│   └── another_a
├── file_a
└── file_b

We can then use this to generate an image file and a set of content-addressed backing files:

# mkcomposefs --digest-store=objects rootfs/ rootfs.img
# ls -l rootfs.img objects/*/*

(quoted diff) +0 -3

-rw-------. 1 root root   10 Nov 18 13:20 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
-rw-------. 1 root root   10 Nov 18 13:20 objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
-rw-r--r--. 1 root root 4228 Nov 18 13:20 rootfs.img

The rootfs.img file contains all information about directory and file metadata plus references to the backing files by name. We can now mount this and look at the result:

# mount -t composefs rootfs.img -o basedir=objects /mnt
# ls  /mnt/
dir  file_a  file_b
# cat /mnt/file_a
content_a

When reading this file the kernel is actually reading the backing file, in a fashion similar to overlayfs. Since the backing file is content-addressed, the objects directory can be shared for multiple images, and any files that happen to have the same content are shared. I refer to this as opportunistic sharing, as it is different than the more course-grained explicit sharing used by e.g. container base images.

The next step is the validation. Note how the object files have fs-verity enabled. In fact, they are named by their fs-verity digest:

# fsverity digest objects/*/*
sha256:02927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4 objects/02/927862b4ab9fb69919187bb78d394e235ce444eeb0a890d37e955827fe4bf4
sha256:cc3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f

The generated filesystm image may contain the expected digest for the backing files. When the backing file digest is incorrect, the open will fail, and if the open succeeds, any other on-disk file-changes will be detected by fs-verity:

# cat objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
content_a
# rm -f objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# echo modified > objects/cc/3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f
# cat /mnt/file_a
WARNING: composefs backing file '3da5b14909626fc99443f580e4d8c9b990e85e0a1d18883dc89b23d43e173f' unexpectedly had no fs-verity digest
cat: /mnt/file_a: Input/output error

This re-uses the existing fs-verity functionallity to protect against changes in file contents, while adding on top of it protection against changes in filesystem metadata and structure. I.e. protecting against replacing a fs-verity enabled file or modifying file permissions or xattrs.

To be fully verified we need another step: we use fs-verity on the image itself. Then we pass the expected digest on the mount command line (which will be verified at mount time):

# fsverity enable rootfs.img
# fsverity digest rootfs.img
sha256:da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 rootfs.img
# mount -t composefs rootfs.img -o basedir=objects,digest=da42003782992856240a3e25264b19601016114775debd80c01620260af86a76 /mnt

So, given a trusted set of mount options (say unlocked from TPM), we have a fully verified filesystem tree mounted, with opportunistic finegrained sharing of identical files.

So, why do we want this? There are two initial users. First of all we want to use the opportunistic sharing for the podman container image baselayer. The idea is to use a composefs mount as the lower directory in an overlay mount, with the upper directory being the container work dir. This will allow automatical file-level disk and page-cache sharning between any two images, independent of details like the permissions and timestamps of the files.

Secondly we are interested in using the verification aspects of composefs in the ostree project. Ostree already supports a content-addressed object store, but it is currently referenced by hardlink farms. The object store and the trees that reference it are signed and verified at download time, but there is no runtime verification. If we replace the hardlink farm with a composefs image that points into the existing object store we can use the verification to implement runtime verification.

In fact, the tooling to create composefs images is 100% reproducible, so all we need is to add the composefs image fs-verity digest into the ostree commit. Then the image can be reconstructed from the ostree commit info, generating a file with the same fs-verity digest.

These are the usecases we're currently interested in, but there seems to be a breadth of other possible uses. For example, many systems use loopback mounts for images (like lxc or snap), and these could take advantage of the opportunistic sharing. We've also talked about using fuse to implement a local cache for the backing files. I.e. you would have the second basedir be a fuse filesystem. On lookup failure in the first basedir it downloads the file and saves it in the first basedir for later lookups. There are many interesting possibilities here.

The patch series contains some documentation on the file format and how to use the filesystem.

The userspace tools (and a standalone kernel module) is available here:
https://github.com/containers/composefs

Initial work on ostree integration is here:

   https://github.com/ostreedev/ostree/pull/2640

Quoted 9 lines

Changes since v2:
- Simplified filesystem format to use fixed size inodes. This resulted in simpler (now < 2k lines) code as well as higher performance at the cost of slightly (~40%) larger images.
- We now use multi-page mappings from the page cache, which removes limits on sizes of xattrs and makes the dirent handling code simpler.
- Added more documentation about the on-disk file format.
- General cleanups based on review comments.

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

I think Dave explained quite well why using overlay is not comparable to what composefs does.

Where? Can I get a link please?

I am referring to this message: https://lore.kernel.org/lkml/[email protected]/

That is a good explanation why the current container runtime overlay storage driver is inadequate, because the orchestration requires untar of OCI tarball image before mounting overlayfs.

It is not a kernel issue, it is a userspace issue, because userspace does not utilize overlayfs driver features that are now 6 years old (redirect_dir) and 4 years old (metacopy).

I completely agree that reflink and hardlinks are not a viable solution to ephemeral containers.

Quoted 25 lines

Quoted 21 lines

If there are good reasons why composefs is superior to erofs+overlayfs Please include them in the submission, since several developers keep raising the same questions - that is all I ask.

Quoted 8 lines

One big difference is that overlay still requires at least a syscall for each file in the image, and then we need the equivalent of "rm -rf" to clean it up. It is somehow acceptable for long-running services, but it is not for "serverless" containers where images/containers are created and destroyed frequently. So even in the case we already have all the image files available locally, we still need to create a checkout with the final structure we need for the image.

I think you did not understand my suggestion:

overlay read-only mount:
layer 1: erofs mount of a precomposed image (same as mkcomposefs) layer 2: any pre-existing fs path with /blocks repository layer 3: any per-existing fs path with /blocks repository
...

Quoted 3 lines

The mkcomposefs flow is exactly the same in this suggestion the upper layer image is created without any syscalls and removed without any syscalls.

Quoted 8 lines

mkcomposefs is supposed to be used server side, when the image is built. The clients that will mount the image don't have to create it (at least for images that will provide the manifest).

So this is quite different as in the overlay model we must create the layout, that is the equivalent of the composefs manifest, on any node the image is pulled to.

You don't need to re-create the erofs manifest on the client. Unless I am completely missing something, the flow that I am suggesting is drop-in replacement to what you have done.

IIUC, you invented an on-disk format for composefs manifest. Is there anything preventing you from using the existing erofs on-disk format to pack the manifest file?
The files in the manifest would be inodes with no blocks, only with size and attributes and overlay xattrs with references to the real object blocks, same as you would do with mkcomposefs. Is it not?

Maybe what I am missing is how are the blob objects distributed? Are they also shipped as composefs image bundles?
That can still be the case with erofs images that may contain both blobs with data and metadata files referencing blobs in older images.

Quoted 10 lines

Quoted 3 lines

Overlayfs already has the feature of redirecting from upper layer to relative paths in lower layers.

Could you please provide more information on how you would compose the overlay image first?

From what I can see, it still requires at least one syscall for each
file in the image to be created and these images are not portable to a different machine.

Terminology nuance - you do not create an overlayfs image on the server you create an erofs image on the server, exactly as you would create a composefs image on the server.

The shipped overlay "image" would then be the erofs image with references to prereqisite images that contain the blobs and the digest of the erofs image.

# mount -t composefs rootfs.img -o basedir=objects /mnt

client will do:

# mount -t erofs rootfs.img -o digest=da.... /metadata
# mount -t overlay -o ro,metacopy=on,lowerdir=/metadata:/objects /mnt

Quoted 4 lines

Should we always make "/blocks" a whiteout to prevent it is leaked in the container?

That would be the simplest option, yes.
If needed we can also make it a hidden layer whose objects never appear in the namespace and can only be referenced from an upper layer redirection.

Quoted 1 line

Quoted 3 lines

And what prevents files under "/blocks" to be replaced with a different version? I think fs-verity on the EROFS image itself won't cover it.

I think that part should be added to the overlayfs kernel driver. We could enhance overlayfs to include optional "overlay.verity" digest on the metacopy upper files to be fed into fsverity when opening lower blob files that reside on an fsverity supported filesystem.

Thanks,
Amir.

Quoted 7 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we
call composefs. This is the first time we propose this publically and
we would like some feedback on it.

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

Quoted 10 lines · Amir Goldstein

Quoted 7 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we
call composefs. This is the first time we propose this publically and
we would like some feedback on it.

Quoted 5 lines

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

I got lots of good feedback from Dave Chinner on V2 that caused rather large changes to simplify the format. So I wanted the new version with those changes out to continue that review. I think also having that simplified version will be helpful for the general discussion.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

I would say both versions of this can work. There are some minor technical issues with the overlay option:

* To get actual verification of the backing files you would need to add support to overlayfs for an "trusted.overlay.digest" xattrs, with behaviour similar to composefs.

* mkfs.erofs doesn't support sparse files (not sure if the kernel code does), which means it is not a good option for the backing all these sparse files. Squashfs seems to support this though, so that is an option.

However, the main issue I have with the overlayfs approach is that it is sort of clumsy and over-complex. Basically, the composefs approach is laser focused on read-only images, whereas the overlayfs approach just chains together technologies that happen to work, but also do a lot of other stuff. The result is that it is more work to use it, it uses more kernel objects (mounts, dm devices, loopbacks) and it has worse performance.

To measure performance I created a largish image (2.6 GB centos9 rootfs) and mounted it via composefs, as well as overlay-over-squashfs, both backed by the same objects directory (on xfs).

If I clear all caches between each run, a ls -lR run on composefs runs in around 700 msec:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR cfs-mount"
Benchmark 1: ls -lR cfs-mount
   Time (mean ± σ):     701.0 ms ±  21.9 ms    [User: 153.6 ms, System: 373.3 ms]
   Range (min … max):   662.3 ms … 725.3 ms    10 runs

Whereas same with overlayfs takes almost four times as long:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
   Time (mean ± σ):      2.738 s ±  0.029 s    [User: 0.176 s, System: 1.688 s]
   Range (min … max):    2.699 s …  2.787 s    10 runs

With page cache between runs the difference is smaller, but still there:

# hyperfine "ls -lR cfs-mnt"
Benchmark 1: ls -lR cfs-mnt
   Time (mean ± σ):     390.1 ms ±   3.7 ms    [User: 140.9 ms, System: 247.1 ms]
   Range (min … max):   381.5 ms … 393.9 ms    10 runs

vs

# hyperfine -i "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
   Time (mean ± σ):     431.5 ms ±   1.2 ms    [User: 124.3 ms, System: 296.9 ms]
   Range (min … max):   429.4 ms … 433.3 ms    10 runs

This isn't all that strange, as overlayfs does a lot more work for each lookup, including multiple name lookups as well as several xattr lookups, whereas composefs just does a single lookup in a pre-computed table. But, given that we don't need any of the other features of overlayfs here, this performance loss seems rather unnecessary.

Quoted 12 lines · Amir Goldstein

Quoted 7 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project we
call composefs. This is the first time we propose this publically and
we would like some feedback on it.

Hi Alexander,

Quoted 3 lines

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

I got lots of good feedback from Dave Chinner on V2 that caused rather large changes to simplify the format. So I wanted the new version with those changes out to continue that review. I think also having that simplified version will be helpful for the general discussion.

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

So, to be more precise, and so that everyone is on the same page, lemme state the two options in full.

For both options, we have a directory "objects" with content-addressed backing files (i.e. files named by sha256). In this directory all files have fs-verity enabled. Additionally there is an image file which you downloaded to the system that somehow references the objects directory by relative filenames.

Composefs option:

The image file has fs-verity enabled. To use the image, you mount it with options "basedir=objects,digest=$imagedigest".

Overlayfs option:

The image file is a loopback image of a gpt disk with two partitions, one partition contains the dm-verity hashes, and the other contains some read-only filesystem.

The read-only filesystem has regular versions of directories and symlinks, but for regular files it has sparse files with the xattrs "trusted.overlay.metacopy" and "trusted.overlay.redirect" set, the later containing a string like like "/de/adbeef..." referencing a backing file in the "objects" directory. In addition, the image also contains overlayfs whiteouts to cover any toplevel filenames from the objects directory that would otherwise appear if objects is used as a lower dir.

To use this you loopback mount the file, and use dm-verity to set up the combined partitions, which you then mount somewhere. Then you mount an overlayfs with options:
"metacopy=on,redirect_dir=follow,lowerdir=veritydev:objects"

I would say both versions of this can work. There are some minor technical issues with the overlay option:

* To get actual verification of the backing files you would need to add support to overlayfs for an "trusted.overlay.digest" xattrs, with behaviour similar to composefs.

* mkfs.erofs doesn't support sparse files (not sure if the kernel code does), which means it is not a good option for the backing all these sparse files. Squashfs seems to support this though, so that is an option.

However, the main issue I have with the overlayfs approach is that it is sort of clumsy and over-complex. Basically, the composefs approach is laser focused on read-only images, whereas the overlayfs approach just chains together technologies that happen to work, but also do a lot of other stuff. The result is that it is more work to use it, it uses more kernel objects (mounts, dm devices, loopbacks) and it has

worse performance.

To measure performance I created a largish image (2.6 GB centos9 rootfs) and mounted it via composefs, as well as overlay-over-squashfs, both backed by the same objects directory (on xfs).

If I clear all caches between each run, a ls -lR run on composefs runs in around 700 msec:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR cfs-mount"
Benchmark 1: ls -lR cfs-mount
  Time (mean ± σ):     701.0 ms ±  21.9 ms    [User: 153.6 ms, System: 373.3 ms]
  Range (min … max):   662.3 ms … 725.3 ms    10 runs

Whereas same with overlayfs takes almost four times as long:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean ± σ):      2.738 s ±  0.029 s    [User: 0.176 s, System: 1.688 s]
  Range (min … max):    2.699 s …  2.787 s    10 runs

With page cache between runs the difference is smaller, but still there:

# hyperfine "ls -lR cfs-mnt"
Benchmark 1: ls -lR cfs-mnt
  Time (mean ± σ):     390.1 ms ±   3.7 ms    [User: 140.9 ms, System: 247.1 ms]
  Range (min … max):   381.5 ms … 393.9 ms    10 runs

vs

# hyperfine -i "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean ± σ):     431.5 ms ±   1.2 ms    [User: 124.3 ms, System: 296.9 ms]
  Range (min … max):   429.4 ms … 433.3 ms    10 runs

This isn't all that strange, as overlayfs does a lot more work for each lookup, including multiple name lookups as well as several xattr lookups, whereas composefs just does a single lookup in a pre-computed

table. But, given that we don't need any of the other features of overlayfs here, this performance loss seems rather unnecessary.

I understand that there is a cost to adding more code, but efficiently supporting containers and other forms of read-only images is a pretty important usecase for Linux these days, and having something tailored for that seems pretty useful to me, even considering the code duplication.

I also understand Cristians worry about stacking filesystem, having looked a bit more at the overlayfs code. But, since composefs doesn't really expose the metadata or vfs structure of the lower directories it is much simpler in a fundamental way.

Quoted 21 lines · Alexander Larsson

Quoted 13 lines · Amir Goldstein

Quoted 9 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project
we
call composefs. This is the first time we propose this publically
and
we would like some feedback on it.

Quoted 5 lines

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 7 lines

I got lots of good feedback from Dave Chinner on V2 that caused rather
large changes to simplify the format. So I wanted the new version with
those changes out to continue that review. I think also having that simplified version will be helpful for the general discussion.

That's ok.
I was not puzzled about why you posted v3.
I was puzzled by why you did not mention anything about the alternatives to adding a new filesystem that were discussed on v2 and argue in favor of the new filesystem option. If you post another version, please make sure to include a good explanation for that.

Quoted 60 lines

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

So, to be more precise, and so that everyone is on the same page, lemme
state the two options in full.

For both options, we have a directory "objects" with content- addressed
backing files (i.e. files named by sha256). In this directory all files have fs-verity enabled. Additionally there is an image file which you downloaded to the system that somehow references the objects
directory by relative filenames.

Composefs option:

 The image file has fs-verity enabled. To use the image, you mount it
 with options "basedir=objects,digest=$imagedigest".

Overlayfs option:

 The image file is a loopback image of a gpt disk with two partitions,
 one partition contains the dm-verity hashes, and the other contains
 some read-only filesystem.

 The read-only filesystem has regular versions of directories and symlinks, but for regular files it has sparse files with the xattrs
 "trusted.overlay.metacopy" and "trusted.overlay.redirect" set, the later containing a string like like "/de/adbeef..." referencing a backing file in the "objects" directory. In addition, the image also
 contains overlayfs whiteouts to cover any toplevel filenames from the
 objects directory that would otherwise appear if objects is used as
 a lower dir.

 To use this you loopback mount the file, and use dm-verity to set up
 the combined partitions, which you then mount somewhere. Then you mount an overlayfs with options:
  "metacopy=on,redirect_dir=follow,lowerdir=veritydev:objects"

I would say both versions of this can work. There are some minor technical issues with the overlay option:

* To get actual verification of the backing files you would need to add support to overlayfs for an "trusted.overlay.digest" xattrs, with
behaviour similar to composefs.

Quoted 7 lines

* mkfs.erofs doesn't support sparse files (not sure if the kernel code
does), which means it is not a good option for the backing all these
sparse files. Squashfs seems to support this though, so that is an option.

Fair enough.
Wasn't expecting for things to work without any changes. Let's first agree that these alone are not a good enough reason to introduce a new filesystem.
Let's move on..

Quoted 11 lines

However, the main issue I have with the overlayfs approach is that it
is sort of clumsy and over-complex. Basically, the composefs approach
is laser focused on read-only images, whereas the overlayfs approach
just chains together technologies that happen to work, but also do a
lot of other stuff. The result is that it is more work to use it, it
uses more kernel objects (mounts, dm devices, loopbacks) and it has

Up to this point, it's just hand waving, and a bit annoying if I am being honest.
overlayfs+metacopy feature were created for the containers use case for very similar set of requirements - they do not just "happen to work"
for the same use case.
Please stick to technical arguments when arguing in favor of the new "laser focused" filesystem option.

Quoted 17 lines

worse performance.

To measure performance I created a largish image (2.6 GB centos9 rootfs) and mounted it via composefs, as well as overlay-over- squashfs,
both backed by the same objects directory (on xfs).

If I clear all caches between each run, a ls -lR run on composefs runs in around 700 msec:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR cfs-
mount"
Benchmark 1: ls -lR cfs-mount
  Time (mean ± σ):     701.0 ms ±  21.9 ms    [User: 153.6 ms,
System: 373.3 ms]
  Range (min … max):   662.3 ms … 725.3 ms    10 runs

Quoted 1 line

Whereas same with overlayfs takes almost four times as long:

No it is not overlayfs, it is overlayfs+squashfs, please stick to facts.
As Gao wrote, squashfs does not optimize directory lookup. You can run a test with ext4 for POC as Gao suggested.
I am sure that mkfs.erofs sparse file support can be added if needed.

Quoted 7 lines

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR ovl-
mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean ± σ):      2.738 s ±  0.029 s    [User: 0.176 s,
System: 1.688 s]
  Range (min … max):    2.699 s …  2.787 s    10 runs

Quoted 2 lines

With page cache between runs the difference is smaller, but still there:

It is the dentry cache that mostly matters for this test and please use hyerfine -w 1 to warmup dentry cache for correct measurement of warm cache lookup.

I guess these test runs started with warm cache? but it wasn't mentioned explicitly.

Quoted 14 lines

# hyperfine "ls -lR cfs-mnt"
Benchmark 1: ls -lR cfs-mnt
  Time (mean ± σ):     390.1 ms ±   3.7 ms    [User: 140.9 ms,
System: 247.1 ms]
  Range (min … max):   381.5 ms … 393.9 ms    10 runs

vs

# hyperfine -i "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean ± σ):     431.5 ms ±   1.2 ms    [User: 124.3 ms,
System: 296.9 ms]
  Range (min … max):   429.4 ms … 433.3 ms    10 runs

Quoted 5 lines

This isn't all that strange, as overlayfs does a lot more work for each lookup, including multiple name lookups as well as several xattr
lookups, whereas composefs just does a single lookup in a pre- computed

Seriously, "multiple name lookups"?
Overlayfs does exactly one lookup for anything but first level subdirs
and for sparse files it does the exact same lookup in /objects as composefs.
Enough with the hand waving please. Stick to hard facts.

Quoted 14 lines

table. But, given that we don't need any of the other features of overlayfs here, this performance loss seems rather unnecessary.

I understand that there is a cost to adding more code, but efficiently
supporting containers and other forms of read-only images is a pretty
important usecase for Linux these days, and having something tailored
for that seems pretty useful to me, even considering the code duplication.

Quoted 7 lines

I also understand Cristians worry about stacking filesystem, having looked a bit more at the overlayfs code. But, since composefs doesn't
really expose the metadata or vfs structure of the lower directories it
is much simpler in a fundamental way.

I agree that composefs is simpler than overlayfs and that its security
model is simpler, but this is not the relevant question. The question is what are the benefits to the prospect users of composefs
that justify this new filesystem driver if overlayfs already implements
the needed functionality.

The only valid technical argument I could gather from your email is - 10% performance improvement in warm cache ls -lR on a 2.6 GB centos9 rootfs image compared to overlayfs+squashfs.

I am not counting the cold cache results until we see results of a modern ro-image fs.

Considering that most real life workloads include reading the data and that most of the time inodes and dentries are cached, IMO, the 10% ls -lR improvement is not a good enough reason for a new "laser focused" filesystem driver.

Correct me if I am wrong, but isn't the use case of ephemeral containers require that composefs is layered under a writable tmpfs using overlayfs?

If that is the case then the warm cache comparison is incorrect as well. To argue for the new filesystem you will need to compare ls -lR of overlay{tmpfs,composefs,xfs} vs. overlay{tmpfs,erofs,xfs}

Alexander,

On a more personal note, I know this discussion has been a bit stormy, but am not trying to fight you.

I think that {mk,}composefs is a wonderful thing that will improve the life of many users.
But mount -t composefs vs. mount -t overlayfs is insignificant to those users, so we just need to figure out based on facts and numbers, which is the best technical alternative.

Quoted 1 line · Amir Goldstein

On Mon, Jan 23, 2023 at 7:56 PM Alexander Larsson <[email protected]>

Quoted 6 lines

No it is not overlayfs, it is overlayfs+squashfs, please stick to facts.
As Gao wrote, squashfs does not optimize directory lookup. You can run a test with ext4 for POC as Gao suggested.
I am sure that mkfs.erofs sparse file support can be added if needed.

New measurements follow, they now include also erofs over loopback, although that isn't strictly fair, because that image is much larger due to the fact that it didn't store the files sparsely. It also includes a version where the topmost lower is directly on the backing xfs (i.e. not via loopback). I attached the scripts used to create the images and do the profiling in case anyone wants to reproduce.

Here are the results (on x86-64, xfs base fs):

overlayfs + loopback squashfs - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 2.483 s ± 0.029 s [User: 0.167 s, System: 1.656 s]
Range (min … max): 2.427 s … 2.530 s 10 runs

overlayfs + loopback squashfs - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 429.2 ms ± 4.6 ms [User: 123.6 ms, System: 295.0 ms]
Range (min … max): 421.2 ms … 435.3 ms 10 runs

overlayfs + loopback ext4 - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 4.332 s ± 0.060 s [User: 0.204 s, System: 3.150 s]
Range (min … max): 4.261 s … 4.442 s 10 runs

overlayfs + loopback ext4 - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 528.3 ms ± 4.0 ms [User: 143.4 ms, System: 381.2 ms]
Range (min … max): 521.1 ms … 536.4 ms 10 runs

overlayfs + loopback erofs - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 3.045 s ± 0.127 s [User: 0.198 s, System: 1.129 s]
Range (min … max): 2.926 s … 3.338 s 10 runs

overlayfs + loopback erofs - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 516.9 ms ± 5.7 ms [User: 139.4 ms, System: 374.0 ms]
Range (min … max): 503.6 ms … 521.9 ms 10 runs

overlayfs + direct - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 2.562 s ± 0.028 s [User: 0.199 s, System: 1.129 s]
Range (min … max): 2.497 s … 2.585 s 10 runs

overlayfs + direct - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 524.5 ms ± 1.6 ms [User: 148.7 ms, System: 372.2 ms]
Range (min … max): 522.8 ms … 527.8 ms 10 runs

composefs - uncached
Benchmark 1: ls -lR mnt-fs
Time (mean ± σ): 681.4 ms ± 14.1 ms [User: 154.4 ms, System: 369.9 ms]
Range (min … max): 652.5 ms … 703.2 ms 10 runs

composefs - cached
Benchmark 1: ls -lR mnt-fs
Time (mean ± σ): 390.8 ms ± 4.7 ms [User: 144.7 ms, System: 243.7 ms]
Range (min … max): 382.8 ms … 399.1 ms 10 runs

For the uncached case, composefs is still almost four times faster than the fastest overlay combo (squashfs), and the non-squashfs versions are strictly slower. For the cached case the difference is less (10%) but with similar order of performance.

For size comparison, here are the resulting images:

8.6M large.composefs
2.5G large.erofs
200M large.ext4
2.6M large.squashfs

Quoted 31 lines · Amir Goldstein

Quoted 21 lines · Alexander Larsson

Quoted 13 lines · Amir Goldstein

Quoted 9 lines · Alexander Larsson

Giuseppe Scrivano and I have recently been working on a new project
we
call composefs. This is the first time we propose this publically
and
we would like some feedback on it.

Quoted 5 lines

Hi Alexander,

I must say that I am a little bit puzzled by this v3. Gao, Christian and myself asked you questions on v2 that are not mentioned in v3 at all.

Quoted 7 lines

I got lots of good feedback from Dave Chinner on V2 that caused rather
large changes to simplify the format. So I wanted the new version with
those changes out to continue that review. I think also having that simplified version will be helpful for the general discussion.

Quoted 7 lines

That's ok.
I was not puzzled about why you posted v3.
I was puzzled by why you did not mention anything about the alternatives to adding a new filesystem that were discussed on v2 and argue in favor of the new filesystem option. If you post another version, please make sure to include a good explanation for that.

Sure, I will add something to the next version. But like, there was already a discussion about this, duplicating that discussion in the v3 announcement when the v2->v3 changes are unrelated to it doesn't seem like it makes a ton of difference.

Quoted 68 lines

Quoted 60 lines

Quoted 6 lines

To sum it up, please do not propose composefs without explaining what are the barriers for achieving the exact same outcome with the use of a read-only overlayfs with two lower layer - uppermost with erofs containing the metadata files, which include trusted.overlay.metacopy and trusted.overlay.redirect xattrs that refer to the lowermost layer containing the content files.

So, to be more precise, and so that everyone is on the same page, lemme
state the two options in full.

For both options, we have a directory "objects" with content- addressed
backing files (i.e. files named by sha256). In this directory all files have fs-verity enabled. Additionally there is an image file which you downloaded to the system that somehow references the objects
directory by relative filenames.

Composefs option:

The image file has fs-verity enabled. To use the image, you mount
it
with options "basedir=objects,digest=$imagedigest".

Overlayfs option:

The image file is a loopback image of a gpt disk with two
partitions,
one partition contains the dm-verity hashes, and the other
contains
some read-only filesystem.

The read-only filesystem has regular versions of directories and symlinks, but for regular files it has sparse files with the
xattrs
"trusted.overlay.metacopy" and "trusted.overlay.redirect" set, the later containing a string like like "/de/adbeef..." referencing a backing file in the "objects" directory. In addition, the image
also
contains overlayfs whiteouts to cover any toplevel filenames from
the
objects directory that would otherwise appear if objects is used
as
a lower dir.

To use this you loopback mount the file, and use dm-verity to set
up
the combined partitions, which you then mount somewhere. Then you mount an overlayfs with options:
"metacopy=on,redirect_dir=follow,lowerdir=veritydev:objects"

I would say both versions of this can work. There are some minor technical issues with the overlay option:

* To get actual verification of the backing files you would need to add support to overlayfs for an "trusted.overlay.digest" xattrs, with
behaviour similar to composefs.

Quoted 7 lines

* mkfs.erofs doesn't support sparse files (not sure if the kernel code
does), which means it is not a good option for the backing all these
sparse files. Squashfs seems to support this though, so that is an option.

Quoted 5 lines

Fair enough.
Wasn't expecting for things to work without any changes. Let's first agree that these alone are not a good enough reason to introduce a new filesystem.
Let's move on..

Yeah.

Quoted 38 lines

Quoted 11 lines

However, the main issue I have with the overlayfs approach is that it
is sort of clumsy and over-complex. Basically, the composefs approach
is laser focused on read-only images, whereas the overlayfs approach
just chains together technologies that happen to work, but also do a
lot of other stuff. The result is that it is more work to use it, it
uses more kernel objects (mounts, dm devices, loopbacks) and it has

Up to this point, it's just hand waving, and a bit annoying if I am being honest.
overlayfs+metacopy feature were created for the containers use case for very similar set of requirements - they do not just "happen to work"
for the same use case.
Please stick to technical arguments when arguing in favor of the new "laser focused" filesystem option.

Quoted 17 lines

worse performance.

To measure performance I created a largish image (2.6 GB centos9 rootfs) and mounted it via composefs, as well as overlay-over- squashfs,
both backed by the same objects directory (on xfs).

If I clear all caches between each run, a ls -lR run on composefs runs in around 700 msec:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR cfs-
mount"
Benchmark 1: ls -lR cfs-mount
  Time (mean ± σ):     701.0 ms ±  21.9 ms    [User: 153.6 ms,
System: 373.3 ms]
  Range (min … max):   662.3 ms … 725.3 ms    10 runs

Quoted 7 lines

Quoted 1 line

Whereas same with overlayfs takes almost four times as long:

No it is not overlayfs, it is overlayfs+squashfs, please stick to facts.
As Gao wrote, squashfs does not optimize directory lookup. You can run a test with ext4 for POC as Gao suggested.
I am sure that mkfs.erofs sparse file support can be added if needed.

New measurements follow, they now include also erofs over loopback, although that isn't strictly fair, because that image is much larger due to the fact that it didn't store the files sparsely. It also includes a version where the topmost lower is directly on the backing xfs (i.e. not via loopback). I attached the scripts used to create the images and do the profiling in case anyone wants to reproduce.

Here are the results (on x86-64, xfs base fs):

overlayfs + loopback squashfs - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 2.483 s ± 0.029 s [User: 0.167 s, System: 1.656 s]
Range (min … max): 2.427 s … 2.530 s 10 runs

overlayfs + loopback squashfs - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 429.2 ms ± 4.6 ms [User: 123.6 ms, System: 295.0 ms]
Range (min … max): 421.2 ms … 435.3 ms 10 runs

overlayfs + loopback ext4 - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 4.332 s ± 0.060 s [User: 0.204 s, System: 3.150 s]
Range (min … max): 4.261 s … 4.442 s 10 runs

overlayfs + loopback ext4 - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 528.3 ms ± 4.0 ms [User: 143.4 ms, System: 381.2 ms]
Range (min … max): 521.1 ms … 536.4 ms 10 runs

overlayfs + loopback erofs - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 3.045 s ± 0.127 s [User: 0.198 s, System: 1.129 s]
Range (min … max): 2.926 s … 3.338 s 10 runs

overlayfs + loopback erofs - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 516.9 ms ± 5.7 ms [User: 139.4 ms, System: 374.0 ms]
Range (min … max): 503.6 ms … 521.9 ms 10 runs

overlayfs + direct - uncached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 2.562 s ± 0.028 s [User: 0.199 s, System: 1.129 s]
Range (min … max): 2.497 s … 2.585 s 10 runs

overlayfs + direct - cached
Benchmark 1: ls -lR mnt-ovl
Time (mean ± σ): 524.5 ms ± 1.6 ms [User: 148.7 ms, System: 372.2 ms]
Range (min … max): 522.8 ms … 527.8 ms 10 runs

composefs - uncached
Benchmark 1: ls -lR mnt-fs
Time (mean ± σ): 681.4 ms ± 14.1 ms [User: 154.4 ms, System: 369.9 ms]
Range (min … max): 652.5 ms … 703.2 ms 10 runs

composefs - cached
Benchmark 1: ls -lR mnt-fs
Time (mean ± σ): 390.8 ms ± 4.7 ms [User: 144.7 ms, System: 243.7 ms]
Range (min … max): 382.8 ms … 399.1 ms 10 runs

For the uncached case, composefs is still almost four times faster than the fastest overlay combo (squashfs), and the non-squashfs versions are strictly slower. For the cached case the difference is less (10%) but with similar order of performance.

For size comparison, here are the resulting images:

8.6M large.composefs
2.5G large.erofs
200M large.ext4
2.6M large.squashfs

Quoted 7 lines

Quoted 7 lines

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR ovl-
mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean ± σ):      2.738 s ±  0.029 s    [User: 0.176 s,
System: 1.688 s]
  Range (min … max):    2.699 s …  2.787 s    10 runs

Quoted 6 lines

Quoted 2 lines

With page cache between runs the difference is smaller, but still there:

It is the dentry cache that mostly matters for this test and please use hyerfine -w 1 to warmup dentry cache for correct measurement of warm cache lookup.

I'm not sure why the dentry cache case would be more important? Starting a new container will very often not have cached the image.

To me the interesting case is for a new image, but with some existing page cache for the backing files directory. That seems to model staring a new image in an active container host, but its somewhat hard to test that case.

Quoted 2 lines

I guess these test runs started with warm cache? but it wasn't mentioned explicitly.

Yes, they were warm (because I ran the previous test before it). But, the new profile script explicitly adds -w 1.

Quoted 20 lines

Quoted 14 lines

# hyperfine "ls -lR cfs-mnt"
Benchmark 1: ls -lR cfs-mnt
  Time (mean ± σ):     390.1 ms ±   3.7 ms    [User: 140.9 ms,
System: 247.1 ms]
  Range (min … max):   381.5 ms … 393.9 ms    10 runs

vs

# hyperfine -i "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean ± σ):     431.5 ms ±   1.2 ms    [User: 124.3 ms,
System: 296.9 ms]
  Range (min … max):   429.4 ms … 433.3 ms    10 runs

Quoted 5 lines

This isn't all that strange, as overlayfs does a lot more work for each lookup, including multiple name lookups as well as several xattr
lookups, whereas composefs just does a single lookup in a pre- computed

Quoted 6 lines

Seriously, "multiple name lookups"?
Overlayfs does exactly one lookup for anything but first level subdirs
and for sparse files it does the exact same lookup in /objects as composefs.
Enough with the hand waving please. Stick to hard facts.

With the discussed layout, in a stat() call on a regular file, ovl_lookup() will do lookups on both the sparse file and the backing file, whereas cfs_dir_lookup() will just map some page cache pages and do a binary search.

Of course if you actually open the file, then cfs_open_file() would do the equivalent lookups in /objects. But that is often not what happens, for example in "ls -l".

Additionally, these extra lookups will cause extra memory use, as you need dentries and inodes for the erofs/squashfs inodes in addition to the overlay inodes.

Quoted 31 lines

Quoted 14 lines

table. But, given that we don't need any of the other features of overlayfs here, this performance loss seems rather unnecessary.

I understand that there is a cost to adding more code, but efficiently
supporting containers and other forms of read-only images is a pretty
important usecase for Linux these days, and having something tailored
for that seems pretty useful to me, even considering the code duplication.

Quoted 7 lines

I also understand Cristians worry about stacking filesystem, having looked a bit more at the overlayfs code. But, since composefs doesn't
really expose the metadata or vfs structure of the lower directories it
is much simpler in a fundamental way.

I agree that composefs is simpler than overlayfs and that its security
model is simpler, but this is not the relevant question. The question is what are the benefits to the prospect users of composefs
that justify this new filesystem driver if overlayfs already implements
the needed functionality.

Quoted 6 lines

The only valid technical argument I could gather from your email is - 10% performance improvement in warm cache ls -lR on a 2.6 GB centos9 rootfs image compared to overlayfs+squashfs.

I am not counting the cold cache results until we see results of a modern ro-image fs.

They are all strictly worse than squashfs in the above testing.

Quoted 5 lines

Considering that most real life workloads include reading the data and that most of the time inodes and dentries are cached, IMO, the 10% ls -lR improvement is not a good enough reason for a new "laser focused" filesystem driver.

Quoted 7 lines

Correct me if I am wrong, but isn't the use case of ephemeral containers require that composefs is layered under a writable tmpfs using overlayfs?

If that is the case then the warm cache comparison is incorrect as well. To argue for the new filesystem you will need to compare ls -lR of overlay{tmpfs,composefs,xfs} vs. overlay{tmpfs,erofs,xfs}

That very much depends. For the ostree rootfs uscase there would be no writable layer, and for containers I'm personally primarily interested in "--readonly" containers (i.e. without an writable layer) in my current automobile/embedded work. For many container cases however, that is true, and no doubt that would make the overhead of overlayfs less of a issue.

Quoted 4 lines

Alexander,

On a more personal note, I know this discussion has been a bit stormy, but am not trying to fight you.

I'm overall not getting a warm fuzzy feeling from this discussion. Getting weird complaints that I'm somehow "stealing" functions or weird "who did $foo first" arguments for instance. You haven't personally attacked me like that, but some of your comments can feel rather pointy, especially in the context of a stormy thread like this. I'm just not used to kernel development workflows, so have patience with me if I do things wrong.

Quoted 5 lines

I think that {mk,}composefs is a wonderful thing that will improve the life of many users.
But mount -t composefs vs. mount -t overlayfs is insignificant to those users, so we just need to figure out based on facts and numbers, which is the best technical alternative.

In reality things are never as easy as one thing strictly being technically best. There is always a multitude of considerations. Is composefs technically better if it uses less memory and performs better for a particular usecase? Or is overlayfs technically better because it is useful for more usecases and already exists? A judgement needs to be made depending on things like complexity/maintainability of the new fs, ease of use, measured performance differences, relative importance of particular performance measurements, and importance of the specific usecase.

It is my belief that the advantages of composefs outweight the cost of the code duplication, but I understand the point of view of a maintainer of an existing codebase and that saying "no" is often the right thing. I will continue to try to argue for my point of view, but will try to make it as factual as possible.

Quoted 7 lines · Alexander Larsson

Quoted 6 lines · Amir Goldstein

Quoted 4 lines · Alexander Larsson

Quoted 3 lines · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson
<[email protected]>
wrote:

Quoted 8 lines

I'm not sure why the dentry cache case would be more important? Starting a new container will very often not have cached the image.

To me the interesting case is for a new image, but with some existing page cache for the backing files directory. That seems to model staring a new image in an active container host, but its somewhat hard to test that case.

ok, you can argue that faster cold cache ls -lR is important for starting new images.
I think you will be asked to show a real life container use case where that benchmark really matters.

Quoted 22 lines

Quoted 6 lines

Quoted 5 lines

This isn't all that strange, as overlayfs does a lot more work for each lookup, including multiple name lookups as well as several xattr
lookups, whereas composefs just does a single lookup in a pre- computed

Quoted 6 lines

Seriously, "multiple name lookups"?
Overlayfs does exactly one lookup for anything but first level subdirs
and for sparse files it does the exact same lookup in /objects as composefs.
Enough with the hand waving please. Stick to hard facts.

With the discussed layout, in a stat() call on a regular file, ovl_lookup() will do lookups on both the sparse file and the backing file, whereas cfs_dir_lookup() will just map some page cache pages and do a binary search.

Of course if you actually open the file, then cfs_open_file() would do the equivalent lookups in /objects. But that is often not what happens, for example in "ls -l".

Quoted 3 lines

Additionally, these extra lookups will cause extra memory use, as you need dentries and inodes for the erofs/squashfs inodes in addition to the overlay inodes.

I see. composefs is really very optimized for ls -lR.

Now only need to figure out if real users start a container and do ls -lR without reading many files is a real life use case.

Quoted 17 lines · Amir Goldstein

Quoted 7 lines · Alexander Larsson

Quoted 6 lines · Amir Goldstein

Quoted 4 lines · Alexander Larsson

Quoted 3 lines · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson
<[email protected]>
wrote:

Quoted 8 lines

I'm not sure why the dentry cache case would be more important? Starting a new container will very often not have cached the image.

To me the interesting case is for a new image, but with some existing page cache for the backing files directory. That seems to model staring a new image in an active container host, but its somewhat hard to test that case.

Quoted 4 lines

ok, you can argue that faster cold cache ls -lR is important for starting new images.
I think you will be asked to show a real life container use case where that benchmark really matters.

I've already described the real world production system bottlenecks that composefs is designed to overcome in a previous thread.

Please go back an read this:

https://lore.kernel.org/linux-fsdevel/[email protected]/

Cold cache performance dominates the runtime of short lived containers as well as high density container hosts being run to their container level memory limits. ls -lR is just a microbenchmark that demonstrates how much better composefs cold cache behaviour is than the alternatives being proposed....

This might also help explain why my initial review comments focussed on getting rid of optional format features, straight lining the processing, changing the format or search algorithms so more sequential cacheline accesses occurred resulting in less memory stalls, etc. i.e. reductions in cold cache lookup overhead will directly translate into faster container workload spin up.

Quoted 22 lines

Quoted 22 lines

Quoted 6 lines

Quoted 5 lines

This isn't all that strange, as overlayfs does a lot more work for each lookup, including multiple name lookups as well as several xattr
lookups, whereas composefs just does a single lookup in a pre- computed

Quoted 6 lines

Seriously, "multiple name lookups"?
Overlayfs does exactly one lookup for anything but first level subdirs
and for sparse files it does the exact same lookup in /objects as composefs.
Enough with the hand waving please. Stick to hard facts.

With the discussed layout, in a stat() call on a regular file, ovl_lookup() will do lookups on both the sparse file and the backing file, whereas cfs_dir_lookup() will just map some page cache pages and do a binary search.

Of course if you actually open the file, then cfs_open_file() would do the equivalent lookups in /objects. But that is often not what happens, for example in "ls -l".

Quoted 5 lines

Quoted 3 lines

Additionally, these extra lookups will cause extra memory use, as you need dentries and inodes for the erofs/squashfs inodes in addition to the overlay inodes.

I see. composefs is really very optimized for ls -lR.

No, composefs is optimised for minimal namespace and inode resolution overhead. 'ls -lR' does a lot of these operations, and therefore you see the efficiency of the design being directly exposed....

Quoted 2 lines

Now only need to figure out if real users start a container and do ls -lR without reading many files is a real life use case.

I've been using 'ls -lR' and 'find . -ctime 1' to benchmark cold cache directory iteration and inode lookup performance for roughly 20 years. The benchmarks I run *never* read file data, nor is that desired - they are pure directory and inode lookup micro-benchmarks used to analyse VFS and filesystem directory and inode lookup performance.

I have been presenting such measurements and patches improving performance of these microbnechmarks to the XFS and fsdevel lists over 15 years and I have *never* had to justify that what I'm measuring is a "real world workload" to anyone. Ever.

Complaining about real world relevancy of the presented benchmark might be considered applying a double standard, wouldn't you agree?

Quoted 10 lines · Dave Chinner

I've already described the real world production system bottlenecks that composefs is designed to overcome in a previous thread.

Please go back an read this:

https://lore.kernel.org/linux-fsdevel/[email protected]/

I've read it and now re-read it.
Most of the post talks about the excess time of creating the namespace,
which is addressed by erofs+overlayfs.

I guess you mean this requirement:
"When you have container instances that might only be needed for a few seconds, taking half a minute to set up the container instance and then another half a minute to tear it down just isn't viable - we need instantiation and teardown times in the order of a second or two."

Forgive for not being part of the containers world, so I have to ask -
Which real life use case requires instantiation and teardown times in the order of a second?

What is the order of number of files in the manifest of those ephemeral
images?

The benchmark was done on a 2.6GB centos9 image.

My very minimal understanding of containers world, is that A large centos9 image would be used quite often on a client so it would be deployed as created inodes in disk filesystem and the ephemeral images are likely to be small changes on top of those large base images.

Furthermore, the ephmeral images would likely be composed of cenos9 + several layers, so the situation of single composefs image as large as centos9 is highly unlikely.

Am I understanding the workflow correctly?

If I am, then I would rather see benchmarks with images that correspond with the real life use case that drives composefs, such as small manifests and/or composefs in combination with overlayfs as it would be used more often.

Quoted 6 lines

Cold cache performance dominates the runtime of short lived containers as well as high density container hosts being run to their container level memory limits. ls -lR is just a microbenchmark that demonstrates how much better composefs cold cache behaviour is than the alternatives being proposed....

Quoted 8 lines

This might also help explain why my initial review comments focussed
on getting rid of optional format features, straight lining the processing, changing the format or search algorithms so more sequential cacheline accesses occurred resulting in less memory stalls, etc. i.e. reductions in cold cache lookup overhead will directly translate into faster container workload spin up.

I agree that this technology is novel and understand why it results in faster cold cache lookup.
I do not know erofs enough to say if similar techniques could be applied to optimize erofs lookup at mkfs.erofs time, but I can guess that this optimization was never attempted.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 30 lines · Amir Goldstein

Quoted 10 lines · Dave Chinner

I've already described the real world production system bottlenecks that composefs is designed to overcome in a previous thread.

Please go back an read this:

https://lore.kernel.org/linux-fsdevel/[email protected]/

I've read it and now re-read it.
Most of the post talks about the excess time of creating the namespace,
which is addressed by erofs+overlayfs.

I guess you mean this requirement:
"When you have container instances that might only be needed for a few seconds, taking half a minute to set up the container instance and then another half a minute to tear it down just isn't viable - we need instantiation and teardown times in the order of a second or two."

Forgive for not being part of the containers world, so I have to ask -
Which real life use case requires instantiation and teardown times in the order of a second?

Quoted 5 lines

What is the order of number of files in the manifest of those ephemeral
images?

The benchmark was done on a 2.6GB centos9 image.

What does this matter? We want to measure a particular kind of operation, so, we use a sample with a lot of those operations. What would it help running some operation on a smaller image that does much less of the critical operations. That would just make it harder to see the data for all the noise. Nobody is saying that reading all the metadata in a 2.6GB image is something a container would do. It is however doing lots of the operations that constrains container startup, and it allows us to compare the performance of these operation between different alternatives.

Quoted 11 lines

My very minimal understanding of containers world, is that A large centos9 image would be used quite often on a client so it would be deployed as created inodes in disk filesystem and the ephemeral images are likely to be small changes on top of those large base images.

Furthermore, the ephmeral images would likely be composed of cenos9 + several layers, so the situation of single composefs image as large as centos9 is highly unlikely.

Am I understanding the workflow correctly?

In a composefs based container storage implementation one would likely not use a layered approach for the "derived" images. Since all file content is shared anyway its more useful to just combine the metadata of the layers into a single composefs image. It is not going to be very large anyway, and it will make lookups much faster as you don't need to do all the negative lookups in the upper layers when looking for files in the base layer.

Quoted 4 lines

If I am, then I would rather see benchmarks with images that correspond with the real life use case that drives composefs, such as small manifests and/or composefs in combination with overlayfs as it would be used more often.

I feel like there is a constant moving of the goal post here. I've provided lots of raw performance numbers, and explained that they are important to our usecases, there has to be an end to how detailed they need to be. I'm not interested in implementing a complete container runtime based on overlayfs just to show that it performs poorly.

Quoted 15 lines

Quoted 6 lines

Cold cache performance dominates the runtime of short lived containers as well as high density container hosts being run to their container level memory limits. ls -lR is just a microbenchmark that demonstrates how much better composefs cold cache behaviour is than the alternatives being proposed....

Quoted 8 lines

This might also help explain why my initial review comments focussed
on getting rid of optional format features, straight lining the processing, changing the format or search algorithms so more sequential cacheline accesses occurred resulting in less memory stalls, etc. i.e. reductions in cold cache lookup overhead will directly translate into faster container workload spin up.

Quoted 7 lines

I agree that this technology is novel and understand why it results in faster cold cache lookup.
I do not know erofs enough to say if similar techniques could be applied to optimize erofs lookup at mkfs.erofs time, but I can guess that this optimization was never attempted.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

On the contrary, erofs lookup is very similar to composefs. There is nothing magical about it, we're talking about pre-computed, static lists of names. What you do is you sort the names, put them in a compact seek-free form, and then you binary search on them. Composefs v3 has some changes to make larger directories slightly more efficient (no chunking), but the general performance should be comparable.

I believe Gao said that mkfs.erofs could do slightly better at how data arranged so that related things are closer to each other. That may help some, but I don't think this is gonna be a massive difference.

Quoted 24 lines · Dave Chinner

Quoted 17 lines · Amir Goldstein

Quoted 7 lines · Alexander Larsson

Quoted 6 lines · Amir Goldstein

Quoted 4 lines · Alexander Larsson

Quoted 3 lines · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson
<[email protected]>
wrote:

Quoted 8 lines

I'm not sure why the dentry cache case would be more important? Starting a new container will very often not have cached the image.

To me the interesting case is for a new image, but with some existing page cache for the backing files directory. That seems to model staring a new image in an active container host, but its somewhat hard to test that case.

Quoted 4 lines

ok, you can argue that faster cold cache ls -lR is important for starting new images.
I think you will be asked to show a real life container use case where that benchmark really matters.

Quoted 7 lines

I've already described the real world production system bottlenecks that composefs is designed to overcome in a previous thread.

Please go back an read this:

https://lore.kernel.org/linux-fsdevel/[email protected]/

I've read it and now re-read it.
Most of the post talks about the excess time of creating the namespace, which is addressed by erofs+overlayfs.

I guess you mean this requirement:
"When you have container instances that might only be needed for a few seconds, taking half a minute to set up the container instance and then another half a minute to tear it down just isn't viable - we need instantiation and teardown times in the order of a second or two."

Forgive for not being part of the containers world, so I have to ask - Which real life use case requires instantiation and teardown times in the order of a second?

What is the order of number of files in the manifest of those ephemeral images?

The benchmark was done on a 2.6GB centos9 image.

My very minimal understanding of containers world, is that A large centos9 image would be used quite often on a client so it would be deployed as created inodes in disk filesystem and the ephemeral images are likely to be small changes on top of those large base images.

Furthermore, the ephmeral images would likely be composed of cenos9 + several layers, so the situation of single composefs image as large as centos9 is highly unlikely.

Am I understanding the workflow correctly?

If I am, then I would rather see benchmarks with images that correspond with the real life use case that drives composefs, such as small manifests and/or composefs in combination with overlayfs as it would be used more often.

Quoted 6 lines

Cold cache performance dominates the runtime of short lived containers as well as high density container hosts being run to their container level memory limits. ls -lR is just a microbenchmark that demonstrates how much better composefs cold cache behaviour is than the alternatives being proposed....

Quoted 7 lines

This might also help explain why my initial review comments focussed on getting rid of optional format features, straight lining the processing, changing the format or search algorithms so more sequential cacheline accesses occurred resulting in less memory stalls, etc. i.e. reductions in cold cache lookup overhead will directly translate into faster container workload spin up.

I agree that this technology is novel and understand why it results in faster cold cache lookup.
I do not know erofs enough to say if similar techniques could be applied to optimize erofs lookup at mkfs.erofs time, but I can guess that this optimization was never attempted.

Quoted 43 lines

Quoted 22 lines

Quoted 22 lines

Quoted 6 lines

Quoted 5 lines

This isn't all that strange, as overlayfs does a lot more work for each lookup, including multiple name lookups as well as several xattr
lookups, whereas composefs just does a single lookup in a pre- computed

Quoted 6 lines

Seriously, "multiple name lookups"?
Overlayfs does exactly one lookup for anything but first level subdirs
and for sparse files it does the exact same lookup in /objects as composefs.
Enough with the hand waving please. Stick to hard facts.

With the discussed layout, in a stat() call on a regular file, ovl_lookup() will do lookups on both the sparse file and the backing file, whereas cfs_dir_lookup() will just map some page cache pages and do a binary search.

Of course if you actually open the file, then cfs_open_file() would do the equivalent lookups in /objects. But that is often not what happens, for example in "ls -l".

Quoted 5 lines

Quoted 3 lines

Additionally, these extra lookups will cause extra memory use, as you need dentries and inodes for the erofs/squashfs inodes in addition to the overlay inodes.

I see. composefs is really very optimized for ls -lR.

No, composefs is optimised for minimal namespace and inode resolution overhead. 'ls -lR' does a lot of these operations, and therefore you see the efficiency of the design being directly exposed....

Quoted 2 lines

Now only need to figure out if real users start a container and do ls -lR without reading many files is a real life use case.

I've been using 'ls -lR' and 'find . -ctime 1' to benchmark cold cache directory iteration and inode lookup performance for roughly 20 years. The benchmarks I run *never* read file data, nor is that desired - they are pure directory and inode lookup micro-benchmarks used to analyse VFS and filesystem directory and inode lookup performance.

Quoted 8 lines

I have been presenting such measurements and patches improving performance of these microbnechmarks to the XFS and fsdevel lists over 15 years and I have *never* had to justify that what I'm measuring is a "real world workload" to anyone. Ever.

Complaining about real world relevancy of the presented benchmark might be considered applying a double standard, wouldn't you agree?

I disagree.
Perhaps my comment was misunderstood.

The cold cache benchmark is certainly relevant for composefs comparison and I expect to see it in future submissions.

The point I am trying to drive is this:
There are two alternatives on the table:
1. Add fs/composefs
2. Improve erofs and overlayfs

Functionally, I think we all agree that both alternatives should work.

Option #1 will take much less effort from composefs authors, so it is understandable that they would do their best to argue in its favor.

Option #2 is prefered for long term maintenance reasons, which is why vfs/erofs/overlayfs developers argue in favor of it.

The only factor that remains that could shift the balance inside this gray area are the actual performance numbers.

And back to my point: the not so simple decision between the two options, by whoever makes this decision, should be based on a real life example of performance improvement and not of a microbenchamk.

In my limited experience, a real life example means composefs as a layer in overlayfs.

I did not see those numbers and it is clear that they will not be as impressive as the bare composefs numbers, so proposing composefs needs to include those numbers as well.

Alexander did claim that he has real life use cases for bare readonly composefs images, but he did not say what the size of the manifests in those images are and he did not say whether these use cases also require startup and teardown in orders of seconds.

It looks like the different POV are now well understood by all parties and that we are in the process of fine tuning the information that needs to be presented for making the best decision based on facts.

This discussion, which was on a collision course at the beginning, looks like it is in a converging course - this makes me happy.

Thanks,
Amir.

Quoted 84 lines · Amir Goldstein

Quoted 24 lines · Dave Chinner

Quoted 17 lines · Amir Goldstein

Quoted 7 lines · Alexander Larsson

Quoted 6 lines · Amir Goldstein

Quoted 4 lines · Alexander Larsson

Quoted 3 lines · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson
<[email protected]>
wrote:

Quoted 8 lines

I'm not sure why the dentry cache case would be more important? Starting a new container will very often not have cached the image.

To me the interesting case is for a new image, but with some existing page cache for the backing files directory. That seems to model staring a new image in an active container host, but its somewhat hard to test that case.

Quoted 4 lines

ok, you can argue that faster cold cache ls -lR is important for starting new images.
I think you will be asked to show a real life container use case where that benchmark really matters.

Quoted 7 lines

I've already described the real world production system bottlenecks that composefs is designed to overcome in a previous thread.

Please go back an read this:

https://lore.kernel.org/linux-fsdevel/[email protected]/

I've read it and now re-read it.
Most of the post talks about the excess time of creating the namespace, which is addressed by erofs+overlayfs.

I guess you mean this requirement:
"When you have container instances that might only be needed for a few seconds, taking half a minute to set up the container instance and then another half a minute to tear it down just isn't viable - we need instantiation and teardown times in the order of a second or two."

Forgive for not being part of the containers world, so I have to ask - Which real life use case requires instantiation and teardown times in the order of a second?

What is the order of number of files in the manifest of those ephemeral images?

The benchmark was done on a 2.6GB centos9 image.

My very minimal understanding of containers world, is that A large centos9 image would be used quite often on a client so it would be deployed as created inodes in disk filesystem and the ephemeral images are likely to be small changes on top of those large base images.

Furthermore, the ephmeral images would likely be composed of cenos9 + several layers, so the situation of single composefs image as large as centos9 is highly unlikely.

Am I understanding the workflow correctly?

If I am, then I would rather see benchmarks with images that correspond with the real life use case that drives composefs, such as small manifests and/or composefs in combination with overlayfs as it would be used more often.

Quoted 13 lines

Cold cache performance dominates the runtime of short lived containers as well as high density container hosts being run to their container level memory limits. ls -lR is just a microbenchmark that demonstrates how much better composefs cold cache behaviour is than the alternatives being proposed....

This might also help explain why my initial review comments focussed on getting rid of optional format features, straight lining the processing, changing the format or search algorithms so more sequential cacheline accesses occurred resulting in less memory stalls, etc. i.e. reductions in cold cache lookup overhead will directly translate into faster container workload spin up.

Quoted 5 lines

I agree that this technology is novel and understand why it results in faster cold cache lookup.
I do not know erofs enough to say if similar techniques could be applied to optimize erofs lookup at mkfs.erofs time, but I can guess that this optimization was never attempted.

As Dave mentioned, containers in a cluster usually run with low memory limits to increase density of how many containers can run on a single

host. I've done some tests to get some numbers on the memory usage.

Please let me know if you've any comment on the method I've used to read the memory usage, if you've any better suggestion please let me know.

I am using a Fedora container image, but I think the image used is not relevant, as the memory used should increase linearly to the image size for both setups.

I am using systemd-run --scope to get a new cgroup, the system uses cgroupv2.

For this first test I am using a RO mount both for composefs and erofs+overlayfs.

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/composefs > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-r482ec1c3024a4a8b9d2a369bf5dc6df3.scope
16367616
0.03user 0.54system 0:00.71elapsed 80%CPU (0avgtext+0avgdata 7552maxresident)k
10592inputs+0outputs (28major+1273minor)pagefaults 0swaps

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/erofs-overlay > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-r5f0f599053c349669e5c1ecacaa037b6.scope
48390144
0.04user 1.03system 0:01.81elapsed 59%CPU (0avgtext+0avgdata 7552maxresident)k
30776inputs+0outputs (28major+1269minor)pagefaults 0swaps

the erofs+overlay setup takes 2.5 times to complete and it uses 3 times the memory used by composefs.

The second test involves a RW mount for composefs.

For the erofs+overlay setup I've just added an upperdir and workdir to the overlay mount, while for composefs I create a completely new overlay mount that uses the composefs mount as the lower layer.

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/composefs-overlay > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-r23519c8048704e5b84a1355f131d9d93.scope
31014912
0.05user 1.15system 0:01.38elapsed 87%CPU (0avgtext+0avgdata 7552maxresident)k
10944inputs+0outputs (28major+1282minor)pagefaults 0swaps

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/erofs-overlay > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-rdbccf045f3124e379cec00273638db08.scope
48308224
0.07user 2.04system 0:03.22elapsed 65%CPU (0avgtext+0avgdata 7424maxresident)k
30720inputs+0outputs (28major+1273minor)pagefaults 0swaps

so the erofs+overlay setup still takes more time (almost 2.5 times) and uses more memory (slightly more than 1.5 times)

Quoted 87 lines · Giuseppe Scrivano

Quoted 84 lines · Amir Goldstein

Quoted 24 lines · Dave Chinner

Quoted 17 lines · Amir Goldstein

Quoted 7 lines · Alexander Larsson

Quoted 6 lines · Amir Goldstein

Quoted 4 lines · Alexander Larsson

Quoted 3 lines · Amir Goldstein

On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson
<[email protected]>
wrote:

Quoted 8 lines

I'm not sure why the dentry cache case would be more important? Starting a new container will very often not have cached the image.

To me the interesting case is for a new image, but with some existing page cache for the backing files directory. That seems to model staring a new image in an active container host, but its somewhat hard to test that case.

Quoted 4 lines

ok, you can argue that faster cold cache ls -lR is important for starting new images.
I think you will be asked to show a real life container use case where that benchmark really matters.

Quoted 7 lines

I've already described the real world production system bottlenecks that composefs is designed to overcome in a previous thread.

Please go back an read this:

https://lore.kernel.org/linux-fsdevel/[email protected]/

I've read it and now re-read it.
Most of the post talks about the excess time of creating the namespace, which is addressed by erofs+overlayfs.

I guess you mean this requirement:
"When you have container instances that might only be needed for a few seconds, taking half a minute to set up the container instance and then another half a minute to tear it down just isn't viable - we need instantiation and teardown times in the order of a second or two."

Forgive for not being part of the containers world, so I have to ask - Which real life use case requires instantiation and teardown times in the order of a second?

What is the order of number of files in the manifest of those ephemeral images?

The benchmark was done on a 2.6GB centos9 image.

My very minimal understanding of containers world, is that A large centos9 image would be used quite often on a client so it would be deployed as created inodes in disk filesystem and the ephemeral images are likely to be small changes on top of those large base images.

Furthermore, the ephmeral images would likely be composed of cenos9 + several layers, so the situation of single composefs image as large as centos9 is highly unlikely.

Am I understanding the workflow correctly?

If I am, then I would rather see benchmarks with images that correspond with the real life use case that drives composefs, such as small manifests and/or composefs in combination with overlayfs as it would be used more often.

Quoted 13 lines

Cold cache performance dominates the runtime of short lived containers as well as high density container hosts being run to their container level memory limits. ls -lR is just a microbenchmark that demonstrates how much better composefs cold cache behaviour is than the alternatives being proposed....

This might also help explain why my initial review comments focussed on getting rid of optional format features, straight lining the processing, changing the format or search algorithms so more sequential cacheline accesses occurred resulting in less memory stalls, etc. i.e. reductions in cold cache lookup overhead will directly translate into faster container workload spin up.

Quoted 8 lines

Quoted 5 lines

I agree that this technology is novel and understand why it results in faster cold cache lookup.
I do not know erofs enough to say if similar techniques could be applied to optimize erofs lookup at mkfs.erofs time, but I can guess that this optimization was never attempted.

As Dave mentioned, containers in a cluster usually run with low memory limits to increase density of how many containers can run on a single

Good selling point.

Quoted 52 lines

host. I've done some tests to get some numbers on the memory usage.

Please let me know if you've any comment on the method I've used to read the memory usage, if you've any better suggestion please let me know.

I am using a Fedora container image, but I think the image used is not relevant, as the memory used should increase linearly to the image size for both setups.

I am using systemd-run --scope to get a new cgroup, the system uses cgroupv2.

For this first test I am using a RO mount both for composefs and erofs+overlayfs.

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/composefs > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-r482ec1c3024a4a8b9d2a369bf5dc6df3.scope
16367616
0.03user 0.54system 0:00.71elapsed 80%CPU (0avgtext+0avgdata 7552maxresident)k
10592inputs+0outputs (28major+1273minor)pagefaults 0swaps

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/erofs-overlay > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-r5f0f599053c349669e5c1ecacaa037b6.scope
48390144
0.04user 1.03system 0:01.81elapsed 59%CPU (0avgtext+0avgdata 7552maxresident)k
30776inputs+0outputs (28major+1269minor)pagefaults 0swaps

the erofs+overlay setup takes 2.5 times to complete and it uses 3 times the memory used by composefs.

The second test involves a RW mount for composefs.

For the erofs+overlay setup I've just added an upperdir and workdir to the overlay mount, while for composefs I create a completely new overlay mount that uses the composefs mount as the lower layer.

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/composefs-overlay > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-r23519c8048704e5b84a1355f131d9d93.scope
31014912
0.05user 1.15system 0:01.38elapsed 87%CPU (0avgtext+0avgdata 7552maxresident)k
10944inputs+0outputs (28major+1282minor)pagefaults 0swaps

# echo 3 > /proc/sys/vm/drop_caches
# \time systemd-run --scope sh -c 'ls -lR /mnt/erofs-overlay > /dev/null; cat $(cat /proc/self/cgroup | sed -e "s|0::|/sys/fs/cgroup|")/memory.peak'
Running scope as unit: run-rdbccf045f3124e379cec00273638db08.scope
48308224
0.07user 2.04system 0:03.22elapsed 65%CPU (0avgtext+0avgdata 7424maxresident)k
30720inputs+0outputs (28major+1273minor)pagefaults 0swaps

Quoted 3 lines

so the erofs+overlay setup still takes more time (almost 2.5 times) and uses more memory (slightly more than 1.5 times)

That's an important comparison. Thanks for running it.

Based on Alexander's explanation about the differences between overlayfs lookup vs. composefs lookup of a regular "metacopy" file, I just need to point out that the same optimization (lazy lookup of the lower data file on open)
can be done in overlayfs as well.
(*) currently, overlayfs needs to lookup the lower file also for st_blocks.

I am not saying that it should be done or that Miklos will agree to make this change in overlayfs, but that seems to be the major difference. getxattr may have some extra cost depending on in-inode xattr format of erofs, but specifically, the metacopy getxattr can be avoided if this is a special overlayfs RO mount that is marked as EVERYTHING IS METACOPY.

I don't expect you guys to now try to hack overlayfs and explore this path to completion.
My expectation is that this information will be clearly visible to anyone reviewing future submission, e.g.:

- This is the comparison we ran...
- This is the reason that composefs gives better results...
- It MAY be possible to optimize erofs/overlayfs to get to similar results, but we did not try to do that

It is especially important IMO to get the ACK of both Gao and Miklos on your analysis, because remember than when this thread started, you did not know about the metacopy option and your main argument was saving the time it takes to create the overlayfs layer files in the filesystem, because you were missing some technical background on overlayfs.

I hope that after you are done being annoyed by all the chores we put you guys up to, you will realize that they help you build your case for the final submission...

Thanks,
Amir.

Quoted 25 lines

Based on Alexander's explanation about the differences between overlayfs lookup vs. composefs lookup of a regular "metacopy" file, I just need to point out that the same optimization (lazy lookup of the lower data file on open)
can be done in overlayfs as well.
(*) currently, overlayfs needs to lookup the lower file also for st_blocks.

I am not saying that it should be done or that Miklos will agree to make this change in overlayfs, but that seems to be the major difference. getxattr may have some extra cost depending on in-inode xattr format of erofs, but specifically, the metacopy getxattr can be avoided if this is a special overlayfs RO mount that is marked as EVERYTHING IS METACOPY.

I don't expect you guys to now try to hack overlayfs and explore this path to completion.
My expectation is that this information will be clearly visible to anyone reviewing future submission, e.g.:

- This is the comparison we ran...
- This is the reason that composefs gives better results...
- It MAY be possible to optimize erofs/overlayfs to get to similar results, but we did not try to do that

Quoted 5 lines

It is especially important IMO to get the ACK of both Gao and Miklos on your analysis, because remember than when this thread started, you did not know about the metacopy option and your main argument was saving the time it takes to create the overlayfs layer files in the filesystem, because you were missing some technical background on overlayfs.

we knew about metacopy, which we already use in our tools to create mapped image copies when idmapped mounts are not available, and also knew about the other new features in overlayfs. For example, the "volatile" feature which was mentioned in your Overlayfs-containers-lpc-2020 talk, was only submitted upstream after begging Miklos and Vivek for months. I had a PoC that I used and tested locally and asked for their help to get it integrated at the file system layer, using seccomp for the same purpose would have been more complex and prone to errors when dealing with external bind mounts containing persistent data.

The only missing bit, at least from my side, was to consider an image that contains only overlay metadata as something we could distribute.

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

Quoted 42 lines

Quoted 25 lines

Based on Alexander's explanation about the differences between overlayfs lookup vs. composefs lookup of a regular "metacopy" file, I just need to point out that the same optimization (lazy lookup of the lower data file on open)
can be done in overlayfs as well.
(*) currently, overlayfs needs to lookup the lower file also for st_blocks.

I am not saying that it should be done or that Miklos will agree to make this change in overlayfs, but that seems to be the major difference. getxattr may have some extra cost depending on in-inode xattr format of erofs, but specifically, the metacopy getxattr can be avoided if this is a special overlayfs RO mount that is marked as EVERYTHING IS METACOPY.

I don't expect you guys to now try to hack overlayfs and explore this path to completion.
My expectation is that this information will be clearly visible to anyone reviewing future submission, e.g.:

- This is the comparison we ran...
- This is the reason that composefs gives better results...
- It MAY be possible to optimize erofs/overlayfs to get to similar results, but we did not try to do that

Quoted 5 lines

It is especially important IMO to get the ACK of both Gao and Miklos on your analysis, because remember than when this thread started, you did not know about the metacopy option and your main argument was saving the time it takes to create the overlayfs layer files in the filesystem, because you were missing some technical background on overlayfs.

we knew about metacopy, which we already use in our tools to create mapped image copies when idmapped mounts are not available, and also knew about the other new features in overlayfs. For example, the "volatile" feature which was mentioned in your Overlayfs-containers-lpc-2020 talk, was only submitted upstream after begging Miklos and Vivek for months. I had a PoC that I used and tested locally and asked for their help to get it integrated at the file system layer, using seccomp for the same purpose would have been more complex and prone to errors when dealing with external bind mounts containing persistent data.

Quoted 3 lines

The only missing bit, at least from my side, was to consider an image that contains only overlay metadata as something we could distribute.

I'm glad that I was able to point this out to you, because now the comparison between the overlayfs and composefs options is more fair.

Quoted 6 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

There is no restriction of metacopy in user namespace. overlayfs needs to be mounted with -o userxattr and the overlay xattrs needs to use user.overlay. prefix.

w.r.t. the implied claim that composefs on-disk format is simple enough so it could be made robust enough to avoid exploits, I will remain silent and let others speak up, but I advise you to take cover, because this is an explosive topic ;)

Thanks,
Amir.

Quoted 10 lines

Quoted 6 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

There is no restriction of metacopy in user namespace. overlayfs needs to be mounted with -o userxattr and the overlay xattrs needs to use user.overlay. prefix.

if I specify both userxattr and metacopy=on then the mount ends up in the following check:

if (config->userxattr) {
        [...]
        if (config->metacopy && metacopy_opt) {
                pr_err("conflicting options: userxattr,metacopy=on\n");
                return -EINVAL;
        }
}

to me it looks like it was done on purpose to prevent metacopy from a user namespace, but I don't know the reason for sure.

Quoted 14 lines

Quoted 10 lines

Quoted 6 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

There is no restriction of metacopy in user namespace. overlayfs needs to be mounted with -o userxattr and the overlay xattrs needs to use user.overlay. prefix.

if I specify both userxattr and metacopy=on then the mount ends up in the following check:

Quoted 8 lines

if (config->userxattr) {
        [...]
        if (config->metacopy && metacopy_opt) {
                pr_err("conflicting options: userxattr,metacopy=on\n");
                return -EINVAL;
        }
}

Right, my bad.

Quoted 3 lines

to me it looks like it was done on purpose to prevent metacopy from a user namespace, but I don't know the reason for sure.

With hand crafted metacopy, an unpriv user can chmod any files to anything by layering another file with different mode on top of it....

Not sure how the composefs security model intends to handle this scenario with userns mount, but it sounds like a similar problem.

Quoted 25 lines · Amir Goldstein

Quoted 14 lines

Quoted 10 lines

Quoted 6 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

There is no restriction of metacopy in user namespace. overlayfs needs to be mounted with -o userxattr and the overlay xattrs needs to use user.overlay. prefix.

if I specify both userxattr and metacopy=on then the mount ends up in the following check:

Quoted 8 lines

if (config->userxattr) {
        [...]
        if (config->metacopy && metacopy_opt) {
                pr_err("conflicting options: userxattr,metacopy=on\n");
                return -EINVAL;
        }
}

Right, my bad.

Quoted 7 lines

Quoted 3 lines

to me it looks like it was done on purpose to prevent metacopy from a user namespace, but I don't know the reason for sure.

With hand crafted metacopy, an unpriv user can chmod any files to anything by layering another file with different mode on top of it....

I might be missing something obvious about metacopy, so please correct me if I am wrong, but I don't see how it is any different than just copying the file and chowning it. Of course, as long as overlay uses the same security model so that a file that wasn't originally possible to access must be still blocked, even if referenced through metacopy.

Quoted 36 lines · Giuseppe Scrivano

Quoted 25 lines · Amir Goldstein

Quoted 14 lines

Quoted 10 lines

Quoted 6 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

There is no restriction of metacopy in user namespace. overlayfs needs to be mounted with -o userxattr and the overlay xattrs needs to use user.overlay. prefix.

if I specify both userxattr and metacopy=on then the mount ends up in the following check:

Quoted 8 lines

if (config->userxattr) {
        [...]
        if (config->metacopy && metacopy_opt) {
                pr_err("conflicting options: userxattr,metacopy=on\n");
                return -EINVAL;
        }
}

Right, my bad.

Quoted 7 lines

Quoted 3 lines

to me it looks like it was done on purpose to prevent metacopy from a user namespace, but I don't know the reason for sure.

With hand crafted metacopy, an unpriv user can chmod any files to anything by layering another file with different mode on top of it....

Quoted 6 lines

I might be missing something obvious about metacopy, so please correct me if I am wrong, but I don't see how it is any different than just copying the file and chowning it. Of course, as long as overlay uses the same security model so that a file that wasn't originally possible to access must be still blocked, even if referenced through metacopy.

You're right.
The reason for mutual exclusion maybe related to the comment in ovl_check_metacopy_xattr() about EACCES. Need to check with Vivek or Miklos.

But get this - you do not need metacopy=on to follow lower inode. It should work without metacopy=on.
metacopy=on only instructs overlayfs whether to copy up data or only metadata when changing metadata of lower object, so it is not relevant for readonly mount.

Quoted 7 lines · Amir Goldstein

Quoted 5 lines · Giuseppe Scrivano

Quoted 2 lines · Amir Goldstein

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I

Quoted 43 lines

Quoted 31 lines

Quoted 23 lines

Quoted 12 lines

Quoted 8 lines

Quoted 4 lines

don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

There is no restriction of metacopy in user namespace. overlayfs needs to be mounted with -o userxattr and the overlay xattrs needs to use user.overlay. prefix.

if I specify both userxattr and metacopy=on then the mount ends up in the following check:

Quoted 8 lines

if (config->userxattr) {
        [...]
        if (config->metacopy && metacopy_opt) {
                pr_err("conflicting options: userxattr,metacopy=on\n");
                return -EINVAL;
        }
}

Right, my bad.

Quoted 7 lines

Quoted 3 lines

to me it looks like it was done on purpose to prevent metacopy from a user namespace, but I don't know the reason for sure.

With hand crafted metacopy, an unpriv user can chmod any files to anything by layering another file with different mode on top of it....

Quoted 6 lines

I might be missing something obvious about metacopy, so please correct me if I am wrong, but I don't see how it is any different than just copying the file and chowning it. Of course, as long as overlay uses the same security model so that a file that wasn't originally possible to access must be still blocked, even if referenced through metacopy.

You're right.
The reason for mutual exclusion maybe related to the comment in ovl_check_metacopy_xattr() about EACCES. Need to check with Vivek or Miklos.

Quoted 6 lines

But get this - you do not need metacopy=on to follow lower inode. It should work without metacopy=on.
metacopy=on only instructs overlayfs whether to copy up data or only metadata when changing metadata of lower object, so it is not relevant for readonly mount.

However, you do need redirect=follow and that one is only mutually exclusive with userxattr.
Again, need to ask Miklos whether that could be relaxed under some conditions.

[spawning overlayfs sub-topic]

Quoted 9 lines · Amir Goldstein

Quoted 7 lines · Amir Goldstein

Quoted 5 lines · Giuseppe Scrivano

Quoted 2 lines · Amir Goldstein

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I

For those who are starting to read here, the context is userns mounting of overlayfs with a lower EROFS layer containing metacopy references to lower data blobs in another fs (a.k.a the composefs model).

IMO, mounting a readonly image of whatever on-disk format is a very high risk for userns mount.
A privileged mount helper that verifies and mounts the EROFS layer sounds like a more feasible solution.

Quoted 36 lines · Giuseppe Scrivano

Quoted 25 lines · Amir Goldstein

Quoted 14 lines

Quoted 10 lines

Quoted 6 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.

There is no restriction of metacopy in user namespace. overlayfs needs to be mounted with -o userxattr and the overlay xattrs needs to use user.overlay. prefix.

if I specify both userxattr and metacopy=on then the mount ends up in the following check:

Quoted 8 lines

if (config->userxattr) {
        [...]
        if (config->metacopy && metacopy_opt) {
                pr_err("conflicting options: userxattr,metacopy=on\n");
                return -EINVAL;
        }
}

Right, my bad.

Quoted 7 lines

Quoted 3 lines

to me it looks like it was done on purpose to prevent metacopy from a user namespace, but I don't know the reason for sure.

With hand crafted metacopy, an unpriv user can chmod any files to anything by layering another file with different mode on top of it....

Quoted 6 lines

I might be missing something obvious about metacopy, so please correct me if I am wrong, but I don't see how it is any different than just copying the file and chowning it. Of course, as long as overlay uses the same security model so that a file that wasn't originally possible to access must be still blocked, even if referenced through metacopy.

You're right.
The reason for mutual exclusion maybe related to the comment in ovl_check_metacopy_xattr() about EACCES. Need to check with Vivek or Miklos.

But get this - you do not need metacopy=on to follow lower inode. It should work without metacopy=on.
metacopy=on only instructs overlayfs whether to copy up data or only metadata when changing metadata of lower object, so it is not relevant for readonly mount.

Quoted 42 lines

Quoted 25 lines

Based on Alexander's explanation about the differences between overlayfs lookup vs. composefs lookup of a regular "metacopy" file, I just need to point out that the same optimization (lazy lookup of the lower data file on open)
can be done in overlayfs as well.
(*) currently, overlayfs needs to lookup the lower file also for st_blocks.

I am not saying that it should be done or that Miklos will agree to make this change in overlayfs, but that seems to be the major difference. getxattr may have some extra cost depending on in-inode xattr format of erofs, but specifically, the metacopy getxattr can be avoided if this is a special overlayfs RO mount that is marked as EVERYTHING IS METACOPY.

I don't expect you guys to now try to hack overlayfs and explore this path to completion.
My expectation is that this information will be clearly visible to anyone reviewing future submission, e.g.:

- This is the comparison we ran...
- This is the reason that composefs gives better results...
- It MAY be possible to optimize erofs/overlayfs to get to similar results, but we did not try to do that

Quoted 5 lines

It is especially important IMO to get the ACK of both Gao and Miklos on your analysis, because remember than when this thread started, you did not know about the metacopy option and your main argument was saving the time it takes to create the overlayfs layer files in the filesystem, because you were missing some technical background on overlayfs.

we knew about metacopy, which we already use in our tools to create mapped image copies when idmapped mounts are not available, and also knew about the other new features in overlayfs. For example, the "volatile" feature which was mentioned in your Overlayfs-containers-lpc-2020 talk, was only submitted upstream after begging Miklos and Vivek for months. I had a PoC that I used and tested locally and asked for their help to get it integrated at the file system layer, using seccomp for the same purpose would have been more complex and prone to errors when dealing with external bind mounts containing persistent data.

Quoted 3 lines

The only missing bit, at least from my side, was to consider an image that contains only overlay metadata as something we could distribute.

I'm glad that I was able to point this out to you, because now the comparison between the overlayfs and composefs options is more fair.

Quoted 5 lines

I previously mentioned my wish of using it from a user namespace, the goal seems more challenging with EROFS or any other block devices. I don't know about the difficulty of getting overlay metacopy working in a user namespace, even though it would be helpful for other use cases as well.