fs-verity: file system-level integrity protection

I'd like to talk about a proposal to implement and upstream something that we've been calling fs-verity, which is something like dm-verity, but implemnted on a per-file basis. It will be implemnted much like fs/crypto, in that most of the code will be in a generic layer, with minimal modifications needed in the file system layer.

The merkle tree will be located after file's normal data, and then after the package manager sets the verity bit, i_size will be updated so that the fs-verity header and merkle tree will be "hidden" from userspace and the file will become immutable.

How does this differ from IMA's file integrity?

*) The pages are verified as they are read, so pages are verified as they are read the storage device; this avoids a large latency hit when the file is first opened or referenced.

*) The design and code are done by file system developers, so it doesn't have the locking problems of the IMA code.

The initial use case of this will be for Android, where the latency concerns of doing the full checksum at file open time is important.

In the future, the fact that a file has been signed using fs-verity, using a PKCS 11 signature with a key on a trusted keyring (possibly the same one used for signed kernel modules, or perhaps a separate keyring) could be used as input into a security policy which requires this for say, setuid executables, setuid shell scripts, etc.

Most of this feature could also be used with a non-cryptographic checksum to provide data checksums for read-only files in a general way for all file systems. It wouldn't be as flexible as btrfs, but for files being stored for backup purposes, it should work quite well.

- Ted

Quoted 7 lines · Theodore Ts'o

I'd like to talk about a proposal to implement and upstream something that we've been calling fs-verity, which is something like dm-verity, but implemnted on a per-file basis. It will be implemnted much like fs/crypto, in that most of the code will be in a generic layer, with minimal modifications needed in the file system layer.

I’m interested in this topic, specifically how we can support binary signing of executable code stored in NFS files.

I’m not expert by any means, but this is also different than IMA in that it avoids the use of a trusted extended attribute. The NFS protocol has no support for system/trusted xattrs, currently.

However, I suspect each file system might prefer to store the tree in its own way. And distributed file systems will have the additional challenge of protecting this metadata as it is in flight from file server to client.

I'd like to talk about a proposal to implement and upstream something that we've been calling fs-verity, which is something like dm-verity, but implemnted on a per-file basis.  It will be implemnted much like fs/crypto, in that most of the code will be in a generic layer, with minimal modifications needed in the file system layer.

The merkle tree will be located after file's normal data, and then after the package manager sets the verity bit, i_size will be updated so that the fs-verity header and merkle tree will be "hidden" from userspace and the file will become immutable.

How does this differ from IMA's file integrity?

*) The pages are verified as they are read, so pages are verified as they are read the storage device; this avoids a large latency hit when the file is first opened or referenced.

*) The design and code are done by file system developers, so it doesn't have the locking problems of the IMA code.

The initial use case of this will be for Android, where the latency concerns of doing the full checksum at file open time is important.

In the future, the fact that a file has been signed using fs-verity, using a PKCS 11

signature with a key on a trusted keyring (possibly
the same one used for signed kernel modules, or perhaps a separate keyring) could be used as input into a security policy which requires this for say, setuid executables, setuid shell scripts, etc.

Most of this feature could also be used with a non-cryptographic checksum to provide data checksums for read-only files in a general way for all file systems.  It wouldn't be as flexible as btrfs, but for files being stored for backup purposes, it should work quite well.

How do you know the file is in this special format? �Would it be a per filesystem flag (so every file) or selectable per-file by some other mechanism. �If it's per-file, why not simply use the existing xattr mechanism?

Quoted 3 lines

*) The pages are verified as they are read, so pages are verified as they are read the storage device; this avoids a large latency hit when the file is first opened or referenced.

The cost of this is presumably one hash per page in the tree, so it costs quite a bit in terms of space. �Presumably the hash tree is also dynamically resident meaning a page fault could now also potentially fault in the hash tree, leading to a lot of sub optimal I/O patterns?

Quoted 2 lines

*) The design and code are done by file system developers, so it doesn't have the locking problems of the IMA code.

That's a bit unfair. �My next question was going to be why not just make this an actual IMA mode (meaning you could choose to have a global hash or a tree hash). �Does this mean that a-priori you've already ruled out IMA integration because you don't want to work with the developers?

PKCS11 is the standard for cryptokeys. �I presume you just mean a message signing standard like PKCS7 or RFC 2315?

Quoted 5 lines

Most of this feature could also be used with a non-cryptographic checksum to provide data checksums for read-only files in a general way for all file systems.��It wouldn't be as flexible as btrfs, but for files being stored for backup purposes, it should work quite well.

I assume the "write" part of this is that the file must be deleted and re-created?

Quoted 6 lines · James Bottomley

How do you know the file is in this special format?  Would it be a per filesystem flag (so every file) or selectable per-file by some other mechanism.  If it's per-file, why not simply use the existing xattr mechanism?

It would be using a per-file flag, just like we do with fscrypt. Given that we only need a single bit of information, using an xattr would be a inefficeint.

Quoted 11 lines

Quoted 5 lines

*) The pages are verified as they are read, so pages are verified as they are read the storage device; this avoids a large latency hit when the file is first opened or referenced.

The cost of this is presumably one hash per page in the tree, so it costs quite a bit in terms of space.  Presumably the hash tree is also dynamically resident meaning a page fault could now also potentially fault in the hash tree, leading to a lot of sub optimal I/O patterns?

This is how dm-verity works, which is used on every single modern Chrome OS and Android phone, with no complaints.  (It doesn't work that way on your phone, unless you've upgraded.  :-)

Quoted 8 lines

Quoted 2 lines

*) The design and code are done by file system developers, so it doesn't have the locking problems of the IMA code.

That's a bit unfair.  My next question was going to be why not just make this an actual IMA mode (meaning you could choose to have a global hash or a tree hash).  Does this mean that a-priori you've already ruled out IMA integration because you don't want to work with the developers?

IMA has a lot of complexity, which I would rather not drag in as a dependency.  Also, having seen some of the ah, "discussions" that Christoph and Dave Chinner have been having with the IMA folks, I'd rather not taint this proposal with IMA's reputation.  :-)

I am completely open to an optional integration with IMA, but I would prefer not to require CONFIG_IMA to be enabled in order to use fs-verity.

Quoted 2 lines

PKCS11 is the standard for cryptokeys.  I presume you just mean a message signing standard like PKCS7 or RFC 2315?

Sorry, I meant PKCS7.  It would be a restricted PKCS7 mode, using a detached signature.  My plan was to reuse the existing code we already have written for signed kernel modules.

Quoted 8 lines

Quoted 5 lines

Most of this feature could also be used with a non-cryptographic checksum to provide data checksums for read-only files in a general way for all file systems.  It wouldn't be as flexible as btrfs, but for files being stored for backup purposes, it should work quite well.

I assume the "write" part of this is that the file must be deleted and re-created?

I'm not sure what you mean.

  If you have an existing file that you
want to protect using fs-verity, it's a matter of appending the fs-verity information onto the end of the file, and then setting the fs-verity flag, at which point all of the fs-verity information disappears from the perspective of stat(2) and read(2) system calls. The verity bit can be examined using FS_IOC_GETFLAGS, and more details about which key was used to sign the file could be examined via some ioctl interface.

In general, though, it's expected that userspace won't care about such details.  Any reads that don't verify will return an error, and if the key used to sign fs-verity information is not trusted by the kernel, the open will return an error.  So all userspace or a security policy would need to take of is that file does have the verity bit set.

Quoted 3 lines

Sorry, I meant PKCS7.��It would be a restricted PKCS7 mode, using a detached signature.��My plan was to reuse the existing code we already have written for signed kernel modules.

OK, so presumably the signature would be over the part of the tree at the end of the file (so the tree is already reduced to a hashable binary representation) and this is verified upon write, after which the hash tree is trusted.

Currently container images are simple tar files and one of the main value adds of docker as a tool is the simplicity of the image creation process. �That process depends on standard tools like tar to create the image, so I was trying to fit this proposal into that process.

OK, so that worries me a bit more. �I assume we could use this ioctl to recreate the file by extracting the tree and thence do a conversion to tar format so that the untarred file has the signed hash, but just doing a tar of the directory won't work, so docker save is going to have to be seriously altered to work with this.

Docker save was going to have to be altered to use IMA, anyway.

 So I don't see that as being any more difficult.  Whether you have to have root to set the magic IMA trusted xattr, or you call a userspace library, there isn't much difference between those two.

- Ted
_______________________________________________
Lsf-pc mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/lsf-pc

Quoted 1 line · Theodore Ts'o

Docker save was going to have to be altered to use IMA, anyway.�

Actually, no, that's not entirely true[1]. �Docker save produces a tar file. �Once the tar on your platform picks up xattrs, docker save just works for container images with IMA hashes and signatures (and selinux labels, which was actually the driver for the change). �The point at which the ecosystem changed to "just work" was the point at which tar understood xattrs. �That's why I was poking on how do we get tar to understand this format, following on the way IMA and selinux did it. �There may be another way of getting this change into the ecosystem, but ecosystem adoption has to be part of the considerations for this.

Quoted 12 lines · James Bottomley

Quoted 1 line · Theodore Ts'o

Docker save was going to have to be altered to use IMA, anyway.

Actually, no, that's not entirely true[1]. Docker save produces a tar file. Once the tar on your platform picks up xattrs, docker save just works for container images with IMA hashes and signatures (and selinux labels, which was actually the driver for the change). The point at which the ecosystem changed to "just work" was the point at which tar understood xattrs. That's why I was poking on how do we get tar to understand this format, following on the way IMA and selinux did it.
There may be another way of getting this change into the ecosystem,
but ecosystem adoption has to be part of the considerations for this.

Oh, I see. You are saying that you want to be able to use tar to backup integrity protected files, and then restore them later.

Yes, that's different from what I was assuming, which is a model where the integrity protect file would be written by some package manager (e.g,. rpm, dpkg, the code that downloads the apk, etc.), and that we would *not* be trying to backup the file with the integrity data, and then restore it later via some kind of untar operation.

The problem here is that a merkle tree simply won't fit inside an xattr for any non-trivail file. And there may be use cases where blocking the open until the integrity is verifeid on the entire file. However, there are uses cases where the a signifcant increase in the open latency can't be tolerated, and wher the file might have might have large portions of dat which will never be read, and thus, don't need to have their integrity verified. (Example: an APK might have megabytes and megabytes of translation resources for N languages, only one of which will normally be used by a particular user on a particular phone. Or as another example, an ELF binary that has huge portions of symbol table and debugging information that is normally not used.)

So the requirement that you must be able to backup an integrity protected file, and then restore it again, without modifying the tool which does the backup and restore, does certainly push you towards using xattrs. But xattrs force the huge open latency, and while Docker is big in some circles, there are lots of use cases where the unmodified backup/restore requiremnt is simply not applicable.

So perhaps there is room for both solutions.

Quoted 50 lines · Theodore Ts'o

Quoted 5 lines · James Bottomley

Quoted 2 lines · Theodore Ts'o

Docker save was going to have to be altered to use IMA, anyway.

Quoted 10 lines

Actually, no, that's not entirely true[1].  Docker save produces a tar file.  Once the tar on your platform picks up xattrs, docker save just works for container images with IMA hashes and signatures (and selinux labels, which was actually the driver for the change).  The point at which the ecosystem changed to "just work" was the point at which tar understood xattrs.  That's why I was poking on how do we get tar to understand this format, following on the way IMA and selinux did it.  There may be another way of getting this change into the ecosystem, but ecosystem adoption has to be part of the considerations for this.

Oh, I see.  You are saying that you want to be able to use tar to backup integrity protected files, and then restore them later.

Yes, that's different from what I was assuming, which is a model where the integrity protect file would be written by some package manager (e.g,. rpm, dpkg, the code that downloads the apk, etc.), and that we would *not* be trying to backup the file with the integrity data, and then restore it later via some kind of untar operation.

The problem here is that a merkle tree simply won't fit inside an xattr for any non-trivail file.  And there may be use cases where blocking the open until the integrity is verifeid on the entire file. However, there are uses cases where the a signifcant increase in the open latency can't be tolerated, and wher the file might have might have large portions of dat which will never be read, and thus, don't need to have their integrity verified.  (Example: an APK might have megabytes and megabytes of translation resources for N languages, only one of which will normally be used by a particular user on a particular phone.  Or as another example, an ELF binary that has huge portions of symbol table and debugging information that is normally not used.)

So the requirement that you must be able to backup an integrity protected file, and then restore it again, without modifying the tool which does the backup and restore, does certainly push you towards using xattrs.  But xattrs force the huge open latency, and while Docker is big in some circles, there are lots of use cases where the unmodified backup/restore requiremnt is simply not applicable.

Quoted 1 line

So perhaps there is room for both solutions.

I think this is relatively straight forward to handle.  The package (tarball, whatever) itself only needs to store the top-level checksum, since this validates the whole Merkle tree, and in turn the integrity of the whole file.  This is exactly what Bittorrent does for files.

When the package is extracted, the Merkle tree can be regenerated and written with the file for random IO access using fs-verity.  When the Merkle tree is written to disk, the top-level checksum is verified against the checksum stored in the package to ensure it was written correctly.  This means only a small checksum needs to be stored in the archive (32 bytes), but an integrated system will have end-to-end data verification.

Quoted 58 lines · Andreas Dilger

Quoted 56 lines · Theodore Ts'o

Quoted 8 lines · James Bottomley

Quoted 4 lines · Theodore Ts'o

Docker save was going to have to be altered to use IMA, anyway.

Quoted 11 lines

Actually, no, that's not entirely true[1].  Docker save produces a tar file.  Once the tar on your platform picks up xattrs, docker save just works for container images with IMA hashes and signatures (and selinux labels, which was actually the driver for the change).  The point at which the ecosystem changed to "just work" was the point at which tar understood xattrs.  That's why I was poking on how do we get tar to understand this format, following on the way IMA and selinux did it.  There may be another way of getting this change into the ecosystem, but ecosystem adoption has to be part of the considerations for this.

Oh, I see.  You are saying that you want to be able to use tar to backup integrity protected files, and then restore them later.

Yes, that's different from what I was assuming, which is a model where the integrity protect file would be written by some package manager (e.g,. rpm, dpkg, the code that downloads the apk, etc.), and that we would *not* be trying to backup the file with the integrity data, and then restore it later via some kind of untar operation.

The problem here is that a merkle tree simply won't fit inside an xattr for any non-trivail file.  And there may be use cases where blocking the open until the integrity is verifeid on the entire file. However, there are uses cases where the a signifcant increase in the open latency can't be tolerated, and wher the file might have might have large portions of dat which will never be read, and thus, don't need to have their integrity verified.  (Example: an APK might have megabytes and megabytes of translation resources for N languages, only one of which will normally be used by a particular user on a particular phone.  Or as another example, an ELF binary that has huge portions of symbol table and debugging information that is normally not used.)

So the requirement that you must be able to backup an integrity protected file, and then restore it again, without modifying the tool which does the backup and restore, does certainly push you towards using xattrs.  But xattrs force the huge open latency, and while Docker is big in some circles, there are lots of use cases where the unmodified backup/restore requiremnt is simply not applicable.

Quoted 7 lines

Quoted 1 line

So perhaps there is room for both solutions.

I think this is relatively straight forward to handle.  The package (tarball, whatever) itself only needs to store the top-level checksum, since this validates the whole Merkle tree, and in turn the integrity of the whole file.  This is exactly what Bittorrent does for files.

Well, not quite: bittorrent doesn't reconstruct the hash from the file, it downloads the hash a piece at a time and uses that to verify the piece of the file it's obtained.  However, I accept that's only because the leechers don't have the whole file from which to reconstruct the hash; seed creation certainly does this.

Quoted 8 lines

When the package is extracted, the Merkle tree can be regenerated and written with the file for random IO access using fs- verity.  When the Merkle tree is written to disk, the top-level checksum is verified against the checksum stored in the package to ensure it was written correctly.  This means only a small checksum needs to be stored in the archive (32 bytes), but an integrated system will have end-to-end data verification.

I certainly buy this approach, and it fits well with the limited data size there is in xattrs but Ted said in the initial proposal the entire tree would be present in the file.  I can't see a need for supplying the entire tree rather than reconstructing it but maybe there's an android use case I'm not seeing (Like not wanting to waste limited CPU power).

Just so I understand the mechanics: The xattr would contain the head node.  When this is written, the tree would be reconstructed from the file and verified.  If it verifies, it must be stored in the filesystem data somehow (or at least the lowest layer), so all subsequent uses of the file can proceed from the per page hash even after unmount and remount?  Then I certainly think it suits both cases.

I certainly buy this approach, and it fits well with the limited data size there is in xattrs but Ted said in the initial proposal the entire tree would be present in the file.��I can't see a need for supplying the entire tree rather than reconstructing it but maybe there's an android use case I'm not seeing (Like not wanting to waste limited CPU power).

Just so I understand the mechanics: The xattr would contain the head node. �When this is written, the tree would be reconstructed from the file and verified. �If it verifies, it must be stored in the filesystem data somehow (or at least the lowest layer), so all subsequent uses of the file can proceed from the per page hash even after unmount and remount? �Then I certainly think it suits both cases.

(And then you can get some of the "the IMA people are insane" taint on yourself.  :-)

Quoted 7 lines · James Bottomley

I certainly buy this approach, and it fits well with the limited data size there is in xattrs but Ted said in the initial proposal the entire tree would be present in the file.  I can't see a need for supplying the entire tree rather than reconstructing it but maybe there's an android use case I'm not seeing (Like not wanting to waste limited CPU power).

You can certainly reconstruct the Merkle tree at install time, but it does need to be saved on disk.  My assumption was that Merkle tree would be written to disk by userspace, along with the fs-verity header, simply because it would make things much simpler for the kernel, and in my view you *have* to modify userspace in some way.

Quoted 7 lines

Just so I understand the mechanics: The xattr would contain the head node.  When this is written, the tree would be reconstructed from the file and verified.  If it verifies, it must be stored in the filesystem data somehow (or at least the lowest layer), so all subsequent uses of the file can proceed from the per page hash even after unmount and remount?  Then I certainly think it suits both cases.

Yes.... in theory we could store the bare root hash (and some other bare minmum information, such as the PKCS7 signature and the elments of a fs-verity header) in an xattr, and setting that xattr would magically cause the merkle tree to be reconstructed and stored on disk.  But most of the file sytem developers have considered the use of setting or clearing xattrs to cause magically code paths to be executed to be an extremely bad idea.

  And since I don't really care about the use case, I can let *you* try to convince Cristoph and Al Viro that his is a good idea, despite the general agreemnt by all file system developers that we've been there, done that, decided it was a really bad idea, and let's never ever do that again.  (And then you can get some of the "the IMA people are insane" taint on yourself.  :-)

                       - Ted

Quoted 9 lines · Theodore Ts'o

Quoted 7 lines · James Bottomley

I certainly buy this approach, and it fits well with the limited data size there is in xattrs but Ted said in the initial proposal the entire tree would be present in the file. I can't see a need for supplying the entire tree rather than reconstructing it but maybe there's an android use case I'm not seeing (Like not wanting to waste limited CPU power).

Quoted 5 lines

You can certainly reconstruct the Merkle tree at install time, but it does need to be saved on disk. My assumption was that Merkle tree would be written to disk by userspace, along with the fs-verity header, simply because it would make things much simpler for the kernel, and in my view you *have* to modify userspace in some way.

OK, so we agree then that what IMA provides: a hash and potential signature over the hash is sufficient input for what you want and does provide existing tooling to achieve it.

Quoted 8 lines

Quoted 7 lines

Just so I understand the mechanics: The xattr would contain the head node. When this is written, the tree would be reconstructed from the file and verified. If it verifies, it must be stored in the filesystem data somehow (or at least the lowest layer), so all subsequent uses of the file can proceed from the per page hash even after unmount and remount? Then I certainly think it suits both cases.

Quoted 7 lines

Yes.... in theory we could store the bare root hash (and some other bare minmum information, such as the PKCS7 signature and the elments of a fs-verity header) in an xattr, and setting that xattr would magically cause the merkle tree to be reconstructed and stored on disk. But most of the file sytem developers have considered the use of setting or clearing xattrs to cause magically code paths to be executed to be an extremely bad idea.

Hang on, you've jumped straight from a discussion of ideas to an insane implementation ... can we take a step back? I'm thinking of three separate things

1. Could the signature piece of this be done in the way IMA currently does file signatures. We all agree "yes" on this, since a signed mekle hash head is the same size as an existing IMA signature and therefore does fit into xattrs.
2. Could IMA use a merkle tree for hash verification a page at a time as part of its implementation? I think the answer to this is yes, except the hook has to be somewhere in the page fault mechanism, so it would need some exploration and prototyping.
3. Could the merkle tree be cached somehow in the filesystem (probably as part of the filesystem implementaiton)? You've already assumed "yes" for this since it's how fs-verity is proposed to work.

So the issue we could discuss is how all this is tied together. The implementation could be your special format file for 3. but with the head hash signature in an xattr. What we get back on tar would be the ordinary file + xattr, but perhaps tar could use this to reconstruct the magic file on untar.

I do think the "special format magic file" is a violation of the unix principles of files being "just files" but if it's the only way, I suppose I could be persuaded. However, this part should also be discussed; it does seem like, to satisfy unix principles, the merkle tree should be provided separately from the file, possibly as a fcntl().

James

Quoted 8 lines

And since I don't really care about the use case, I can let *you*
try to convince Cristoph and Al Viro that his is a good idea, despite the general agreemnt by all file system developers that we've been there, done that, decided it was a really bad idea, and let's never ever do that again. (And then you can get some of the "the IMA people are insane" taint on yourself. :-)

- Ted

Quoted 96 lines

On Jan 28, 2018, at 10:03 AM, James Bottomley <James.Bottomley@Hans enPartnership.com> wrote:

Quoted 15 lines · Theodore Ts'o

Quoted 12 lines · James Bottomley

I certainly buy this approach, and it fits well with the limited
data size there is in xattrs but Ted said in the initial proposal
the entire tree would be present in the file.  I can't see a need
for supplying the entire tree rather than reconstructing it but maybe there's an android use case I'm not seeing (Like not wanting
to waste limited CPU power).

Quoted 8 lines

You can certainly reconstruct the Merkle tree at install time, but it
does need to be saved on disk.  My assumption was that Merkle tree
would be written to disk by userspace, along with the fs-verity header, simply because it would make things much simpler for the kernel, and in my view you *have* to modify userspace in some way.

OK, so we agree then that what IMA provides: a hash and potential signature over the hash is sufficient input for what you want and does
provide existing tooling to achieve it.

Quoted 15 lines

Quoted 14 lines

Just so I understand the mechanics: The xattr would contain the head node.  When this is written, the tree would be reconstructed
from the file and verified.  If it verifies, it must be stored in
the filesystem data somehow (or at least the lowest layer), so all
subsequent uses of the file can proceed from the per page hash even
after unmount and remount?  Then I certainly think it suits both
cases.

Quoted 10 lines

Yes.... in theory we could store the bare root hash (and some other
bare minmum information, such as the PKCS7 signature and the elments
of a fs-verity header) in an xattr, and setting that xattr would magically cause the merkle tree to be reconstructed and stored on disk.  But most of the file sytem developers have considered the use
of setting or clearing xattrs to cause magically code paths to be executed to be an extremely bad idea.

Hang on, you've jumped straight from a discussion of ideas to an insane
implementation ... can we take a step back?  I'm thinking of three separate things

  1. Could the signature piece of this be done in the way IMA currently
     does file signatures.  We all agree "yes" on this, since a signed
     mekle hash head is the same size as an existing IMA signature and
     therefore does fit into xattrs.
  2. Could IMA use a merkle tree for hash verification a page at a time
     as part of its implementation?  I think the answer to this is yes,
     except the hook has to be somewhere in the page fault mechanism, so
     it would need some exploration and prototyping.
  3. Could the merkle tree be cached somehow in the filesystem (probably
     as part of the filesystem implementaiton)?  You've already assumed
     "yes" for this since it's how fs-verity is proposed to work.

So the issue we could discuss is how all this is tied together.  The
implementation could be your special format file for 3. but with the
head hash signature in an xattr.  What we get back on tar would be the
ordinary file + xattr, but perhaps tar could use this to reconstruct
the magic file on untar.

Quoted 8 lines

I do think the "special format magic file" is a violation of the unix
principles of files being "just files" but if it's the only way, I suppose I could be persuaded.  However, this part should also be discussed; it does seem like, to satisfy unix principles, the merkle
tree should be provided separately from the file, possibly as a fcntl().

For NFS, storing the tree or even an IMA signature in a trusted xattr is currently off the table because the protocol does not convey non-user xattrs.

My interest is finding a way to get binary signing
without the need of the xattr.

Interestingly, Solaris puts the signature in an ELF header. That would certainly work for NFS.

Quoted 17 lines

James

Quoted 13 lines

  And since I don't really care about the use case, I can let *you*
try to convince Cristoph and Al Viro that his is a good idea, despite
the general agreemnt by all file system developers that we've been
there, done that, decided it was a really bad idea, and let's never
ever do that again.  (And then you can get some of the "the IMA people are insane" taint on yourself.  :-)

                       - Ted

--
Chuck Lever

Quoted 105 lines · Chuck Lever

Quoted 104 lines

On Jan 28, 2018, at 10:03 AM, James Bottomley <James.Bottomley@Hans enPartnership.com> wrote:

Quoted 15 lines · Theodore Ts'o

Quoted 12 lines · James Bottomley

I certainly buy this approach, and it fits well with the limited
data size there is in xattrs but Ted said in the initial proposal
the entire tree would be present in the file. I can't see a need
for supplying the entire tree rather than reconstructing it but maybe there's an android use case I'm not seeing (Like not wanting
to waste limited CPU power).

Quoted 8 lines

You can certainly reconstruct the Merkle tree at install time, but it
does need to be saved on disk. My assumption was that Merkle tree
would be written to disk by userspace, along with the fs-verity header, simply because it would make things much simpler for the kernel, and in my view you *have* to modify userspace in some way.

OK, so we agree then that what IMA provides: a hash and potential signature over the hash is sufficient input for what you want and does
provide existing tooling to achieve it.

Quoted 15 lines

Quoted 14 lines

Just so I understand the mechanics: The xattr would contain the head node. When this is written, the tree would be reconstructed
from the file and verified. If it verifies, it must be stored in
the filesystem data somehow (or at least the lowest layer), so all
subsequent uses of the file can proceed from the per page hash even
after unmount and remount? Then I certainly think it suits both
cases.

Quoted 10 lines

Yes.... in theory we could store the bare root hash (and some other
bare minmum information, such as the PKCS7 signature and the elments
of a fs-verity header) in an xattr, and setting that xattr would magically cause the merkle tree to be reconstructed and stored on disk. But most of the file sytem developers have considered the use
of setting or clearing xattrs to cause magically code paths to be executed to be an extremely bad idea.

Hang on, you've jumped straight from a discussion of ideas to an insane
implementation ... can we take a step back? I'm thinking of three separate things

1. Could the signature piece of this be done in the way IMA
currently
does file signatures. We all agree "yes" on this, since a
signed
mekle hash head is the same size as an existing IMA signature
and
therefore does fit into xattrs.
2. Could IMA use a merkle tree for hash verification a page at a
time
as part of its implementation? I think the answer to this is
yes,
except the hook has to be somewhere in the page fault
mechanism, so
it would need some exploration and prototyping.
3. Could the merkle tree be cached somehow in the filesystem
(probably
as part of the filesystem implementaiton)? You've already
assumed
"yes" for this since it's how fs-verity is proposed to work.

So the issue we could discuss is how all this is tied together. The
implementation could be your special format file for 3. but with the
head hash signature in an xattr. What we get back on tar would be the
ordinary file + xattr, but perhaps tar could use this to reconstruct
the magic file on untar.

I do think the "special format magic file" is a violation of the unix
principles of files being "just files" but if it's the only way, I suppose I could be persuaded. However, this part should also be discussed; it does seem like, to satisfy unix principles, the merkle
tree should be provided separately from the file, possibly as a fcntl().

Quoted 3 lines

For NFS, storing the tree or even an IMA signature in a trusted xattr is currently off the table because the protocol does not convey non-user xattrs.

OK, but are you sure fixing that with a binary format interpreter is the correct thing to do? I think it should work both for IMA and selinux if you go this road, right?

Quoted 5 lines

My interest is finding a way to get binary signing without the need of the xattr.

Interestingly, Solaris puts the signature in an ELF header. That would certainly work for NFS.

That would work ... sort of the ELF equivalent of authenticode.
However, it only works for binaries ... what about the IMA uses of non
binary files?

James

Quoted 20 lines

Quoted 17 lines

James

Quoted 13 lines

And since I don't really care about the use case, I can let
*you*
try to convince Cristoph and Al Viro that his is a good idea, despite
the general agreemnt by all file system developers that we've been
there, done that, decided it was a really bad idea, and let's never
ever do that again. (And then you can get some of the "the IMA people are insane" taint on yourself. :-)

- Ted

--
Chuck Lever

Quoted 3 lines

On Jan 29, 2018, at 1:39 AM, James Bottomley <James.Bottomley@Hanse nPartnership.com> wrote:
On Sun, 2018-01-28 at 10:19 -0800, Chuck Lever wrote:

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

I do think the "special format magic file" is a violation of the unix principles of files being "just files" but if it's the only way, I suppose I could be persuaded.  However, this part should also be discussed; it does seem like, to satisfy unix principles, the merkle tree should be provided separately from the file, possibly as a fcntl().

Quoted 7 lines

Quoted 3 lines

For NFS, storing the tree or even an IMA signature in a trusted xattr is currently off the table because the protocol does not convey non-user xattrs.

OK, but are you sure fixing that with a binary format interpreter is the correct thing to do?  I think it should work both for IMA and selinux if you go this road, right?

NFSv4.2 does have the ability to convey security labels between NFS client and server. One thing that has been suggested is to define a security label that can store information such as file capabilities. The same tactic could be used to store a limited size item such as the signed root hash.

Given the ability to reconstitute the Merkle tree instead of storing it, that might be enough to allow NFS to play (using the mechanism that you and Ted are developing here).

Quoted 6 lines

Quoted 5 lines

My interest is finding a way to get binary signing without the need of the xattr.

Interestingly, Solaris puts the signature in an ELF header. That would certainly work for NFS.

Quoted 3 lines

That would work ... sort of the ELF equivalent of authenticode. However, it only works for binaries ... what about the IMA uses of non binary files?

Fair enough, though I'm not sure Oracle (for example) has a non- executable use case. We are focusing largely on similar virtualization use cases as yours.

OK, so we agree then that what IMA provides: a hash and potential signature over the hash is sufficient input for what you want and does provide existing tooling to achieve it.

1. Could the signature piece of this be done in the way IMA currently does file signatures. �We all agree "yes" on this, since a signed mekle hash head is the same size as an existing IMA signature and therefore does fit into xattrs.

2. Could IMA use a merkle tree for hash verification a page at a time as part of its implementation? �I think the answer to this is yes, except the hook has to be somewhere in the page fault mechanism, so it would need some exploration and prototyping.

3. Could the merkle tree be cached somehow in the filesystem (probably as part of the filesystem implementaiton)? �You've already assumed "yes" for this since it's how fs-verity is proposed to work.

Speaking as a kernel developer, it makes a more sense to keep things in the kernel,

means that the Docker client (or the package manager, etc.) needs a minor change to call the userspace library, that's infinitely preferable to keeping huge amounts of complexity in non-swappable kernel memory --- which increases the attack surface of the kernel, and so on.

So in my opinion, clean design of the kernel trumps the requirement of "not one change, not one jot, in the Docker client".

It could be that the requiremnts of "keep the kernel changes simple" and "no massive latency at file open time", means that requiremnt sets of fs-verity and IMA are irreconcilable. Which is fine as far as I'm concerned. Maybe IMA and fs-verity should be considered orthogonal solutions.

- Ted

Quoted 4 lines · James Bottomley

OK, so we agree then that what IMA provides: a hash and potential signature over the hash is sufficient input for what you want and does provide existing tooling to achieve it.

Well, that's not all IMA provides. Remmber, the 'M' in IMA is *measurement*. IMA was originally about checksuming every single file that is opened and writing it to a long. You could suppress it by using SELinux to tag files (using xattrs which are *not* signed) and then writing an IMA policy which tells IMA to ignore files that belong to a certain SE Linux tag. This is something that I *don't* want and today, enabling CONFIG_IMA drags all of this (including doing text parsing of the IMA policy in the kernel, SELinux, etc.) into the kernel.

One of my other complaints about IMA is that it's an integrated solution, with a huge amount of complexity. If it was only about file signing, that would be one thing, but that's actually not the case today. Hence my comment about I don't want fs-verity to have a dependency on IMA, such that we are forced to drag in all of IMA and SELinux for anyone who wants to use use fs-verity.

This is fine; I don't want to have to dictate changes to IMA; I'd much prefer to avoid the complexity instead of trying to reform it, since I'm sure the IMA folks will be happy to explain why there are all sorts of reasons why things have to be done the way it has to be done. For example, the assertion that the latency hit at open(2) *must* be there in order to kowtow to Microsoft because of its Trusted Boot policies; fair enough, but *I* don't care about Trusted Boot, and it's not fair to impose penalties on all scenarios because of a desire to keep Microsoft happy just for one particular use case.

Quoted 5 lines

1. Could the signature piece of this be done in the way IMA currently does file signatures.  We all agree "yes" on this, since a signed mekle hash head is the same size as an existing IMA signature and therefore does fit into xattrs.

Well.... not exactly.

It is fair to say that there are two parts of integrity metadata; and the Merkle tree can be reconstructed at file install time.

Quoted 4 lines

2. Could IMA use a merkle tree for hash verification a page at a time as part of its implementation?  I think the answer to this is yes, except the hook has to be somewhere in the page fault mechanism, so it would need some exploration and prototyping.

The way you are formulating things presumes that all of IMA has to be dragged into any file integrity solution. That's begging the question. As I've mentioned above, IMA has all sorts of complexity which is currently mandatory, and I'm not volunteering to disentagling the mess to make it be sane. (And if you don't like that word, how about, "designed with good taste?")

Quoted 4 lines

3. Could the merkle tree be cached somehow in the filesystem (probably as part of the filesystem implementaiton)?  You've already assumed "yes" for this since it's how fs-verity is proposed to work.

I don't consider this the Merkel tree to a *cache*, however because if you don't mind a massive latency at file open tme, you can just use the existing IMA mechanism. So a core part of the design is that the Merkle file is stored permanently (*not* as a cache) alongside the file. And if the file is renamed, the Merkle tree should come along for the ride.

Whether the Merkle tree is reconstructed as part of the file / package installation process, or whether the Merkle tree stored as part of the package or streamed from the app store, etc., is an implementation detail, and I don't think we need to prescribe one way of doing things. I *do* think though we should allow for possibility where limitations on the local CPU power is such that it would be preferable for the Merkle tree to be supplied from a remote server instead of generated on the local system.

The problem is you have a specific use case in mind, involving the Docker client, where you want to store the signature in an xattr, and then not require any local changes to the Docker client --- and that's not in my requirements set, and it was *your* statement that the Docker client MUST NOT be modified which forced where (a) the Merkle tree must be reconstructed in the kernel, and (b) it must be triggered by setting the xattr.

My goal is to keep things simple, which means

* No parsing of the IMA policy as a text input in the kernel
* No Merkle tree construction in the kernel (which is also true of dm-verity)
* No magic xattr triggering

Speaking as a kernel developer, it makes a more sense to keep things in the kernel, and do as much in userspace as possible --- and if that means that the Docker client (or the package manager, etc.) needs a minor change to call the userspace library, that's infinitely preferable to keeping huge amounts of complexity in non-swappable kernel memory --- which increases the attack surface of the kernel, and so on.

So in my opinion, clean design of the kernel trumps the requirement of "not one change, not one jot, in the Docker client".

It could be that the requiremnts of "keep the kernel changes simple" and "no massive latency at file open time", means that requiremnt sets of fs-verity and IMA are irreconcilable. Which is fine as far as I'm concerned. Maybe IMA and fs-verity should be considered orthogonal solutions.

- Ted

Sigh, there seems to be some confusion. �Initially, when the Integrity Measurement Architecture (IMA) was first upstreamed, it only extended the trusted boot concept of measuring files before use up to the OS, but that was a long time ago. �Since then, IMA-appraisal was upstreamed, which extends the secure boot concept of verifying signatures up to the running OS.

Enabling IMA doesn't automatically require SELinux or any other LSM labels. �The rule granularity is up to you.

Quoted 7 lines · Mimi Zohar

Sigh, there seems to be some confusion.  Initially, when the Integrity Measurement Architecture (IMA) was first upstreamed, it only extended the trusted boot concept of measuring files before use up to the OS, but that was a long time ago.  Since then, IMA-appraisal was upstreamed, which extends the secure boot concept of verifying signatures up to the running OS.

Part of the problem is the documentation doesn't make any of this at all clear.

Indeed, I'll note that Documentation/ABI/testing/ima_policy
still talks about using file system magic numbers to determine whether or not files should be measured, and I've been given to understand you're using a per-filesystem flag now.

The documentation seems to strongly imply that in order to be secure, you can't use IMA by itself, you have to use EVM as well. Exactly which components can be used independently is not clear, and apparently I made the wrong guesses when trying to read through the Linux-IMA wiki pages as well as the Gentoo pages on IMA and EVM. Maybe it's just the documentation is badly written, but it leaves the impression of *extreme* complexity.

I did try to play with it, but ima-evm-utils aren't packaged for Debian, and when I tried building from source, they apparently don't even build on Debian Testing. (Sorry, I don't use RHEL for my development systems.)

And I'll note the Gentoo pages warn, "don't use
on production systems". All of which do not make for a good first look for IMA.

Quoted 3 lines

Enabling IMA doesn't automatically require SELinux or any other LSM labels.  The rule granularity is up to you.

Yes, but if I only want to have a dozen or so files to be data integrity protect, it appears that it's not simple to do, *without* using SELinux. And anytime SELinux and "simple" go in the same sentence, I weep a little. Every few years I try configuring SELinux on Debian development laptop. And I conclude that I'm too stupid for to configure SELinux.

A lot of people have requested being able to identify files based on pathnames. �I don't need to tell you this isn't safe from a security perspective. �So how would you identify these few files? �I doubt you are planning on hard coding them. �If you have a generic solution, I would really be interested in hearing it.

Quoted 5 lines · Mimi Zohar

A lot of people have requested being able to identify files based on pathnames.  I don't need to tell you this isn't safe from a security perspective.  So how would you identify these few files?  I doubt you are planning on hard coding them.  If you have a generic solution, I would really be interested in hearing it.

So what we are implementing with fs-verity is that verification is enabled by setting a flag in the inode ("the verity bit") which causes the file system to enforce the data integrity checks. This bit can be checked by using the FS_IOC_GETFLAGS ioctl (like any of the other existing file system flags, such as immutable, append-only, no-COW, etc.)

What this means is that it would be possible for userspace application to simply open a file (which might, for example, be a privileged APK) and before using it, check the verity bit via the open file descriptor. If the verity bit is set, then the userspace application can safely read from the file, and no that it hasn't been tampered with, even via an off-line evil maid attack. In this particular case case, there isn't even a need to use SELinux, or indeed, any LSM at all. No need to compile in EVM, no need to compile in IMA, no need to compile in SELinux, etc.

In other use cases, whether or not a file has the verity bit set could be used by an LSM that wishes to make policy restrictions --- for
example: "if the file is setuid, then the verity bit *must* be set". Or there could be a policy where all executables *must* have the verity bit set. This model has the advantage of a very clean separation between the policy and the mechanism, where the mechanism exports a single bit, "is this file one which is protected by the verity bit"?

Granted, it is a different model than what IMA/EVM use. But it is much simpler, and it is optimized for use cases where most of the files might not be data integrity protected (perhaps because most of then security-critical files are located on a read-only volume being protected using dm-verity).

Because we use a Merkle tree, we are also making the tradeoff between a complete verification of the entire contents of the file at file open time (which imposes a file open latency, and means that if the file can be tampered after it is opened, IMA won't detect the problem), versus verification at readpage time (which means that you might fail while reading the file, instead of finding out at open time). This is again consistent with dm-verity, where we do not attempt to verify the checksum of the entire block device at system startup; instead we check on each block read, and if the verification fails, we fail the read in question.

For some use cases, the use of a full-file hash ala today's IMA-Appraisal might be a better choice. I have never claimed that fs-verity was intended to be a replacemnt for IMA.

Cheers,

- Ted

P.S. I wonder if it was a mistake to not choose a whole new name for IMA-Appraisal. There are lots of documentations on the web which talk about "IMA", and it's not clear if it supposed to mean "IMA-Measure", or a generic term encompassing "IMA-Appraisal" and "IMA-Meaure". One might be able to guess based on how out-of-date the rest of web page happens to be, but it's really not clear.

Also, the two concepts are quite different, and data integrity via checking a digitally signed hash is only partially related to "measuring" a file. Perhaps it's related by virtue of the fact that you have to calculate a cryptographic checksum over the entire file.

But once you get to data integrity protected via a Merkle tree at file read time, this is extremely quite far away from any traditional definition of "measurement". So purely from a naming convention, perhaps trying to take data integrity verification using Merkle trees should forcing it into the IMA framework might not be such a great fit.

So the filesystem is enforcing a policy set by userspace. �What is protecting that policy? �Can the verity bit be unset once set?

Quoted 6 lines

But once you get to data integrity protected via a Merkle tree at file read time, this is extremely quite far away from any traditional definition of "measurement". So purely from a naming convention, perhaps trying to take data integrity verification using Merkle trees should forcing it into the IMA framework might not be such a great fit.

At what point is the signature on the Merkle tree hash verified? �I can't imagine it being done every time a page is read. �It must be done and the result cached at file open.

At what point is the signature on the Merkle tree hash verified? �I can't imagine it being done every time a page is read. �It must be done and the result cached at file open.

Quoted 3 lines · Mimi Zohar

At what point is the signature on the Merkle tree hash verified?  I can't imagine it being done every time a page is read.  It must be done and the result cached at file open.

Sorry, I misread your question. The signature on the Merkle tree hash is verified the file is opened, and then validated Merkle tree hash is cached in the in-memory inode data structure.

This is similar to how we cache the per-file key in fscrypt; once the key is derived, we keep it in the inode cache until the inode is dropped from the inode cache, or after a userspace request to revoke all keys derived from a user's login key (which is triggered when the user logs out of their ChromeOS session).

- Ted

Bringing it all together, what is needed?
- the signature of the Merkle tree hash
- a method for validating the signature
- a method for knowing if fs-verity is enabled on the system A mode where fs-verity can not be disabled on the local, running system, once enabled.
- lastly, a policy.  Just because a file has a signature, does not necessarily imply that it should be verified.

The LSM security_file_open hook is where fs-verity and IMA meet. �The fs-verity Merkle tree hash signature would be another IMA-appraisal integrity verification method.

For those that are interested in attesting to the measurement list or including the file hash/signatures in the audit log, the same mechanisms that currently exist would be in place for using the fs- verity merkle tree signature. �The same mechanisms that are in place for including file signatures in software packages could be re-used.

Bringing it all together, what is needed?
- the signature of the Merkle tree hash
- a method for validating the signature
- a method for knowing if fs-verity is enabled on the system A mode where fs-verity can not be disabled on the local, running system, once enabled.
- lastly, a policy. �Just because a file has a signature, does not necessarily imply that it should be verified.

Quoted 4 lines · Mimi Zohar

The LSM security_file_open hook is where fs-verity and IMA meet.  The fs-verity Merkle tree hash signature would be another IMA-appraisal integrity verification method.

I wasn't planning on using the file open hook for fs-verity at all, since the merkle hash signature is going to be cached in the per-file system inode --- e.g., for ext4, it would be stored in EXT4_I(inode), aka "struct ext4_inode_info".

Quoted 5 lines

For those that are interested in attesting to the measurement list or including the file hash/signatures in the audit log, the same mechanisms that currently exist would be in place for using the fs- verity merkle tree signature.  The same mechanisms that are in place for including file signatures in software packages could be re-used.

This is *your* set of requiremnets, not mine. If it's easy to do, that's fine. But if it doesn't, piling on extra requirements which fs-verity can't meet is not a proof of fs-verity being not fit for purpose, at least some use cases (e.g,. the ones we are envisioning for fs-verity).

Again, I'm not the one arguing for IMA/fs-verity integration, and while I am happy to work with IMA if at all possible, I am *not* interested in compromising the fs-verity use case, or adding vast amounts of complexity to fs-verity just to confirm to the IMA architecture. (That is, any complexity will need to be in optional userspace components, such as what I've been discussing with James to support his docker use case, or in the security/integrity subtree. I really *don't* want to deal with unnecessary complexity into fs/ext4, fs/f2fs, or fs/verity.)

Quoted 8 lines

Bringing it all together, what is needed?
- the signature of the Merkle tree hash
- a method for validating the signature
- a method for knowing if fs-verity is enabled on the system A mode where fs-verity can not be disabled on the local, running system, once enabled.
- lastly, a policy.  Just because a file has a signature, does not necessarily imply that it should be verified.

So I want to support a very simple case where the policy is simply "the public key needed to verify the PKCS7 signing block is in a trusted keyring".

If someone needs something more complex than the simple "key is in the trusted keyring", I'm happy to say that the right answer is to use the LSM hooks, and it should be straight forward to provide interfaces so that the LSM can determine that file system supports verity and the file has the verity bit set. The LSM could also apply additional restrictions (if the SELinux file type is supersekrit_t, then only a smaller set of keys will be allowed to be used to sign the PKCS7 signature block).

Is this sufficient to allow fs-verity the ability to use IMA, if the policy so requests it?

Quoted 6 lines · Theodore Ts'o

Quoted 4 lines · Mimi Zohar

The LSM security_file_open hook is where fs-verity and IMA meet. �The fs-verity Merkle tree hash signature would be another IMA-appraisal integrity verification method.

Quoted 4 lines

I wasn't planning on using the file open hook for fs-verity at all, since the merkle hash signature is going to be cached in the per-file system inode --- e.g., for ext4, it would be stored in EXT4_I(inode), aka "struct ext4_inode_info".

Ok, so if not at open, at what point do you plan on validating the merkle hash signature?

I agree with you that the integrity architecture needs to be easily extendsible to support different integrity authentication methods. �The list, below, is a set of fs-verity functions needed to be exported for IMA. �Hopefully I didn't miss anything.

Quoted 3 lines

So I want to support a very simple case where the policy is simply "the public key needed to verify the PKCS7 signing block is in a trusted keyring".

Which trusted keyring? �How is the keyring populated? �What prevents other keys from being added to that keyring?

So the implied policy is that no signatures will be verified unless userspace turns the bit on. �So you're trust is based in userspace.

Quoted 8 lines

If someone needs something more complex than the simple "key is in the trusted keyring", I'm happy to say that the right answer is to use the LSM hooks, and it should be straight forward to provide interfaces so that the LSM can determine that file system supports verity and the file has the verity bit set. The LSM could also apply additional restrictions (if the SELinux file type is supersekrit_t, then only a smaller set of keys will be allowed to be used to sign the PKCS7 signature block).

Interesting, so there is an additional policy limiting which keys may be used based on an LSM labels, if so desired.

Quoted 10 lines · Mimi Zohar

Quoted 8 lines · Theodore Ts'o

Quoted 5 lines · Mimi Zohar

The LSM security_file_open hook is where fs-verity and IMA meet.  The fs-verity Merkle tree hash signature would be another IMA-appraisal integrity verification method.

Quoted 7 lines

Quoted 4 lines

I wasn't planning on using the file open hook for fs-verity at all, since the merkle hash signature is going to be cached in the per-file system inode --- e.g., for ext4, it would be stored in EXT4_I(inode), aka "struct ext4_inode_info".

Ok, so if not at open, at what point do you plan on validating the merkle hash signature?

I said we would not using the LSM file_open hook.  Instead it will be done in ext4_file_open() or f2fs_file_open().  That's because the place where the key is cached is in the file system specific portion of the inode.  This is what I meant by EXT4_I(inode) aka struct ext4_inode_info.  So it does happen at file open time; it just doesn't use the LSM file_open hook.  One of the features of fs-verity is that it does *not* require the use of the LSM infrastructure.

This is all sounding appallingly ext4/f2fs specific. �What about other filesystems that might want this feature, how would they play?

I assume also that a write of the magic file updates the key and signature in the inode metadata? �I suppose this also avoids the original IMA locking problem by sorting it out below the VFS, but it also means you have to invent mechanisms to query the key (user space might want to know for audit purposes) and to update the key (in case the original is compromised).

Also when you say "key" presumably you mean pointer to x509 public certificate in a keyring somewhere, say by DN and Version or SKID?

I really think some time needs to be spent figuring out how it should be supported in a fs generic way (at least for the user visible API) otherwise every fs will grow its own version and we'll have a user tooling nightmare on our hands.

Quoted 5 lines · James Bottomley

This is all sounding appallingly ext4/f2fs specific.  What about other filesystems that might want this feature, how would they play?

Like fscrypto, where most of the code is in fs/crypto, most of the fs-verity will be in fs/verity.  There will be minimal hooks in a particular file system, so if another file system wants to play, then can do so relatively easily.

Quoted 6 lines

I assume also that a write of the magic file updates the key and signature in the inode metadata?  I suppose this also avoids the original IMA locking problem by sorting it out below the VFS, but it also means you have to invent mechanisms to query the key (user space might want to know for audit purposes) and to update the key (in case the original is compromised).

Like dm-verity, fs-verity only supports the read-only case.  So we don't need to worry about write updates.

Quoted 2 lines

Also when you say "key" presumably you mean pointer to x509 public certificate in a keyring somewhere, say by DN and Version or SKID?

Yes.  The cert might be hard-coded public key in the kernel sources; or there might be a hard-coded CA public key, and userspace is allowed to add new certs to the keyring so long as they are signed by a CA cert in the system keyring.  Again, I refer you to how how signed kernel modules are managed.

It would be possible to add something more complicated, involving SELinux policies, or IMA policies --- but that's not something that I need, so I view that as something that can be added later, by those who need it.

Quoted 4 lines

I really think some time needs to be spent figuring out how it should be supported in a fs generic way (at least for the user visible API) otherwise every fs will grow its own version and we'll have a user tooling nightmare on our hands.

Like with fscrypt, there will be a standard set of user interfaces, most of which will be implemented in file sytem generic code.  In the case of fscrypt, the same userspace is used for ext4 and f2fs in AOSP.

For the generic desktop case, there is a generic userspace tooling[1] which we are hoping will be picked up by Ubuntu as a replacement for ecryptfs in the next year or so.  (Discussions are in process; it is not plan of record yet as far as I know.)

[1] https://github.com/google/fscrypt

The goal for fs-verity to keep the userspace interface as simple as possible, and of *course* it will be file system generic (it will work for ext4 and f2fs, and it will be set up to be easy for other interested file systems to add support for it).

There are admittedly tradeoffs from the model model used by IMA/EVM where the security feature is imposed on the file system code without its knowledge or consent, and that is some minor changes are needed in the file system code.  But there are some efficiencies and some approaches that it allows that aren't possible with one where new file system functionality is shoehorned into an LSM.

Quoted 4 lines

Like fscrypto, where most of the code is in fs/crypto, most of the fs-verity will be in fs/verity.��There will be minimal hooks in a particular file system, so if another file system wants to play, then can do so relatively easily.

OK, sounds good ... I notice, now I look, that fscrypt uses xattrs (albeit hidden under the covers of get/set_context), will dm-verity use the same trick or do people really need space in the inode?

Quoted 2 lines

Like dm-verity, fs-verity only supports the read-only case.��So we don't need to worry about write updates.

OK, but we still need to retrieve the key (if only for the "I just found this image what did I sign the file with?" case). �I assume, in- line with fscrypt, you'll use ioctls for that.

One thought that strikes is that since you're already using xattrs under the covers, they could simply be exposed, at least for the new VERITY index I assume you'll be using. �That way most archive systems would pick up the signature. �If you're worried about locking inversions and other problems on write, the xattrs could be read only, so the archive would need extra help being rewritten.

No, as long as you do cert to signature matching by DN/Serial or SKID it's fine if the key policy is must match and be in system keyring. �The reason being we're inching towards dumping all the UEFI certs into the system keyring which wouldn't enhance my view of security if I knew some random OEM could also produce acceptable signatures for my filesystem.

The implementations in fscrypt look pretty close to a lot of what IMA does, just coded under the VFS.

�Assuming fs-verity follows the model, it may be possible to have a generic case that just works with IMA and a specific case for a fs where the presence of the new verity superblock operations overrides the IMA attachment. �This might actually allow building a model where either the FS just takes care of integrity via the verity sops or we use the generic IMA. �The only complexity might be archive restore of the image.

Quoted 8 lines · James Bottomley

Quoted 4 lines

Like fscrypto, where most of the code is in fs/crypto, most of the fs-verity will be in fs/verity.��There will be minimal hooks in a particular file system, so if another file system wants to play, then can do so relatively easily.

OK, sounds good ... I notice, now I look, that fscrypt uses xattrs (albeit hidden under the covers of get/set_context), will dm-verity use the same trick or do people really need space in the inode?

I assume you mean fs-verity above, and no, we aren't going to use xattrs because the Merkle tree won't fit in the xattr. So the plan was to put the fs-verity header, the PKCS7 signature, and the Merkle tree after i_size (rounded to a blocksize boundary). Remember, the fs-verity case we only worry about the read-ony case.

As I stated above, we need to put the Merkle tree after i_size anyway, so the current plan doesn't use xattrs at all. Xattr storage space is also precious (especially if you are trying to keep all of the xattrs

Quoted 10 lines · Theodore Ts'o

Quoted 8 lines · James Bottomley

Quoted 4 lines

Like fscrypto, where most of the code is in fs/crypto, most of the fs-verity will be in fs/verity. There will be minimal hooks in a particular file system, so if another file system wants to play, then can do so relatively easily.

OK, sounds good ... I notice, now I look, that fscrypt uses xattrs (albeit hidden under the covers of get/set_context), will dm-verity use the same trick or do people really need space in the inode?

Quoted 5 lines

I assume you mean fs-verity above, and no, we aren't going to use xattrs because the Merkle tree won't fit in the xattr. So the plan was to put the fs-verity header, the PKCS7 signature, and the Merkle tree after i_size (rounded to a blocksize boundary). Remember, the fs-verity case we only worry about the read-ony case.

I think putting valid data beyond EOF is going to be problematic for many filesystems. Getting things like truncate right are hard enough without having to special case a bunch of new functionality that specifically allows IO access beyond EOF. Indeed, how does "truncate isize but leave special data behind" work and what's the userspace API to drive it? And how does it interact with all the page cache code that checks for page->index beyond EOF to detect a truncated page that should not be accessed?

There's also further complications for filesystems like XFS e.g. how do we tell the difference between valid data beyond EOF and speculative allocation (done by delalloc) beyond EOF that contains no data and can be removed if it is not written to in a short while?

This just seems like a horrible can of worms to me and is not something we should be building generic infrastructure around.

Just how big do these merkle trees get, anyway?

Quoted 3 lines

As I stated above, we need to put the Merkle tree after i_size anyway, so the current plan doesn't use xattrs at all. Xattr storage space is also precious (especially if you are trying to keep all of the xattrs

No it's not. xattr space is specifically designed for uses like this, and if you have to take an extra IO to read it then that's the cost of storing large chunks of non-userdata data on a file. You;ve got to take extra IOs to read the merkle tree if it's stored beyond EOF anyway, so it doesn't matter if we take extra IOs to read it from an xattr....

Quoted 32 lines · Dave Chinner

Quoted 10 lines · Theodore Ts'o

Quoted 8 lines · James Bottomley

Quoted 4 lines

Like fscrypto, where most of the code is in fs/crypto, most of the fs-verity will be in fs/verity. There will be minimal hooks in a particular file system, so if another file system wants to play, then can do so relatively easily.

OK, sounds good ... I notice, now I look, that fscrypt uses xattrs (albeit hidden under the covers of get/set_context), will dm-verity use the same trick or do people really need space in the inode?

Quoted 5 lines

I assume you mean fs-verity above, and no, we aren't going to use xattrs because the Merkle tree won't fit in the xattr. So the plan was to put the fs-verity header, the PKCS7 signature, and the Merkle tree after i_size (rounded to a blocksize boundary). Remember, the fs-verity case we only worry about the read-ony case.

I think putting valid data beyond EOF is going to be problematic for many filesystems. Getting things like truncate right are hard enough without having to special case a bunch of new functionality that specifically allows IO access beyond EOF. Indeed, how does "truncate isize but leave special data behind" work and what's the userspace API to drive it? And how does it interact with all the page cache code that checks for page->index beyond EOF to detect a truncated page that should not be accessed?

There's also further complications for filesystems like XFS e.g. how do we tell the difference between valid data beyond EOF and speculative allocation (done by delalloc) beyond EOF that contains no data and can be removed if it is not written to in a short while?

Quoted 4 lines

This just seems like a horrible can of worms to me and is not something we should be building generic infrastructure around.

Just how big do these merkle trees get, anyway?

The Merkle tree will have one checksum per "leaf block" of the filesystem (though I'd recommend to use a fixed-size checksum leaf block like 4KB so that userspace doesn't need to care about the actual filesystem blocksize on disk).

Since the tree size depends on file size, it would hit the 64KB xattr size limit at 64MB (CRC32) or 8MB (SHA256), unless we also allow larger xattrs to userspace.

There was an ext4 feature landed in 4.13 to allow larger on-disk xattrs than the previous 4KB (single block) limit (essentially any size xattr could be stored), so that wouldn't be a problem if the userspace xattr API limit was removed.

Quoted 39 lines · Dave Chinner

Quoted 14 lines · Theodore Ts'o

Quoted 11 lines · James Bottomley

Quoted 6 lines

Like fscrypto, where most of the code is in fs/crypto, most of the fs-verity will be in fs/verity.  There will be minimal hooks in a particular file system, so if another file system wants to play, then can do so relatively easily.

OK, sounds good ... I notice, now I look, that fscrypt uses xattrs (albeit hidden under the covers of get/set_context), will dm-verity use the same trick or do people really need space in the inode?

Quoted 6 lines

I assume you mean fs-verity above, and no, we aren't going to use xattrs because the Merkle tree won't fit in the xattr.  So the plan was to put the fs-verity header, the PKCS7 signature, and the Merkle tree after i_size (rounded to a blocksize boundary).  Remember, the fs-verity case we only worry about the read-ony case.

I think putting valid data beyond EOF is going to be problematic for many filesystems. Getting things like truncate right are hard enough without having to special case a bunch of new functionality that specifically allows IO access beyond EOF. Indeed, how does "truncate isize but leave special data behind" work and what's the userspace API to drive it? And how does it interact with all the page cache code that checks for page->index beyond EOF to detect a truncated page that should not be accessed?

There's also further complications for filesystems like XFS e.g. how do we tell the difference between valid data beyond EOF and speculative allocation (done by delalloc) beyond EOF that contains no data and can be removed if it is not written to in a short while?

Quoted 4 lines

This just seems like a horrible can of worms to me and is not something we should be building generic infrastructure around.

Just how big do these merkle trees get, anyway?

The Merkle tree will have one checksum per "leaf block" of the filesystem (though I'd recommend to use a fixed-size checksum leaf block like 4KB so that userspace doesn't need to care about the actual filesystem blocksize on disk).  After that, there is a tree of checksums from the leaf blocks up to the root.  If there was a weak checksum like CRC32 (4 bytes/leaf) then the tree size would be somewhat over 0.1% of the file size.  If the tree has a strong checksum like SHA256 (32 bytes/leaf) then the overhead is over 0.8%.

I think putting valid data beyond EOF is going to be problematic for many filesystems. Getting things like truncate right are hard enough without having to special case a bunch of new functionality that specifically allows IO access beyond EOF. Indeed, how does "truncate isize but leave special data behind" work and what's the userspace API to drive it? And how does it interact with all the page cache code that checks for page->index beyond EOF to detect a truncated page that should not be accessed?

Quoted 8 lines · Dave Chinner

I think putting valid data beyond EOF is going to be problematic for many filesystems. Getting things like truncate right are hard enough without having to special case a bunch of new functionality that specifically allows IO access beyond EOF. Indeed, how does "truncate isize but leave special data behind" work and what's the userspace API to drive it? And how does it interact with all the page cache code that checks for page->index beyond EOF to detect a truncated page that should not be accessed?

This isn't an issue because once fs-verity protection is enabled, the file becomes immutable.

That is, you can delete the file, but you
can't modify it afterwards --- so you don't have to truncate it.

We don't actually want userspace to be able to access pages beyond i_size via read(2) or mmap(2), so I don't believe this to be a problem. It's only kernel code that needs to access the pages via find_get_page.

Quoted 7 lines · Mimi Zohar

Quoted 6 lines · Theodore Ts'o

Quoted 4 lines · Mimi Zohar

The LSM security_file_open hook is where fs-verity and IMA meet.  The fs-verity Merkle tree hash signature would be another IMA-appraisal integrity verification method.

Quoted 7 lines

Quoted 4 lines

I wasn't planning on using the file open hook for fs-verity at all, since the merkle hash signature is going to be cached in the per-file system inode --- e.g., for ext4, it would be stored in EXT4_I(inode), aka "struct ext4_inode_info".

Ok, so if not at open, at what point do you plan on validating the merkle hash signature?

I said we would not using the LSM file_open hook. Instead it will be done in ext4_file_open() or f2fs_file_open(). That's because the place where the key is cached is in the file system specific portion of the inode. This is what I meant by EXT4_I(inode) aka struct ext4_inode_info. So it does happen at file open time; it just doesn't use the LSM file_open hook. One of the features of fs-verity is that it does *not* require the use of the LSM infrastructure.

Quoted 4 lines

I agree with you that the integrity architecture needs to be easily extendsible to support different integrity authentication methods. The list, below, is a set of fs-verity functions needed to be exported for IMA.  Hopefully I didn't miss anything.

Just to be clear, who is signing up to do all of this work to enable fs-verity for IMA? In particular, if you and James are asking *me* to do all of this development work, when I have no need for any of these benefits, I'm going to have to decline the honor. I'm happy to try to modify the design to make it easier for IMA to use fs-verity. That's different from actually implementing it.

Quoted 6 lines

Quoted 3 lines

So I want to support a very simple case where the policy is simply "the public key needed to verify the PKCS7 signing block is in a trusted keyring".

Which trusted keyring?  How is the keyring populated?  What prevents other keys from being added to that keyring?

It will probably be a separate keyring, but in theory we could use the keyring used to validate signed kernel modules.

There are a number of
different ways the policy for signed kernel modules can be configured[1], and the same would be true for fs-verity. (The reason for a separate kernel keyring is to allow for a separation between those keys trusted to sign kernel modules, and those trusted to sign for fs-verity.)

[1] https://01.org/linuxgraphics/gfx-docs/drm/admin-guide/module-signing.html

Quoted 2 lines

So the implied policy is that no signatures will be verified unless userspace turns the bit on.  So you're trust is based in userspace.

For the initial use case that we're interested in, yes. The check to make sure the verity bit is enabled for a small set of privileged APK's will be done by the AOSP userspace, and the AOSP userspace lives in a read-only system partition which is protected using dm-verity. (And the keys for dm-verity are protected using asigned bootloader; it is possible to do data integrity without using IMA/EVM, and millions and millions of Android devices are doing in this way.)

The problem we're trying to solve is that in order to allow for the privileged APK's to be upgraded more frequently, they are stored in the /data partition, which is encrypted; however, fs-crypt does not provide data integrity guarntees. fs-verity allows us to protect those APK's (which are equivalent in power to software in the system partition, being privileged APK's).

Note also that an APK will often have large number of translations that will never be used on a particular device, so doing full check of the entire APK will mean doing unneeded work. Obviously you have to trade off the time of doing sequential reads versus the random reads for doing on-demand Merkle tree checks, but if you have translation resources for Russian, Chinese, French, Spanish, etc, and the only language used on a particular device is English, then the on-demand checks can often be far more efficient, especially if the one of the figures of merit is how long it takes for an application to start painting the first pixel after the user clicks on its icon.

Quoted 9 lines

Quoted 8 lines

If someone needs something more complex than the simple "key is in the trusted keyring", I'm happy to say that the right answer is to use the LSM hooks, and it should be straight forward to provide interfaces so that the LSM can determine that file system supports verity and the file has the verity bit set. The LSM could also apply additional restrictions (if the SELinux file type is supersekrit_t, then only a smaller set of keys will be allowed to be used to sign the PKCS7 signature block).

Quoted 2 lines

Interesting, so there is an additional policy limiting which keys may be used based on an LSM labels, if so desired.

If someone implements it, yes. If there is a business case where the benefits outweighs the costs, and the headcount can be funded, it would happen. However, having spent time trying to understand the IMA/EVM/LSM/SELinux architecture, I've spent enough time to decide it's not something I would do on my own time, for fun. :-)

I see. �So anyone else adding security features is required to use the LSM hooks, but your use case is different? (rhetorical statement)

Quoted 5 lines

If someone implements it, yes. If there is a business case where the benefits outweighs the costs, and the headcount can be funded, it would happen. However, having spent time trying to understand the IMA/EVM/LSM/SELinux architecture, I've spent enough time to decide it's not something I would do on my own time, for fun. :-)

I understand you have lots of experience reading and understanding kernel code, but understanding another subsystem, without ever having been involved, no matter how well written that code is, takes time to understand. �There is some really well written code in the security tree, especially considering the constraints put on it.

dm-verity does not use LSM. Signed kernel-modules does not use LSM.

Quoted 6 lines · James Bottomley

OK, so we agree then that what IMA provides: a hash and potential signature over the hash is sufficient input for what you want and does
provide existing tooling to achieve it.

Well, that's not all IMA provides.  Remmber, the 'M' in IMA is *measurement*.  IMA was originally about checksuming every single file that is opened and writing it to a long.  You could suppress it by using SELinux to tag files (using xattrs which are *not* signed) and then writing an IMA policy which tells IMA to ignore files that belong to a certain SE Linux tag.  This is something that I *don't* want and today, enabling CONFIG_IMA drags all of this (including doing text parsing of the IMA policy in the kernel, SELinux, etc.) into the kernel.

One of my other complaints about IMA is that it's an integrated solution, with a huge amount of complexity.  If it was only about file signing, that would be one thing, but that's actually not the case today.

  Hence my comment about I don't want fs-verity to have a dependency on IMA, such that we are forced to drag in all of IMA and SELinux for anyone who wants to use use fs-verity.

This is fine; I don't want to have to dictate changes to IMA; I'd much prefer to avoid the complexity instead of trying to reform it, since I'm sure the IMA folks will be happy to explain why there are all sorts of reasons why things have to be done the way it has to be done. For example, the assertion that the latency hit at open(2) *must* be there in order to kowtow to Microsoft because of its Trusted Boot policies; fair enough, but *I* don't care about Trusted Boot, and it's not fair to impose penalties on all scenarios because of a desire to keep Microsoft happy just for one particular use case.

Quoted 7 lines

   1. Could the signature piece of this be done in the way IMA currently
      does file signatures.  We all agree "yes" on this, since a signed
      mekle hash head is the same size as an existing IMA signature and
      therefore does fit into xattrs.

Well.... not exactly.

It is fair to say that there are two parts of integrity metadata; and the Merkle tree can be reconstructed at file install time.

Quoted 7 lines

   2. Could IMA use a merkle tree for hash verification a page at a time
      as part of its implementation?  I think the answer to this is yes,
      except the hook has to be somewhere in the page fault mechanism, so
      it would need some exploration and prototyping.

The way you are formulating things presumes that all of IMA has to be dragged into any file integrity solution.  That's begging the question.

  As I've mentioned above, IMA has all sorts of complexity which is currently mandatory, and I'm not volunteering to disentagling the mess to make it be sane.  (And if you don't like that word, how about, "designed with good taste?")

Quoted 5 lines

   3. Could the merkle tree be cached somehow in the filesystem (probably
      as part of the filesystem implementaiton)?  You've already assumed
      "yes" for this since it's how fs-verity is proposed to work.

I don't consider this the Merkel tree to a *cache*, however because if you don't mind a massive latency at file open tme, you can just use the existing IMA mechanism.  So a core part of the design is that the Merkle file is stored permanently (*not* as a cache) alongside the file.

  And if the file is renamed, the Merkle tree should come along for the ride.

Whether the Merkle tree is reconstructed as part of the file / package installation process, or whether the Merkle tree stored as part of the package or streamed from the app store, etc., is an implementation detail, and I don't think we need to prescribe one way of doing things.

  I *do* think though we should allow for possibility where limitations on the local CPU power is such that it would be preferable for the Merkle tree to be supplied from a remote server instead of generated on the local system.

The problem is you have a specific use case in mind, involving the Docker client, where you want to store the signature in an xattr, and then not require any local changes to the Docker client --- and that's not in my requirements set, and it was *your* statement that the Docker client MUST NOT be modified which forced where (a) the Merkle tree must be reconstructed in the kernel, and (b) it must be triggered by setting the xattr.

My goal is to keep things simple, which means

* No parsing of the IMA policy as a text input in the kernel
* No Merkle tree construction in the kernel (which is also true of dm-verity)
* No magic xattr triggering

Speaking as a kernel developer, it makes a more sense to keep things in the kernel, and do as much in userspace as possible --- and if that means that the Docker client (or the package manager, etc.) needs a minor change to call the userspace library, that's infinitely preferable to keeping huge amounts of complexity in non-swappable kernel memory --- which increases the attack surface of the kernel, and so on.

So in my opinion, clean design of the kernel trumps the requirement of "not one change, not one jot, in the Docker client".

It could be that the requiremnts of "keep the kernel changes simple" and "no massive latency at file open time", means that requiremnt sets of fs-verity and IMA are irreconcilable.  Which is fine as far as I'm concerned.  Maybe IMA and fs-verity should be considered orthogonal solutions.

- Ted

OK, so I don't believe that to be true either. �Secure Boot was something we did based on MS mandated technologies and something *some* people though we had to impose strange policies over to please MS. �However, IMA was never part of that secure boot solution, so trying to tar it with the same brush is unfair (and inaccurate).

The question I'm poking at is how integrity is enforced. �Right at the moment it's a small number of security hooks but they're on the fops gates (exec and the like). �To verify a per page hash, they'd have to be in the mm subsystem as well (regardless of whether it's IMA or fs- verity doing it) ... unless you're planning to ignore all the security hooks as well.

Quoted 2 lines

So in my opinion, clean design of the kernel trumps the requirement of "not one change, not one jot, in the Docker client".

OK, bad example on my part, thanks to runc and containerd I don't give much of a toss about the docker client. �I care much more about compliance with the container runtime standard. �At it's base that has anything you can do to tar is fine because it uses tar to define the image. �I buy that we can modify tools easily, but the same doesn't apply to standards.

So if there as a solution which enapculated the information needed to create the fs-verity header and the PKCS7 signature in an xattr --- which is how you carry it around in the tar image --- and when the tarfile is unpacked, the software which does the unpacking calls a library which checks for the xattr, removes it, writes out the fsverity header and Merkle tree, and then calls the ioctl which sets the "verity" bit, thus instantiating the data integrity protection, would this meet your requirements?

In other words, the xattr in the tar file is just the method for carrying the information; and (a) it is not how the information would be stored in the underlying file system when it is actually used, and
(b) it requires the userspace code to do this transformation, so we don't have to build the Merkle tree in the kernel. Is this sufficient for your container use case?

Quoted 5 lines

Currently container images are simple tar files and one of the main value adds of docker as a tool is the simplicity of the image creation process.  That process depends on standard tools like tar to create the image, so I was trying to fit this proposal into that process.

The goal of fs-verity is protect arbitrary files without breaking the backwards compatibility of those that *read* the file. So for example, an APK file is downloaded and verified at the intial download --- and never verified again. It is *used* many times after the initial download (and hence, potentially after an evil maid attack), and this is done by many different tools, including by code inside the APK itself. So it is not feasible to change the tools that read the APK, which means we can't make any backwards-incompatible changes to the file itself.

However, changing the tools that *establish* the locally verified copy is considered completely fair game. I'll note that this is true no matter what system you use. If you are using IMA, after the file is downloaded, you still have to set the magic "trusted xattr" (which, I will note, requires root access since it is a trusted root access). So you have to make at least *some* changes to the code path which writes the file that you want to be integrity protected. In the case of IMA, this includes code that has to run as root.

The cost of this is presumably one hash per page in the tree, so it costs quite a bit in terms of space. �Presumably the hash tree is also dynamically resident meaning a page fault could now also potentially fault in the hash tree, leading to a lot of sub optimal I/O patterns?

I'd like to talk about a proposal to implement and upstream something that we've been calling fs-verity,

... read-only files in a general way for all file systems

Hi, it's me again! I was summoned by this mention of read-only files. You may recall our previous conversation:
https://www.spinics.net/lists/linux-fsdevel/msg75086.html
(I also cleverly tried to work it in to a tangentially related DAX
discussion in https://marc.info/?l=linux-fsdevel&m=150152316817001&w=2 )

So given this proposal will require read-only files, can we do
the interface bikeshedding soon around what the userspace API
will look like, and have it be orthogonal to (but a prerequisite for) fs-verity?

The problem is not the userspace API, it's the bike-shedding over all of the different ways we could *do* immutability, all of which would require separate bits in the on-disk representation of the inode. You can have any combination of:

* Immutable data
* Immutable metadata
* Immutable xattrs

* Immutable links_counts (don't allow hard links / don't allow deletes)
* Immutable timestamps

* etc.
* Who is allowed to set the immutable bit?

* Is the immutable bit allowed to be cleared? If so, by who?

We picked a particular set of combinations for the current immutable bit, and they clearly aren't to your liking; so you are essentially trying to propose another bit, with a different set of the above combinations. The problem is this doesn't scale.

Given that this *is* such a fringe use case,

I wonder if a better way
of satisfying your wishlist is to have a mount option which changes how the immutable bit is enforced; essentially from reading the previous mail thread, you want to (a) allow non-root users to set the immutable bit, (b) you want to disable the immutable bit from being cleared, and (c) allow immutable files to be deleted and hard linked. Yes?

P.S. The reason why it would be tricky to set the immutable bit on a directory

P.P.S. You could also implement your own custom LSM which enforces your particular use case. That might be a better way to go at least in the short-term, especially since we now have stable LSM support.

Quoted 6 lines · Theodore Ts'o

The problem is not the userspace API, it's the bike-shedding over all of the different ways we could *do* immutability, all of which would require separate bits in the on-disk representation of the inode.  You can have any combination of:

Quoted 3 lines

* Immutable data
* Immutable metadata
   * Immutable xattrs

Everyone here wants immutable data (*all* of the data I hope),

Quoted 11 lines · Colin Walters

Quoted 9 lines · Theodore Ts'o

The problem is not the userspace API, it's the bike-shedding over all of the different ways we could *do* immutability, all of which would require separate bits in the on-disk representation of the inode.  You can have any combination of:

* Immutable data
* Immutable metadata
   * Immutable xattrs

Quoted 1 line

Everyone here wants immutable data (*all* of the data I hope),

No, no, I don't.   In the world today most linux distributions from which we produce containers do have the annoying property of writing stuff where they shouldn't (mostly into /etc).

I'd like to talk about a proposal to implement and upstream something that we've been calling fs-verity, which is something like dm-verity, but implemnted on a per-file basis. It will be implemnted much like fs/crypto, in that most of the code will be in a generic layer, with minimal modifications needed in the file system layer.

The merkle tree will be located after file's normal data, and then after the package manager sets the verity bit, i_size will be updated so that the fs-verity header and merkle tree will be "hidden" from userspace and the file will become immutable.

How does this differ from IMA's file integrity?

*) The pages are verified as they are read, so pages are verified as they are read the storage device; this avoids a large latency hit when the file is first opened or referenced.

*) The design and code are done by file system developers, so it doesn't have the locking problems of the IMA code.

The initial use case of this will be for Android, where the latency concerns of doing the full checksum at file open time is important.

In the future, the fact that a file has been signed using fs-verity, using a PKCS 11 signature with a key on a trusted keyring (possibly the same one used for signed kernel modules, or perhaps a separate keyring) could be used as input into a security policy which requires this for say, setuid executables, setuid shell scripts, etc.

Most of this feature could also be used with a non-cryptographic checksum to provide data checksums for read-only files in a general way for all file systems. It wouldn't be as flexible as btrfs, but for files being stored for backup purposes, it should work quite well.

I'd like to talk about a proposal to implement and upstream something that we've been calling fs-verity, which is something like dm-verity, but implemnted on a per-file basis. It will be implemnted much like fs/crypto, in that most of the code will be in a generic layer, with minimal modifications needed in the file system layer.

The merkle tree will be located after file's normal data, and then after the package manager sets the verity bit, i_size will be updated so that the fs-verity header and merkle tree will be "hidden" from userspace and the file will become immutable.

How does this differ from IMA's file integrity?

*) The pages are verified as they are read, so pages are verified as they are read the storage device; this avoids a large latency hit when the file is first opened or referenced.

*) The design and code are done by file system developers, so it doesn't have the locking problems of the IMA code.

The initial use case of this will be for Android, where the latency concerns of doing the full checksum at file open time is important.

In the future, the fact that a file has been signed using fs-verity, using a PKCS 11 signature with a key on a trusted keyring (possibly the same one used for signed kernel modules, or perhaps a separate keyring) could be used as input into a security policy which requires this for say, setuid executables, setuid shell scripts, etc.

Most of this feature could also be used with a non-cryptographic checksum to provide data checksums for read-only files in a general way for all file systems. It wouldn't be as flexible as btrfs, but for files being stored for backup purposes, it should work quite well.

- Ted

I'd like to talk about a proposal to implement and upstream something that we've been calling fs-verity, which is something like dm-verity, but implemnted on a per-file basis. It will be implemnted much like fs/crypto, in that most of the code will be in a generic layer, with minimal modifications needed in the file system layer.

The merkle tree will be located after file's normal data, and then after the package manager sets the verity bit, i_size will be updated so that the fs-verity header and merkle tree will be "hidden" from userspace and the file will become immutable.

How does this differ from IMA's file integrity?

*) The pages are verified as they are read, so pages are verified as they are read the storage device; this avoids a large latency hit when the file is first opened or referenced.

*) The design and code are done by file system developers, so it doesn't have the locking problems of the IMA code.