fsverity: support for non-4K pages
Subsystems (7)
Eric Biggers
maintainerv2 cover fsverity: support for non-4K pages
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]
Currently, filesystems (ext4, f2fs, and btrfs) only support fsverity when the Merkle tree block size, filesystem block size, and page size are all the same. In practice that means 4K, since increasing the page size, e.g. to 16K, forces the Merkle tree block size and filesystem block size to be increased accordingly. That can be impractical; for one, users want the same file signatures to work on all systems.
Therefore, this patchset reduces the coupling between these sizes.
First, patches 1-4 are cleanups.
Second, patches 5-9 allow the Merkle tree block size to be less than the page size or filesystem block size, provided that it's not larger than either one. This involves, among other things, changing the way that fs/verity/verify.c tracks which hash blocks have been verified.
Finally, patches 10-11 make ext4 support fsverity when the filesystem block size is less than the page size. Note, f2fs doesn't need similar changes because f2fs always assumes that the filesystem block size and page size are the same anyway. I haven't looked into btrfs yet.
I've tested this patchset using the "verity" group of tests in xfstests with the following xfstests patchset applied:
"[PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size"
(https://lore.kernel.org/fstests/[email protected]/T/#u)
Note: on the thread "[RFC PATCH 00/11] fs-verity support for XFS"
(https://lore.kernel.org/linux-xfs/[email protected]/T/#u)
there have been many requests for other things to support, including:
* folios in the pagecache
* alternative Merkle tree caching methods
* direct I/O
* merkle_tree_block_size > page_size
* extremely large files, using a reclaimable bitmap
We shouldn't try to boil the ocean, though, so to keep the scope of this patchset manageable I haven't changed it significantly from v1. This patchset does bring us closer to many of the above, just not all the way there. I'd like to follow up this patchset with a change to support folios, which should be straightforward. Next, we can do a change to generalize the Merkle tree interface to allow XFS to use an alternative caching method, as that sounds like the highest priority item for XFS.
Anyway, the changelog is:
Changed in v2:
- Rebased onto the recent fsverity cleanups.
- Split some parts of the big "support verification" patch into separate patches.
- Passed the data_pos to verify_data_block() instead of computing it
using page->index, to make it ready for folio and DIO support.
- Eliminated some unnecessary arithmetic in verify_data_block().
- Changed the log_* fields in merkle_tree_params to u8.
- Restored PageLocked and !PageUptodate checks for pagecache pages.
- Eliminated the change to fsverity_hash_buffer().
- Other small cleanups
Eric Biggers (11):
fsverity: use unsigned long for level_start
fsverity: simplify Merkle tree readahead size calculation
fsverity: store log2(digest_size) precomputed
fsverity: use EFBIG for file too large to enable verity
fsverity: replace fsverity_hash_page() with fsverity_hash_block()
fsverity: support verification with tree block size < PAGE_SIZE
fsverity: support enabling with tree block size < PAGE_SIZE
ext4: simplify ext4_readpage_limit()
f2fs: simplify f2fs_readpage_limit()
fs/buffer.c: support fsverity in block_read_full_folio()
ext4: allow verity with fs block size < PAGE_SIZE
Documentation/filesystems/fsverity.rst | 76 +++--- fs/buffer.c | 67 ++++- fs/ext4/readpage.c | 3 +- fs/ext4/super.c | 5 - fs/f2fs/data.c | 3 +- fs/verity/enable.c | 260 ++++++++++---------- fs/verity/fsverity_private.h | 20 +- fs/verity/hash_algs.c | 24 +- fs/verity/open.c | 98 ++++++-- fs/verity/verify.c | 325 +++++++++++++++++-------- include/linux/fsverity.h | 14 +- 11 files changed, 565 insertions(+), 330 deletions(-)
Signature · 1 line
2.39.0
Eric Biggers
maintainerPATCH v2 1/11 fsverity: use unsigned long for level_start
From: Eric Biggers <[email protected]>
fs/verity/ isn't consistent with whether Merkle tree block indices are 'unsigned long' or 'u64'. There's no real point to using u64 for them, though, since (a) a Merkle tree with over ULONG_MAX blocks would only be needed for a file larger than MAX_LFS_FILESIZE, and (b) for reads, the status of all Merkle tree blocks has to be tracked in memory.
Therefore, let's make things a bit more efficient on 32-bit systems by using 'unsigned long[]' for merkle_tree_params::level_start, instead of 'u64[]'. Also, to be extra safe, explicitly check that there aren't more than ULONG_MAX Merkle tree blocks.
--- fs/verity/fsverity_private.h | 2 +- fs/verity/open.c | 20 +++++++++++++++----- 2 files changed, 16 insertions(+), 6 deletions(-)
fs/verity/fsverity_private.h +1 -1
@@ -52,7 +52,7 @@ struct merkle_tree_params {* Starting block index for each tree level, ordered from leaf level (0) * to root level ('num_levels - 1') */ -u64 level_start[FS_VERITY_MAX_LEVELS ]; +unsigned long level_start[FS_VERITY_MAX_LEVELS ]; };/*
fs/verity/open.c +15 -5
@@ -34,6 +34,7 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params,struct fsverity_hash_alg *hash_alg;int err;u64 blocks; +u64 blocks_in_level[FS_VERITY_MAX_LEVELS ];u64 offset;int level;
@@ -94,17 +95,26 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params, } blocks = (blocks + params->hashes_per_block -1 ) >> params->log_arity; -/* temporarily using level_start[] to store blocks in level */ - params->level_start[params->num_levels++] = blocks; + blocks_in_level[params->num_levels++] = blocks; } - params->level0_blocks = params->level_start[0 ]; + params->level0_blocks = blocks_in_level[0 ];/* Compute the starting block of each level */ offset =0 ;for (level = (int )params->num_levels -1 ; level >=0 ; level--) { - blocks = params->level_start[level]; params->level_start[level] = offset; - offset += blocks; + offset += blocks_in_level[level]; + } + +/* +* Since the data, and thus also the Merkle tree, cannot have more than +* ULONG_MAX pages, hash block indices can always fit in an +* 'unsigned long'. To be safe, explicitly check for it too. +*/ +if (offset >ULONG_MAX ) { +fsverity_err (inode,"Too many blocks in Merkle tree" ); + err = -EFBIG ; + goto out_err; } params->tree_size = offset << log_blocksize;
Signature · 1 line
2.39.0
Eric Biggers
maintainerPATCH v2 2/11 fsverity: simplify Merkle tree readahead size calculation
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
First, calculate max_ra_pages more efficiently by using the bio size.
Second, calculate the number of readahead pages from the hash page index, instead of calculating it ahead of time using the data page index. This ends up being a bit simpler, especially since level 0 is last in the tree, so we can just limit the readahead to the tree size.
--- fs/verity/fsverity_private.h | 2 +- fs/verity/open.c | 3 ++- fs/verity/verify.c | 21 +++++++-------------- 3 files changed, 10 insertions(+), 16 deletions(-)
fs/verity/fsverity_private.h +1 -1
@@ -46,7 +46,7 @@ struct merkle_tree_params {unsigned int log_arity;/* log2(hashes_per_block) */ unsigned int num_levels;/* number of levels in Merkle tree */ u64 tree_size;/* Merkle tree size in bytes */ -unsigned long level0_blocks;/* number of blocks in tree level 0 */ +unsigned long tree_pages;/* Merkle tree size in pages */ /* * Starting block index for each tree level, ordered from leaf level (0)
fs/verity/open.c +2 -1
@@ -7,6 +7,7 @@#include "fsverity_private.h" +#include <linux/mm.h> #include <linux/slab.h> static struct kmem_cache *fsverity_info_cachep;
@@ -97,7 +98,6 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params, params->log_arity; blocks_in_level[params->num_levels++] = blocks; } - params->level0_blocks = blocks_in_level[0 ];/* Compute the starting block of each level */ offset =0 ;
@@ -118,6 +118,7 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params, } params->tree_size = offset << log_blocksize; + params->tree_pages =PAGE_ALIGN (params->tree_size) >>PAGE_SHIFT ;return 0 ; out_err:
fs/verity/verify.c +7 -14
@@ -74,7 +74,7 @@ static inline int cmp_hashes(const struct fsverity_info *vi,*/ static bool verify_page (struct inode *inode,const struct fsverity_info *vi,struct ahash_request *req,struct page *data_page, -unsigned long level0_ra_pages) +unsigned long max_ra_pages) {const struct merkle_tree_params *params = &vi->tree_params;const unsigned int hsize = params->digest_size;
@@ -103,7 +103,8 @@ static bool verify_page(struct inode *inode, const struct fsverity_info *vi,hash_at_level (params, index, level, &hindex, &hoffset); hpage = inode->i_sb->s_vop->read_merkle_tree_page (inode, hindex, - level ==0 ? level0_ra_pages :0 ); + level ==0 ?min (max_ra_pages, + params->tree_pages - hindex) :0 );if (IS_ERR (hpage)) { err =PTR_ERR (hpage); fsverity_err(inode,
@@ -199,14 +200,13 @@ void fsverity_verify_bio(struct bio *bio) {struct inode *inode =bio_first_page_all (bio)->mapping->host;const struct fsverity_info *vi = inode->i_verity_info; -const struct merkle_tree_params *params = &vi->tree_params;struct ahash_request *req;struct bio_vec *bv;struct bvec_iter_all iter_all;unsigned long max_ra_pages =0 ;/* This allocation never fails, since it's mempool-backed. */ - req =fsverity_alloc_hash_request (params->hash_alg,GFP_NOFS ); + req =fsverity_alloc_hash_request (vi->tree_params.hash_alg,GFP_NOFS );if (bio->bi_opf &REQ_RAHEAD ) {/*
@@ -218,24 +218,17 @@ void fsverity_verify_bio(struct bio *bio)* This improves sequential read performance, as it greatly * reduces the number of I/O requests made to the Merkle tree. */ -bio_for_each_segment_all (bv, bio, iter_all) - max_ra_pages++; - max_ra_pages /= 4; + max_ra_pages = bio->bi_iter.bi_size >> (PAGE_SHIFT + 2); }bio_for_each_segment_all (bv, bio, iter_all) { -struct page *page = bv->bv_page; -unsigned long level0_index = page->index >> params->log_arity; -unsigned long level0_ra_pages = -min (max_ra_pages, params->level0_blocks - level0_index); - -if (!verify_page (inode, vi, req, page, level0_ra_pages)) { +if (!verify_page (inode, vi, req, bv->bv_page, max_ra_pages)) { bio->bi_status =BLK_STS_IOERR ;break ; } } -fsverity_free_hash_request (params->hash_alg, req); +fsverity_free_hash_request (vi->tree_params.hash_alg, req); }EXPORT_SYMBOL_GPL (fsverity_verify_bio);#endif /* CONFIG_BLOCK */
Signature · 1 line
2.39.0
Eric Biggers
maintainerPATCH v2 3/11 fsverity: store log2(digest_size) precomputed
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add log_digestsize to struct merkle_tree_params so that it can be used in verify.c. Also save memory by using u8 for all the log_* fields.
--- fs/verity/fsverity_private.h | 5 +++-- fs/verity/open.c | 3 ++- fs/verity/verify.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-)
fs/verity/fsverity_private.h +3 -2
@@ -42,8 +42,9 @@ struct merkle_tree_params {unsigned int digest_size;/* same as hash_alg->digest_size */ unsigned int block_size;/* size of data and tree blocks */ unsigned int hashes_per_block;/* number of hashes per tree block */ -unsigned int log_blocksize;/* log2(block_size) */ -unsigned int log_arity;/* log2(hashes_per_block) */ +u8 log_digestsize;/* log2(digest_size) */ +u8 log_blocksize;/* log2(block_size) */ +u8 log_arity;/* log2(hashes_per_block) */ unsigned int num_levels;/* number of levels in Merkle tree */ u64 tree_size;/* Merkle tree size in bytes */ unsigned long tree_pages;/* Merkle tree size in pages */
fs/verity/open.c +2 -1
@@ -76,7 +76,8 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params, err = -EINVAL ; goto out_err; } - params->log_arity = params->log_blocksize -ilog2 (params->digest_size); + params->log_digestsize =ilog2 (params->digest_size); + params->log_arity = log_blocksize - params->log_digestsize; params->hashes_per_block =1 << params->log_arity;/*
fs/verity/verify.c +1 -1
@@ -35,7 +35,7 @@ static void hash_at_level(const struct merkle_tree_params *params,/* Offset of the wanted hash (in bytes) within the hash block */ *hoffset = (position & ((1 << params->log_arity) -1 )) << - (params->log_blocksize - params->log_arity); + params->log_digestsize; } static inline int cmp_hashes(const struct fsverity_info *vi,
Signature · 8 lines
2.39.0 _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerPATCH v2 4/11 fsverity: use EFBIG for file too large to enable verity
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Currently, there is an implementation limit where files can't have more than 8 Merkle tree levels. With SHA-256 and 4K blocks, this limit is never reached, since a file would need to be larger than 2**64 bytes to need 9 levels. However, with SHA-512, 9 levels are needed for files larger than about 1.15 EB, which is possible on btrfs. Therefore, this limit technically became reachable when btrfs added fsverity support.
Meanwhile, support for merkle_tree_block_size < PAGE_SIZE will introduce another implementation limit on file size, resulting from the use of an in-memory bitmap to track which Merkle tree blocks have been verified.
In any case, currently FS_IOC_ENABLE_VERITY fails with EINVAL when the file is too large. This is undocumented, and also ambiguous since EINVAL can mean other things too. Let's change the error code to EFBIG, which is much clearer, and document it.
--- Documentation/filesystems/fsverity.rst | 1 + fs/verity/open.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
Documentation/filesystems/fsverity.rst +1 -0
@@ -161,6 +161,7 @@ FS_IOC_ENABLE_VERITY can fail with the following errors: - ``EBUSY``: this ioctl is already running on the file - ``EEXIST``: the file already has verity enabled - ``EFAULT``: the caller provided inaccessible memory +- ``EFBIG``: the file is too large to enable verity on - ``EINTR``: the operation was interrupted by a fatal signal - ``EINVAL``: unsupported version, hash algorithm, or block size; or reserved bits are set; or the file descriptor refers to neither a
fs/verity/open.c +1 -1
@@ -92,7 +92,7 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params,while (blocks >1 ) {if (params->num_levels >=FS_VERITY_MAX_LEVELS ) {fsverity_err (inode,"Too many levels in Merkle tree" ); - err = -EINVAL ; + err = -EFBIG ; goto out_err; } blocks = (blocks + params->hashes_per_block - 1) >>
Signature · 1 line
2.39.0
Eric Biggers
maintainerPATCH v2 5/11 fsverity: replace fsverity_hash_page() with fsverity_hash_block()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
In preparation for allowing the Merkle tree block size to differ from PAGE_SIZE, replace fsverity_hash_page() with fsverity_hash_block(). The new function is similar to the old one, but it operates on the block at the given offset in the page instead of on the full page.
(For now, all callers still pass a full page.)
--- fs/verity/enable.c | 4 ++-- fs/verity/fsverity_private.h | 6 +++--- fs/verity/hash_algs.c | 24 +++++++++++------------- fs/verity/verify.c | 9 +++++---- 4 files changed, 21 insertions(+), 22 deletions(-)
fs/verity/enable.c +2 -2
@@ -99,8 +99,8 @@ static int build_merkle_tree_level(struct file *filp, unsigned int level, } } - err =fsverity_hash_page (params, inode, req, src_page, - &pending_hashes[pending_size]); + err =fsverity_hash_block (params, inode, req, src_page,0 , + &pending_hashes[pending_size]);put_page (src_page);if (err)return err;
fs/verity/fsverity_private.h +3 -3
@@ -88,9 +88,9 @@ void fsverity_free_hash_request(struct fsverity_hash_alg *alg, struct ahash_request *req);const u8 *fsverity_prepare_hash_state (struct fsverity_hash_alg *alg,const u8 *salt,size_t salt_size); -int fsverity_hash_page (const struct merkle_tree_params *params, -const struct inode *inode, -struct ahash_request *req,struct page *page,u8 *out); +int fsverity_hash_block (const struct merkle_tree_params *params, +const struct inode *inode,struct ahash_request *req, +struct page *page,unsigned int offset,u8 *out);int fsverity_hash_buffer (struct fsverity_hash_alg *alg,const void *data,size_t size,u8 *out);void __init fsverity_check_hash_algs (void );
fs/verity/hash_algs.c +11 -13
@@ -220,35 +220,33 @@ const u8 *fsverity_prepare_hash_state(struct fsverity_hash_alg *alg, }/** -* fsverity_hash_page() - hash a single data or hash page +* fsverity_hash_block() - hash a single data or hash block * @params: the Merkle tree's parameters * @inode: inode for which the hashing is being done * @req: preallocated hash request -* @page: the page to hash +* @page: the page containing the block to hash +* @offset: the offset of the block within @page * @out: output digest, size 'params->digest_size' bytes * -* Hash a single data or hash block, assuming block_size == PAGE_SIZE. -* The hash is salted if a salt is specified in the Merkle tree parameters. +* Hash a single data or hash block. The hash is salted if a salt is specified +* in the Merkle tree parameters. * * Return: 0 on success, -errno on failure */ -int fsverity_hash_page (const struct merkle_tree_params *params, -const struct inode *inode, -struct ahash_request *req,struct page *page,u8 *out) +int fsverity_hash_block (const struct merkle_tree_params *params, +const struct inode *inode,struct ahash_request *req, +struct page *page,unsigned int offset,u8 *out) {struct scatterlist sg;DECLARE_CRYPTO_WAIT (wait);int err; -if (WARN_ON (params->block_size !=PAGE_SIZE )) -return -EINVAL ; -sg_init_table (&sg,1 ); -sg_set_page (&sg, page,PAGE_SIZE ,0 ); +sg_set_page (&sg, page, params->block_size, offset);ahash_request_set_callback (req,CRYPTO_TFM_REQ_MAY_SLEEP |CRYPTO_TFM_REQ_MAY_BACKLOG , crypto_req_done, &wait); -ahash_request_set_crypt (req, &sg, out,PAGE_SIZE ); +ahash_request_set_crypt (req, &sg, out, params->block_size);if (params->hashstate) { err =crypto_ahash_import (req, params->hashstate);
@@ -264,7 +262,7 @@ int fsverity_hash_page(const struct merkle_tree_params *params, err =crypto_wait_req (err, &wait);if (err) -fsverity_err (inode,"Error %d computing page hash" , err); +fsverity_err (inode,"Error %d computing block hash" , err);return err; }
fs/verity/verify.c +5 -4
@@ -125,12 +125,13 @@ static bool verify_page(struct inode *inode, const struct fsverity_info *vi, want_hash = vi->root_hash; descend: -/* Descend the tree verifying hash pages */ +/* Descend the tree verifying hash blocks. */ for (; level >0 ; level--) {struct page *hpage = hpages[level -1 ];unsigned int hoffset = hoffsets[level -1 ]; - err =fsverity_hash_page (params, inode, req, hpage, real_hash); + err =fsverity_hash_block (params, inode, req, hpage,0 , + real_hash);if (err) goto out; err =cmp_hashes (vi, want_hash, real_hash, index, level -1 );
@@ -142,8 +143,8 @@ static bool verify_page(struct inode *inode, const struct fsverity_info *vi,put_page (hpage); } -/* Finally, verify the data page */ - err =fsverity_hash_page (params, inode, req, data_page, real_hash); +/* Finally, verify the data block. */ + err =fsverity_hash_block (params, inode, req, data_page,0 , real_hash);if (err) goto out; err =cmp_hashes (vi, want_hash, real_hash, index,-1 );
Signature · 1 line
2.39.0
Eric Biggers
maintainerPATCH v2 6/11 fsverity: support verification with tree block size < PAGE_SIZE
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add support for verifying data from verity files whose Merkle tree block size is less than the page size. The main use case for this is to allow a single Merkle tree block size to be used across all systems, so that only one set of fsverity file digests and signatures is needed.
To do this, eliminate various assumptions that the Merkle tree block size and the page size are the same:
- Make fsverity_verify_page() a wrapper around a new function fsverity_verify_blocks() which verifies one or more blocks in a page.
- When a Merkle tree block is needed, get the corresponding page and only verify and use the needed portion. (The Merkle tree continues to be read and cached in page-sized chunks; that doesn't need to change.)
- When the Merkle tree block size and page size differ, use a bitmap fsverity_info::hash_block_verified to keep track of which Merkle tree blocks have been verified, as PageChecked cannot be used directly.
--- Documentation/filesystems/fsverity.rst | 49 ++-- fs/verity/fsverity_private.h | 5 +- fs/verity/open.c | 80 ++++++- fs/verity/verify.c | 309 ++++++++++++++++++------- include/linux/fsverity.h | 11 +- 5 files changed, 328 insertions(+), 126 deletions(-)
Documentation/filesystems/fsverity.rst +23 -26
@@ -572,47 +572,44 @@ For filesystems using Linux's pagecache, the ``->read_folio()`` and are marked Uptodate. Merely hooking ``->read_iter()`` would be insufficient, since ``->read_iter()`` is not used for memory maps. -Therefore, fs/verity/ provides a function fsverity_verify_page() which -verifies a page that has been read into the pagecache of a verity -inode, but is still locked and not Uptodate, so it's not yet readable -by userspace. As needed to do the verification, -fsverity_verify_page() will call back into the filesystem to read -Merkle tree pages via fsverity_operations::read_merkle_tree_page(). - -fsverity_verify_page() returns false if verification failed; in this +Therefore, fs/verity/ provides the function fsverity_verify_blocks() +which verifies data that has been read into the pagecache of a verity +inode. The containing page must still be locked and not Uptodate, so +it's not yet readable by userspace. As needed to do the verification, +fsverity_verify_blocks() will call back into the filesystem to read +hash blocks via fsverity_operations::read_merkle_tree_page(). + +fsverity_verify_blocks() returns false if verification failed; in this case, the filesystem must not set the page Uptodate. Following this, as per the usual Linux pagecache behavior, attempts by userspace to read() from the part of the file containing the page will fail with EIO, and accesses to the page within a memory map will raise SIGBUS. -fsverity_verify_page() currently only supports the case where the -Merkle tree block size is equal to PAGE_SIZE (often 4096 bytes). - -In principle, fsverity_verify_page() verifies the entire path in the -Merkle tree from the data page to the root hash. However, for -efficiency the filesystem may cache the hash pages. Therefore, -fsverity_verify_page() only ascends the tree reading hash pages until -an already-verified hash page is seen, as indicated by the PageChecked -bit being set. It then verifies the path to that page. +In principle, verifying a data block requires verifying the entire +path in the Merkle tree from the data block to the root hash. +However, for efficiency the filesystem may cache the hash blocks. +Therefore, fsverity_verify_blocks() only ascends the tree reading hash +blocks until an already-verified hash block is seen. It then verifies +the path to that block. This optimization, which is also used by dm-verity, results in excellent sequential read performance. This is because usually (e.g. -127 in 128 times for 4K blocks and SHA-256) the hash page from the +127 in 128 times for 4K blocks and SHA-256) the hash block from the bottom level of the tree will already be cached and checked from -reading a previous data page. However, random reads perform worse. +reading a previous data block. However, random reads perform worse. Block device based filesystems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Block device based filesystems (e.g. ext4 and f2fs) in Linux also use the pagecache, so the above subsection applies too. However, they -also usually read many pages from a file at once, grouped into a +also usually read many data blocks from a file at once, grouped into a structure called a "bio". To make it easier for these types of filesystems to support fs-verity, fs/verity/ also provides a function -fsverity_verify_bio() which verifies all pages in a bio. +fsverity_verify_bio() which verifies all data blocks in a bio. ext4 and f2fs also support encryption. If a verity file is also -encrypted, the pages must be decrypted before being verified. To +encrypted, the data must be decrypted before being verified. To support this, these filesystems allocate a "post-read context" for each bio and store it in ``->bi_private``::
@@ -631,10 +628,10 @@ verification. Finally, pages where no decryption or verity error occurred are marked Uptodate, and the pages are unlocked. On many filesystems, files can contain holes. Normally, -``->readahead()`` simply zeroes holes and sets the corresponding pages -Uptodate; no bios are issued. To prevent this case from bypassing -fs-verity, these filesystems use fsverity_verify_page() to verify hole -pages. +``->readahead()`` simply zeroes hole blocks and considers the +corresponding data to be up-to-date; no bios are issued. To prevent +this case from bypassing fs-verity, filesystems use +fsverity_verify_blocks() to verify hole blocks. Filesystems also disable direct I/O on verity files, since otherwise direct I/O would bypass fs-verity.
fs/verity/fsverity_private.h +4 -1
@@ -42,9 +42,11 @@ struct merkle_tree_params {unsigned int digest_size;/* same as hash_alg->digest_size */ unsigned int block_size;/* size of data and tree blocks */ unsigned int hashes_per_block;/* number of hashes per tree block */ +unsigned int blocks_per_page;/* PAGE_SIZE / block_size */ u8 log_digestsize;/* log2(digest_size) */ u8 log_blocksize;/* log2(block_size) */ u8 log_arity;/* log2(hashes_per_block) */ +u8 log_blocks_per_page;/* log2(blocks_per_page) */ unsigned int num_levels;/* number of levels in Merkle tree */ u64 tree_size;/* Merkle tree size in bytes */ unsigned long tree_pages;/* Merkle tree size in pages */
@@ -70,9 +72,10 @@ struct fsverity_info {u8 root_hash[FS_VERITY_MAX_DIGEST_SIZE ];u8 file_digest[FS_VERITY_MAX_DIGEST_SIZE ];const struct inode *inode; +unsigned long *hash_block_verified; +spinlock_t hash_page_init_lock; }; -#define FS_VERITY_MAX_SIGNATURE_SIZE (FS_VERITY_MAX_DESCRIPTOR_SIZE - \ sizeof(struct fsverity_descriptor))
fs/verity/open.c +69 -11
123 lines not on this page · raw message
fs/verity/verify.c +223 -86
420 lines not on this page · raw message
include/linux/fsverity.h +9 -2
@@ -170,7 +170,8 @@ int fsverity_ioctl_read_metadata(struct file *filp, const void __user *uarg);/* verify.c */ -bool fsverity_verify_page (struct page *page); +bool fsverity_verify_blocks (struct page *page,unsigned int len, +unsigned int offset);void fsverity_verify_bio (struct bio *bio);void fsverity_enqueue_verify_work (struct work_struct *work);
@@ -230,7 +231,8 @@ static inline int fsverity_ioctl_read_metadata(struct file *filp,/* verify.c */ -static inline bool fsverity_verify_page (struct page *page) +static inline bool fsverity_verify_blocks (struct page *page,unsigned int len, +unsigned int offset) {WARN_ON (1 );return false;
@@ -248,6 +250,11 @@ static inline void fsverity_enqueue_verify_work(struct work_struct *work)#endif /* !CONFIG_FS_VERITY */ +static inline bool fsverity_verify_page (struct page *page) +{ +return fsverity_verify_blocks (page,PAGE_SIZE ,0 ); +} +/** * fsverity_active() - do reads from the inode need to go through fs-verity? * @inode: inode to check
Signature · 1 line
2.39.0
Eric Biggers
maintainerPATCH v2 7/11 fsverity: support enabling with tree block size < PAGE_SIZE
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Make FS_IOC_ENABLE_VERITY support values of fsverity_enable_arg::block_size other than PAGE_SIZE.
To make this possible, rework build_merkle_tree(), which was reading data and hash pages from the file and assuming that they were the same thing as "blocks".
For reading the data blocks, just replace the direct pagecache access with __kernel_read(), to naturally read one block at a time.
(A disadvantage of the above is that we lose the two optimizations of hashing the pagecache pages in-place and forcing the maximum readahead. That shouldn't be very important, though.)
The hash block reads are a bit more difficult to handle, as the only way to do them is through fsverity_operations::read_merkle_tree_page().
Instead, let's switch to the single-pass tree construction algorithm that fsverity-utils uses. This eliminates the need to read back any hash blocks while the tree is being built, at the small cost of an extra block-sized memory buffer per Merkle tree level. This is probably what I should have done originally.
Taken together, the above two changes result in page-size independent code that is also a bit simpler than what we had before.
--- Documentation/filesystems/fsverity.rst | 20 +- fs/verity/enable.c | 260 ++++++++++++------------- include/linux/fsverity.h | 3 +- 3 files changed, 134 insertions(+), 149 deletions(-)
Documentation/filesystems/fsverity.rst +9 -11
@@ -118,10 +118,11 @@ as follows: - ``hash_algorithm`` must be the identifier for the hash algorithm to use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256. See ``include/uapi/linux/fsverity.h`` for the list of possible values. -- ``block_size`` must be the Merkle tree block size. Currently, this - must be equal to the system page size, which is usually 4096 bytes. - Other sizes may be supported in the future. This value is not - necessarily the same as the filesystem block size. +- ``block_size`` is the Merkle tree block size, in bytes. In Linux + v6.3 and later, this can be any power of 2 between (inclusively) + 1024 and the minimum of the system page size and the filesystem + block size. In earlier versions, the page size was the only allowed + value. - ``salt_size`` is the size of the salt in bytes, or 0 if no salt is provided. The salt is a value that is prepended to every hashed block; it can be used to personalize the hashing for a particular
@@ -519,9 +520,7 @@ support paging multi-gigabyte xattrs into memory, and to support encrypting xattrs. Note that the verity metadata *must* be encrypted when the file is, since it contains hashes of the plaintext data. -Currently, ext4 verity only supports the case where the Merkle tree -block size, filesystem block size, and page size are all the same. It -also only supports extent-based files. +ext4 only allows verity on extent-based files. f2fs ----
@@ -539,11 +538,10 @@ Like ext4, f2fs stores the verity metadata (Merkle tree and fsverity_descriptor) past the end of the file, starting at the first 64K boundary beyond i_size. See explanation for ext4 above. Moreover, f2fs supports at most 4096 bytes of xattr entries per inode -which wouldn't be enough for even a single Merkle tree block. +which usually wouldn't be enough for even a single Merkle tree block. -Currently, f2fs verity only supports a Merkle tree block size of 4096. -Also, f2fs doesn't support enabling verity on files that currently -have atomic or volatile writes pending. +f2fs doesn't support enabling verity on files that currently have +atomic or volatile writes pending. btrfs -----
fs/verity/enable.c +124 -136
305 lines not on this page · raw message
include/linux/fsverity.h +1 -2
@@ -93,8 +93,7 @@ struct fsverity_operations {* isn't already cached. Implementations may ignore this * argument; it's only a performance optimization. * -* This can be called at any time on an open verity file, as well as -* between ->begin_enable_verity() and ->end_enable_verity(). It may be +* This can be called at any time on an open verity file. It may be * called by multiple processes concurrently, even with the same page. * * Note that this must retrieve a *page*, not necessarily a *block*.
Signature · 8 lines
2.39.0 _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerPATCH v2 8/11 ext4: simplify ext4_readpage_limit()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Now that the implementation of FS_IOC_ENABLE_VERITY has changed to not involve reading back Merkle tree blocks that were previously written, there is no need for ext4_readpage_limit() to allow for this case.
--- fs/ext4/readpage.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/ext4/readpage.c +1 -2
@@ -211,8 +211,7 @@ static void ext4_set_bio_post_read_ctx(struct bio *bio,static inline loff_t ext4_readpage_limit (struct inode *inode) { -if (IS_ENABLED (CONFIG_FS_VERITY ) && - (IS_VERITY (inode) ||ext4_verity_in_progress (inode))) +if (IS_ENABLED (CONFIG_FS_VERITY ) &&IS_VERITY (inode))return inode->i_sb->s_maxbytes;return i_size_read (inode);
Signature · 8 lines
2.39.0 _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerPATCH v2 9/11 f2fs: simplify f2fs_readpage_limit()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Now that the implementation of FS_IOC_ENABLE_VERITY has changed to not involve reading back Merkle tree blocks that were previously written, there is no need for f2fs_readpage_limit() to allow for this case.
--- fs/f2fs/data.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/f2fs/data.c +1 -2
@@ -2053,8 +2053,7 @@ int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,static inline loff_t f2fs_readpage_limit (struct inode *inode) { -if (IS_ENABLED (CONFIG_FS_VERITY ) && - (IS_VERITY (inode) ||f2fs_verity_in_progress (inode))) +if (IS_ENABLED (CONFIG_FS_VERITY ) &&IS_VERITY (inode))return inode->i_sb->s_maxbytes;return i_size_read (inode);
Signature · 1 line
2.39.0
Eric Biggers
maintainerPATCH v2 10/11 fs/buffer.c: support fsverity in block_read_full_folio()
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
After each filesystem block (as represented by a buffer_head) has been read from disk by block_read_full_folio(), verify it if needed. The verification is done on the fsverity_read_workqueue. Also allow reads of verity metadata past i_size, as required by ext4.
This is needed to support fsverity on ext4 filesystems where the filesystem block size is less than the page size.
The new code is compiled away when CONFIG_FS_VERITY=n.
--- fs/buffer.c | 67 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 57 insertions(+), 10 deletions(-)
fs/buffer.c +57 -10
@@ -48,6 +48,7 @@#include <linux/sched/mm.h> #include <trace/events/block.h> #include <linux/fscrypt.h> +#include <linux/fsverity.h> #include "internal.h"
@@ -295,20 +296,52 @@ static void end_buffer_async_read(struct buffer_head *bh, int uptodate)return ; } -struct decrypt_bh_ctx { +struct postprocess_bh_ctx {struct work_struct work;struct buffer_head *bh; }; +static void verify_bh (struct work_struct *work) +{ +struct postprocess_bh_ctx *ctx = + container_of(work, struct postprocess_bh_ctx, work); +struct buffer_head *bh = ctx->bh; +bool valid; + + valid =fsverity_verify_blocks (bh->b_page, bh->b_size,bh_offset (bh)); +end_buffer_async_read (bh, valid); +kfree (ctx); +} + +static bool need_fsverity (struct buffer_head *bh) +{ +struct page *page = bh->b_page; +struct inode *inode = page->mapping->host; + +return fsverity_active (inode) && +/* needed by ext4 */ + page->index <DIV_ROUND_UP (inode->i_size,PAGE_SIZE ); +} +static void decrypt_bh (struct work_struct *work) { -struct decrypt_bh_ctx *ctx = - container_of(work, struct decrypt_bh_ctx, work); +struct postprocess_bh_ctx *ctx = + container_of(work, struct postprocess_bh_ctx, work);struct buffer_head *bh = ctx->bh;int err; err =fscrypt_decrypt_pagecache_blocks (bh->b_page, bh->b_size,bh_offset (bh)); +if (err ==0 &&need_fsverity (bh)) { +/* +* We use different work queues for decryption and for verity +* because verity may require reading metadata pages that need +* decryption, and we shouldn't recurse to the same workqueue. +*/ +INIT_WORK (&ctx->work, verify_bh); +fsverity_enqueue_verify_work (&ctx->work); +return ; + }end_buffer_async_read (bh, err ==0 );kfree (ctx); }
@@ -319,15 +352,24 @@ static void decrypt_bh(struct work_struct *work)*/ static void end_buffer_async_read_io (struct buffer_head *bh,int uptodate) { -/* Decrypt if needed */ -if (uptodate && -fscrypt_inode_uses_fs_layer_crypto (bh->b_page->mapping->host)) { -struct decrypt_bh_ctx *ctx =kmalloc (sizeof (*ctx),GFP_ATOMIC ); +struct inode *inode = bh->b_page->mapping->host; +bool decrypt =fscrypt_inode_uses_fs_layer_crypto (inode); +bool verify =need_fsverity (bh); + +/* Decrypt (with fscrypt) and/or verify (with fsverity) if needed. */ +if (uptodate && (decrypt || verify)) { +struct postprocess_bh_ctx *ctx = +kmalloc (sizeof (*ctx),GFP_ATOMIC );if (ctx) { -INIT_WORK (&ctx->work, decrypt_bh); ctx->bh = bh; -fscrypt_enqueue_decrypt_work (&ctx->work); +if (decrypt) { +INIT_WORK (&ctx->work, decrypt_bh); +fscrypt_enqueue_decrypt_work (&ctx->work); + }else { +INIT_WORK (&ctx->work, verify_bh); +fsverity_enqueue_verify_work (&ctx->work); + }return ; } uptodate =0 ;
@@ -2245,6 +2287,11 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block)int nr, i;int fully_mapped =1 ;bool page_error = false; +loff_t limit =i_size_read (inode); + +/* This is needed for ext4. */ +if (IS_ENABLED (CONFIG_FS_VERITY ) &&IS_VERITY (inode)) + limit = inode->i_sb->s_maxbytes;VM_BUG_ON_FOLIO (folio_test_large (folio), folio);
@@ -2253,7 +2300,7 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) bbits =block_size_bits (blocksize); iblock = (sector_t )folio->index << (PAGE_SHIFT - bbits); - lblock = (i_size_read (inode)+blocksize-1 ) >> bbits; + lblock = (limit+blocksize-1 ) >> bbits; bh = head; nr =0 ; i =0 ;
Signature · 1 line
2.39.0
Re: [PATCH v2 10/11] fs/buffer.c: support fsverity in block_read_full_folio()
Quoted 4 lines · On Fri, 23 Dec 2022 12:36:37 -0800 Eric Biggers <[email protected]> wrote:
After each filesystem block (as represented by a
buffer_head) has been read from disk byblock_read_full_folio(), verify it if needed. The verification is done on thefsverity_read_workqueue. Also allow reads of verity metadata pasti_size, as required by ext4.
Sigh. Do we reeeeealy need to mess with buffer.c in this fashion? Did any other subsystems feel a need to do this?
Quoted 2 lines
This is needed to support fsverity on ext4 filesystems where the filesystem block size is less than the page size.
Does any real person actually do this?
Eric Biggers
maintainerRe: [PATCH v2 10/11] fs/buffer.c: support fsverity in block_read_full_folio()
Quoted 9 lines · Andrew Morton
Quoted 4 lines · On Fri, 23 Dec 2022 12:36:37 -0800 Eric Biggers <[email protected]> wrote:
After each filesystem block (as represented by a
buffer_head) has been read from disk byblock_read_full_folio(), verify it if needed. The verification is done on thefsverity_read_workqueue. Also allow reads of verity metadata pasti_size, as required by ext4.Sigh. Do we reeeeealy need to mess with
buffer.cin this fashion? Did any other subsystems feel a need to do this?
ext4 is currently the only filesystem that uses block_read_full_folio() and that supports fsverity. However, since fsverity has a common infrastructure across filesystems, in fs/verity/, it makes sense to support it in the other filesystem infrastructure so that things aren't mutually exclusive for no reason.
Note that this applies to fscrypt too, which block_read_full_folio() (previously block_read_full_page()) already supports since v5.5.
If you'd prefer that block_read_full_folio() be copied into ext4, then modified to support fscrypt and fsverity, and then the fscrypt support removed from the original copy, we could do that. That seems more like a workaround to avoid modifying certain files than an actually better solution, but it could be done.
Quoted 5 lines
Quoted 3 lines
This is needed to support fsverity on ext4 filesystems where the filesystem block size is less than the page size.
Does any real person actually do this?
Yes, on systems with the page size larger than 4K, the ext4 filesystem block size is often smaller than the page size. ext4 encryption (fscrypt) originally had the same limitation, and Chandan Rajendra from IBM did significant work to solve it a few years ago, with the changes landing in v5.5.
- Eric
Eric Biggers
maintainerRe: [PATCH v2 10/11] fs/buffer.c: support fsverity in block_read_full_folio()
Quoted 28 lines · Eric Biggers
Quoted 9 lines · Andrew Morton
Quoted 4 lines · On Fri, 23 Dec 2022 12:36:37 -0800 Eric Biggers <[email protected]> wrote:
After each filesystem block (as represented by a
buffer_head) has been read from disk byblock_read_full_folio(), verify it if needed. The verification is done on thefsverity_read_workqueue. Also allow reads of verity metadata pasti_size, as required by ext4.Sigh. Do we reeeeealy need to mess with
buffer.cin this fashion? Did any other subsystems feel a need to do this?ext4 is currently the only filesystem that uses
block_read_full_folio()and that supports fsverity. However, since fsverity has a common infrastructure across filesystems, infs/verity/, it makes sense to support it in the other filesystem infrastructure so that things aren't mutually exclusive for no reason.Note that this applies to fscrypt too, which
block_read_full_folio()(previouslyblock_read_full_page()) already supports since v5.5.If you'd prefer that
block_read_full_folio()be copied into ext4, then modified to support fscrypt and fsverity, and then the fscrypt support removed from the original copy, we could do that. That seems more like a workaround to avoid modifying certain files than an actually better solution, but it could be done.Quoted 4 lines
Quoted 3 lines
This is needed to support fsverity on ext4 filesystems where the filesystem block size is less than the page size.
Quoted 8 lines
Quoted 1 line
Does any real person actually do this?
Yes, on systems with the page size larger than 4K, the ext4 filesystem block size is often smaller than the page size. ext4 encryption (fscrypt) originally had the same limitation, and Chandan Rajendra from IBM did significant work to solve it a few years ago, with the changes landing in v5.5.
- Eric
Any more thoughts on this from Andrew, the ext4 maintainers, or anyone else?
- Eric
Re: [PATCH v2 10/11] fs/buffer.c: support fsverity in block_read_full_folio()
Quoted 1 line · Eric Biggers
Any more thoughts on this from Andrew, the ext4 maintainers, or anyone else?
As someone else: I relaly much prefer to support common functionality (fsverity) in common helpers rather than copy and pasting them into various file systems. The copy common helper and slightly modify it is a cancer infecting various file systems that makes it really hard to maintain the kernel.
Eric Biggers
maintainerPATCH v2 11/11 ext4: allow verity with fs block size < PAGE_SIZE
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Now that the needed changes have been made to fs/buffer.c, ext4 is ready to support the verity feature when the filesystem block size is less than the page size. So remove the mount-time check that prevented this.
--- Documentation/filesystems/fsverity.rst | 8 +++++--- fs/ext4/super.c | 5 ----- 2 files changed, 5 insertions(+), 8 deletions(-)
Documentation/filesystems/fsverity.rst +5 -3
@@ -497,9 +497,11 @@ To create verity files on an ext4 filesystem, the filesystem must have been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on it. "verity" is an RO_COMPAT filesystem feature, so once set, old kernels will only be able to mount the filesystem readonly, and old -versions of e2fsck will be unable to check the filesystem. Moreover, -currently ext4 only supports mounting a filesystem with the "verity" -feature when its block size is equal to PAGE_SIZE (often 4096 bytes). +versions of e2fsck will be unable to check the filesystem. + +Originally, an ext4 filesystem with the "verity" feature could only be +mounted when its block size was equal to the system page size +(typically 4096 bytes). In Linux v6.3, this limitation was removed. ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files. It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared.
fs/ext4/super.c +0 -5
@@ -5336,11 +5336,6 @@ static int __ext4_fill_super(struct fs_context *fc, struct super_block *sb) } } -if (ext4_has_feature_verity (sb) && sb->s_blocksize !=PAGE_SIZE ) { -ext4_msg (sb,KERN_ERR ,"Unsupported blocksize for fs-verity" ); - goto failed_mount_wq; - } -/* * Get the # of file system overhead blocks from the * superblock if present.
Signature · 1 line
2.39.0
Hi Eric,
I have roughly gone through the series and run the (patched) xfstests on this patchset on a powerpc machine with 64k pagesize and 64k,4k and 1k merkle tree size on EXT4 and everything seems to work correctly.
Just for records, test generic/692 takes a lot of time to complete with 64k merkel tree size due to the calculations assuming it to be 4k, however I was able to manually test that particular scenario. (I'll try to send a patch to fix the fstest later).
Anyways, feel free to add:
Since I was not very familiar with the fsverty codebase, I'll try to take some more time to review the code and get back with any comments/RVBs.
Regards,
ojaswin
Quoted 91 lines · Eric Biggers
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]Currently, filesystems (ext4, f2fs, and btrfs) only support fsverity when the Merkle tree block size, filesystem block size, and page size are all the same. In practice that means 4K, since increasing the page size, e.g. to 16K, forces the Merkle tree block size and filesystem block size to be increased accordingly. That can be impractical; for one, users want the same file signatures to work on all systems.
Therefore, this patchset reduces the coupling between these sizes.
First, patches 1-4 are cleanups.
Second, patches 5-9 allow the Merkle tree block size to be less than the page size or filesystem block size, provided that it's not larger than either one. This involves, among other things, changing the way that
fs/verity/verify.ctracks which hash blocks have been verified.Finally, patches 10-11 make ext4 support fsverity when the filesystem block size is less than the page size. Note, f2fs doesn't need similar changes because f2fs always assumes that the filesystem block size and page size are the same anyway. I haven't looked into btrfs yet.
I've tested this patchset using the "verity" group of tests in xfstests with the following xfstests patchset applied:
"[PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size"
(https://lore.kernel.org/fstests/[email protected]/T/#u)Note: on the thread "[RFC PATCH 00/11] fs-verity support for XFS"
(https://lore.kernel.org/linux-xfs/[email protected]/T/#u)
there have been many requests for other things to support, including:* folios in the pagecache
* alternative Merkle tree caching methods
* direct I/O
*merkle_tree_block_size>page_size
* extremely large files, using a reclaimable bitmapWe shouldn't try to boil the ocean, though, so to keep the scope of this patchset manageable I haven't changed it significantly from v1. This patchset does bring us closer to many of the above, just not all the way there. I'd like to follow up this patchset with a change to support folios, which should be straightforward. Next, we can do a change to generalize the Merkle tree interface to allow XFS to use an alternative caching method, as that sounds like the highest priority item for XFS.
Anyway, the changelog is:
Changed in v2:
- Rebased onto the recent fsverity cleanups.
- Split some parts of the big "support verification" patch into separate patches.
- Passed thedata_postoverify_data_block()instead of computing itusing page->index, to make it ready for folio and DIO support.- Eliminated some unnecessary arithmetic in
verify_data_block().
- Changed the log_* fields inmerkle_tree_paramsto u8.
- Restored PageLocked and !PageUptodate checks for pagecache pages.
- Eliminated the change tofsverity_hash_buffer().
- Other small cleanupsEric Biggers (11):
fsverity: use unsigned long forlevel_start
fsverity: simplify Merkle tree readahead size calculation
fsverity: storelog2(digest_size)precomputed
fsverity: use EFBIG for file too large to enable verity
fsverity: replacefsverity_hash_page()withfsverity_hash_block()
fsverity: support verification with tree block size <PAGE_SIZE
fsverity: support enabling with tree block size <PAGE_SIZE
ext4: simplifyext4_readpage_limit()
f2fs: simplifyf2fs_readpage_limit()
fs/buffer.c: support fsverity inblock_read_full_folio()
ext4: allow verity with fs block size <PAGE_SIZEDocumentation/filesystems/fsverity.rst | 76 +++--- fs/buffer.c | 67 ++++- fs/ext4/readpage.c | 3 +- fs/ext4/super.c | 5 - fs/f2fs/data.c | 3 +- fs/verity/enable.c | 260 ++++++++++---------- fs/verity/fsverity_private.h | 20 +- fs/verity/hash_algs.c | 24 +- fs/verity/open.c | 98 ++++++-- fs/verity/verify.c | 325 +++++++++++++++++-------- include/linux/fsverity.h | 14 +- 11 files changed, 565 insertions(+), 330 deletions(-)Signature · 1 line
2.39.0
Eric Biggers
maintainerQuoted 13 lines · Ojaswin Mujoo
Hi Eric,
I have roughly gone through the series and run the (patched) xfstests on this patchset on a powerpc machine with 64k pagesize and 64k,4k and 1k merkle tree size on EXT4 and everything seems to work correctly.
Just for records, test generic/692 takes a lot of time to complete with 64k merkel tree size due to the calculations assuming it to be 4k, however I was able to manually test that particular scenario. (I'll try to send a patch to fix the fstest later).
Anyways, feel free to add:
Quoted 8 lines
Tested-by Ojaswin Mujoo <[email protected]>Since I was not very familiar with the fsverty codebase, I'll try to take some more time to review the code and get back with any comments/RVBs.
Regards,
ojaswin
Thanks Ojaswin! That's a good point about generic/692. The right fix for it is to make it use $FSV_BLOCK_SIZE instead of 4K in its calculations.
I suppose you saw that issue by running the test on ext4 with fs_block_size == page_size == 64K, causing xfstests to use merkle_tree_block_size == 64K by default. Thanks for doing that; that's something I haven't been able to test yet. My focus has been on merkle_tree_block_size < page_size. merkle_tree_block_size > 4K should just work, though, assuming merkle_tree_block_size <= min(fs_block_size, page_size). (Or merkle_tree_block_size == fs_block_size == page_size before this patch series.)
- Eric
Ojaswin Mujoo
reviewerQuoted 1 line · Eric Biggers
Hi Eric,
Quoted 2 lines
Thanks Ojaswin! That's a good point about generic/692. The right fix for it is to make it use $FSV_BLOCK_SIZE instead of 4K in its calculations.
Yes, that should fix the issue, I'll try to send in a patch for this when I find some time.
Quoted 5 lines
I suppose you saw that issue by running the test on ext4 with
fs_block_size==page_size== 64K, causing xfstests to usemerkle_tree_block_size== 64K by default. Thanks for doing that; that's something I haven't been able to test yet. My focus has been onmerkle_tree_block_size<page_size.
Correct, I was testing "everything = 64k" scenario when I noticed the slowdown.
Quoted 3 lines
merkle_tree_block_size> 4K should just work, though, assumingmerkle_tree_block_size<=min(fs_block_size, page_size). (Ormerkle_tree_block_size==fs_block_size==page_sizebefore this patch series.)
Yes true, I still tested them just in case :)
Regards,
Ojaswin
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerQuoted 3 lines · Eric Biggers
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]
I've applied this patchset for 6.3, but I'd still greatly appreciate reviews and acks, especially on the last 4 patches, which touch files outside fs/verity/.
(I applied it to
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git/log/?h=fsverity for now, but there might be a new git repo soon, as is being discussed elsewhere.)
- Eric
Andrey Albershteyn
Reviewed-byQuoted 5 lines · Eric Biggers
Quoted 3 lines · Eric Biggers
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]
Quoted 9 lines
I've applied this patchset for 6.3, but I'd still greatly appreciate reviews and acks, especially on the last 4 patches, which touch files outside
fs/verity/.(I applied it to
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git/log/?h=fsverity for now, but there might be a new git repo soon, as is being discussed elsewhere.)- Eric
The fs/verity patches look good to me, I've checked them but forgot to send RVB :( Haven't tested them yet though
Signature · 8 lines
- Andrey _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerQuoted 13 lines · Andrey Albershteyn
Quoted 12 lines · Eric Biggers
Quoted 3 lines · Eric Biggers
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]I've applied this patchset for 6.3, but I'd still greatly appreciate reviews and acks, especially on the last 4 patches, which touch files outside
fs/verity/.(I applied it to
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git/log/?h=fsverity for now, but there might be a new git repo soon, as is being discussed elsewhere.)
Quoted 8 lines
Quoted 2 lines
- Eric
The fs/verity patches look good to me, I've checked them but forgot to send RVB :( Haven't tested them yet though
Reviewed-by Andrey Albershteyn <[email protected]>
Thanks Andrey! I added your Reviewed-by to patches 1-7 only, since you said "the fs/verity patches". Let me know if I can add it to patches 8-11 too.
- Eric
Eric Biggers
maintainerQuoted 4 lines · Eric Biggers
ext4: simplify
ext4_readpage_limit()
f2fs: simplifyf2fs_readpage_limit()fs/buffer.c: support fsverity inblock_read_full_folio()
ext4: allow verity with fs block size <PAGE_SIZE
I'd still appreciate acks from the other ext4 and f2fs developers on these!
- Eric
This series was applied to jaegeuk/f2fs.git (dev) by Eric Biggers <[email protected]>:
Hello:
This series was applied to jaegeuk/f2fs.git (dev)
by Eric Biggers <[email protected]>:
Quoted 4 lines · On Fri, 23 Dec 2022 12:36:27 -0800 you wrote:
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]
Quoted 8 lines
Currently, filesystems (ext4, f2fs, and btrfs) only support fsverity when the Merkle tree block size, filesystem block size, and page size are all the same. In practice that means 4K, since increasing the page size, e.g. to 16K, forces the Merkle tree block size and filesystem block size to be increased accordingly. That can be impractical; for one, users want the same file signatures to work on all systems.
[...]
Here is the summary with links:
- [f2fs-dev,v2,01/11] fsverity: use unsigned long for level_start
https://git.kernel.org/jaegeuk/f2fs/c/284d5db5f99e - [f2fs-dev,v2,02/11] fsverity: simplify Merkle tree readahead size calculation https://git.kernel.org/jaegeuk/f2fs/c/9098f36b739d - [f2fs-dev,v2,03/11] fsverity: store log2(digest_size) precomputed https://git.kernel.org/jaegeuk/f2fs/c/579a12f78d88 - [f2fs-dev,v2,04/11] fsverity: use EFBIG for file too large to enable verity https://git.kernel.org/jaegeuk/f2fs/c/55eed69cc8fd - [f2fs-dev,v2,05/11] fsverity: replace fsverity_hash_page() with fsverity_hash_block() https://git.kernel.org/jaegeuk/f2fs/c/f45555bf23cf - [f2fs-dev,v2,06/11] fsverity: support verification with tree block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/5306892a50bf - [f2fs-dev,v2,07/11] fsverity: support enabling with tree block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/56124d6c87fd - [f2fs-dev,v2,08/11] ext4: simplify ext4_readpage_limit() https://git.kernel.org/jaegeuk/f2fs/c/5e122148a3d5 - [f2fs-dev,v2,09/11] f2fs: simplify f2fs_readpage_limit() https://git.kernel.org/jaegeuk/f2fs/c/feb0576a361a - [f2fs-dev,v2,10/11] fs/buffer.c: support fsverity in block_read_full_folio() https://git.kernel.org/jaegeuk/f2fs/c/4fa512ce7051 - [f2fs-dev,v2,11/11] ext4: allow verity with fs block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/db85d14dc5c5
You are awesome, thank you!
Signature · 10 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Eric Biggers
maintainerQuoted 43 lines · On Tue, Feb 28, 2023 at 01:01:54AM +0000, [email protected] wrote:
Hello:
This series was applied to jaegeuk/f2fs.git (dev)
by Eric Biggers <[email protected]>:Quoted 4 lines · On Fri, 23 Dec 2022 12:36:27 -0800 you wrote:
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]Quoted 8 lines
Currently, filesystems (ext4, f2fs, and btrfs) only support fsverity when the Merkle tree block size, filesystem block size, and page size are all the same. In practice that means 4K, since increasing the page size, e.g. to 16K, forces the Merkle tree block size and filesystem block size to be increased accordingly. That can be impractical; for one, users want the same file signatures to work on all systems.
[...]
Here is the summary with links:
- [f2fs-dev,v2,01/11] fsverity: use unsigned long forlevel_starthttps://git.kernel.org/jaegeuk/f2fs/c/284d5db5f99e - [f2fs-dev,v2,02/11] fsverity: simplify Merkle tree readahead size calculation https://git.kernel.org/jaegeuk/f2fs/c/9098f36b739d - [f2fs-dev,v2,03/11] fsverity: store log2(digest_size) precomputed https://git.kernel.org/jaegeuk/f2fs/c/579a12f78d88 - [f2fs-dev,v2,04/11] fsverity: use EFBIG for file too large to enable verity https://git.kernel.org/jaegeuk/f2fs/c/55eed69cc8fd - [f2fs-dev,v2,05/11] fsverity: replace fsverity_hash_page() with fsverity_hash_block() https://git.kernel.org/jaegeuk/f2fs/c/f45555bf23cf - [f2fs-dev,v2,06/11] fsverity: support verification with tree block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/5306892a50bf - [f2fs-dev,v2,07/11] fsverity: support enabling with tree block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/56124d6c87fd - [f2fs-dev,v2,08/11] ext4: simplify ext4_readpage_limit() https://git.kernel.org/jaegeuk/f2fs/c/5e122148a3d5 - [f2fs-dev,v2,09/11] f2fs: simplify f2fs_readpage_limit() https://git.kernel.org/jaegeuk/f2fs/c/feb0576a361a - [f2fs-dev,v2,10/11] fs/buffer.c: support fsverity in block_read_full_folio() https://git.kernel.org/jaegeuk/f2fs/c/4fa512ce7051 - [f2fs-dev,v2,11/11] ext4: allow verity with fs block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/db85d14dc5c5
Quoted 5 lines
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
These commits reached the f2fs tree through mainline, not through being applied to the f2fs tree. So this email shouldn't have been sent. Jaegeuk, can you look into fixing the configuration of the f2fs patchwork bot to prevent this?
- Eric
Jaegeuk Kim
maintainerQuoted 50 lines · Eric Biggers
Quoted 43 lines · On Tue, Feb 28, 2023 at 01:01:54AM +0000, [email protected] wrote:
Hello:
This series was applied to jaegeuk/f2fs.git (dev)
by Eric Biggers <[email protected]>:Quoted 4 lines · On Fri, 23 Dec 2022 12:36:27 -0800 you wrote:
[This patchset applies to mainline + some fsverity cleanups I sent out
recently. You can get everything from tag "fsverity-non4k-v2" of
https://git.kernel.org/pub/scm/fs/fscrypt/fscrypt.git ]Quoted 8 lines
Currently, filesystems (ext4, f2fs, and btrfs) only support fsverity when the Merkle tree block size, filesystem block size, and page size are all the same. In practice that means 4K, since increasing the page size, e.g. to 16K, forces the Merkle tree block size and filesystem block size to be increased accordingly. That can be impractical; for one, users want the same file signatures to work on all systems.
[...]
Here is the summary with links:
- [f2fs-dev,v2,01/11] fsverity: use unsigned long forlevel_starthttps://git.kernel.org/jaegeuk/f2fs/c/284d5db5f99e - [f2fs-dev,v2,02/11] fsverity: simplify Merkle tree readahead size calculation https://git.kernel.org/jaegeuk/f2fs/c/9098f36b739d - [f2fs-dev,v2,03/11] fsverity: store log2(digest_size) precomputed https://git.kernel.org/jaegeuk/f2fs/c/579a12f78d88 - [f2fs-dev,v2,04/11] fsverity: use EFBIG for file too large to enable verity https://git.kernel.org/jaegeuk/f2fs/c/55eed69cc8fd - [f2fs-dev,v2,05/11] fsverity: replace fsverity_hash_page() with fsverity_hash_block() https://git.kernel.org/jaegeuk/f2fs/c/f45555bf23cf - [f2fs-dev,v2,06/11] fsverity: support verification with tree block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/5306892a50bf - [f2fs-dev,v2,07/11] fsverity: support enabling with tree block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/56124d6c87fd - [f2fs-dev,v2,08/11] ext4: simplify ext4_readpage_limit() https://git.kernel.org/jaegeuk/f2fs/c/5e122148a3d5 - [f2fs-dev,v2,09/11] f2fs: simplify f2fs_readpage_limit() https://git.kernel.org/jaegeuk/f2fs/c/feb0576a361a - [f2fs-dev,v2,10/11] fs/buffer.c: support fsverity in block_read_full_folio() https://git.kernel.org/jaegeuk/f2fs/c/4fa512ce7051 - [f2fs-dev,v2,11/11] ext4: allow verity with fs block size < PAGE_SIZE https://git.kernel.org/jaegeuk/f2fs/c/db85d14dc5c5Quoted 5 lines
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 3 lines
These commits reached the f2fs tree through mainline, not through being applied to the f2fs tree. So this email shouldn't have been sent. Jaegeuk, can you look into fixing the configuration of the f2fs patchwork bot to prevent this?
Hmm, not sure how to fix that, since it seems patchwork bot reports this, once I pulled mainline into f2fs/dev branch.
Quoted 2 lines
- Eric