Improve simple directory offset wrap behavior
Subsystems (4)
v7 cover Improve simple directory offset wrap behavior
From: Chuck Lever <[email protected]>
The purpose of this series is to construct a set of upstream fixes that can be backported to v6.6 to address CVE-2024-46701.
In response to a reported failure of libhugetlbfs-test.32bit.gethugepagesizes:
https://lore.kernel.org/linux-fsdevel/[email protected]/T/#t
I've narrowed the range of directory offset values returned by simple_offset_add() to 3 .. (S32_MAX - 1) on all platforms. This means the allocation behavior is identical on 32-bit systems, 64-bit systems, and 32-bit user space on 64-bit kernels. The new range still permits over 2 billion concurrent entries per directory.
Changes since v6:
- Restrict the directory offset value range to S32_MAX on all platforms
This series (against v6.13-rc4) has been pushed to:
https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/log/?h=tmpfs-fixes
Chuck Lever (5):
libfs: Return ENOSPC when the directory offset range is exhausted Revert "libfs: Add simple_offset_empty()"
Revert "libfs: fix infinite directory reads for offset dir"
libfs: Replace simple_offset end-of-directory detection
libfs: Use d_children list to iterate simple_offset directories
fs/libfs.c | 162 +++++++++++++++++++++------------------------ include/linux/fs.h | 1 - mm/shmem.c | 4 +- 3 files changed, 79 insertions(+), 88 deletions(-)
Signature · 1 line
2.47.0
PATCH v7 1/5 libfs: Return ENOSPC when the directory offset range is exhausted
From: Chuck Lever <[email protected]>
Testing shows that the EBUSY error return from mtree_alloc_cyclic() leaks into user space. The ERRORS section of "man creat(2)" says:
Quoted 3 lines
EBUSY
O_EXCLwas specified in flags and pathname refers
to a block device that is in use by the system (e.g., it is mounted).
ENOSPC is closer to what applications expect in this situation.
Note that the normal range of simple directory offset values is 2..2^63, so hitting this error is going to be rare to impossible.
--- fs/libfs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/libfs.c +2 -2
@@ -292,8 +292,8 @@ int simple_offset_add(struct offset_ctx *octx, struct dentry *dentry) ret =mtree_alloc_cyclic (&octx->mt, &offset, dentry,DIR_OFFSET_MIN ,LONG_MAX , &octx->next_offset,GFP_KERNEL ); -if (ret <0 ) -return ret; +if (unlikely (ret <0 )) +return ret == -EBUSY ? -ENOSPC : ret;offset_set (dentry, offset);return 0 ;
Signature · 1 line
2.47.0
PATCH v7 2/5 Revert "libfs: Add simple_offset_empty()"
In reply to [email protected]From: Chuck Lever <[email protected]>
simple_empty() and simple_offset_empty() perform the same task. The latter's use as a canary to find bugs has not found any new issues. A subsequent patch will remove the use of the mtree for iterating directory contents, so revert back to using a similar mechanism for determining whether a directory is indeed empty.
Only one such mechanism is ever needed.
--- fs/libfs.c | 32 -------------------------------- include/linux/fs.h | 1 - mm/shmem.c | 4 ++-- 3 files changed, 2 insertions(+), 35 deletions(-)
fs/libfs.c +0 -32
@@ -329,38 +329,6 @@ void simple_offset_remove(struct offset_ctx *octx, struct dentry *dentry)offset_set (dentry,0 ); } -/** -* simple_offset_empty - Check if a dentry can be unlinked -* @dentry: dentry to be tested -* -* Returns 0 if @dentry is a non-empty directory; otherwise returns 1. -*/ -int simple_offset_empty (struct dentry *dentry) -{ -struct inode *inode =d_inode (dentry); -struct offset_ctx *octx; -struct dentry *child; -unsigned long index; -int ret =1 ; - -if (!inode || !S_ISDIR (inode->i_mode)) -return ret; - - index =DIR_OFFSET_MIN ; - octx = inode->i_op->get_offset_ctx (inode); -mt_for_each (&octx->mt, child, index,LONG_MAX ) { -spin_lock (&child->d_lock); -if (simple_positive (child)) { -spin_unlock (&child->d_lock); - ret =0 ; -break ; - } -spin_unlock (&child->d_lock); - } - -return ret; -} -/** * simple_offset_rename - handle directory offsets for rename * @old_dir: parent directory of source entry
include/linux/fs.h +0 -1
@@ -3468,7 +3468,6 @@ struct offset_ctx {void simple_offset_init (struct offset_ctx *octx);int simple_offset_add (struct offset_ctx *octx,struct dentry *dentry);void simple_offset_remove (struct offset_ctx *octx,struct dentry *dentry); -int simple_offset_empty (struct dentry *dentry);int simple_offset_rename (struct inode *old_dir,struct dentry *old_dentry,struct inode *new_dir,struct dentry *new_dentry); int simple_offset_rename_exchange(struct inode *old_dir,
mm/shmem.c +2 -2
@@ -3820,7 +3820,7 @@ static int shmem_unlink(struct inode *dir, struct dentry *dentry)static int shmem_rmdir (struct inode *dir,struct dentry *dentry) { -if (!simple_offset_empty (dentry)) +if (!simple_empty (dentry))return -ENOTEMPTY ;drop_nlink (d_inode (dentry));
@@ -3877,7 +3877,7 @@ static int shmem_rename2(struct mnt_idmap *idmap,return simple_offset_rename_exchange (old_dir, old_dentry, new_dir, new_dentry); -if (!simple_offset_empty (new_dentry)) +if (!simple_empty (new_dentry))return -ENOTEMPTY ;if (flags &RENAME_WHITEOUT ) {
Signature · 1 line
2.47.0
PATCH v7 3/5 Revert "libfs: fix infinite directory reads for offset dir"
In reply to [email protected]From: Chuck Lever <[email protected]>
The current directory offset allocator (based on mtree_alloc_cyclic) stores the next offset value to return in octx->next_offset. This mechanism typically returns values that increase monotonically over time. Eventually, though, the newly allocated offset value wraps back to a low number (say, 2) which is smaller than other already- allocated offset values.
Yu Kuai <[email protected]> reports that, after commit 64a7ce76fb90 ("libfs: fix infinite directory reads for offset dir"), if a directory's offset allocator wraps, existing entries are no longer visible via readdir/getdents because offset_readdir() stops listing entries once an entry's offset is larger than octx->next_offset. These entries vanish persistently -- they can be looked up, but will never again appear in readdir(3) output.
The reason for this is that the commit treats directory offsets as monotonically increasing integer values rather than opaque cookies, and introduces this comparison:
if (dentry2offset(dentry) >= last_index) {On 64-bit platforms, the directory offset value upper bound is 2^63 - 1. Directory offsets will monotonically increase for millions of years without wrapping.
On 32-bit platforms, however, LONG_MAX is 2^31 - 1. The allocator can wrap after only a few weeks (at worst).
Revert commit 64a7ce76fb90 ("libfs: fix infinite directory reads for offset dir") to prepare for a fix that can work properly on 32-bit systems and might apply to recent LTS kernels where shmem employs the simple_offset mechanism.
--- fs/libfs.c | 35 +++++++++++------------------------ 1 file changed, 11 insertions(+), 24 deletions(-)
fs/libfs.c +11 -24
@@ -422,14 +422,6 @@ void simple_offset_destroy(struct offset_ctx *octx)mtree_destroy (&octx->mt); } -static int offset_dir_open (struct inode *inode,struct file *file) -{ -struct offset_ctx *ctx = inode->i_op->get_offset_ctx (inode); - - file->private_data = (void *)ctx->next_offset; -return 0 ; -} -/** * offset_dir_llseek - Advance the read position of a directory descriptor * @file: an open directory whose position is to be updated
@@ -443,9 +435,6 @@ static int offset_dir_open(struct inode *inode, struct file *file)*/ static loff_t offset_dir_llseek (struct file *file,loff_t offset,int whence) { -struct inode *inode = file->f_inode; -struct offset_ctx *ctx = inode->i_op->get_offset_ctx (inode); -switch (whence) {case SEEK_CUR : offset += file->f_pos;
@@ -459,8 +448,7 @@ static loff_t offset_dir_llseek(struct file *file, loff_t offset, int whence) }/* In this case, ->private_data is protected by f_pos_lock */ -if (!offset) - file->private_data = (void *)ctx->next_offset; + file->private_data =NULL ;return vfs_setpos (file, offset,LONG_MAX ); }
@@ -491,7 +479,7 @@ static bool offset_dir_emit(struct dir_context *ctx, struct dentry *dentry) inode->i_ino, fs_umode_to_dtype(inode->i_mode)); } -static void offset_iterate_dir (struct inode *inode,struct dir_context *ctx,long last_index) +static void *offset_iterate_dir (struct inode *inode,struct dir_context *ctx) {struct offset_ctx *octx = inode->i_op->get_offset_ctx (inode);struct dentry *dentry;
@@ -499,21 +487,17 @@ static void offset_iterate_dir(struct inode *inode, struct dir_context *ctx, lonwhile (true) { dentry =offset_find_next (octx, ctx->pos);if (!dentry) -return ; - -if (dentry2offset (dentry) >= last_index) { -dput (dentry); -return ; - } +return ERR_PTR (-ENOENT );if (!offset_dir_emit (ctx, dentry)) {dput (dentry); -return ; +break ; } ctx->pos =dentry2offset (dentry) +1 ;dput (dentry); } +return NULL ; }/**
@@ -540,19 +524,22 @@ static void offset_iterate_dir(struct inode *inode, struct dir_context *ctx, lonstatic int offset_readdir (struct file *file,struct dir_context *ctx) {struct dentry *dir = file->f_path.dentry; -long last_index = (long )file->private_data;lockdep_assert_held (&d_inode (dir)->i_rwsem);if (!dir_emit_dots (file, ctx))return 0 ; -offset_iterate_dir (d_inode (dir), ctx, last_index); +/* In this case, ->private_data is protected by f_pos_lock */ +if (ctx->pos ==DIR_OFFSET_MIN ) + file->private_data =NULL ; +else if (file->private_data ==ERR_PTR (-ENOENT )) +return 0 ; + file->private_data =offset_iterate_dir (d_inode (dir), ctx);return 0 ; }const struct file_operations simple_offset_dir_operations = { - .open = offset_dir_open, .llseek = offset_dir_llseek, .iterate_shared = offset_readdir, .read = generic_read_dir,
Signature · 1 line
2.47.0
PATCH v7 4/5 libfs: Replace simple_offset end-of-directory detection
In reply to [email protected]From: Chuck Lever <[email protected]>
According to getdents(3), the d_off field in each returned directory entry points to the next entry in the directory. The d_off field in the last returned entry in the readdir buffer must contain a valid offset value, but if it points to an actual directory entry, then readdir/getdents can loop.
This patch introduces a specific fixed offset value that is placed in the d_off field of the last entry in a directory. Some user space applications assume that the EOD offset value is larger than the offsets of real directory entries, so the largest valid offset value is reserved for this purpose. This new value is never allocated by simple_offset_add().
When ->iterate_dir() returns, getdents{64} inserts the ctx->pos value into the d_off field of the last valid entry in the readdir buffer. When it hits EOD, offset_readdir() sets ctx->pos to the EOD offset value so the last entry is updated to point to the EOD marker.
When trying to read the entry at the EOD offset, offset_readdir() terminates immediately.
It is worth noting that using a Maple tree for directory offset value allocation does not guarantee a 63-bit range of values -- on platforms where "long" is a 32-bit type, the directory offset value range is still 0..(2^31 - 1). For broad compatibility with 32-bit user space, the largest tmpfs directory cookie value is now S32_MAX.
--- fs/libfs.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-)
fs/libfs.c +21 -16
@@ -245,9 +245,15 @@ const struct inode_operations simple_dir_inode_operations = { };EXPORT_SYMBOL (simple_dir_inode_operations); -/* 0 is '.', 1 is '..', so always start with offset 2 or more */ +/* simple_offset_add() never assigns these to a dentry */ enum { -DIR_OFFSET_MIN =2 , +DIR_OFFSET_EOD =S32_MAX , +}; + +/* simple_offset_add() allocation range */ +enum { +DIR_OFFSET_MIN =2 , +DIR_OFFSET_MAX =DIR_OFFSET_EOD -1 , };static void offset_set (struct dentry *dentry,long offset)
@@ -291,7 +297,8 @@ int simple_offset_add(struct offset_ctx *octx, struct dentry *dentry)return -EBUSY ; ret =mtree_alloc_cyclic (&octx->mt, &offset, dentry,DIR_OFFSET_MIN , -LONG_MAX , &octx->next_offset,GFP_KERNEL ); +DIR_OFFSET_MAX , &octx->next_offset, +GFP_KERNEL );if (unlikely (ret <0 ))return ret == -EBUSY ? -ENOSPC : ret;
@@ -447,8 +454,6 @@ static loff_t offset_dir_llseek(struct file *file, loff_t offset, int whence)return -EINVAL ; } -/* In this case, ->private_data is protected by f_pos_lock */ - file->private_data =NULL ;return vfs_setpos (file, offset,LONG_MAX ); }
@@ -458,7 +463,7 @@ static struct dentry *offset_find_next(struct offset_ctx *octx, loff_t offset)struct dentry *child, *found =NULL ;rcu_read_lock (); - child =mas_find (&mas,LONG_MAX ); + child =mas_find (&mas,DIR_OFFSET_MAX );if (!child) goto out;spin_lock (&child->d_lock);
@@ -479,7 +484,7 @@ static bool offset_dir_emit(struct dir_context *ctx, struct dentry *dentry) inode->i_ino, fs_umode_to_dtype(inode->i_mode)); } -static void *offset_iterate_dir (struct inode *inode,struct dir_context *ctx) +static void offset_iterate_dir (struct inode *inode,struct dir_context *ctx) {struct offset_ctx *octx = inode->i_op->get_offset_ctx (inode);struct dentry *dentry;
@@ -487,7 +492,7 @@ static void *offset_iterate_dir(struct inode *inode, struct dir_context *ctx)while (true) { dentry =offset_find_next (octx, ctx->pos);if (!dentry) -return ERR_PTR (-ENOENT ); + goto out_eod;if (!offset_dir_emit (ctx, dentry)) {dput (dentry);
@@ -497,7 +502,10 @@ static void *offset_iterate_dir(struct inode *inode, struct dir_context *ctx) ctx->pos =dentry2offset (dentry) +1 ;dput (dentry); } -return NULL ; +return ; + +out_eod: + ctx->pos =DIR_OFFSET_EOD ; }/**
@@ -517,6 +525,8 @@ static void *offset_iterate_dir(struct inode *inode, struct dir_context *ctx)* * On return, @ctx->pos contains an offset that will read the next entry * in this directory when offset_readdir() is called again with @ctx. +* Caller places this value in the d_off field of the last entry in the +* user's buffer. * * Return values: * %0 - Complete
@@ -529,13 +539,8 @@ static int offset_readdir(struct file *file, struct dir_context *ctx)if (!dir_emit_dots (file, ctx))return 0 ; - -/* In this case, ->private_data is protected by f_pos_lock */ -if (ctx->pos ==DIR_OFFSET_MIN ) - file->private_data =NULL ; -else if (file->private_data ==ERR_PTR (-ENOENT )) -return 0 ; - file->private_data =offset_iterate_dir (d_inode (dir), ctx); +if (ctx->pos !=DIR_OFFSET_EOD ) +offset_iterate_dir (d_inode (dir), ctx);return 0 ; }
Signature · 1 line
2.47.0
PATCH v7 5/5 libfs: Use d_children list to iterate simple_offset directories
In reply to [email protected]From: Chuck Lever <[email protected]>
The mtree mechanism has been effective at creating directory offsets that are stable over multiple opendir instances. However, it has not been able to handle the subtleties of renames that are concurrent with readdir.
Instead of using the mtree to emit entries in the order of their offset values, use it only to map incoming ctx->pos to a starting entry. Then use the directory's d_children list, which is already maintained properly by the dcache, to find the next child to emit.
One of the sneaky things about this is that when the mtree-allocated offset value wraps (which is very rare), looking up ctx->pos++ is not going to find the next entry; it will return NULL. Instead, by following the d_children list, the offset values can appear in any order but all of the entries in the directory will be visited eventually.
Note also that the readdir() is guaranteed to reach the tail of this list. Entries are added only at the head of d_children, and readdir walks from its current position in that list towards its tail.
--- fs/libfs.c | 84 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 58 insertions(+), 26 deletions(-)
fs/libfs.c +58 -26
@@ -247,12 +247,13 @@ EXPORT_SYMBOL(simple_dir_inode_operations);/* simple_offset_add() never assigns these to a dentry */ enum { +DIR_OFFSET_FIRST =2 ,/* Find first real entry */ DIR_OFFSET_EOD =S32_MAX , };/* simple_offset_add() allocation range */ enum { -DIR_OFFSET_MIN =2 , +DIR_OFFSET_MIN =DIR_OFFSET_FIRST +1 ,DIR_OFFSET_MAX =DIR_OFFSET_EOD -1 , };
@@ -457,51 +458,82 @@ static loff_t offset_dir_llseek(struct file *file, loff_t offset, int whence)return vfs_setpos (file, offset,LONG_MAX ); } -static struct dentry *offset_find_next (struct offset_ctx *octx,loff_t offset) +static struct dentry *find_positive_dentry (struct dentry *parent, +struct dentry *dentry, +bool next) { -MA_STATE (mas, &octx->mt, offset, offset); +struct dentry *found =NULL ; + +spin_lock (&parent->d_lock); +if (next) + dentry =d_next_sibling (dentry); +else if (!dentry) + dentry =d_first_child (parent); +hlist_for_each_entry_from (dentry, d_sib) { +if (!simple_positive (dentry)) +continue ; +spin_lock_nested (&dentry->d_lock,DENTRY_D_LOCK_NESTED ); +if (simple_positive (dentry)) + found =dget_dlock (dentry); +spin_unlock (&dentry->d_lock); +if (likely (found)) +break ; + } +spin_unlock (&parent->d_lock); +return found; +} + +static noinline_for_stack struct dentry * +offset_dir_lookup (struct dentry *parent,loff_t offset) +{ +struct inode *inode =d_inode (parent); +struct offset_ctx *octx = inode->i_op->get_offset_ctx (inode);struct dentry *child, *found =NULL ; -rcu_read_lock (); - child =mas_find (&mas,DIR_OFFSET_MAX ); -if (!child) - goto out; -spin_lock (&child->d_lock); -if (simple_positive (child)) - found =dget_dlock (child); -spin_unlock (&child->d_lock); -out: -rcu_read_unlock (); +MA_STATE (mas, &octx->mt, offset, offset); + +if (offset ==DIR_OFFSET_FIRST ) + found =find_positive_dentry (parent,NULL , false); +else { +rcu_read_lock (); + child =mas_find (&mas,DIR_OFFSET_MAX ); + found =find_positive_dentry (parent, child, false); +rcu_read_unlock (); + }return found; }static bool offset_dir_emit (struct dir_context *ctx,struct dentry *dentry) {struct inode *inode =d_inode (dentry); -long offset =dentry2offset (dentry); -return ctx->actor (ctx, dentry->d_name.name, dentry->d_name.len, offset, - inode->i_ino,fs_umode_to_dtype (inode->i_mode)); +return dir_emit (ctx, dentry->d_name.name, dentry->d_name.len, + inode->i_ino,fs_umode_to_dtype (inode->i_mode)); } -static void offset_iterate_dir (struct inode *inode,struct dir_context *ctx) +static void offset_iterate_dir (struct file *file,struct dir_context *ctx) { -struct offset_ctx *octx = inode->i_op->get_offset_ctx (inode); +struct dentry *dir = file->f_path.dentry;struct dentry *dentry; + dentry =offset_dir_lookup (dir, ctx->pos); +if (!dentry) + goto out_eod;while (true) { - dentry =offset_find_next (octx, ctx->pos); -if (!dentry) - goto out_eod; +struct dentry *next; -if (!offset_dir_emit (ctx, dentry)) { -dput (dentry); + ctx->pos =dentry2offset (dentry); +if (!offset_dir_emit (ctx, dentry))break ; - } - ctx->pos =dentry2offset (dentry) +1 ; + next =find_positive_dentry (dir, dentry, true);dput (dentry); + +if (!next) + goto out_eod; + dentry = next; } +dput (dentry);return ; out_eod:
@@ -540,7 +572,7 @@ static int offset_readdir(struct file *file, struct dir_context *ctx)if (!dir_emit_dots (file, ctx))return 0 ;if (ctx->pos !=DIR_OFFSET_EOD ) -offset_iterate_dir (d_inode (dir), ctx); +offset_iterate_dir (file, ctx);return 0 ; }
Signature · 1 line
2.47.0
Quoted 8 lines · On Sat, 28 Dec 2024 12:55:16 -0500, [email protected] wrote:
From: Chuck Lever <[email protected]>
The purpose of this series is to construct a set of upstream fixes that can be backported to v6.6 to address CVE-2024-46701.
In response to a reported failure of libhugetlbfs-test.32bit.gethugepagesizes:
[...]
Applied to the vfs-6.14.libfs branch of the vfs/vfs.git tree. Patches in the vfs-6.14.libfs branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-6.14.libfs
[1/5] libfs: Return ENOSPC when the directory offset range is exhausted
https://git.kernel.org/vfs/vfs/c/903dc9c43a15 [2/5] Revert "libfs: Add simple_offset_empty()" https://git.kernel.org/vfs/vfs/c/d7bde4f27cee [3/5] Revert "libfs: fix infinite directory reads for offset dir" https://git.kernel.org/vfs/vfs/c/b662d858131d [4/5] libfs: Replace simple_offset end-of-directory detection https://git.kernel.org/vfs/vfs/c/68a3a6500314 [5/5] libfs: Use d_children list to iterate simple_offset directories https://git.kernel.org/vfs/vfs/c/b9b588f22a0c