simple xattr improvements
Subsystems (6)
v4 cover simple xattr improvements
v4:
- yet another accounting fix for shmem_initxattrs
v3:
- need per kernfs_node locking for simple_xattr_set
- another accounting fix for shmem_initxattrs
v2:
- don't use xxhash
- fix lazy alloc concurrency
- fix rcu freeing on evict
- fix rcu walk of per-inode list
- don't double account freed space
---
Miklos Szeredi (4):
kernfs: fix xattr race condition with multiple superblocks
tmpfs: simplify constructing "security.foo" xattr names
simple_xattr: change interface to pass struct simple_xattrs **
simpe_xattr: use per-sb cache
fs/kernfs/dir.c | 13 +- fs/kernfs/file.c | 13 +- fs/kernfs/inode.c | 36 ++--- fs/kernfs/kernfs-internal.h | 24 +++- fs/kernfs/mount.c | 2 +- fs/pidfs.c | 45 ++----- fs/xattr.c | 261 +++++++++++++++++++----------------- include/linux/kernfs.h | 11 +- include/linux/shmem_fs.h | 3 +- include/linux/xattr.h | 35 +++-- mm/shmem.c | 50 +++---- net/socket.c | 30 ++--- 12 files changed, 252 insertions(+), 271 deletions(-)
Signature · 1 line
2.54.0
PATCH v4 1/4 kernfs: fix xattr race condition with multiple superblocks
Multiple superblocks with different namespaces can share the same kernfs_node when kernfs_test_super() finds a matching root but different namespace. This means multiple inodes from different superblocks can reference the same kernfs_node->iattr->xattrs structure.
The VFS layer only holds per-inode locks during xattr operations, which is insufficient to serialize concurrent xattr modifications on the shared kernfs_node. This can lead to race conditions in simple_xattr_set() where the lookup->replace/remove sequence is not atomic with respect to operations from other superblocks.
Fix this by protecting xattr operations with the existing hashed kernfs_locks->open_file_mutex[] array, which is already used to protect per-node open file data. The hashed mutex array provides scalable per-node serialization (scaled by CPU count, up to 1024 locks on 32+ CPU systems) with zero memory overhead.
Changes:
- Rename open_file_mutex[] to node_mutex[] to reflect dual purpose
- Add kernfs_node_lock_ptr() and kernfs_node_lock() helpers
- Protect simple_xattr_set() calls in kernfs_xattr_set() and kernfs_vfs_user_xattr_set() with the hashed mutex
- Update file.c to use new helpers via compatibility wrappers
- Update documentation to explain the extended lock usage
--- fs/kernfs/file.c | 13 +++---------- fs/kernfs/inode.c | 12 ++++++++++++ fs/kernfs/kernfs-internal.h | 20 ++++++++++++++++++++ fs/kernfs/mount.c | 2 +- include/linux/kernfs.h | 11 ++++++++--- 5 files changed, 44 insertions(+), 14 deletions(-)
fs/kernfs/file.c +3 -10
@@ -40,22 +40,15 @@ struct kernfs_open_node {static DEFINE_SPINLOCK (kernfs_notify_lock);static struct kernfs_node *kernfs_notify_list =KERNFS_NOTIFY_EOL ; +/* Compatibility wrappers - use the common hashed node lock */ static inline struct mutex *kernfs_open_file_mutex_ptr (struct kernfs_node *kn) { -int idx =hash_ptr (kn,NR_KERNFS_LOCK_BITS ); - -return &kernfs_locks->open_file_mutex[idx]; +return kernfs_node_lock_ptr (kn); }static inline struct mutex *kernfs_open_file_mutex_lock (struct kernfs_node *kn) { -struct mutex *lock; - - lock =kernfs_open_file_mutex_ptr (kn); - -mutex_lock (lock); - -return lock; +return kernfs_node_lock (kn); }/**
fs/kernfs/inode.c +12 -0
@@ -320,6 +320,15 @@ int kernfs_xattr_set(struct kernfs_node *kn, const char *name,if (!attrs)return -ENOMEM ; +/* +* Protect xattr modifications with the hashed per-node mutex. +* Multiple superblocks (with different namespaces) can share the same +* kernfs_node, so inode locking alone is insufficient. The hashed mutex +* ensures serialization of concurrent xattr operations on the same node, +* including the lazy allocation of the xattrs structure itself. +*/ +CLASS (kernfs_node_lock, lock)(kn); + xattrs =simple_xattrs_lazy_alloc (&attrs->xattrs, value, flags);if (IS_ERR_OR_NULL (xattrs))return PTR_ERR (xattrs);
@@ -372,6 +381,9 @@ static int kernfs_vfs_user_xattr_set(const struct xattr_handler *handler,if (!attrs)return -ENOMEM ; +/* See comment in kernfs_xattr_set() about locking. */ +CLASS (kernfs_node_lock, lock)(kn); + xattrs =simple_xattrs_lazy_alloc (&attrs->xattrs, value, flags);if (IS_ERR_OR_NULL (xattrs))return PTR_ERR (xattrs);
fs/kernfs/kernfs-internal.h +20 -0
@@ -211,4 +211,24 @@ extern const struct inode_operations kernfs_symlink_iops;* kernfs locks */ extern struct kernfs_global_locks *kernfs_locks; + +/* Hashed mutex helpers - protect per-node data structures */ +static inline struct mutex *kernfs_node_lock_ptr (struct kernfs_node *kn) +{ +int idx =hash_ptr (kn,NR_KERNFS_LOCK_BITS ); + +return &kernfs_locks->node_mutex[idx]; +} + +static inline struct mutex *kernfs_node_lock (struct kernfs_node *kn) +{ +struct mutex *lock =kernfs_node_lock_ptr (kn); + +mutex_lock (lock); +return lock; +} + +DEFINE_CLASS(kernfs_node_lock, struct mutex *, + mutex_unlock(_T), kernfs_node_lock(kn), struct kernfs_node *kn) +#endif /* __KERNFS_INTERNAL_H */
fs/kernfs/mount.c +1 -1
@@ -446,7 +446,7 @@ static void __init kernfs_mutex_init(void)int count;for (count =0 ; count <NR_KERNFS_LOCKS ; count++) -mutex_init (&kernfs_locks->open_file_mutex[count]); +mutex_init (&kernfs_locks->node_mutex[count]); }static void __init kernfs_lock_init (void )
include/linux/kernfs.h +8 -3
@@ -76,20 +76,25 @@ struct kernfs_iattrs;* kernfs_open_file. * * kernfs_open_files are chained at kernfs_open_node->files, which is -* protected by kernfs_global_locks.open_file_mutex[i]. +* protected by kernfs_global_locks.node_mutex[i]. * * To reduce possible contention in sysfs access, arising due to single -* locks, use an array of locks (e.g. open_file_mutex) and use kernfs_node +* locks, use an array of locks (e.g. node_mutex) and use kernfs_node * object address as hash keys to get the index of these locks. * * Hashed mutexes are safe to use here because operations using these don't * rely on global exclusion. * +* The hashed mutex array protects per-node data: the kernfs_open_node for +* open file management, and kernfs_node xattr operations (necessary because +* multiple superblocks with different namespaces can share the same +* kernfs_node, making per-inode locking insufficient). +* * In future we intend to replace other global locks with hashed ones as well. * kernfs_global_locks acts as a holder for all such hash tables. */ struct kernfs_global_locks { -struct mutex open_file_mutex[NR_KERNFS_LOCKS ]; +struct mutex node_mutex[NR_KERNFS_LOCKS ]; };enum kernfs_node_type {
Signature · 1 line
2.54.0
PATCH v4 2/4 tmpfs: simplify constructing "security.foo" xattr names
In reply to Miklos SzerediUse kasprintf() instead of doing it with kmalloc() + 2 x memcpy().
--- mm/shmem.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-)
mm/shmem.c +2 -9
@@ -4232,7 +4232,6 @@ static int shmem_initxattrs(struct inode *inode,struct shmem_sb_info *sbinfo =SHMEM_SB (inode->i_sb);const struct xattr *xattr;size_t ispace =0 ; -size_t len;CLASS (simple_xattrs, xattrs)();if (IS_ERR (xattrs))
@@ -4260,17 +4259,11 @@ static int shmem_initxattrs(struct inode *inode,if (IS_ERR (new_xattr))break ; - len =strlen (xattr->name) +1 ; - new_xattr->name =kmalloc (XATTR_SECURITY_PREFIX_LEN + len, -GFP_KERNEL_ACCOUNT ); + new_xattr->name =kasprintf (GFP_KERNEL_ACCOUNT , +XATTR_SECURITY_PREFIX "%s" , xattr->name);if (!new_xattr->name)break ; -memcpy (new_xattr->name,XATTR_SECURITY_PREFIX , -XATTR_SECURITY_PREFIX_LEN ); -memcpy (new_xattr->name +XATTR_SECURITY_PREFIX_LEN , - xattr->name, len); -if (simple_xattr_add (xattrs, new_xattr))break ;retain_and_null_ptr (new_xattr);
Signature · 1 line
2.54.0
PATCH v4 3/4 simple_xattr: change interface to pass struct simple_xattrs **
In reply to Miklos SzerediChange the simple_xattr API to accept pointer-to-pointer (struct simple_xattrs **) instead of pointer. This allows the functions to handle lazy allocation internally without requiring callers to use simple_xattrs_lazy_alloc().
The simple_xattr_set(), simple_xattr_set_limited() and simple_xattr_add() functions now handle allocation when xattrs is NULL. simple_xattrs_free() now also frees the xattrs structure itself and sets the pointer to NULL.
This simplifies callers and removes the need for most callers to explicitly manage xattrs allocation and lifetime.
In shmem_initxattrs(), the total required space for all initial xattrs (ispace) is pre-calculated and deducted from sbinfo->free_ispace.
Since this patch modifies the function to add new xattrs directly to the inode's &info->xattrs list rather than using a local temporary variable, a failure means that the partially populated info->xattrs list remains attached to the inode.
When the VFS caller handles the -ENOMEM error, it drops the newly created inode via iput(), shmem_free_inode() adds freed to sbinfo->free_ispace a second time, permanently inflating the tmpfs free space quota.
Fix by substracting already added xattrs from ispace.
--- fs/kernfs/dir.c | 12 +++-------- fs/kernfs/inode.c | 24 ++++------------------ fs/pidfs.c | 37 ++++++---------------------------- fs/xattr.c | 46 ++++++++++++++++++++++++++++++++++--------- include/linux/xattr.h | 21 ++++++-------------- mm/shmem.c | 34 ++++++++++---------------------- net/socket.c | 24 +++++----------------- 7 files changed, 71 insertions(+), 127 deletions(-)
fs/kernfs/dir.c +3 -9
@@ -605,11 +605,8 @@ void kernfs_put(struct kernfs_node *kn)if (kernfs_type (kn) ==KERNFS_LINK )kernfs_put (kn->symlink.target_kn); -if (kn->iattr && kn->iattr->xattrs) { -simple_xattrs_free (kn->iattr->xattrs,NULL ); -kfree (kn->iattr->xattrs); - kn->iattr->xattrs =NULL ; - } +if (kn->iattr) +simple_xattrs_free (&kn->iattr->xattrs,NULL );spin_lock (&root->kernfs_idr_lock);idr_remove (&root->ino_idr, (u32 )kernfs_ino (kn));
@@ -709,10 +706,7 @@ static struct kernfs_node *__kernfs_new_node(struct kernfs_root *root, err_out4:if (kn->iattr) { -if (kn->iattr->xattrs) { -simple_xattrs_free (kn->iattr->xattrs,NULL ); -kfree (kn->iattr->xattrs); - } +simple_xattrs_free (&kn->iattr->xattrs,NULL );kmem_cache_free (kernfs_iattrs_cache, kn->iattr); } err_out3:
fs/kernfs/inode.c +4 -20
@@ -144,8 +144,7 @@ ssize_t kernfs_iop_listxattr(struct dentry *dentry, char *buf, size_t size)if (!attrs)return -ENOMEM ; -return simple_xattr_list (d_inode (dentry),READ_ONCE (attrs->xattrs), - buf, size); +return simple_xattr_list (d_inode (dentry), &attrs->xattrs, buf, size); }static inline void set_default_inode_attr (struct inode *inode,umode_t mode)
@@ -297,23 +296,17 @@ int kernfs_xattr_get(struct kernfs_node *kn, const char *name, void *value, size_t size) {struct kernfs_iattrs *attrs =kernfs_iattrs_noalloc (kn); -struct simple_xattrs *xattrs;if (!attrs)return -ENODATA ; - xattrs =READ_ONCE (attrs->xattrs); -if (!xattrs) -return -ENODATA ; - -return simple_xattr_get (xattrs, name, value, size); +return simple_xattr_get (&attrs->xattrs, name, value, size); }int kernfs_xattr_set (struct kernfs_node *kn,const char *name,const void *value,size_t size,int flags) {struct simple_xattr *old_xattr; -struct simple_xattrs *xattrs;struct kernfs_iattrs *attrs; attrs =kernfs_iattrs (kn);
@@ -329,11 +322,7 @@ int kernfs_xattr_set(struct kernfs_node *kn, const char *name,*/ CLASS (kernfs_node_lock, lock)(kn); - xattrs =simple_xattrs_lazy_alloc (&attrs->xattrs, value, flags); -if (IS_ERR_OR_NULL (xattrs)) -return PTR_ERR (xattrs); - - old_xattr =simple_xattr_set (xattrs, name, value, size, flags); + old_xattr =simple_xattr_set (&attrs->xattrs, name, value, size, flags);if (IS_ERR (old_xattr))return PTR_ERR (old_xattr);
@@ -371,7 +360,6 @@ static int kernfs_vfs_user_xattr_set(const struct xattr_handler *handler, {const char *full_name =xattr_full_name (handler, suffix);struct kernfs_node *kn = inode->i_private; -struct simple_xattrs *xattrs;struct kernfs_iattrs *attrs;if (!(kernfs_root (kn)->flags &KERNFS_ROOT_SUPPORT_USER_XATTR ))
@@ -384,11 +372,7 @@ static int kernfs_vfs_user_xattr_set(const struct xattr_handler *handler,/* See comment in kernfs_xattr_set() about locking. */ CLASS (kernfs_node_lock, lock)(kn); - xattrs =simple_xattrs_lazy_alloc (&attrs->xattrs, value, flags); -if (IS_ERR_OR_NULL (xattrs)) -return PTR_ERR (xattrs); - -return simple_xattr_set_limited (xattrs, &attrs->xattr_limits, +return simple_xattr_set_limited (&attrs->xattrs, &attrs->xattr_limits, full_name, value, size, flags); }
fs/pidfs.c +6 -31
@@ -196,12 +196,7 @@ static void pidfs_free_attr_work(struct work_struct *work) head =llist_del_all (&pidfs_free_list);llist_for_each_entry_safe (attr, next, head, pidfs_llist) { -struct simple_xattrs *xattrs = attr->xattrs; - -if (xattrs) { -simple_xattrs_free (xattrs,NULL ); -kfree (xattrs); - } +simple_xattrs_free (&attr->xattrs,NULL );kfree (attr); } }
@@ -815,14 +810,8 @@ static ssize_t pidfs_listxattr(struct dentry *dentry, char *buf, size_t size) {struct inode *inode =d_inode (dentry);struct pid *pid = inode->i_private; -struct pidfs_attr *attr = pid->attr; -struct simple_xattrs *xattrs; - - xattrs =READ_ONCE (attr->xattrs); -if (!xattrs) -return 0 ; -return simple_xattr_list (inode, xattrs, buf, size); +return simple_xattr_list (inode, &pid->attr->xattrs, buf, size); }static const struct inode_operations pidfs_inode_operations = {
@@ -1057,16 +1046,9 @@ static int pidfs_xattr_get(const struct xattr_handler *handler, const char *suffix, void *value, size_t size) { struct pid *pid = inode->i_private; -struct pidfs_attr *attr = pid->attr; -const char *name; -struct simple_xattrs *xattrs; - - xattrs =READ_ONCE (attr->xattrs); -if (!xattrs) -return -ENODATA ; +const char *name =xattr_full_name (handler, suffix); - name =xattr_full_name (handler, suffix); -return simple_xattr_get (xattrs, name, value, size); +return simple_xattr_get (&pid->attr->xattrs, name, value, size); } static int pidfs_xattr_set(const struct xattr_handler *handler,
@@ -1075,20 +1057,13 @@ static int pidfs_xattr_set(const struct xattr_handler *handler, const void *value, size_t size, int flags) { struct pid *pid = inode->i_private; -struct pidfs_attr *attr = pid->attr; -const char *name; -struct simple_xattrs *xattrs; +const char *name =xattr_full_name (handler, suffix);struct simple_xattr *old_xattr;/* Ensure we're the only one to set @attr->xattrs. */ WARN_ON_ONCE (!inode_is_locked (inode)); - xattrs =simple_xattrs_lazy_alloc (&attr->xattrs, value, flags); -if (IS_ERR_OR_NULL (xattrs)) -return PTR_ERR (xattrs); - - name =xattr_full_name (handler, suffix); - old_xattr =simple_xattr_set (xattrs, name, value, size, flags); + old_xattr =simple_xattr_set (&pid->attr->xattrs, name, value, size, flags);if (IS_ERR (old_xattr))return PTR_ERR (old_xattr);
fs/xattr.c +37 -9
127 lines not on this page · raw message
include/linux/xattr.h +6 -15
@@ -133,26 +133,23 @@ static inline void simple_xattr_limits_init(struct simple_xattr_limits *limits) }int simple_xattrs_init (struct simple_xattrs *xattrs); -struct simple_xattrs *simple_xattrs_alloc (void ); -struct simple_xattrs *simple_xattrs_lazy_alloc (struct simple_xattrs **xattrsp, -const void *value,int flags); -void simple_xattrs_free (struct simple_xattrs *xattrs,size_t *freed_space); +void simple_xattrs_free (struct simple_xattrs **xattrs,size_t *freed_space);size_t simple_xattr_space (const char *name,size_t size);struct simple_xattr *simple_xattr_alloc (const void *value,size_t size);void simple_xattr_free (struct simple_xattr *xattr);void simple_xattr_free_rcu (struct simple_xattr *xattr); -int simple_xattr_get (struct simple_xattrs *xattrs,const char *name, +int simple_xattr_get (struct simple_xattrs **xattrs,const char *name,void *buffer,size_t size); -struct simple_xattr *simple_xattr_set (struct simple_xattrs *xattrs, +struct simple_xattr *simple_xattr_set (struct simple_xattrs **xattrs,const char *name,const void *value,size_t size,int flags); -int simple_xattr_set_limited (struct simple_xattrs *xattrs, +int simple_xattr_set_limited (struct simple_xattrs **xattrs,struct simple_xattr_limits *limits,const char *name,const void *value,size_t size,int flags); -ssize_t simple_xattr_list (struct inode *inode,struct simple_xattrs *xattrs, +ssize_t simple_xattr_list (struct inode *inode,struct simple_xattrs **xattrs,char *buffer,size_t size); -int simple_xattr_add (struct simple_xattrs *xattrs, +int simple_xattr_add (struct simple_xattrs **xattrs,struct simple_xattr *new_xattr);int xattr_list_one (char **buffer,ssize_t *remaining_size,const char *name);
@@ -162,10 +159,4 @@ DEFINE_CLASS(simple_xattr, simple_xattr_alloc(value, size), const void *value, size_t size) -DEFINE_CLASS (simple_xattrs, - struct simple_xattrs *, - if (!IS_ERR_OR_NULL(_T)) { simple_xattrs_free(_T, NULL); kfree(_T); }, -simple_xattrs_alloc (), - void) -#endif /* _LINUX_XATTR_H */
mm/shmem.c +10 -24
@@ -1425,10 +1425,8 @@ static void shmem_evict_inode(struct inode *inode) } } -if (info->xattrs) { -simple_xattrs_free (info->xattrs, sbinfo->max_inodes ? &freed :NULL ); -kfree (info->xattrs); - } +simple_xattrs_free (&info->xattrs, sbinfo->max_inodes ? &freed :NULL ); +shmem_free_inode (inode->i_sb, freed);WARN_ON (inode->i_blocks);clear_inode (inode);
@@ -4233,10 +4231,6 @@ static int shmem_initxattrs(struct inode *inode,const struct xattr *xattr;size_t ispace =0 ; -CLASS (simple_xattrs, xattrs)(); -if (IS_ERR (xattrs)) -return PTR_ERR (xattrs); -if (sbinfo->max_inodes) {for (xattr = xattr_array; xattr->name !=NULL ; xattr++) { ispace += simple_xattr_space(xattr->name,
@@ -4264,8 +4258,11 @@ static int shmem_initxattrs(struct inode *inode,if (!new_xattr->name)break ; -if (simple_xattr_add (xattrs, new_xattr)) +if (simple_xattr_add (&info->xattrs, new_xattr))break ; + +if (sbinfo->max_inodes) + ispace -=simple_xattr_space (new_xattr->name, new_xattr->size);retain_and_null_ptr (new_xattr); }
@@ -4277,8 +4274,8 @@ static int shmem_initxattrs(struct inode *inode, }return -ENOMEM ; } +WARN_ON (ispace); -smp_store_release (&info->xattrs,no_free_ptr (xattrs));return 0 ; }
@@ -4287,14 +4284,9 @@ static int shmem_xattr_handler_get(const struct xattr_handler *handler, const char *name, void *buffer, size_t size) { struct shmem_inode_info *info = SHMEM_I(inode); -struct simple_xattrs *xattrs; - - xattrs =READ_ONCE (info->xattrs); -if (!xattrs) -return -ENODATA ; name =xattr_full_name (handler, name); -return simple_xattr_get (xattrs, name, buffer, size); +return simple_xattr_get (&info->xattrs, name, buffer, size); } static int shmem_xattr_handler_set(const struct xattr_handler *handler,
@@ -4305,16 +4297,11 @@ static int shmem_xattr_handler_set(const struct xattr_handler *handler, {struct shmem_inode_info *info =SHMEM_I (inode);struct shmem_sb_info *sbinfo =SHMEM_SB (inode->i_sb); -struct simple_xattrs *xattrs;struct simple_xattr *old_xattr;size_t ispace =0 ; name =xattr_full_name (handler, name); - xattrs =simple_xattrs_lazy_alloc (&info->xattrs, value, flags); -if (IS_ERR_OR_NULL (xattrs)) -return PTR_ERR (xattrs); -if (value && sbinfo->max_inodes) { ispace =simple_xattr_space (name, size);raw_spin_lock (&sbinfo->stat_lock);
@@ -4327,7 +4314,7 @@ static int shmem_xattr_handler_set(const struct xattr_handler *handler,return -ENOSPC ; } - old_xattr =simple_xattr_set (xattrs, name, value, size, flags); + old_xattr =simple_xattr_set (&info->xattrs, name, value, size, flags);if (!IS_ERR (old_xattr)) { ispace =0 ;if (old_xattr && sbinfo->max_inodes)
@@ -4375,8 +4362,7 @@ static ssize_t shmem_listxattr(struct dentry *dentry, char *buffer, size_t size) {struct shmem_inode_info *info =SHMEM_I (d_inode (dentry)); -return simple_xattr_list (d_inode (dentry),READ_ONCE (info->xattrs), - buffer, size); +return simple_xattr_list (d_inode (dentry), &info->xattrs, buffer, size); }#endif /* CONFIG_TMPFS_XATTR */
net/socket.c +5 -19
@@ -347,12 +347,8 @@ static struct inode *sock_alloc_inode(struct super_block *sb)static void sock_evict_inode (struct inode *inode) {struct sockfs_inode *si =SOCKFS_I (inode); -struct simple_xattrs *xattrs = si->xattrs; -if (xattrs) { -simple_xattrs_free (xattrs,NULL ); -kfree (xattrs); - } +simple_xattrs_free (&si->xattrs,NULL );clear_inode (inode); }
@@ -443,13 +439,9 @@ static int sockfs_user_xattr_get(const struct xattr_handler *handler, const char *suffix, void *value, size_t size) { const char *name = xattr_full_name(handler, suffix); -struct simple_xattrs *xattrs; - - xattrs =READ_ONCE (SOCKFS_I (inode)->xattrs); -if (!xattrs) -return -ENODATA ; +struct sockfs_inode *si =SOCKFS_I (inode); -return simple_xattr_get (xattrs, name, value, size); +return simple_xattr_get (&si->xattrs, name, value, size); } static int sockfs_user_xattr_set(const struct xattr_handler *handler,
@@ -460,13 +452,8 @@ static int sockfs_user_xattr_set(const struct xattr_handler *handler, {const char *name =xattr_full_name (handler, suffix);struct sockfs_inode *si =SOCKFS_I (inode); -struct simple_xattrs *xattrs; - - xattrs =simple_xattrs_lazy_alloc (&si->xattrs, value, flags); -if (IS_ERR_OR_NULL (xattrs)) -return PTR_ERR (xattrs); -return simple_xattr_set_limited (xattrs, &si->xattr_limits, +return simple_xattr_set_limited (&si->xattrs, &si->xattr_limits, name, value, size, flags); }
@@ -635,8 +622,7 @@ static ssize_t sockfs_listxattr(struct dentry *dentry, char *buffer,struct sockfs_inode *si =SOCKFS_I (d_inode (dentry));ssize_t len, used; - len =simple_xattr_list (d_inode (dentry),READ_ONCE (si->xattrs), - buffer, size); + len =simple_xattr_list (d_inode (dentry), &si->xattrs, buffer, size);if (len <0 )return len;
Signature · 1 line
2.54.0
PATCH v4 4/4 simpe_xattr: use per-sb cache
In reply to Miklos SzerediMove the hash table to the super block to remove excessive overhead in case of small number of xattrs per inode.
Add linked list to the inode, used for listxattr and eviction. Listxattr uses rcu protection to iterate the list of xattrs.
Before being made per-sb, lazy allocation was protected by inode lock. Now inode lock no longer provides sufficient exclusion, so use cmpxchg() to ensure atomicity.
Though I haven't found a description of this pattern, after some research it seems that cmpxchg_release() and READ_ONCE() should provide the necessary memory barriers.
Use simple_xattr_free_rcu() in simple_xattrs_free(). This is needed because the hash table is now shared between inodes and lookup on a different inode might be running the compare function on the just freed element within the RCU grace period.
Following stats are based on slabinfo diff, after creating 100k empty files, then adding a "user.test=foo" xattr to each:
v7.0 (no rhashtable):
File creation: 993.40 bytes/file
Xattr addition: 79.99 bytes/file
v7.1-rc2 (per-inode rhashtable):
File creation: 939.73 bytes/file
Xattr addition: 1296.08 bytes/file
v7.1-rc2 + this patch (per-sb rhashtable)
File creation: 946.84 bytes/file
Xattr addition: 111.86 bytes/file
The overhead of a single xattr is reduced to nearly v7.0 levels. The per xattr overhead is slightly larger due to the addition of three pointers to struct simple_xattr.
--- fs/kernfs/dir.c | 5 +- fs/kernfs/inode.c | 10 +- fs/kernfs/kernfs-internal.h | 4 +- fs/pidfs.c | 14 +- fs/xattr.c | 283 +++++++++++++++++------------------- include/linux/shmem_fs.h | 3 +- include/linux/xattr.h | 26 ++-- mm/shmem.c | 13 +- net/socket.c | 12 +- 9 files changed, 192 insertions(+), 178 deletions(-)
fs/kernfs/dir.c +3 -2
@@ -606,7 +606,7 @@ void kernfs_put(struct kernfs_node *kn)kernfs_put (kn->symlink.target_kn);if (kn->iattr) -simple_xattrs_free (&kn->iattr->xattrs,NULL ); +simple_xattrs_free (&root->xa_cache, &kn->iattr->xattrs,NULL );spin_lock (&root->kernfs_idr_lock);idr_remove (&root->ino_idr, (u32 )kernfs_ino (kn));
@@ -621,6 +621,7 @@ void kernfs_put(struct kernfs_node *kn) }else {/* just released the root kn, free @root too */ idr_destroy (&root->ino_idr); +simple_xattr_cache_cleanup (&root->xa_cache);kfree_rcu (root, rcu); } }
@@ -706,7 +707,7 @@ static struct kernfs_node *__kernfs_new_node(struct kernfs_root *root, err_out4:if (kn->iattr) { -simple_xattrs_free (&kn->iattr->xattrs,NULL ); +simple_xattrs_free (&root->xa_cache, &kn->iattr->xattrs,NULL );kmem_cache_free (kernfs_iattrs_cache, kn->iattr); } err_out3:
fs/kernfs/inode.c +7 -3
@@ -37,6 +37,7 @@ static struct kernfs_iattrs *__kernfs_iattrs(struct kernfs_node *kn, bool alloc)if (!ret)return NULL ; +INIT_LIST_HEAD_RCU (&ret->xattrs);/* assign default attributes */ ret->ia_uid =GLOBAL_ROOT_UID ; ret->ia_gid =GLOBAL_ROOT_GID ;
@@ -296,11 +297,12 @@ int kernfs_xattr_get(struct kernfs_node *kn, const char *name, void *value, size_t size) {struct kernfs_iattrs *attrs =kernfs_iattrs_noalloc (kn); +struct simple_xattr_cache *cache = &kernfs_root (kn)->xa_cache;if (!attrs)return -ENODATA ; -return simple_xattr_get (&attrs->xattrs, name, value, size); +return simple_xattr_get (cache, &attrs->xattrs, name, value, size); } int kernfs_xattr_set(struct kernfs_node *kn, const char *name,
@@ -308,6 +310,7 @@ int kernfs_xattr_set(struct kernfs_node *kn, const char *name, {struct simple_xattr *old_xattr;struct kernfs_iattrs *attrs; +struct simple_xattr_cache *cache = &kernfs_root (kn)->xa_cache; attrs =kernfs_iattrs (kn);if (!attrs)
@@ -322,7 +325,7 @@ int kernfs_xattr_set(struct kernfs_node *kn, const char *name,*/ CLASS (kernfs_node_lock, lock)(kn); - old_xattr =simple_xattr_set (&attrs->xattrs, name, value, size, flags); + old_xattr =simple_xattr_set (cache, &attrs->xattrs, name, value, size, flags);if (IS_ERR (old_xattr))return PTR_ERR (old_xattr);
@@ -372,7 +375,8 @@ static int kernfs_vfs_user_xattr_set(const struct xattr_handler *handler,/* See comment in kernfs_xattr_set() about locking. */ CLASS (kernfs_node_lock, lock)(kn); -return simple_xattr_set_limited (&attrs->xattrs, &attrs->xattr_limits, +return simple_xattr_set_limited (&kernfs_root (kn)->xa_cache, + &attrs->xattrs, &attrs->xattr_limits, full_name, value, size, flags); }
fs/kernfs/kernfs-internal.h +3 -1
@@ -26,7 +26,7 @@ struct kernfs_iattrs {struct timespec64 ia_mtime;struct timespec64 ia_ctime; -struct simple_xattrs *xattrs; +struct list_head xattrs;struct simple_xattr_limits xattr_limits; };
@@ -54,6 +54,8 @@ struct kernfs_root {rwlock_t kernfs_rename_lock;struct rcu_head rcu; + +struct simple_xattr_cache xa_cache; };/* +1 to avoid triggering overflow warning when negating it */
fs/pidfs.c +9 -5
@@ -37,6 +37,8 @@ static struct kmem_cache *pidfs_attr_cachep __ro_after_init;static struct path pidfs_root_path = {}; +static struct simple_xattr_cache pidfs_xa_cache; +void pidfs_get_root (struct path *path) { *path = pidfs_root_path;
@@ -96,7 +98,7 @@ static const struct rhashtable_params pidfs_ino_ht_params = {* use file handles. */ struct pidfs_attr { -struct simple_xattrs *xattrs; +struct list_head xattrs;union {struct pidfs_anon_attr ;struct llist_node pidfs_llist;
@@ -196,7 +198,7 @@ static void pidfs_free_attr_work(struct work_struct *work) head =llist_del_all (&pidfs_free_list);llist_for_each_entry_safe (attr, next, head, pidfs_llist) { -simple_xattrs_free (&attr->xattrs,NULL ); +simple_xattrs_free (&pidfs_xa_cache, &attr->xattrs,NULL );kfree (attr); } }
@@ -224,7 +226,7 @@ void pidfs_free_pid(struct pid *pid)if (IS_ERR (attr))return ; -if (likely (!attr->xattrs)) +if (likely (list_empty (&attr->xattrs)))kfree (attr);else if (llist_add (&attr->pidfs_llist, &pidfs_free_list))schedule_work (&pidfs_free_work);
@@ -1007,6 +1009,8 @@ int pidfs_register_pid(struct pid *pid)if (!new_attr)return -ENOMEM ; +INIT_LIST_HEAD_RCU (&new_attr->xattrs); +/* Synchronize with pidfs_exit(). */ guard (spinlock_irq)(&pid->wait_pidfd.lock);
@@ -1048,7 +1052,7 @@ static int pidfs_xattr_get(const struct xattr_handler *handler,struct pid *pid = inode->i_private;const char *name =xattr_full_name (handler, suffix); -return simple_xattr_get (&pid->attr->xattrs, name, value, size); +return simple_xattr_get (&pidfs_xa_cache, &pid->attr->xattrs, name, value, size); } static int pidfs_xattr_set(const struct xattr_handler *handler,
@@ -1063,7 +1067,7 @@ static int pidfs_xattr_set(const struct xattr_handler *handler,/* Ensure we're the only one to set @attr->xattrs. */ WARN_ON_ONCE (!inode_is_locked (inode)); - old_xattr =simple_xattr_set (&pid->attr->xattrs, name, value, size, flags); + old_xattr =simple_xattr_set (&pidfs_xa_cache, &pid->attr->xattrs, name, value, size, flags);if (IS_ERR (old_xattr))return PTR_ERR (old_xattr);
fs/xattr.c +136 -147
455 lines not on this page · raw message
include/linux/shmem_fs.h +2 -1
@@ -48,7 +48,7 @@ struct shmem_inode_info { };struct timespec64 i_crtime;/* file creation time */ struct shared_policy policy;/* NUMA memory alloc policy */ -struct simple_xattrs *xattrs;/* list of xattrs */ +struct list_head xattrs;/* list of xattrs */ pgoff_t fallocend;/* highest fallocate endindex */ unsigned int fsflags;/* for FS_IOC_[SG]ETFLAGS */ atomic_t stop_eviction;/* hold when working on inode */
@@ -89,6 +89,7 @@ struct shmem_sb_info {struct list_head shrinklist;/* List of shinkable inodes */ unsigned long shrinklist_len;/* Length of shrinklist */ struct shmem_quota_limits qlimits;/* Default quota limits */ +struct simple_xattr_cache xa_cache; };static inline struct shmem_inode_info *SHMEM_I (struct inode *inode)
include/linux/xattr.h +16 -10
@@ -106,12 +106,14 @@ static inline const char *xattr_prefix(const struct xattr_handler *handler)return handler->prefix ?: handler->name; } -struct simple_xattrs { -struct rhashtable ht; +struct simple_xattr_cache { +struct rhashtable *ht; };struct simple_xattr {struct rhash_head hash_node; +struct list_head *parent; +struct list_head node;struct rcu_head rcu;char *name;size_t size;
@@ -132,27 +134,31 @@ static inline void simple_xattr_limits_init(struct simple_xattr_limits *limits)atomic_set (&limits->xattr_size,0 ); } -int simple_xattrs_init (struct simple_xattrs *xattrs); -void simple_xattrs_free (struct simple_xattrs **xattrs,size_t *freed_space); +void simple_xattrs_free (struct simple_xattr_cache *cache,struct list_head *xattrs, +size_t *freed_space);size_t simple_xattr_space (const char *name,size_t size);struct simple_xattr *simple_xattr_alloc (const void *value,size_t size);void simple_xattr_free (struct simple_xattr *xattr);void simple_xattr_free_rcu (struct simple_xattr *xattr); -int simple_xattr_get (struct simple_xattrs **xattrs,const char *name, -void *buffer,size_t size); -struct simple_xattr *simple_xattr_set (struct simple_xattrs **xattrs, +int simple_xattr_get (struct simple_xattr_cache *cache,struct list_head *xattrs, +const char *name,void *buffer,size_t size); +struct simple_xattr *simple_xattr_set (struct simple_xattr_cache *cache, +struct list_head *xattrs,const char *name,const void *value,size_t size,int flags); -int simple_xattr_set_limited (struct simple_xattrs **xattrs, +int simple_xattr_set_limited (struct simple_xattr_cache *cache, +struct list_head *xattrs,struct simple_xattr_limits *limits,const char *name,const void *value,size_t size,int flags); -ssize_t simple_xattr_list (struct inode *inode,struct simple_xattrs **xattrs, +ssize_t simple_xattr_list (struct inode *inode,struct list_head *xattrs,char *buffer,size_t size); -int simple_xattr_add (struct simple_xattrs **xattrs, +int simple_xattr_add (struct simple_xattr_cache *cache,struct list_head *xattrs,struct simple_xattr *new_xattr);int xattr_list_one (char **buffer,ssize_t *remaining_size,const char *name); +void simple_xattr_cache_cleanup (struct simple_xattr_cache *cache); + DEFINE_CLASS(simple_xattr, struct simple_xattr *, if (!IS_ERR_OR_NULL(_T)) simple_xattr_free(_T),
mm/shmem.c +9 -4
@@ -1425,7 +1425,7 @@ static void shmem_evict_inode(struct inode *inode) } } -simple_xattrs_free (&info->xattrs, sbinfo->max_inodes ? &freed :NULL ); +simple_xattrs_free (&sbinfo->xa_cache, &info->xattrs, sbinfo->max_inodes ? &freed :NULL );shmem_free_inode (inode->i_sb, freed);WARN_ON (inode->i_blocks);
@@ -3084,6 +3084,7 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap, inode->i_generation =get_random_u32 (); info =SHMEM_I (inode);memset (info,0 , (char *)inode - (char *)info); +INIT_LIST_HEAD_RCU (&info->xattrs);spin_lock_init (&info->lock);atomic_set (&info->stop_eviction,0 ); info->seals =F_SEAL_SEAL ;
@@ -4258,7 +4259,7 @@ static int shmem_initxattrs(struct inode *inode,if (!new_xattr->name)break ; -if (simple_xattr_add (&info->xattrs, new_xattr)) +if (simple_xattr_add (&sbinfo->xa_cache, &info->xattrs, new_xattr))break ;if (sbinfo->max_inodes)
@@ -4283,10 +4284,11 @@ static int shmem_xattr_handler_get(const struct xattr_handler *handler, struct dentry *unused, struct inode *inode, const char *name, void *buffer, size_t size) { + struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);struct shmem_inode_info *info =SHMEM_I (inode); name =xattr_full_name (handler, name); -return simple_xattr_get (&info->xattrs, name, buffer, size); +return simple_xattr_get (&sbinfo->xa_cache, &info->xattrs, name, buffer, size); } static int shmem_xattr_handler_set(const struct xattr_handler *handler,
@@ -4314,7 +4316,7 @@ static int shmem_xattr_handler_set(const struct xattr_handler *handler,return -ENOSPC ; } - old_xattr =simple_xattr_set (&info->xattrs, name, value, size, flags); + old_xattr =simple_xattr_set (&sbinfo->xa_cache, &info->xattrs, name, value, size, flags);if (!IS_ERR (old_xattr)) { ispace =0 ;if (old_xattr && sbinfo->max_inodes)
@@ -4963,6 +4965,9 @@ static void shmem_put_super(struct super_block *sb)free_percpu (sbinfo->ino_batch);percpu_counter_destroy (&sbinfo->used_blocks);mpol_put (sbinfo->mpol); +#ifdef CONFIG_TMPFS_XATTR +simple_xattr_cache_cleanup (&sbinfo->xa_cache); +#endif kfree (sbinfo); sb->s_fs_info =NULL ; }
net/socket.c +7 -5
@@ -310,8 +310,10 @@ static int move_addr_to_user(struct sockaddr_storage *kaddr, int klen,static struct kmem_cache *sock_inode_cachep__ro_after_init ; +static struct simple_xattr_cache sockfs_xa_cache; +struct sockfs_inode { -struct simple_xattrs *xattrs; +struct list_head xattrs;struct simple_xattr_limits xattr_limits;struct socket_alloc ; };
@@ -328,7 +330,7 @@ static struct inode *sock_alloc_inode(struct super_block *sb) si =alloc_inode_sb (sb, sock_inode_cachep,GFP_KERNEL );if (!si)return NULL ; - si->xattrs =NULL ; +INIT_LIST_HEAD_RCU (&si->xattrs);simple_xattr_limits_init (&si->xattr_limits);init_waitqueue_head (&si->socket.wq.wait);
@@ -348,7 +350,7 @@ static void sock_evict_inode(struct inode *inode) {struct sockfs_inode *si =SOCKFS_I (inode); -simple_xattrs_free (&si->xattrs,NULL ); +simple_xattrs_free (&sockfs_xa_cache, &si->xattrs,NULL );clear_inode (inode); }
@@ -441,7 +443,7 @@ static int sockfs_user_xattr_get(const struct xattr_handler *handler,const char *name =xattr_full_name (handler, suffix);struct sockfs_inode *si =SOCKFS_I (inode); -return simple_xattr_get (&si->xattrs, name, value, size); +return simple_xattr_get (&sockfs_xa_cache, &si->xattrs, name, value, size); } static int sockfs_user_xattr_set(const struct xattr_handler *handler,
@@ -453,7 +455,7 @@ static int sockfs_user_xattr_set(const struct xattr_handler *handler,const char *name =xattr_full_name (handler, suffix);struct sockfs_inode *si =SOCKFS_I (inode); -return simple_xattr_set_limited (&si->xattrs, &si->xattr_limits, +return simple_xattr_set_limited (&sockfs_xa_cache, &si->xattrs, &si->xattr_limits, name, value, size, flags); }
Signature · 1 line
2.54.0
Quoted 8 lines · Miklos Szeredi
v4:
- yet another accounting fix forshmem_initxattrsv3:
- need perkernfs_nodelocking forsimple_xattr_set
- another accounting fix forshmem_initxattrs[...]
Applied to the vfs-7.2.xattr branch of the vfs/vfs.git tree. Patches in the vfs-7.2.xattr branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-7.2.xattr
[1/4] kernfs: fix xattr race condition with multiple superblocks
https://git.kernel.org/vfs/vfs/c/6a07814ff643 [2/4] tmpfs: simplify constructing "security.foo" xattr names https://git.kernel.org/vfs/vfs/c/832f4de4c8ba [3/4] simple_xattr: change interface to pass struct simple_xattrs ** https://git.kernel.org/vfs/vfs/c/076e5cef28e2 [4/4] simpe_xattr: use per-sb cache https://git.kernel.org/vfs/vfs/c/1e7cd8a53b72