overlayfs: Optimize override/revert creds
Subsystems (4)
v2 cover overlayfs: Optimize override/revert creds
Hi,
It was noticed that some workloads suffer from contention on increasing/decrementing the ->usage counter in their credentials, those refcount operations are associated with overriding/reverting the current task credentials. (the linked thread adds more context)
In some specialized cases, overlayfs is one of them, the credentials in question have a longer lifetime than the override/revert "critical section". In the overlayfs case, the credentials are created when the fs is mounted and destroyed when it's unmounted. In this case of long lived credentials, the usage counter doesn't need to be incremented/decremented.
Add a lighter version of credentials override/revert to be used in these specialized cases. To make sure that the override/revert calls are paired, add a cleanup guard macro. This was suggested here:
https://lore.kernel.org/all/20231219-marken-pochen-26d888fb9bb9@brauner/
With a small number of tweaks:
- Used inline functions instead of macros;
- A small change to store the credentials into the passed argument, the guard is now defined as (note the added '_T ='):
DEFINE_GUARD(cred, const struct cred *, _T = override_creds_light(_T),
revert_creds_light(_T));
- Allow "const" arguments to be used with these kind of guards;
Some comments:
- If patch 1/4 is not a good idea (adding the cast), the alternative I can see is using some kind of container for the credentials;
- The only user for the backing file ops is overlayfs, so these changes make sense, but may not make sense in the most general case;
For the numbers, some from 'perf c2c', before this series:
(edited to fit)
#
# ----- HITM ----- Shared
# Num RmtHitm LclHitm Symbol Object Source:Line Node
# ..... ....... ....... .......................... ................ .................. ....
#
-------------------------
0 412 1028
-------------------------
41.50% 42.22% [k] revert_creds [kernel.vmlinux] atomic64_64.h:39 0 1
15.05% 10.60% [k] override_creds [kernel.vmlinux] atomic64_64.h:25 0 1
0.73% 0.58% [k] init_file [kernel.vmlinux] atomic64_64.h:25 0 1
0.24% 0.10% [k] revert_creds [kernel.vmlinux] cred.h:266 0 1
32.28% 37.16% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1
9.47% 8.75% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1
0.49% 0.58% [k] inode_owner_or_capable [kernel.vmlinux] mnt_idmapping.h:81 0 1
0.24% 0.00% [k] generic_permission [kernel.vmlinux] namei.c:354 0-------------------------
1 50 103
-------------------------
100.00% 100.00% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1-------------------------
2 50 98
-------------------------
96.00% 96.94% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1
2.00% 1.02% [k] update_load_avg [kernel.vmlinux] atomic64_64.h:25 0 1
0.00% 2.04% [k] update_load_avg [kernel.vmlinux] fair.c:4118 0
2.00% 0.00% [k] update_cfs_group [kernel.vmlinux] fair.c:3932 0 1after this series:
#
# ----- HITM ----- Shared
# Num RmtHitm LclHitm Symbol Object Source:Line Node
# ..... ....... ....... .................... ................ ................ ....
#
-------------------------
0 54 88
-------------------------
100.00% 100.00% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1-------------------------
1 48 83
-------------------------
97.92% 97.59% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1
2.08% 1.20% [k] update_load_avg [kernel.vmlinux] atomic64_64.h:25 0 1
0.00% 1.20% [k] update_load_avg [kernel.vmlinux] fair.c:4118 0 1-------------------------
2 28 44
-------------------------
85.71% 79.55% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1
14.29% 20.45% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1The contention is practically gone.
Vinicius Costa Gomes (4):
cleanup: Fix discarded const warning when defining guards
cred: Add a light version of override/revert_creds()
overlayfs: Optimize credentials usage
fs: Optimize credentials reference count for backing file ops
fs/backing-file.c | 124 +++++++++++++++++++--------------------- fs/overlayfs/copy_up.c | 4 +- fs/overlayfs/dir.c | 22 +++---- fs/overlayfs/file.c | 70 ++++++++++++----------- fs/overlayfs/inode.c | 60 ++++++++++--------- fs/overlayfs/namei.c | 21 ++++--- fs/overlayfs/readdir.c | 18 +++--- fs/overlayfs/util.c | 23 ++++---- fs/overlayfs/xattrs.c | 34 +++++------ include/linux/cleanup.h | 2 +- include/linux/cred.h | 21 +++++++ kernel/cred.c | 6 +- 12 files changed, 215 insertions(+), 190 deletions(-)
Signature · 1 line
2.43.0
RFC v2 1/4 cleanup: Fix discarded const warning when defining guards
When defining guards for const types the void* return implicitly discards the const modifier. Be explicit about it.
Compiler warning (gcc 13.2.1):
./include/linux/cleanup.h:154:18: warning: return discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
154 | { return *_T; }
| ^~~./include/linux/cred.h:193:1: note: in expansion of macro ‘DEFINE_GUARD’
193 | DEFINE_GUARD(cred, const struct cred *, _T = override_creds_light(_T),
| ^~~~~~~~~~~~
--- include/linux/cleanup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/cleanup.h +1 -1
@@ -151,7 +151,7 @@ static inline class_##_name##_t class_##_name##ext##_constructor(_init_args) \#define DEFINE_GUARD (_name, _type, _lock, _unlock) \ DEFINE_CLASS(_name, _type, if (_T) { _unlock; }, ({ _lock; _T; }), _type _T); \ static inline void * class_##_name##_lock_ptr(class_##_name##_t *_T) \ - { return *_T; } + { return (void *)*_T; }#define DEFINE_GUARD_COND (_name, _ext, _condlock) \ EXTEND_CLASS(_name, _ext, \
Signature · 1 line
2.43.0
Amir Goldstein
maintainerRe: [RFC v2 1/4] cleanup: Fix discarded const warning when defining guards
Quoted 4 lines · Vinicius Costa Gomes
When defining guards for const types the void* return implicitly discards the const modifier. Be explicit about it.
Quoted 9 lines
Compiler warning (gcc 13.2.1):
./include/linux/cleanup.h:154:18: warning: return discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
154 | { return *_T; }
| ^~~./include/linux/cred.h:193:1: note: in expansion of macro ‘DEFINE_GUARD’
193 | DEFINE_GUARD(cred, const struct cred *, _T =override_creds_light(_T),| ^~~~~~~~~~~~
I did not look closely, but can't you use DEFINE_LOCK_GUARD_1()?
Thanks,
Amir.
RFC v2 2/4 cred: Add a light version of override/revert_creds()
In reply to Vinicius Costa GomesAdd a light version of override/revert_creds(), this should only be used when the credentials in question will outlive the critical section and the critical section doesn't change the ->usage of the credentials.
To make their usage less error prone, introduce cleanup guards asto be used like this:
guard(cred)(credentials_to_override_and_restore);
or this:
scoped_guard(cred, credentials_to_override_and_restore) {
/* with credentials overridden */
}--- include/linux/cred.h | 21 +++++++++++++++++++++ kernel/cred.c | 6 +++--- 2 files changed, 24 insertions(+), 3 deletions(-)
include/linux/cred.h +21 -0
@@ -172,6 +172,27 @@ static inline bool cap_ambient_invariant_ok(const struct cred *cred) cred->cap_inheritable)); } +/* +* Override creds without bumping reference count. Caller must ensure +* reference remains valid or has taken reference. Almost always not the +* interface you want. Use override_creds()/revert_creds() instead. +*/ +static inline const struct cred *override_creds_light (const struct cred *override_cred) +{ +const struct cred *old = current->cred; + +rcu_assign_pointer (current->cred, override_cred); +return old; +} + +static inline void revert_creds_light (const struct cred *revert_cred) +{ +rcu_assign_pointer (current->cred, revert_cred); +} + +DEFINE_GUARD(cred, const struct cred *, _T = override_creds_light(_T), +revert_creds_light (_T)); +/** * get_new_cred_many - Get references on a new set of credentials * @cred: The new credentials to reference
kernel/cred.c +3 -3
@@ -485,7 +485,7 @@ EXPORT_SYMBOL(abort_creds);*/ const struct cred *override_creds (const struct cred *new) { -const struct cred *old = current->cred; +const struct cred *old;kdebug ("override_creds(%p{%ld})" , new,atomic_long_read (&new->usage));
@@ -499,7 +499,7 @@ const struct cred *override_creds(const struct cred *new)* visible to other threads under RCU. */ get_new_cred ((struct cred *)new); -rcu_assign_pointer (current->cred, new); + old =override_creds_light (new);kdebug ("override_creds() = %p{%ld}" , old,atomic_long_read (&old->usage));
@@ -521,7 +521,7 @@ void revert_creds(const struct cred *old)kdebug ("revert_creds(%p{%ld})" , old,atomic_long_read (&old->usage)); -rcu_assign_pointer (current->cred, old); +revert_creds_light (old);put_cred (override); }EXPORT_SYMBOL (revert_creds);
Signature · 1 line
2.43.0
Re: [RFC v2 2/4] cred: Add a light version of override/revert_creds()
Quoted 11 lines · Vinicius Costa Gomes
Add a light version of override/revert_creds(), this should only be used when the credentials in question will outlive the critical section and the critical section doesn't change the ->usage of the credentials.
To make their usage less error prone, introduce cleanup guards asto be used like this:
guard(cred)(credentials_to_override_and_restore);
Quoted 7 lines
or this:
scoped_guard(cred, credentials_to_override_and_restore) { /* with credentials overridden */ }Signed-off-by Vinicius Costa Gomes <[email protected]>
You may add:
I would also add:
Thanks,
Amir.
Quoted 71 lines
--- include/linux/cred.h | 21 +++++++++++++++++++++ kernel/cred.c | 6 +++--- 2 files changed, 24 insertions(+), 3 deletions(-)include/linux/cred.h +21
-0@@ -172,6 +172,27 @@ static inline bool cap_ambient_invariant_ok(const struct cred *cred) cred->cap_inheritable)); } +/* +* Override creds without bumping reference count. Caller must ensure +* reference remains valid or has taken reference. Almost always not the +* interface you want. Use override_creds()/revert_creds() instead. +*/ +static inline const struct cred *override_creds_light (const struct cred *override_cred) +{ +const struct cred *old = current->cred; + +rcu_assign_pointer (current->cred, override_cred); +return old; +} + +static inline void revert_creds_light (const struct cred *revert_cred) +{ +rcu_assign_pointer (current->cred, revert_cred); +} + +DEFINE_GUARD(cred, const struct cred *, _T = override_creds_light(_T), +revert_creds_light (_T)); +/** * get_new_cred_many - Get references on a new set of credentials * @cred: The new credentials to reference kernel/cred.c +3
-4@@ -485,7 +485,7 @@ EXPORT_SYMBOL(abort_creds);*/ const struct cred *override_creds (const struct cred *new) { -const struct cred *old = current->cred; +const struct cred *old;kdebug ("override_creds(%p{%ld})" , new,atomic_long_read (&new->usage));@@ -499,7 +499,7 @@ const struct cred *override_creds(const struct cred *new)* visible to other threads under RCU. */ get_new_cred ((struct cred *)new); -rcu_assign_pointer (current->cred, new); + old =override_creds_light (new);kdebug ("override_creds() = %p{%ld}" , old,atomic_long_read (&old->usage));@@ -521,7 +521,7 @@ void revert_creds(const struct cred *old)kdebug ("revert_creds(%p{%ld})" , old,atomic_long_read (&old->usage)); -rcu_assign_pointer (current->cred, old); +revert_creds_light (old);put_cred (override); }EXPORT_SYMBOL (revert_creds); --2.43.0
Re: [RFC v2 2/4] cred: Add a light version of override/revert_creds()
Quoted 19 lines · Amir Goldstein
Quoted 17 lines · Vinicius Costa Gomes
Add a light version of override/revert_creds(), this should only be used when the credentials in question will outlive the critical section and the critical section doesn't change the ->usage of the credentials.
To make their usage less error prone, introduce cleanup guards asto be used like this:
guard(cred)(credentials_to_override_and_restore);or this:
scoped_guard(cred, credentials_to_override_and_restore) { /* with credentials overridden */ }
Quoted 8 lines
Quoted 1 line
Signed-off-by Vinicius Costa Gomes <[email protected]>You may add:
Reviewed-by Amir Goldstein <[email protected]>I would also add:
Suggested-by Christian Brauner <[email protected]>
Forgot about that one.
Cheers,
Signature · 1 line
Vinicius
RFC v2 3/4 overlayfs: Optimize credentials usage
In reply to Vinicius Costa GomesFile operations in overlayfs also check against the credentials of the mounter task, stored in the superblock, this credentials will outlive most of the operations. For these cases, use the recently introduced guard statements to guarantee that override/revert_creds() are paired.
--- fs/overlayfs/copy_up.c | 4 +-- fs/overlayfs/dir.c | 22 +++++++------ fs/overlayfs/file.c | 70 ++++++++++++++++++++++-------------------- fs/overlayfs/inode.c | 60 +++++++++++++++++++----------------- fs/overlayfs/namei.c | 21 ++++++------- fs/overlayfs/readdir.c | 18 +++++------ fs/overlayfs/util.c | 23 +++++++------- fs/overlayfs/xattrs.c | 34 ++++++++++---------- 8 files changed, 130 insertions(+), 122 deletions(-)
fs/overlayfs/copy_up.c +2 -2
@@ -1202,7 +1202,8 @@ static int ovl_copy_up_flags(struct dentry *dentry, int flags)if (err)return err; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred);while (!err) {struct dentry *next;struct dentry *parent =NULL ;
@@ -1227,7 +1228,6 @@ static int ovl_copy_up_flags(struct dentry *dentry, int flags)dput (parent);dput (next); } -revert_creds (old_cred);return err; }
fs/overlayfs/dir.c +12 -10
@@ -687,9 +687,9 @@ static int ovl_set_link_redirect(struct dentry *dentry)const struct cred *old_cred;int err; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred); err =ovl_set_redirect (dentry, false); -revert_creds (old_cred);return err; }
@@ -894,12 +894,13 @@ static int ovl_do_remove(struct dentry *dentry, bool is_dir)if (err) goto out; - old_cred =ovl_override_creds (dentry->d_sb); -if (!lower_positive) - err =ovl_remove_upper (dentry, is_dir, &list); -else - err =ovl_remove_and_whiteout (dentry, &list); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); + scoped_guard(cred, old_cred) { +if (!lower_positive) + err =ovl_remove_upper (dentry, is_dir, &list); +else + err =ovl_remove_and_whiteout (dentry, &list); + }if (!err) {if (is_dir)clear_nlink (dentry->d_inode);
@@ -1146,7 +1147,8 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, goto out; } - old_cred =ovl_override_creds (old->d_sb); + old_cred =ovl_creds (old->d_sb); + old_cred =override_creds_light (old_cred);if (!list_empty (&list)) { opaquedir =ovl_clear_empty (new, &list);
@@ -1279,7 +1281,7 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, out_unlock:unlock_rename (new_upperdir, old_upperdir); out_revert_creds: -revert_creds (old_cred); +revert_creds_light (old_cred);if (update_nlink)ovl_nlink_end (new);else
fs/overlayfs/file.c +36 -34
134 lines not on this page · raw message
fs/overlayfs/inode.c +32 -28
143 lines not on this page · raw message
fs/overlayfs/namei.c +10 -11
@@ -946,13 +946,12 @@ static int ovl_maybe_validate_verity(struct dentry *dentry)if (!ovl_test_flag (OVL_VERIFIED_DIGEST , inode)) {const struct cred *old_cred; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred); err =ovl_validate_verity (ofs, &metapath, &datapath);if (err ==0 )ovl_set_flag (OVL_VERIFIED_DIGEST , inode); - -revert_creds (old_cred); }ovl_inode_unlock (inode);
@@ -984,9 +983,10 @@ static int ovl_maybe_lookup_lowerdata(struct dentry *dentry)if (ovl_dentry_lowerdata (dentry)) goto out; - old_cred =ovl_override_creds (dentry->d_sb); - err =ovl_lookup_data_layers (dentry, redirect, &datapath); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); +scoped_guard (cred, old_cred) + err = ovl_lookup_data_layers(dentry, redirect, &datapath); +if (err) goto out_err;
@@ -1052,7 +1052,8 @@ struct dentry *ovl_lookup(struct inode *dir, struct dentry *dentry,if (dentry->d_name.len > ofs->namelen)return ERR_PTR (-ENAMETOOLONG ); - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred); upperdir =ovl_dentry_upper (dentry->d_parent);if (upperdir) { d.mnt =ovl_upper_mnt (ofs);
@@ -1331,7 +1332,6 @@ struct dentry *ovl_lookup(struct inode *dir, struct dentry *dentry,ovl_dentry_init_reval (dentry, upperdentry,OVL_I_E (inode)); -revert_creds (old_cred);if (origin_path) {dput (origin_path->dentry);kfree (origin_path);
@@ -1355,7 +1355,6 @@ struct dentry *ovl_lookup(struct inode *dir, struct dentry *dentry,kfree (upperredirect); out:kfree (d.redirect); -revert_creds (old_cred);return ERR_PTR (err); }
@@ -1379,7 +1378,8 @@ bool ovl_lower_positive(struct dentry *dentry)if (!ovl_dentry_upper (dentry))return true; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred);/* Positive upper -> have to look up lower to see whether it exists */ for (i =0 ; !done && !positive && i <ovl_numlower (poe); i++) {struct dentry *this;
@@ -1412,7 +1412,6 @@ bool ovl_lower_positive(struct dentry *dentry)dput (this); } } -revert_creds (old_cred);return positive; }
fs/overlayfs/readdir.c +9 -9
@@ -275,7 +275,8 @@ static int ovl_check_whiteouts(const struct path *path, struct ovl_readdir_datastruct dentry *dentry, *dir = path->dentry;const struct cred *old_cred; - old_cred =ovl_override_creds (rdd->dentry->d_sb); + old_cred =ovl_creds (rdd->dentry->d_sb); +guard (cred)(old_cred); err =down_write_killable (&dir->d_inode->i_rwsem);if (!err) {
@@ -290,7 +291,6 @@ static int ovl_check_whiteouts(const struct path *path, struct ovl_readdir_data }inode_unlock (dir->d_inode); } -revert_creds (old_cred);return err; }
@@ -755,7 +755,8 @@ static int ovl_iterate(struct file *file, struct dir_context *ctx)const struct cred *old_cred;int err; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred);if (!ctx->pos)ovl_dir_reset (file);
@@ -807,7 +808,6 @@ static int ovl_iterate(struct file *file, struct dir_context *ctx) } err =0 ; out: -revert_creds (old_cred);return err; }
@@ -857,9 +857,9 @@ static struct file *ovl_dir_open_realfile(const struct file *file,struct file *res;const struct cred *old_cred; - old_cred =ovl_override_creds (file_inode (file)->i_sb); + old_cred =ovl_creds (file_inode (file)->i_sb); +guard (cred)(old_cred); res =ovl_path_open (realpath,O_RDONLY | (file->f_flags &O_LARGEFILE )); -revert_creds (old_cred);return res; }
@@ -984,9 +984,9 @@ int ovl_check_empty_dir(struct dentry *dentry, struct list_head *list)struct rb_root root =RB_ROOT ;const struct cred *old_cred; - old_cred =ovl_override_creds (dentry->d_sb); - err =ovl_dir_read_merged (dentry, list, &root); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); +scoped_guard (cred, old_cred) + err = ovl_dir_read_merged(dentry, list, &root);if (err)return err;
fs/overlayfs/util.c +12 -11
@@ -1157,15 +1157,16 @@ int ovl_nlink_start(struct dentry *dentry)if (d_is_dir (dentry) || !ovl_test_flag (OVL_INDEX , inode))return 0 ; - old_cred =ovl_override_creds (dentry->d_sb); -/* -* The overlay inode nlink should be incremented/decremented IFF the -* upper operation succeeds, along with nlink change of upper inode. -* Therefore, before link/unlink/rename, we store the union nlink -* value relative to the upper inode nlink in an upper inode xattr. -*/ - err =ovl_set_nlink_upper (dentry); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); + scoped_guard(cred, old_cred) { +/* +* The overlay inode nlink should be incremented/decremented IFF the +* upper operation succeeds, along with nlink change of upper inode. +* Therefore, before link/unlink/rename, we store the union nlink +* value relative to the upper inode nlink in an upper inode xattr. +*/ + err =ovl_set_nlink_upper (dentry); + }if (err) goto out_drop_write;
@@ -1188,9 +1189,9 @@ void ovl_nlink_end(struct dentry *dentry)if (ovl_test_flag (OVL_INDEX , inode) && inode->i_nlink ==0 ) {const struct cred *old_cred; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred);ovl_cleanup_index (dentry); -revert_creds (old_cred); }ovl_inode_unlock (inode);
fs/overlayfs/xattrs.c +17 -17
@@ -45,9 +45,9 @@ static int ovl_xattr_set(struct dentry *dentry, struct inode *inode, const charif (!value && !upperdentry) {ovl_path_lower (dentry, &realpath); - old_cred =ovl_override_creds (dentry->d_sb); - err =vfs_getxattr (mnt_idmap (realpath.mnt), realdentry, name,NULL ,0 ); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); +scoped_guard (cred, old_cred) + err = vfs_getxattr(mnt_idmap(realpath.mnt), realdentry, name, NULL, 0);if (err <0 ) goto out; }
@@ -64,15 +64,16 @@ static int ovl_xattr_set(struct dentry *dentry, struct inode *inode, const charif (err) goto out; - old_cred =ovl_override_creds (dentry->d_sb); -if (value) { - err =ovl_do_setxattr (ofs, realdentry, name, value, size, - flags); - }else { -WARN_ON (flags !=XATTR_REPLACE ); - err =ovl_do_removexattr (ofs, realdentry, name); + old_cred =ovl_creds (dentry->d_sb); + scoped_guard(cred, old_cred) { +if (value) { + err =ovl_do_setxattr (ofs, realdentry, name, value, size, + flags); + }else { +WARN_ON (flags !=XATTR_REPLACE ); + err =ovl_do_removexattr (ofs, realdentry, name); + } } -revert_creds (old_cred);ovl_drop_write (dentry);/* copy c/mtime */
@@ -89,9 +90,9 @@ static int ovl_xattr_get(struct dentry *dentry, struct inode *inode, const charstruct path realpath;ovl_i_path_real (inode, &realpath); - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred); res =vfs_getxattr (mnt_idmap (realpath.mnt), realpath.dentry, name, value, size); -revert_creds (old_cred);return res; }
@@ -119,9 +120,9 @@ ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)const struct cred *old_cred;size_t prefix_len, name_len; - old_cred =ovl_override_creds (dentry->d_sb); - res =vfs_listxattr (realdentry, list, size); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); +scoped_guard (cred, old_cred) + res = vfs_listxattr(realdentry, list, size);if (res <=0 || size ==0 )return res;
@@ -268,4 +269,3 @@ const struct xattr_handler * const *ovl_xattr_handlers(struct ovl_fs *ofs)return ofs->config.userxattr ? ovl_user_xattr_handlers : ovl_trusted_xattr_handlers; } -
Signature · 1 line
2.43.0
Amir Goldstein
maintainerRe: [RFC v2 3/4] overlayfs: Optimize credentials usage
Quoted 87 lines · Vinicius Costa Gomes
File operations in overlayfs also check against the credentials of the mounter task, stored in the superblock, this credentials will outlive most of the operations. For these cases, use the recently introduced guard statements to guarantee that override/revert_creds() are paired.
Signed-off-by Vinicius Costa Gomes <[email protected]>--- fs/overlayfs/copy_up.c | 4 +-- fs/overlayfs/dir.c | 22 +++++++------ fs/overlayfs/file.c | 70 ++++++++++++++++++++++-------------------- fs/overlayfs/inode.c | 60 +++++++++++++++++++----------------- fs/overlayfs/namei.c | 21 ++++++------- fs/overlayfs/readdir.c | 18 +++++------ fs/overlayfs/util.c | 23 +++++++------- fs/overlayfs/xattrs.c | 34 ++++++++++---------- 8 files changed, 130 insertions(+), 122 deletions(-)fs/overlayfs/copy_up.c +2
-2@@ -1202,7 +1202,8 @@ static int ovl_copy_up_flags(struct dentry *dentry, int flags)if (err)return err; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred);while (!err) {struct dentry *next;struct dentry *parent =NULL ;@@ -1227,7 +1228,6 @@ static int ovl_copy_up_flags(struct dentry *dentry, int flags)dput (parent);dput (next); } -revert_creds (old_cred);return err; }fs/overlayfs/dir.c +11
-9@@ -687,9 +687,9 @@ static int ovl_set_link_redirect(struct dentry *dentry)const struct cred *old_cred;int err; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred); err =ovl_set_redirect (dentry, false); -revert_creds (old_cred);return err; }@@ -894,12 +894,13 @@ static int ovl_do_remove(struct dentry *dentry, bool is_dir)if (err) goto out; - old_cred =ovl_override_creds (dentry->d_sb); -if (!lower_positive) - err =ovl_remove_upper (dentry, is_dir, &list); -else - err =ovl_remove_and_whiteout (dentry, &list); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); + scoped_guard(cred, old_cred) { +if (!lower_positive) + err =ovl_remove_upper (dentry, is_dir, &list); +else + err =ovl_remove_and_whiteout (dentry, &list); + }if (!err) {if (is_dir)clear_nlink (dentry->d_inode);@@ -1146,7 +1147,8 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, goto out; } - old_cred =ovl_override_creds (old->d_sb); + old_cred =ovl_creds (old->d_sb); + old_cred =override_creds_light (old_cred);if (!list_empty (&list)) { opaquedir =ovl_clear_empty (new, &list);
fs/overlayfs/dir.c static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, · line 1287
Quoted 8 lines
fs/overlayfs/dir.c +1
-1@@ -1279,7 +1281,7 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, out_unlock:unlock_rename (new_upperdir, old_upperdir); out_revert_creds: -revert_creds (old_cred); +revert_creds_light (old_cred);if (update_nlink)ovl_nlink_end (new);else
Most of my comments on this patch are identical to the ones I have made on backing file, so rather complete that review before moving on to this bigger patch.
I even wonder if we need a specialized macro for overlayfs guard(ovl_creds, ofs); or ifguard(cred, ovl_override_creds(dentry->d_sb));
is good enough.
One thing that stands out in functions like ovl_rename() is that, understandably, you tried to preserve logic, but in fact, the scope of override_creds/revert_creds() in some of the overlayfs functions ir rather arbitrary.
The simplest solution for functions like the above is to use guard(cred, .. and extend the scope till the end of the function.
This needs more careful review, but the end result will be much cleaner.
Quoted 116 lines
fs/overlayfs/file.c +26
-30@@ -42,7 +42,8 @@ static struct file *ovl_open_realfile(const struct file *file,if (flags &O_APPEND ) acc_mode |=MAY_APPEND ; - old_cred =ovl_override_creds (inode->i_sb); + old_cred =ovl_creds (inode->i_sb); +guard (cred)(old_cred); real_idmap =mnt_idmap (realpath->mnt); err =inode_permission (real_idmap, realinode,MAY_OPEN | acc_mode);if (err) {@@ -54,7 +55,6 @@ static struct file *ovl_open_realfile(const struct file *file, realfile =backing_file_open (&file->f_path, flags, realpath,current_cred ()); } -revert_creds (old_cred); pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n", file, file, ovl_whatisit(inode, realinode), file->f_flags,@@ -214,9 +214,9 @@ static loff_t ovl_llseek(struct file *file, loff_t offset, int whence)ovl_inode_lock (inode); real.file->f_pos = file->f_pos; - old_cred =ovl_override_creds (inode->i_sb); - ret =vfs_llseek (real.file, offset, whence); -revert_creds (old_cred); + old_cred =ovl_creds (inode->i_sb); +scoped_guard (cred, old_cred) + ret = vfs_llseek(real.file, offset, whence); file->f_pos = real.file->f_pos;ovl_inode_unlock (inode);@@ -388,7 +388,6 @@ static ssize_t ovl_splice_write(struct pipe_inode_info *pipe, struct file *out,static int ovl_fsync (struct file *file,loff_t start,loff_t end,int datasync) {struct fd real; -const struct cred *old_cred;int ret; ret =ovl_sync_status (OVL_FS (file_inode (file)->i_sb));@@ -401,9 +400,11 @@ static int ovl_fsync(struct file *file, loff_t start, loff_t end, int datasync)/* Don't sync lower file for fear of receiving EROFS error */ if (file_inode (real.file) ==ovl_inode_upper (file_inode (file))) { - old_cred =ovl_override_creds (file_inode (file)->i_sb); +const struct cred *old_cred; + + old_cred =ovl_creds (file_inode (file)->i_sb); +guard (cred)(old_cred); ret =vfs_fsync_range (real.file, start, end, datasync); -revert_creds (old_cred); }fdput (real);@@ -441,9 +442,9 @@ static long ovl_fallocate(struct file *file, int mode, loff_t offset, loff_t lenif (ret) goto out_unlock; - old_cred =ovl_override_creds (file_inode (file)->i_sb); - ret =vfs_fallocate (real.file, mode, offset, len); -revert_creds (old_cred); + old_cred =ovl_creds (file_inode (file)->i_sb); +scoped_guard (cred, old_cred) + ret = vfs_fallocate(real.file, mode, offset, len);/* Update size */ ovl_file_modified (file);@@ -466,9 +467,9 @@ static int ovl_fadvise(struct file *file, loff_t offset, loff_t len, int advice)if (ret)return ret; - old_cred =ovl_override_creds (file_inode (file)->i_sb); - ret =vfs_fadvise (real.file, offset, len, advice); -revert_creds (old_cred); + old_cred =ovl_creds (file_inode (file)->i_sb); +scoped_guard (cred, old_cred) + ret = vfs_fadvise(real.file, offset, len, advice);fdput (real);@@ -509,25 +510,25 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in, goto out_unlock; } - old_cred =ovl_override_creds (file_inode (file_out)->i_sb); -switch (op) { -case OVL_COPY : - ret =vfs_copy_file_range (real_in.file, pos_in, - real_out.file, pos_out, len, flags); -break ; + old_cred =ovl_creds (file_inode (file_out)->i_sb); + scoped_guard(cred, old_cred) +switch (op) { +case OVL_COPY : + ret =vfs_copy_file_range (real_in.file, pos_in, + real_out.file, pos_out, len, flags); +break ; -case OVL_CLONE : - ret =vfs_clone_file_range (real_in.file, pos_in, - real_out.file, pos_out, len, flags); -break ; +case OVL_CLONE : + ret =vfs_clone_file_range (real_in.file, pos_in, + real_out.file, pos_out, len, flags); +break ; -case OVL_DEDUPE : - ret =vfs_dedupe_file_range_one (real_in.file, pos_in, - real_out.file, pos_out, len, - flags); -break ; - }
fs/overlayfs/file.c static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in, · line 534
Quoted 10 lines
fs/overlayfs/file.c +6
-1-revert_creds (old_cred); +case OVL_DEDUPE : + ret =vfs_dedupe_file_range_one (real_in.file, pos_in, + real_out.file, pos_out, len, + flags); +break ; + }/* Update size */ ovl_file_modified (file_out);
This is another case where extending the scope to the end of the function is the simpler/cleaner solution.
Thanks,
Amir.
Re: [RFC v2 3/4] overlayfs: Optimize credentials usage
Quoted 102 lines · Amir Goldstein
Quoted 87 lines · Vinicius Costa Gomes
File operations in overlayfs also check against the credentials of the mounter task, stored in the superblock, this credentials will outlive most of the operations. For these cases, use the recently introduced guard statements to guarantee that override/revert_creds() are paired.
Signed-off-by Vinicius Costa Gomes <[email protected]>--- fs/overlayfs/copy_up.c | 4 +-- fs/overlayfs/dir.c | 22 +++++++------ fs/overlayfs/file.c | 70 ++++++++++++++++++++++-------------------- fs/overlayfs/inode.c | 60 +++++++++++++++++++----------------- fs/overlayfs/namei.c | 21 ++++++------- fs/overlayfs/readdir.c | 18 +++++------ fs/overlayfs/util.c | 23 +++++++------- fs/overlayfs/xattrs.c | 34 ++++++++++---------- 8 files changed, 130 insertions(+), 122 deletions(-)fs/overlayfs/copy_up.c +2
-2@@ -1202,7 +1202,8 @@ static int ovl_copy_up_flags(struct dentry *dentry, int flags)if (err)return err; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred);while (!err) {struct dentry *next;struct dentry *parent =NULL ;@@ -1227,7 +1228,6 @@ static int ovl_copy_up_flags(struct dentry *dentry, int flags)dput (parent);dput (next); } -revert_creds (old_cred);return err; }fs/overlayfs/dir.c +11
-9@@ -687,9 +687,9 @@ static int ovl_set_link_redirect(struct dentry *dentry)const struct cred *old_cred;int err; - old_cred =ovl_override_creds (dentry->d_sb); + old_cred =ovl_creds (dentry->d_sb); +guard (cred)(old_cred); err =ovl_set_redirect (dentry, false); -revert_creds (old_cred);return err; }@@ -894,12 +894,13 @@ static int ovl_do_remove(struct dentry *dentry, bool is_dir)if (err) goto out; - old_cred =ovl_override_creds (dentry->d_sb); -if (!lower_positive) - err =ovl_remove_upper (dentry, is_dir, &list); -else - err =ovl_remove_and_whiteout (dentry, &list); -revert_creds (old_cred); + old_cred =ovl_creds (dentry->d_sb); + scoped_guard(cred, old_cred) { +if (!lower_positive) + err =ovl_remove_upper (dentry, is_dir, &list); +else + err =ovl_remove_and_whiteout (dentry, &list); + }if (!err) {if (is_dir)clear_nlink (dentry->d_inode);@@ -1146,7 +1147,8 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, goto out; } - old_cred =ovl_override_creds (old->d_sb); + old_cred =ovl_creds (old->d_sb); + old_cred =override_creds_light (old_cred);if (!list_empty (&list)) { opaquedir =ovl_clear_empty (new, &list);fs/overlayfs/dir.c static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, · line 1287
Quoted 8 lines
fs/overlayfs/dir.c +1
-1@@ -1279,7 +1281,7 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir, out_unlock:unlock_rename (new_upperdir, old_upperdir); out_revert_creds: -revert_creds (old_cred); +revert_creds_light (old_cred);if (update_nlink)ovl_nlink_end (new);else Most of my comments on this patch are identical to the ones I have made on backing file, so rather complete that review before moving on to this bigger patch.
Quoted 5 lines
I even wonder if we need a specialized macro for overlayfs
guard(ovl_creds, ofs); or ifguard(cred, ovl_override_creds(dentry->d_sb));
is good enough.
I think that if the DEFINE_LOCK_GUARD_1() idea works, that might be unecessary. Let's see.
Quoted 5 lines
One thing that stands out in functions like
ovl_rename()is that, understandably, you tried to preserve logic, but in fact, the scope of override_creds/revert_creds() in some of the overlayfs functions ir rather arbitrary.
That's very good to learn.
Quoted 4 lines
The simplest solution for functions like the above is to use guard(cred, .. and extend the scope till the end of the function.
This needs more careful review, but the end result will be much cleaner.
Yeah, increasing the indentation level of whole blocks cause the whole patch to be much harder to review.
Using more guard() statements will certainly help.
Quoted 132 lines
Quoted 116 lines
fs/overlayfs/file.c +26
-30@@ -42,7 +42,8 @@ static struct file *ovl_open_realfile(const struct file *file,if (flags &O_APPEND ) acc_mode |=MAY_APPEND ; - old_cred =ovl_override_creds (inode->i_sb); + old_cred =ovl_creds (inode->i_sb); +guard (cred)(old_cred); real_idmap =mnt_idmap (realpath->mnt); err =inode_permission (real_idmap, realinode,MAY_OPEN | acc_mode);if (err) {@@ -54,7 +55,6 @@ static struct file *ovl_open_realfile(const struct file *file, realfile =backing_file_open (&file->f_path, flags, realpath,current_cred ()); } -revert_creds (old_cred); pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n", file, file, ovl_whatisit(inode, realinode), file->f_flags,@@ -214,9 +214,9 @@ static loff_t ovl_llseek(struct file *file, loff_t offset, int whence)ovl_inode_lock (inode); real.file->f_pos = file->f_pos; - old_cred =ovl_override_creds (inode->i_sb); - ret =vfs_llseek (real.file, offset, whence); -revert_creds (old_cred); + old_cred =ovl_creds (inode->i_sb); +scoped_guard (cred, old_cred) + ret = vfs_llseek(real.file, offset, whence); file->f_pos = real.file->f_pos;ovl_inode_unlock (inode);@@ -388,7 +388,6 @@ static ssize_t ovl_splice_write(struct pipe_inode_info *pipe, struct file *out,static int ovl_fsync (struct file *file,loff_t start,loff_t end,int datasync) {struct fd real; -const struct cred *old_cred;int ret; ret =ovl_sync_status (OVL_FS (file_inode (file)->i_sb));@@ -401,9 +400,11 @@ static int ovl_fsync(struct file *file, loff_t start, loff_t end, int datasync)/* Don't sync lower file for fear of receiving EROFS error */ if (file_inode (real.file) ==ovl_inode_upper (file_inode (file))) { - old_cred =ovl_override_creds (file_inode (file)->i_sb); +const struct cred *old_cred; + + old_cred =ovl_creds (file_inode (file)->i_sb); +guard (cred)(old_cred); ret =vfs_fsync_range (real.file, start, end, datasync); -revert_creds (old_cred); }fdput (real);@@ -441,9 +442,9 @@ static long ovl_fallocate(struct file *file, int mode, loff_t offset, loff_t lenif (ret) goto out_unlock; - old_cred =ovl_override_creds (file_inode (file)->i_sb); - ret =vfs_fallocate (real.file, mode, offset, len); -revert_creds (old_cred); + old_cred =ovl_creds (file_inode (file)->i_sb); +scoped_guard (cred, old_cred) + ret = vfs_fallocate(real.file, mode, offset, len);/* Update size */ ovl_file_modified (file);@@ -466,9 +467,9 @@ static int ovl_fadvise(struct file *file, loff_t offset, loff_t len, int advice)if (ret)return ret; - old_cred =ovl_override_creds (file_inode (file)->i_sb); - ret =vfs_fadvise (real.file, offset, len, advice); -revert_creds (old_cred); + old_cred =ovl_creds (file_inode (file)->i_sb); +scoped_guard (cred, old_cred) + ret = vfs_fadvise(real.file, offset, len, advice);fdput (real);@@ -509,25 +510,25 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in, goto out_unlock; } - old_cred =ovl_override_creds (file_inode (file_out)->i_sb); -switch (op) { -case OVL_COPY : - ret =vfs_copy_file_range (real_in.file, pos_in, - real_out.file, pos_out, len, flags); -break ; + old_cred =ovl_creds (file_inode (file_out)->i_sb); + scoped_guard(cred, old_cred) +switch (op) { +case OVL_COPY : + ret =vfs_copy_file_range (real_in.file, pos_in, + real_out.file, pos_out, len, flags); +break ; -case OVL_CLONE : - ret =vfs_clone_file_range (real_in.file, pos_in, - real_out.file, pos_out, len, flags); -break ; +case OVL_CLONE : + ret =vfs_clone_file_range (real_in.file, pos_in, + real_out.file, pos_out, len, flags); +break ; -case OVL_DEDUPE : - ret =vfs_dedupe_file_range_one (real_in.file, pos_in, - real_out.file, pos_out, len, - flags); -break ; - }fs/overlayfs/file.c static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in, · line 534
Quoted 10 lines
fs/overlayfs/file.c +6
-1-revert_creds (old_cred); +case OVL_DEDUPE : + ret =vfs_dedupe_file_range_one (real_in.file, pos_in, + real_out.file, pos_out, len, + flags); +break ; + }/* Update size */ ovl_file_modified (file_out);This is another case where extending the scope to the end of the function is the simpler/cleaner solution.
Thanks,
Amir.
Signature · 1 line
Vinicius
RFC v2 4/4 fs: Optimize credentials reference count for backing file ops
In reply to Vinicius Costa GomesFor backing file operations, users are expected to pass credentials that will outlive the backing file common operations.
Use the specialized guard statements to override/revert the credentials.
--- fs/backing-file.c | 124 ++++++++++++++++++++++------------------------ 1 file changed, 60 insertions(+), 64 deletions(-)
fs/backing-file.c +60 -64
@@ -140,7 +140,7 @@ ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, struct backing_file_ctx *ctx) {struct backing_aio *aio =NULL ; -const struct cred *old_cred; +const struct cred *old_cred = ctx->cred;ssize_t ret;if (WARN_ON_ONCE (!(file->f_mode &FMODE_BACKING )))
@@ -153,29 +153,28 @@ ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, !(file->f_mode & FMODE_CAN_ODIRECT)) return -EINVAL; - old_cred =override_creds (ctx->cred); -if (is_sync_kiocb (iocb)) { -rwf_t rwf =iocb_to_rw_flags (flags); + scoped_guard(cred, old_cred) { +if (is_sync_kiocb (iocb)) { +rwf_t rwf =iocb_to_rw_flags (flags); - ret =vfs_iter_read (file, iter, &iocb->ki_pos, rwf); - }else { - ret = -ENOMEM ; - aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); -if (!aio) - goto out; + ret =vfs_iter_read (file, iter, &iocb->ki_pos, rwf); + }else { + ret = -ENOMEM ; + aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); +if (!aio) + goto out; - aio->orig_iocb = iocb; -kiocb_clone (&aio->iocb, iocb,get_file (file)); - aio->iocb.ki_complete = backing_aio_rw_complete; -refcount_set (&aio->ref,2 ); - ret =vfs_iocb_iter_read (file, &aio->iocb, iter); -backing_aio_put (aio); -if (ret != -EIOCBQUEUED ) -backing_aio_cleanup (aio, ret); + aio->orig_iocb = iocb; +kiocb_clone (&aio->iocb, iocb,get_file (file)); + aio->iocb.ki_complete = backing_aio_rw_complete; +refcount_set (&aio->ref,2 ); + ret =vfs_iocb_iter_read (file, &aio->iocb, iter); +backing_aio_put (aio); +if (ret != -EIOCBQUEUED ) +backing_aio_cleanup (aio, ret); + } } out: -revert_creds (old_cred); -if (ctx->accessed) ctx->accessed (ctx->user_file);
@@ -187,7 +186,7 @@ ssize_t backing_file_write_iter(struct file *file, struct iov_iter *iter, struct kiocb *iocb, int flags, struct backing_file_ctx *ctx) { - const struct cred *old_cred; + const struct cred *old_cred = ctx->cred;ssize_t ret;if (WARN_ON_ONCE (!(file->f_mode &FMODE_BACKING )))
@@ -210,39 +209,37 @@ ssize_t backing_file_write_iter(struct file *file, struct iov_iter *iter,*/ flags &= ~IOCB_DIO_CALLER_COMP ; - old_cred =override_creds (ctx->cred); -if (is_sync_kiocb (iocb)) { -rwf_t rwf =iocb_to_rw_flags (flags); + scoped_guard(cred, old_cred) { +if (is_sync_kiocb (iocb)) { +rwf_t rwf =iocb_to_rw_flags (flags); - ret =vfs_iter_write (file, iter, &iocb->ki_pos, rwf); -if (ctx->end_write) - ctx->end_write (ctx->user_file); - }else { -struct backing_aio *aio; + ret =vfs_iter_write (file, iter, &iocb->ki_pos, rwf); +if (ctx->end_write) + ctx->end_write (ctx->user_file); + }else { +struct backing_aio *aio; - ret =backing_aio_init_wq (iocb); -if (ret) - goto out; + ret =backing_aio_init_wq (iocb); +if (ret) +return ret; - ret = -ENOMEM ; - aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); -if (!aio) - goto out; + ret = -ENOMEM ; + aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); +if (!aio) +return ret; - aio->orig_iocb = iocb; - aio->end_write = ctx->end_write; -kiocb_clone (&aio->iocb, iocb,get_file (file)); - aio->iocb.ki_flags = flags; - aio->iocb.ki_complete = backing_aio_queue_completion; -refcount_set (&aio->ref,2 ); - ret =vfs_iocb_iter_write (file, &aio->iocb, iter); -backing_aio_put (aio); -if (ret != -EIOCBQUEUED ) -backing_aio_cleanup (aio, ret); + aio->orig_iocb = iocb; + aio->end_write = ctx->end_write; +kiocb_clone (&aio->iocb, iocb,get_file (file)); + aio->iocb.ki_flags = flags; + aio->iocb.ki_complete = backing_aio_queue_completion; +refcount_set (&aio->ref,2 ); + ret =vfs_iocb_iter_write (file, &aio->iocb, iter); +backing_aio_put (aio); +if (ret != -EIOCBQUEUED ) +backing_aio_cleanup (aio, ret); + } } -out: -revert_creds (old_cred); -return ret; }EXPORT_SYMBOL_GPL (backing_file_write_iter);
@@ -252,15 +249,15 @@ ssize_t backing_file_splice_read(struct file *in, loff_t *ppos,unsigned int flags, struct backing_file_ctx *ctx) { - const struct cred *old_cred; + const struct cred *old_cred = ctx->cred;ssize_t ret;if (WARN_ON_ONCE (!(in->f_mode &FMODE_BACKING )))return -EIO ; - old_cred =override_creds (ctx->cred); - ret =vfs_splice_read (in, ppos, pipe, len, flags); -revert_creds (old_cred); + scoped_guard(cred, old_cred) { + ret =vfs_splice_read (in, ppos, pipe, len, flags); + }if (ctx->accessed) ctx->accessed (ctx->user_file);
@@ -274,7 +271,7 @@ ssize_t backing_file_splice_write(struct pipe_inode_info *pipe,unsigned int flags, struct backing_file_ctx *ctx) { - const struct cred *old_cred; + const struct cred *old_cred = ctx->cred;ssize_t ret;if (WARN_ON_ONCE (!(out->f_mode &FMODE_BACKING )))
@@ -284,12 +281,11 @@ ssize_t backing_file_splice_write(struct pipe_inode_info *pipe,if (ret)return ret; - old_cred =override_creds (ctx->cred); -file_start_write (out); - ret =iter_file_splice_write (pipe, out, ppos, len, flags); -file_end_write (out); -revert_creds (old_cred); - + scoped_guard(cred, old_cred) { +file_start_write (out); + ret =iter_file_splice_write (pipe, out, ppos, len, flags); +file_end_write (out); + }if (ctx->end_write) ctx->end_write (ctx->user_file);
@@ -300,7 +296,7 @@ EXPORT_SYMBOL_GPL(backing_file_splice_write);int backing_file_mmap (struct file *file,struct vm_area_struct *vma,struct backing_file_ctx *ctx) { -const struct cred *old_cred; +const struct cred *old_cred = ctx->cred;int ret; if (WARN_ON_ONCE(!(file->f_mode & FMODE_BACKING)) ||
@@ -312,9 +308,9 @@ int backing_file_mmap(struct file *file, struct vm_area_struct *vma,vma_set_file (vma, file); - old_cred =override_creds (ctx->cred); - ret =call_mmap (vma->vm_file, vma); -revert_creds (old_cred); + scoped_guard(cred, old_cred) { + ret =call_mmap (vma->vm_file, vma); + }if (ctx->accessed) ctx->accessed (ctx->user_file);
Signature · 1 line
2.43.0
Amir Goldstein
maintainerRe: [RFC v2 4/4] fs: Optimize credentials reference count for backing file ops
Quoted 7 lines · Vinicius Costa Gomes
For backing file operations, users are expected to pass credentials that will outlive the backing file common operations.
Use the specialized guard statements to override/revert the credentials.
As I wrote before, I prefer to see this patch gets reviewed and merged before the overlayfs large patch, so please reorder the series.
Quoted 19 lines
Signed-off-by Vinicius Costa Gomes <[email protected]>--- fs/backing-file.c | 124 ++++++++++++++++++++++------------------------ 1 file changed, 60 insertions(+), 64 deletions(-)fs/backing-file.c +1
-1@@ -140,7 +140,7 @@ ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, struct backing_file_ctx *ctx) {struct backing_aio *aio =NULL ; -const struct cred *old_cred; +const struct cred *old_cred = ctx->cred;ssize_t ret;if (WARN_ON_ONCE (!(file->f_mode &FMODE_BACKING )))
fs/backing-file.c ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, · line 156
Quoted 7 lines
fs/backing-file.c +1
-3@@ -153,29 +153,28 @@ ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, !(file->f_mode & FMODE_CAN_ODIRECT)) return -EINVAL; - old_cred =override_creds (ctx->cred); -if (is_sync_kiocb (iocb)) { -rwf_t rwf =iocb_to_rw_flags (flags); + scoped_guard(cred, old_cred) {
This reads very strage.
Also, I see that e.g. scoped_guard(spinlock_irqsave, ... hides the local var used for save/restore of flags inside the macro.
Perhaps you use the same technique for scoped_guard(cred, .. loose the local old_cred variable in all those functions and then the code will read:
scoped_guard(cred, ctx->cred) {
which is nicer IMO.
Quoted 23 lines
fs/backing-file.c +8
-13+if (is_sync_kiocb (iocb)) { +rwf_t rwf =iocb_to_rw_flags (flags); - ret =vfs_iter_read (file, iter, &iocb->ki_pos, rwf); - }else { - ret = -ENOMEM ; - aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); -if (!aio) - goto out; + ret =vfs_iter_read (file, iter, &iocb->ki_pos, rwf); + }else { + ret = -ENOMEM ; + aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); +if (!aio) + goto out; - aio->orig_iocb = iocb; -kiocb_clone (&aio->iocb, iocb,get_file (file)); - aio->iocb.ki_complete = backing_aio_rw_complete; -refcount_set (&aio->ref,2 ); - ret =vfs_iocb_iter_read (file, &aio->iocb, iter); -backing_aio_put (aio); -if (ret != -EIOCBQUEUED )
fs/backing-file.c ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, · line 175
Quoted 10 lines
fs/backing-file.c +9
-1-backing_aio_cleanup (aio, ret); + aio->orig_iocb = iocb; +kiocb_clone (&aio->iocb, iocb,get_file (file)); + aio->iocb.ki_complete = backing_aio_rw_complete; +refcount_set (&aio->ref,2 ); + ret =vfs_iocb_iter_read (file, &aio->iocb, iter); +backing_aio_put (aio); +if (ret != -EIOCBQUEUED ) +backing_aio_cleanup (aio, ret); + }
if possible, I would rather avoid all this churn in functions that mostly do work with the new cred, so either use guard(cred, ) directly or split a helper that uses guard(cred, ) form the rest.
Thanks,
Amir.
Re: [RFC v2 4/4] fs: Optimize credentials reference count for backing file ops
Quoted 6 lines · Amir Goldstein
Quoted 4 lines · Vinicius Costa Gomes
For backing file operations, users are expected to pass credentials that will outlive the backing file common operations.
Quoted 7 lines
Quoted 3 lines
Use the specialized guard statements to override/revert the credentials.
As I wrote before, I prefer to see this patch gets reviewed and merged before the overlayfs large patch, so please reorder the series.
Sure. Will do.
Quoted 32 lines
Quoted 19 lines
Signed-off-by Vinicius Costa Gomes <[email protected]>--- fs/backing-file.c | 124 ++++++++++++++++++++++------------------------ 1 file changed, 60 insertions(+), 64 deletions(-)fs/backing-file.c +1
-1@@ -140,7 +140,7 @@ ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, struct backing_file_ctx *ctx) {struct backing_aio *aio =NULL ; -const struct cred *old_cred; +const struct cred *old_cred = ctx->cred;ssize_t ret;if (WARN_ON_ONCE (!(file->f_mode &FMODE_BACKING )))fs/backing-file.c ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, · line 156
Quoted 7 lines
fs/backing-file.c +1
-3@@ -153,29 +153,28 @@ ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, !(file->f_mode & FMODE_CAN_ODIRECT)) return -EINVAL; - old_cred =override_creds (ctx->cred); -if (is_sync_kiocb (iocb)) { -rwf_t rwf =iocb_to_rw_flags (flags); + scoped_guard(cred, old_cred) {This reads very strage.
Also, I see that e.g. scoped_guard(spinlock_irqsave, ... hides the local var used for save/restore of flags inside the macro.
Quoted 7 lines
Perhaps you use the same technique for scoped_guard(cred, .. loose the local
old_credvariable in all those functions and then the code will read:
scoped_guard(cred, ctx->cred){which is nicer IMO.
Most likely using DEFINE_LOCK_GUARD_1() would allow us to use the nicer version.
Quoted 35 lines
Quoted 24 lines
fs/backing-file.c +8
-13+if (is_sync_kiocb (iocb)) { +rwf_t rwf =iocb_to_rw_flags (flags); - ret =vfs_iter_read (file, iter, &iocb->ki_pos, rwf); - }else { - ret = -ENOMEM ; - aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); -if (!aio) - goto out; + ret =vfs_iter_read (file, iter, &iocb->ki_pos, rwf); + }else { + ret = -ENOMEM ; + aio =kmem_cache_zalloc (backing_aio_cachep,GFP_KERNEL ); +if (!aio) + goto out; - aio->orig_iocb = iocb; -kiocb_clone (&aio->iocb, iocb,get_file (file)); - aio->iocb.ki_complete = backing_aio_rw_complete; -refcount_set (&aio->ref,2 ); - ret =vfs_iocb_iter_read (file, &aio->iocb, iter); -backing_aio_put (aio); -if (ret != -EIOCBQUEUED )fs/backing-file.c ssize_t backing_file_read_iter(struct file *file, struct iov_iter *iter, · line 175
Quoted 10 lines
fs/backing-file.c +9
-1-backing_aio_cleanup (aio, ret); + aio->orig_iocb = iocb; +kiocb_clone (&aio->iocb, iocb,get_file (file)); + aio->iocb.ki_complete = backing_aio_rw_complete; +refcount_set (&aio->ref,2 ); + ret =vfs_iocb_iter_read (file, &aio->iocb, iter); +backing_aio_put (aio); +if (ret != -EIOCBQUEUED ) +backing_aio_cleanup (aio, ret); + }
Quoted 4 lines
if possible, I would rather avoid all this churn in functions that mostly do work with the new cred, so either use
guard(cred, )directly or split a helper that usesguard(cred, )form the rest.
Yeah, I think what happened is that I tried to keep the scope of the guard to be as close as possible to override/revert (as you said), and that caused the churn.
Probably using guard() more will reduce these confusing code changes. I am going to try that.
Quoted 2 lines
Thanks,
Amir.
Cheers,
Signature · 1 line
Vinicius
Amir Goldstein
maintainerQuoted 3 lines · Vinicius Costa Gomes
Hi,
Hi Vinicius,
I have some specific comments about the overlayfs patch, but first I prefer to provide higher level feedback on the series.
Quoted 87 lines
It was noticed that some workloads suffer from contention on increasing/decrementing the ->usage counter in their credentials, those refcount operations are associated with overriding/reverting the current task credentials. (the linked thread adds more context)
In some specialized cases, overlayfs is one of them, the credentials in question have a longer lifetime than the override/revert "critical section". In the overlayfs case, the credentials are created when the fs is mounted and destroyed when it's unmounted. In this case of long lived credentials, the usage counter doesn't need to be incremented/decremented.
Add a lighter version of credentials override/revert to be used in these specialized cases. To make sure that the override/revert calls are paired, add a cleanup guard macro. This was suggested here:
https://lore.kernel.org/all/20231219-marken-pochen-26d888fb9bb9@brauner/
With a small number of tweaks:
- Used inline functions instead of macros;
- A small change to store the credentials into the passed argument, the guard is now defined as (note the added '_T ='):DEFINE_GUARD(cred, const struct cred *, _T =
override_creds_light(_T),
revert_creds_light(_T));- Allow "const" arguments to be used with these kind of guards;
Some comments:
- If patch 1/4 is not a good idea (adding the cast), the alternative I can see is using some kind of container for the credentials;
- The only user for the backing file ops is overlayfs, so these changes make sense, but may not make sense in the most general case;For the numbers, some from 'perf c2c', before this series:
(edited to fit)#
# ----- HITM ----- Shared # Num RmtHitm LclHitm Symbol Object Source:Line Node # ..... ....... ....... .......................... ................ .................. .... # ------------------------- 0 412 1028 ------------------------- 41.50% 42.22% [k] revert_creds [kernel.vmlinux] atomic64_64.h:39 0 1 15.05% 10.60% [k] override_creds [kernel.vmlinux] atomic64_64.h:25 0 1 0.73% 0.58% [k] init_file [kernel.vmlinux] atomic64_64.h:25 0 1 0.24% 0.10% [k] revert_creds [kernel.vmlinux] cred.h:266 0 1 32.28% 37.16% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1 9.47% 8.75% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1 0.49% 0.58% [k] inode_owner_or_capable [kernel.vmlinux] mnt_idmapping.h:81 0 1 0.24% 0.00% [k] generic_permission [kernel.vmlinux] namei.c:354 0-------------------------
1 50 103 ------------------------- 100.00% 100.00% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1-------------------------
2 50 98 ------------------------- 96.00% 96.94% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1 2.00% 1.02% [k] update_load_avg [kernel.vmlinux] atomic64_64.h:25 0 1 0.00% 2.04% [k] update_load_avg [kernel.vmlinux] fair.c:4118 0 2.00% 0.00% [k] update_cfs_group [kernel.vmlinux] fair.c:3932 0 1after this series:
#
# ----- HITM ----- Shared # Num RmtHitm LclHitm Symbol Object Source:Line Node # ..... ....... ....... .................... ................ ................ .... # ------------------------- 0 54 88 ------------------------- 100.00% 100.00% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1-------------------------
1 48 83 ------------------------- 97.92% 97.59% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1 2.08% 1.20% [k] update_load_avg [kernel.vmlinux] atomic64_64.h:25 0 1 0.00% 1.20% [k] update_load_avg [kernel.vmlinux] fair.c:4118 0 1
Quoted 8 lines
-------------------------
2 28 44 ------------------------- 85.71% 79.55% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1 14.29% 20.45% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1The contention is practically gone.
That is very impressive.
Can you say which workloads were running during this test? Specifically, I am wondering how much of the improvement came from backing_file.c and how much from overlayfs/*.c.
The reason I am asking is because the overlayfs patch is quite large and can take more time to review, so I am wondering out loud if we are not better off this
course of action:
1. convert backing_file.c to use new helpers/guards
2. convert overlayfs to use new helpers/guards
#1 should definitely go in via Christian's tree and should get a wider review from fsdevel (please CC fsdevel next time)
#2 is contained for overlayfs reviewers. Once the helpers are merged and used by backing_file helpers, overlayfs can be converted independently.
#1 and #2 could both be merged in the same merge cycle, or not, it does not matter. Most likely, #2 will go through Christian's tree as well, but I think we need to work according to this merge order.
We can also work on the review in parallel and you may keep the overlayfs patch in following posts, just wanted us to be on the same page w.r.t to the process.
Thanks,
Amir.
Hi Amir,
Quoted 107 lines · Amir Goldstein
cc fsdevelQuoted 3 lines · Vinicius Costa Gomes
Hi,
Hi Vinicius,
I have some specific comments about the overlayfs patch, but first I prefer to provide higher level feedback on the series.
Quoted 94 lines
It was noticed that some workloads suffer from contention on increasing/decrementing the ->usage counter in their credentials, those refcount operations are associated with overriding/reverting the current task credentials. (the linked thread adds more context)
In some specialized cases, overlayfs is one of them, the credentials in question have a longer lifetime than the override/revert "critical section". In the overlayfs case, the credentials are created when the fs is mounted and destroyed when it's unmounted. In this case of long lived credentials, the usage counter doesn't need to be incremented/decremented.
Add a lighter version of credentials override/revert to be used in these specialized cases. To make sure that the override/revert calls are paired, add a cleanup guard macro. This was suggested here:
https://lore.kernel.org/all/20231219-marken-pochen-26d888fb9bb9@brauner/
With a small number of tweaks:
- Used inline functions instead of macros;
- A small change to store the credentials into the passed argument, the guard is now defined as (note the added '_T ='):DEFINE_GUARD(cred, const struct cred *, _T =
override_creds_light(_T),
revert_creds_light(_T));- Allow "const" arguments to be used with these kind of guards;
Some comments:
- If patch 1/4 is not a good idea (adding the cast), the alternative I can see is using some kind of container for the credentials;
- The only user for the backing file ops is overlayfs, so these changes make sense, but may not make sense in the most general case;For the numbers, some from 'perf c2c', before this series:
(edited to fit)#
# ----- HITM ----- Shared # Num RmtHitm LclHitm Symbol Object Source:Line Node # ..... ....... ....... .......................... ................ .................. .... # ------------------------- 0 412 1028 ------------------------- 41.50% 42.22% [k] revert_creds [kernel.vmlinux] atomic64_64.h:39 0 1 15.05% 10.60% [k] override_creds [kernel.vmlinux] atomic64_64.h:25 0 1 0.73% 0.58% [k] init_file [kernel.vmlinux] atomic64_64.h:25 0 1 0.24% 0.10% [k] revert_creds [kernel.vmlinux] cred.h:266 0 1 32.28% 37.16% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1 9.47% 8.75% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1 0.49% 0.58% [k] inode_owner_or_capable [kernel.vmlinux] mnt_idmapping.h:81 0 1 0.24% 0.00% [k] generic_permission [kernel.vmlinux] namei.c:354 0-------------------------
1 50 103 ------------------------- 100.00% 100.00% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1-------------------------
2 50 98 ------------------------- 96.00% 96.94% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1 2.00% 1.02% [k] update_load_avg [kernel.vmlinux] atomic64_64.h:25 0 1 0.00% 2.04% [k] update_load_avg [kernel.vmlinux] fair.c:4118 0 2.00% 0.00% [k] update_cfs_group [kernel.vmlinux] fair.c:3932 0 1after this series:
#
# ----- HITM ----- Shared # Num RmtHitm LclHitm Symbol Object Source:Line Node # ..... ....... ....... .................... ................ ................ .... # ------------------------- 0 54 88 ------------------------- 100.00% 100.00% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1-------------------------
1 48 83 ------------------------- 97.92% 97.59% [k] update_cfs_group [kernel.vmlinux] atomic64_64.h:15 0 1 2.08% 1.20% [k] update_load_avg [kernel.vmlinux] atomic64_64.h:25 0 1 0.00% 1.20% [k] update_load_avg [kernel.vmlinux] fair.c:4118 0 1-------------------------
2 28 44 ------------------------- 85.71% 79.55% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1 14.29% 20.45% [k] generic_permission [kernel.vmlinux] mnt_idmapping.h:81 0 1
Quoted 7 lines
Quoted 1 line
The contention is practically gone.
That is very impressive.
Can you say which workloads were running during this test? Specifically, I am wondering how much of the improvement came frombacking_file.cand how much from overlayfs/*.c.
I received the workload packaged from one of our customer teams, it's a docker image to run a wordpress/php/nginx thing, totally not my area and not sure that I can give much more details. The only think that I know is that this workload does *a lot* of faccessat().
Anyway, I did a experiment removing the backing ops patch, got this numbers (edited for clarity):
#
# ----- HITM ----- ------- Store Refs ------ Shared # Num RmtHitm LclHitm L1 Hit L1 Miss N/A Symbol Object Source:Line Node # ..... ....... ....... ....... ....... ....... .......................... ................. .................. ....
#
---------------------------------------------------
0 79 97 0 0 0
---------------------------------------------------
0.00% 1.03% 0.00% 0.00% 0.00% [k] revert_creds [kernel.kallsyms] atomic64_64.h:39 0 1
1.27% 0.00% 0.00% 0.00% 0.00% [k] init_file [kernel.kallsyms] atomic64_64.h:25 0 1
62.03% 71.13% 0.00% 0.00% 0.00% [k] generic_permission [kernel.kallsyms] mnt_idmapping.h:81 0 1
35.44% 26.80% 0.00% 0.00% 0.00% [k] generic_permission [kernel.kallsyms] mnt_idmapping.h:81 0 1
1.27% 0.00% 0.00% 0.00% 0.00% [k] generic_permission [kernel.kallsyms] mnt_idmapping.h:81 0
0.00% 1.03% 0.00% 0.00% 0.00% [k] generic_permission [kernel.kallsyms] namei.c:354 0 1 ---------------------------------------------------
1 52 103 0 0 0
---------------------------------------------------
98.08% 98.06% 0.00% 0.00% 0.00% [k] update_cfs_group [kernel.kallsyms] atomic64_64.h:15 0 1
0.00% 1.94% 0.00% 0.00% 0.00% [k] update_load_avg [kernel.kallsyms] atomic64_64.h:25 0 1
1.92% 0.00% 0.00% 0.00% 0.00% [k] update_cfs_group [kernel.kallsyms] fair.c:3932 0 ---------------------------------------------------
2 59 77 0 0 0
---------------------------------------------------
93.22% 98.70% 0.00% 0.00% 0.00% [k] update_cfs_group [kernel.kallsyms] atomic64_64.h:15 0 1
5.08% 1.30% 0.00% 0.00% 0.00% [k] update_cfs_group [kernel.kallsyms] fair.c:3932 0 1
1.69% 0.00% 0.00% 0.00% 0.00% [k] update_load_avg [kernel.kallsyms] atomic64_64.h:25 0 1So, the main source of contention is not in the backing file ops (but there is still some). That seems to align with the numbers that Chen Hu provided[1], that most of the contention was in ovl_permission().
[1] https://lore.kernel.org/all/[email protected]/
Quoted 8 lines
The reason I am asking is because the overlayfs patch is quite large and can take more time to review, so I am wondering out loud if we are not better off this
course of action:1. convert
backing_file.cto use new helpers/guards
2. convert overlayfs to use new helpers/guards
For this particular workload, (2) is more important. But I am open to propose (1) first, no problem at all.
Also, if you think that some other way of spliting the series, for example, one patch per function being converted, would be easier/better, I can do that too.
Quoted 3 lines
#1 should definitely go in via Christian's tree and should get a wider review from fsdevel (please CC fsdevel next time)
Of course. Will CC fsdevel.
Quoted 7 lines
#2 is contained for overlayfs reviewers. Once the helpers are merged and used by
backing_filehelpers, overlayfs can be converted independently.#1 and #2 could both be merged in the same merge cycle, or not, it does not matter. Most likely, #2 will go through Christian's tree as well, but I think we need to work according to this merge order.
Quoted 6 lines
We can also work on the review in parallel and you may keep the overlayfs patch in following posts, just wanted us to be on the same page w.r.t to the process.
Thanks,
Amir.
Cheers,
Signature · 1 line
Vinicius