fscrypt: improve encrypted symlink performance
Subsystems (6)
Eric Biggers
maintainerv2 cover fscrypt: improve encrypted symlink performance
This series improves the performance of following encrypted symlinks on ext4, f2fs, and ubifs.
Eric Biggers (2):
vfs: use READ_ONCE() to access ->i_link
fscrypt: cache decrypted symlink target in ->i_link
fs/crypto/hooks.c | 40 +++++++++++++++++++++++++++++++++------- fs/crypto/keyinfo.c | 21 +++++++++++++++++++++ fs/ext4/super.c | 3 +++ fs/f2fs/super.c | 3 +++ fs/namei.c | 4 ++-- fs/ubifs/super.c | 3 +++ include/linux/fscrypt.h | 5 +++++ 7 files changed, 70 insertions(+), 9 deletions(-)
Signature · 6 lines
2.21.0.392.gf8f6787159e-goog ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
Eric Biggers
maintainerPATCH v2 1/2 vfs: use READ_ONCE() to access ->i_link
From: Eric Biggers <[email protected]>
Use 'READ_ONCE(inode->i_link)' to explicitly support filesystems caching the symlink target in ->i_link later if it was unavailable at iget() time, or wasn't easily available. I'll be doing this in fscrypt, to improve the performance of encrypted symlinks on ext4, f2fs, and ubifs.
->i_link will start NULL and may later be set to a non-NULL value by a smp_store_release() or cmpxchg_release(). READ_ONCE() is needed on the read side. smp_load_acquire() is unnecessary because only a data dependency barrier is required. (Thanks to Al for pointing this out.)
--- fs/namei.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/namei.c +2 -2
@@ -1066,7 +1066,7 @@ const char *get_link(struct nameidata *nd)return ERR_PTR (error); nd->last_type =LAST_BIND ; - res = inode->i_link; + res =READ_ONCE (inode->i_link);if (!res) {const char * (*get)(struct dentry *,struct inode *,struct delayed_call *);
@@ -4729,7 +4729,7 @@ int vfs_readlink(struct dentry *dentry, char __user *buffer, int buflen)spin_unlock (&inode->i_lock); } - link = inode->i_link; + link =READ_ONCE (inode->i_link);if (!link) { link = inode->i_op->get_link (dentry, inode, &done);if (IS_ERR (link))
Signature · 8 lines
2.21.0.392.gf8f6787159e-goog _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Al Viro
maintainerRe: [PATCH v2 1/2] vfs: use READ_ONCE() to access ->i_link
Quoted 11 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Use '
READ_ONCE(inode->i_link)' to explicitly support filesystems caching the symlink target in ->i_link later if it was unavailable atiget()time, or wasn't easily available. I'll be doing this in fscrypt, to improve the performance of encrypted symlinks on ext4, f2fs, and ubifs.->i_link will start NULL and may later be set to a non-NULL value by a
smp_store_release()orcmpxchg_release().READ_ONCE()is needed on the read side.smp_load_acquire()is unnecessary because only a data dependency barrier is required. (Thanks to Al for pointing this out.)
You've missed vfs_readlink()...
Eric Biggers
maintainerRe: [PATCH v2 1/2] vfs: use READ_ONCE() to access ->i_link
Quoted 8 lines · Al Viro
Quoted 7 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Use '
READ_ONCE(inode->i_link)' to explicitly support filesystems caching the symlink target in ->i_link later if it was unavailable atiget()time, or wasn't easily available. I'll be doing this in fscrypt, to improve the performance of encrypted symlinks on ext4, f2fs, and ubifs.
Quoted 6 lines
Quoted 4 lines
->i_link will start NULL and may later be set to a non-NULL value by a
smp_store_release()orcmpxchg_release().READ_ONCE()is needed on the read side.smp_load_acquire()is unnecessary because only a data dependency barrier is required. (Thanks to Al for pointing this out.)You've missed
vfs_readlink()...
How so? The patch already updates vfs_readlink().
- Eric
Al Viro
maintainerRe: [PATCH v2 1/2] vfs: use READ_ONCE() to access ->i_link
Quoted 9 lines · Eric Biggers
Quoted 8 lines · Al Viro
Quoted 7 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Use '
READ_ONCE(inode->i_link)' to explicitly support filesystems caching the symlink target in ->i_link later if it was unavailable atiget()time, or wasn't easily available. I'll be doing this in fscrypt, to improve the performance of encrypted symlinks on ext4, f2fs, and ubifs.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
->i_link will start NULL and may later be set to a non-NULL value by a
smp_store_release()orcmpxchg_release().READ_ONCE()is needed on the read side.smp_load_acquire()is unnecessary because only a data dependency barrier is required. (Thanks to Al for pointing this out.)You've missed
vfs_readlink()...How so? The patch already updates
vfs_readlink().
Huh? Right you are - fsck knows how have I managed to misread that... Anyway, I can pick that READ_ONCE() patch through vfs.git if you wish, or it can go through your tree with my Acked-by - up to you...
Eric Biggers
maintainerRe: [PATCH v2 1/2] vfs: use READ_ONCE() to access ->i_link
Quoted 15 lines · Al Viro
Quoted 14 lines · Eric Biggers
Quoted 13 lines · Al Viro
Quoted 11 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Use '
READ_ONCE(inode->i_link)' to explicitly support filesystems caching the symlink target in ->i_link later if it was unavailable atiget()time, or wasn't easily available. I'll be doing this in fscrypt, to improve the performance of encrypted symlinks on ext4, f2fs, and ubifs.->i_link will start NULL and may later be set to a non-NULL value by a
smp_store_release()orcmpxchg_release().READ_ONCE()is needed on the read side.smp_load_acquire()is unnecessary because only a data dependency barrier is required. (Thanks to Al for pointing this out.)
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
You've missed
vfs_readlink()...How so? The patch already updates
vfs_readlink().Huh? Right you are - fsck knows how have I managed to misread that... Anyway, I can pick that
READ_ONCE()patch through vfs.git if you wish, or it can go through your tree with my Acked-by - up to you...
Unless you expect merge conflicts, it's easier for me to take it through the fscrypt tree, since it's a small patch and the second patch technically isn't correct without it. I'll add your Acked-by.
Are you planning to review the second patch too?
Thanks!
- Eric
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
Theodore Ts'o
maintainerRe: [PATCH v2 1/2] vfs: use READ_ONCE() to access ->i_link
In reply to Al ViroQuoted 15 lines · Al Viro
Quoted 14 lines · Eric Biggers
Quoted 13 lines · Al Viro
Quoted 11 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Use '
READ_ONCE(inode->i_link)' to explicitly support filesystems caching the symlink target in ->i_link later if it was unavailable atiget()time, or wasn't easily available. I'll be doing this in fscrypt, to improve the performance of encrypted symlinks on ext4, f2fs, and ubifs.->i_link will start NULL and may later be set to a non-NULL value by a
smp_store_release()orcmpxchg_release().READ_ONCE()is needed on the read side.smp_load_acquire()is unnecessary because only a data dependency barrier is required. (Thanks to Al for pointing this out.)
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
You've missed
vfs_readlink()...How so? The patch already updates
vfs_readlink().Huh? Right you are - fsck knows how have I managed to misread that... Anyway, I can pick that
READ_ONCE()patch through vfs.git if you wish, or it can go through your tree with my Acked-by - up to you...
Al, I'll take it through the fscrypt tree with your Acked-by, if you don't mind.
- Ted
Eric Biggers
maintainerPATCH v2 2/2 fscrypt: cache decrypted symlink target in ->i_link
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Path lookups that traverse encrypted symlink(s) are very slow because each encrypted symlink needs to be decrypted each time it's followed. This also involves dropping out of rcu-walk mode.
Make encrypted symlinks faster by caching the decrypted symlink target in ->i_link. The first call to fscrypt_get_symlink() sets it. Then, the existing VFS path lookup code uses the non-NULL ->i_link to take the fast path where ->get_link() isn't called, and lookups in rcu-walk mode remain in rcu-walk mode.
Also set ->i_link immediately when a new encrypted symlink is created.
To safely free the symlink target after an RCU grace period has elapsed, introduce a new function fscrypt_free_inode(), and make the relevant filesystems call it just before actually freeing the inode.
--- fs/crypto/hooks.c | 40 +++++++++++++++++++++++++++++++++------- fs/crypto/keyinfo.c | 21 +++++++++++++++++++++ fs/ext4/super.c | 3 +++ fs/f2fs/super.c | 3 +++ fs/ubifs/super.c | 3 +++ include/linux/fscrypt.h | 5 +++++ 6 files changed, 68 insertions(+), 7 deletions(-)
fs/crypto/hooks.c +33 -7
@@ -179,11 +179,9 @@ int __fscrypt_encrypt_symlink(struct inode *inode, const char *target, sd->len =cpu_to_le16 (ciphertext_len); err =fname_encrypt (inode, &iname, sd->encrypted_path, ciphertext_len); -if (err) { -if (!disk_link->name) -kfree (sd); -return err; - } +if (err) + goto err_free_sd; +/* * Null-terminating the ciphertext doesn't make sense, but we still * count the null terminator in the length, so we might as well
@@ -191,9 +189,20 @@ int __fscrypt_encrypt_symlink(struct inode *inode, const char *target,*/ sd->encrypted_path[ciphertext_len] ='\0' ; +/* Cache the plaintext symlink target for later use by get_link() */ + err = -ENOMEM ; + inode->i_link =kmemdup (target, len +1 ,GFP_NOFS ); +if (!inode->i_link) + goto err_free_sd; +if (!disk_link->name) disk_link->name = (unsigned char *)sd;return 0 ; + +err_free_sd: +if (!disk_link->name) +kfree (sd); +return err; }EXPORT_SYMBOL_GPL (__fscrypt_encrypt_symlink);
@@ -202,7 +211,7 @@ EXPORT_SYMBOL_GPL(__fscrypt_encrypt_symlink);* @inode: the symlink inode * @caddr: the on-disk contents of the symlink * @max_size: size of @caddr buffer -* @done: if successful, will be set up to free the returned target +* @done: if successful, will be set up to free the returned target if needed * * If the symlink's encryption key is available, we decrypt its target. * Otherwise, we encode its target for presentation.
@@ -217,12 +226,18 @@ const char *fscrypt_get_symlink(struct inode *inode, const void *caddr, {const struct fscrypt_symlink_data *sd;struct fscrypt_str cstr, pstr; +bool has_key;int err;/* This is for encrypted symlinks only */ if (WARN_ON (!IS_ENCRYPTED (inode)))return ERR_PTR (-EINVAL ); +/* If the decrypted target is already cached, just return it. */ + pstr.name =READ_ONCE (inode->i_link); +if (pstr.name) +return pstr.name; +/* * Try to set up the symlink's encryption key, but we can continue * regardless of whether the key is available or not.
@@ -230,6 +245,7 @@ const char *fscrypt_get_symlink(struct inode *inode, const void *caddr, err =fscrypt_get_encryption_info (inode);if (err)return ERR_PTR (err); + has_key =fscrypt_has_encryption_key (inode);/* * For historical reasons, encrypted symlink targets are prefixed with
@@ -261,7 +277,17 @@ const char *fscrypt_get_symlink(struct inode *inode, const void *caddr, goto err_kfree; pstr.name[pstr.len] ='\0' ; -set_delayed_call (done, kfree_link, pstr.name); + +/* +* Cache decrypted symlink targets in i_link for later use. Don't cache +* symlink targets encoded without the key, since those become outdated +* once the key is added. This pairs with the READ_ONCE() above and in +* the VFS path lookup code. +*/ +if (!has_key || +cmpxchg_release (&inode->i_link,NULL , pstr.name) !=NULL ) +set_delayed_call (done, kfree_link, pstr.name); +return pstr.name; err_kfree:
fs/crypto/keyinfo.c +21 -0
@@ -584,9 +584,30 @@ int fscrypt_get_encryption_info(struct inode *inode) }EXPORT_SYMBOL (fscrypt_get_encryption_info); +/** +* fscrypt_put_encryption_info - free most of an inode's fscrypt data +* +* Free the inode's fscrypt_info. Filesystems must call this when the inode is +* being evicted. An RCU grace period need not have elapsed yet. +*/ void fscrypt_put_encryption_info (struct inode *inode) {put_crypt_info (inode->i_crypt_info); inode->i_crypt_info =NULL ; }EXPORT_SYMBOL (fscrypt_put_encryption_info); + +/** +* fscrypt_free_inode - free an inode's fscrypt data requiring RCU delay +* +* Free the inode's cached decrypted symlink target, if any. Filesystems must +* call this after an RCU grace period, just before they free the inode. +*/ +void fscrypt_free_inode (struct inode *inode) +{ +if (IS_ENCRYPTED (inode) &&S_ISLNK (inode->i_mode)) { +kfree (inode->i_link); + inode->i_link =NULL ; + } +} +EXPORT_SYMBOL (fscrypt_free_inode);
fs/ext4/super.c +3 -0
@@ -1110,6 +1110,9 @@ static int ext4_drop_inode(struct inode *inode)static void ext4_i_callback (struct rcu_head *head) { struct inode *inode = container_of(head, struct inode, i_rcu); + +fscrypt_free_inode (inode); +kmem_cache_free (ext4_inode_cachep,EXT4_I (inode)); }
fs/f2fs/super.c +3 -0
@@ -1003,6 +1003,9 @@ static void f2fs_dirty_inode(struct inode *inode, int flags)static void f2fs_i_callback (struct rcu_head *head) { struct inode *inode = container_of(head, struct inode, i_rcu); + +fscrypt_free_inode (inode); +kmem_cache_free (f2fs_inode_cachep,F2FS_I (inode)); }
fs/ubifs/super.c +3 -0
@@ -276,7 +276,10 @@ static void ubifs_i_callback(struct rcu_head *head) { struct inode *inode = container_of(head, struct inode, i_rcu);struct ubifs_inode *ui =ubifs_inode (inode); +kfree (ui->data); +fscrypt_free_inode (inode); +kmem_cache_free (ubifs_inode_slab, ui); }
include/linux/fscrypt.h +5 -0
@@ -114,6 +114,7 @@ extern int fscrypt_inherit_context(struct inode *, struct inode *,/* keyinfo.c */ extern int fscrypt_get_encryption_info (struct inode *);extern void fscrypt_put_encryption_info (struct inode *); +extern void fscrypt_free_inode (struct inode *);/* fname.c */ extern int fscrypt_setup_filename(struct inode *, const struct qstr *,
@@ -322,6 +323,10 @@ static inline void fscrypt_put_encryption_info(struct inode *inode)return ; } +static inline void fscrypt_free_inode (struct inode *inode) +{ +} +/* fname.c */ static inline int fscrypt_setup_filename(struct inode *dir, const struct qstr *iname,
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
Theodore Ts'o
maintainerRe: [PATCH v2 2/2] fscrypt: cache decrypted symlink target in ->i_link
Quoted 12 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Path lookups that traverse encrypted symlink(s) are very slow because each encrypted symlink needs to be decrypted each time it's followed. This also involves dropping out of rcu-walk mode.
Make encrypted symlinks faster by caching the decrypted symlink target in ->i_link. The first call to
fscrypt_get_symlink()sets it. Then, the existing VFS path lookup code uses the non-NULL ->i_link to take the fast path where ->get_link() isn't called, and lookups in rcu-walk mode remain in rcu-walk mode.
Quoted 8 lines
Also set ->i_link immediately when a new encrypted symlink is created.
To safely free the symlink target after an RCU grace period has elapsed, introduce a new function
fscrypt_free_inode(), and make the relevant filesystems call it just before actually freeing the inode.Cc Al Viro <[email protected]>Signed-off-by Eric Biggers <[email protected]>
Looks good, applied.
- Ted