dcache: ensure d_flags & d_inode are consistent in lookup_fast()
PATCH 1/1 dcache: ensure d_flags & d_inode are consistent in lookup_fast()
After extending the size of dentry from 192-bytes to 208-bytes under aarch64, we got oops during the running of xfstests generic/429:
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000002 CPU: 3 PID: 2725 Comm: t_encrypted_d_r Tainted: G D 5.1.0-rc4 pc : inode_permission+0x28/0x160 lr : link_path_walk.part.11+0x27c/0x528 ...... Call trace: inode_permission+0x28/0x160 link_path_walk.part.11+0x27c/0x528 path_lookupat+0x64/0x208 filename_lookup+0xa0/0x178 user_path_at_empty+0x58/0x70 vfs_statx+0x94/0x118 __se_sys_newfstatat+0x58/0x98 __arm64_sys_newfstatat+0x24/0x30 el0_svc_common+0x7c/0x148 el0_svc_handler+0x38/0x88 el0_svc+0x8/0xc
If we revert the size extension of dentry, the oops will be gone. However if we just move the d_inode field from the begin of dentry struct to the end of dentry struct (poorly simulate the way how __randomize_layout works), the oops will reoccur.
The following scenario illustrates the problem:
precondition:
* dentry A has just been unlinked and becomes a negative dentry
* dentry A is encrypted, so it has d_revalidate hook: fscrypt_d_revalidate()
* lookup process is looking A/file, and creation process is creating A
lookup process: creation process:
lookup_fast
__d_lookup_rcu returns dentry A
d_revalidate returns -ECHILD
d_revalidate again succeed
__d_set_inode_and_type dentry->d_inode = inode WRITE_ONCE(dentry->d_flags, flags)
d_is_negative(dentry) return false
follow_managed doesn't nothing
// inconsistent with d_flags
d_backing_inode() return NULL
nd->inode = NULL
may_lookup()
// oops occurs
inode_permission(nd->inodeThe root cause is the inconsistency between d_flags & d_inode during the REF-walk in lookup_fast(): d_is_negative(dentry) returns false, but d_backing_inode() still returns a NULL pointer.
The RCU-walk path in lookup_fast() uses d_seq to ensure d_flags & d_inode are consistent, and lookup_slow() use inode lock to ensure that, so only the REF-walk path in lookup_fast() is problematic.
Fixing it by adding a paired smp_rmb/smp_wmb between the reading/writing of d_inode & d_flags to ensure the consistency.
--- fs/dcache.c | 2 ++ fs/namei.c | 7 +++++++ 2 files changed, 9 insertions(+)
fs/dcache.c +2 -0
@@ -316,6 +316,8 @@ static inline void __d_set_inode_and_type(struct dentry *dentry,unsigned flags; dentry->d_inode = inode; +/* paired with smp_rmb() in lookup_fast() */ +smp_wmb (); flags =READ_ONCE (dentry->d_flags); flags &= ~(DCACHE_ENTRY_TYPE |DCACHE_FALLTHRU ); flags |= type_flags;
fs/namei.c +7 -0
@@ -1628,6 +1628,13 @@ static int lookup_fast(struct nameidata *nd,return -ENOENT ; } +/* +* Paired with smp_wmb() in __d_set_inode_and_type() to ensure +* d_backing_inode is not NULL after the checking of d_flags +* in d_is_negative() completes. +*/ +smp_rmb (); + path->mnt = mnt; path->dentry = dentry; err =follow_managed (path, nd);
Signature · 1 line
2.16.2.dirty
ping ?
Quoted 104 lines · Hou Tao
After extending the size of dentry from 192-bytes to 208-bytes under aarch64, we got oops during the running of xfstests generic/429:
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000002 CPU: 3 PID: 2725 Comm: t_encrypted_d_r Tainted: G D 5.1.0-rc4 pc : inode_permission+0x28/0x160 lr : link_path_walk.part.11+0x27c/0x528 ...... Call trace: inode_permission+0x28/0x160 link_path_walk.part.11+0x27c/0x528 path_lookupat+0x64/0x208 filename_lookup+0xa0/0x178 user_path_at_empty+0x58/0x70 vfs_statx+0x94/0x118 __se_sys_newfstatat+0x58/0x98 __arm64_sys_newfstatat+0x24/0x30 el0_svc_common+0x7c/0x148 el0_svc_handler+0x38/0x88 el0_svc+0x8/0xcIf we revert the size extension of dentry, the oops will be gone. However if we just move the
d_inodefield from the begin of dentry struct to the end of dentry struct (poorly simulate the way how__randomize_layoutworks), the oops will reoccur.The following scenario illustrates the problem:
precondition:
* dentry A has just been unlinked and becomes a negative dentry
* dentry A is encrypted, so it hasd_revalidatehook:fscrypt_d_revalidate()
* lookup process is looking A/file, and creation process is creating Alookup process: creation process:
lookup_fast
__d_lookup_rcureturns dentry A
d_revalidatereturns-ECHILD
d_revalidateagain succeed
__d_set_inode_and_typedentry->d_inode= inodeWRITE_ONCE(dentry->d_flags, flags)
d_is_negative(dentry)return false
follow_manageddoesn't nothing
// inconsistent withd_flags
d_backing_inode()return NULL
nd->inode= NULL
may_lookup()// oops occurs inode_permission(nd->inodeThe root cause is the inconsistency between
d_flags&d_inodeduring the REF-walk inlookup_fast():d_is_negative(dentry)returns false, butd_backing_inode()still returns a NULL pointer.The RCU-walk path in
lookup_fast()usesd_seqto ensured_flags&d_inodeare consistent, andlookup_slow()use inode lock to ensure that, so only the REF-walk path inlookup_fast()is problematic.Fixing it by adding a paired smp_rmb/smp_wmb between the reading/writing of
d_inode&d_flagsto ensure the consistency.Signed-off-by Hou Tao <[email protected]>--- fs/dcache.c | 2 ++ fs/namei.c | 7 +++++++ 2 files changed, 9 insertions(+)fs/dcache.c +2
-0@@ -316,6 +316,8 @@ static inline void __d_set_inode_and_type(struct dentry *dentry,unsigned flags; dentry->d_inode = inode; +/* paired with smp_rmb() in lookup_fast() */ +smp_wmb (); flags =READ_ONCE (dentry->d_flags); flags &= ~(DCACHE_ENTRY_TYPE |DCACHE_FALLTHRU ); flags |= type_flags;fs/namei.c +7
-0@@ -1628,6 +1628,13 @@ static int lookup_fast(struct nameidata *nd,return -ENOENT ; } +/* +* Paired with smp_wmb() in __d_set_inode_and_type() to ensure +* d_backing_inode is not NULL after the checking of d_flags +* in d_is_negative() completes. +*/ +smp_rmb (); + path->mnt = mnt; path->dentry = dentry; err =follow_managed (path, nd);
Al Viro
maintainerQuoted 10 lines · Hou Tao
The root cause is the inconsistency between
d_flags&d_inodeduring the REF-walk inlookup_fast():d_is_negative(dentry)returns false, butd_backing_inode()still returns a NULL pointer.The RCU-walk path in
lookup_fast()usesd_seqto ensured_flags&d_inodeare consistent, andlookup_slow()use inode lock to ensure that, so only the REF-walk path inlookup_fast()is problematic.Fixing it by adding a paired smp_rmb/smp_wmb between the reading/writing of
d_inode&d_flagsto ensure the consistency.
The problem is real, but I'm not sure I like the proposed fix ;-/ We could simply use d_really_is_negative() there, avoiding all that mess. If and when we get around to whiteouts-in-dcache (i.e. if unionfs series gets resurrected), we can revisit that...