ceph: Fix use-after-free in __ceph_remove_cap
PATCH 1/1 ceph: Fix use-after-free in __ceph_remove_cap
KASAN reports a use-after-free when running xfstest generic/531, with the following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9
This happens because __ceph_remove_cap() may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list with
rb_erase(&cap->ci_node, &ci->i_caps);
And, when this finally happens, the use-after-free will occur.
This can be fixed by protecting the rb_erase with the s_cap_lock spinlock, which is used by ceph_send_cap_releases(), before the cap is freed.
--- fs/ceph/caps.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/ceph/caps.c +2 -2
@@ -1089,13 +1089,13 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) } cap->cap_ino = ci->i_vino.ino; -spin_unlock (&session->s_cap_lock); -/* remove from inode list */ rb_erase (&cap->ci_node, &ci->i_caps);if (ci->i_auth_cap == cap) ci->i_auth_cap =NULL ; +spin_unlock (&session->s_cap_lock); +if (removed)ceph_put_cap (mdsc, cap);
Quoted 43 lines · Luis Henriques
KASAN reports a use-after-free when running xfstest generic/531, with the following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9This happens because
__ceph_remove_cap()may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list withrb_erase(&cap->ci_node, &ci->i_caps);And, when this finally happens, the use-after-free will occur.
This can be fixed by protecting the
rb_erasewith thes_cap_lockspinlock, which is used byceph_send_cap_releases(), before the cap is freed.Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/ceph/caps.c +0
-2@@ -1089,13 +1089,13 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) } cap->cap_ino = ci->i_vino.ino; -spin_unlock (&session->s_cap_lock); -
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) · line 1101
Quoted 10 lines
fs/ceph/caps.c +2
-0/* remove from inode list */ rb_erase (&cap->ci_node, &ci->i_caps);if (ci->i_auth_cap == cap) ci->i_auth_cap =NULL ; +spin_unlock (&session->s_cap_lock); +if (removed)ceph_put_cap (mdsc, cap);
Is there any reason we need to wait until this point to remove it from the rbtree? ISTM that we ought to just do that at the beginning of the function, before we take the s_cap_lock.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 49 lines · Jeff Layton
Quoted 48 lines · Luis Henriques
KASAN reports a use-after-free when running xfstest generic/531, with the following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9This happens because
__ceph_remove_cap()may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list withrb_erase(&cap->ci_node, &ci->i_caps);And, when this finally happens, the use-after-free will occur.
This can be fixed by protecting the
rb_erasewith thes_cap_lockspinlock, which is used byceph_send_cap_releases(), before the cap is freed.Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/ceph/caps.c +0
-2@@ -1089,13 +1089,13 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) } cap->cap_ino = ci->i_vino.ino; -spin_unlock (&session->s_cap_lock); -/* remove from inode list */ rb_erase (&cap->ci_node, &ci->i_caps);if (ci->i_auth_cap == cap) ci->i_auth_cap =NULL ;
Quoted 9 lines
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) · line 1101
Quoted 5 lines
fs/ceph/caps.c +2
-0+spin_unlock (&session->s_cap_lock); +if (removed)ceph_put_cap (mdsc, cap);Is there any reason we need to wait until this point to remove it from the rbtree? ISTM that we ought to just do that at the beginning of the function, before we take the
s_cap_lock.
That sounds good to me, at least at a first glace. I spent some time looking for any possible issues in the code, and even run a few tests.
However, looking at git log I found commit f818a73674c5 ("ceph: fix cap removal races"), which moved that rb_erase from the beginning of the function to it's current position. So, unless the race mentioned in this commit has disappeared in the meantime (which is possible, this commit is from 2010!), this rbtree operation shouldn't be changed.
And I now wonder if my patch isn't introducing a race too... __ceph_remove_cap() is supposed to always be called with the session mutex held, except for the ceph_evict_inode() path. Which is where I'm seeing the UAF. So, maybe what's missing here is the s_mutex. Hmm...
Cheers,
--
Luis
A message that is not in the archive (sent off-list or not yet received): CAAM7YA=dg8ufUWqrD_V8pSdvxrnU+knOW4uW4io_b=Lwjhpg5Q@mail.gmail.com
Quoted 76 lines · Zheng
Quoted 74 lines · Luis Henriques
Quoted 2 lines · Jeff Layton
Quoted 1 line · Luis Henriques
KASAN reports a use-after-free when running xfstest generic/531, with
the
Quoted 20 lines
Quoted 20 lines
following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9This happens because
__ceph_remove_cap()may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list withQuoted 5 lines
Quoted 5 lines
rb_erase(&cap->ci_node, &ci->i_caps);And, when this finally happens, the use-after-free will occur.
This can be fixed by protecting the
rb_erasewith thes_cap_lockspinlock,
Quoted 7 lines
Quoted 7 lines
which is used by
ceph_send_cap_releases(), before the cap is freed.Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Quoted 5 lines
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap,
Quoted 5 lines
fs/ceph/caps.c +0
-0@@ -1089,13 +1089,13 @@ void __ceph_remove_cap(struct ceph_cap *cap,bool
queue_release)Quoted 10 lines
Quoted 10 lines
fs/ceph/caps.c +0
-2} cap->cap_ino = ci->i_vino.ino; -spin_unlock (&session->s_cap_lock); -/* remove from inode list */ rb_erase (&cap->ci_node, &ci->i_caps);if (ci->i_auth_cap == cap) ci->i_auth_cap =NULL ;Quoted 9 lines
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) · line 1101
Quoted 5 lines
fs/ceph/caps.c +2
-0+spin_unlock (&session->s_cap_lock); +if (removed)ceph_put_cap (mdsc, cap);Is there any reason we need to wait until this point to remove it from the rbtree? ISTM that we ought to just do that at the beginning of the function, before we take the
s_cap_lock.That sounds good to me, at least at a first glace. I spent some time looking for any possible issues in the code, and even run a few tests.
However, looking at git log I found commit
f818a73674c5("ceph: fix cap removal races"), which moved thatrb_erasefrom the beginning of the function to it's current position. So, unless the race mentioned in this commit has disappeared in the meantime (which is possible, this commit is from 2010!), this rbtree operation shouldn't be changed.
Quoted 7 lines
Quoted 6 lines
And I now wonder if my patch isn't introducing a race too...
__ceph_remove_cap()is supposed to always be called with the session mutex held, except for theceph_evict_inode()path. Which is where I'm seeing the UAF. So, maybe what's missing here is thes_mutex. Hmm...we can't lock
s_mutexhere, becausei_ceph_lockis locked
Well, my idea wasn't to get s_mutex here but earlier in the stack. Maybe in ceph_evict_inode, protecting the call to __ceph_remove_caps. But I didn't really looked into that yet, so I'm not really sure if that's feasible (or even if that would fix this UAF). I suspect that's not possible anyway, due to the comment above __ceph_remove_cap:
caller will not hold session s_mutex if called from destroy_inode.
Cheers,
--
Luís
Quoted 83 lines · Luis Henriques
Quoted 82 lines · Zheng
Quoted 80 lines · Luis Henriques
Quoted 2 lines · Jeff Layton
Quoted 1 line · Luis Henriques
KASAN reports a use-after-free when running xfstest generic/531, with
the
Quoted 20 lines
Quoted 20 lines
following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9This happens because
__ceph_remove_cap()may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list withQuoted 5 lines
Quoted 5 lines
rb_erase(&cap->ci_node, &ci->i_caps);And, when this finally happens, the use-after-free will occur.
This can be fixed by protecting the
rb_erasewith thes_cap_lockspinlock,
Quoted 7 lines
Quoted 7 lines
which is used by
ceph_send_cap_releases(), before the cap is freed.Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Quoted 5 lines
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap,
Quoted 5 lines
fs/ceph/caps.c +0
-0@@ -1089,13 +1089,13 @@ void __ceph_remove_cap(struct ceph_cap *cap,bool
queue_release)Quoted 10 lines
Quoted 10 lines
fs/ceph/caps.c +0
-2} cap->cap_ino = ci->i_vino.ino; -spin_unlock (&session->s_cap_lock); -/* remove from inode list */ rb_erase (&cap->ci_node, &ci->i_caps);if (ci->i_auth_cap == cap) ci->i_auth_cap =NULL ;Quoted 9 lines
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) · line 1101
Quoted 5 lines
fs/ceph/caps.c +2
-0+spin_unlock (&session->s_cap_lock); +if (removed)ceph_put_cap (mdsc, cap);Is there any reason we need to wait until this point to remove it from the rbtree? ISTM that we ought to just do that at the beginning of the function, before we take the
s_cap_lock.That sounds good to me, at least at a first glace. I spent some time looking for any possible issues in the code, and even run a few tests.
However, looking at git log I found commit
f818a73674c5("ceph: fix cap removal races"), which moved thatrb_erasefrom the beginning of the function to it's current position. So, unless the race mentioned in this commit has disappeared in the meantime (which is possible, this commit is from 2010!), this rbtree operation shouldn't be changed.And I now wonder if my patch isn't introducing a race too...
__ceph_remove_cap()is supposed to always be called with the session mutex held, except for theceph_evict_inode()path. Which is where I'm seeing the UAF. So, maybe what's missing here is thes_mutex. Hmm...
Quoted 5 lines
Quoted 1 line
we can't lock
s_mutexhere, becausei_ceph_lockis lockedWell, my idea wasn't to get
s_mutexhere but earlier in the stack. Maybe inceph_evict_inode, protecting the call to__ceph_remove_caps. But I didn't really looked into that yet, so I'm not really sure if
Ok, I looked into that now and obviously that's not possible. So, I guess my original patch is still the best option.
Cheers,
Signature · 1 line
Luis
Quoted 4 lines
that's feasible (or even if that would fix this UAF). I suspect that's not possible anyway, due to the comment above
__ceph_remove_cap:caller will not hold session
s_mutexif called fromdestroy_inode.
Quoted 70 lines · Luis Henriques
Quoted 49 lines · Jeff Layton
Quoted 48 lines · Luis Henriques
KASAN reports a use-after-free when running xfstest generic/531, with the following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9This happens because
__ceph_remove_cap()may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list withrb_erase(&cap->ci_node, &ci->i_caps);And, when this finally happens, the use-after-free will occur.
This can be fixed by protecting the
rb_erasewith thes_cap_lockspinlock, which is used byceph_send_cap_releases(), before the cap is freed.Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/ceph/caps.c +0
-2@@ -1089,13 +1089,13 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) } cap->cap_ino = ci->i_vino.ino; -spin_unlock (&session->s_cap_lock); -/* remove from inode list */ rb_erase (&cap->ci_node, &ci->i_caps);if (ci->i_auth_cap == cap) ci->i_auth_cap =NULL ;Quoted 9 lines
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) · line 1101
Quoted 5 lines
fs/ceph/caps.c +2
-0+spin_unlock (&session->s_cap_lock); +if (removed)ceph_put_cap (mdsc, cap);Is there any reason we need to wait until this point to remove it from the rbtree? ISTM that we ought to just do that at the beginning of the function, before we take the
s_cap_lock.That sounds good to me, at least at a first glace. I spent some time looking for any possible issues in the code, and even run a few tests.
However, looking at git log I found commit
f818a73674c5("ceph: fix cap removal races"), which moved thatrb_erasefrom the beginning of the function to it's current position. So, unless the race mentioned in this commit has disappeared in the meantime (which is possible, this commit is from 2010!), this rbtree operation shouldn't be changed.
Quoted 5 lines
And I now wonder if my patch isn't introducing a race too...
__ceph_remove_cap()is supposed to always be called with the session mutex held, except for theceph_evict_inode()path. Which is where I'm seeing the UAF. So, maybe what's missing here is thes_mutex. Hmm...
I don't get it. That commit log talks about needing to ensure that the backpointer is cleared under the lock which is fine, but I don't see why we need to keep it in the inode's rbtree until that point.
Unhashing an object before you potentially free it is just good practice, IMO. If we need to do something different here, then I think it'd be good to add a comment explaining why.
Signature · 1 line
Jeff Layton <[email protected]>
PATCH v2 1/1 ceph: Fix use-after-free in __ceph_remove_cap
KASAN reports a use-after-free when running xfstest generic/531, with the following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9
This happens because __ceph_remove_cap() may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list with
rb_erase(&cap->ci_node, &ci->i_caps);
And, when this finally happens, the use-after-free will occur.
This can be fixed by removing the cap from the inode list before being removed from the session list, and thus eliminating the risk of an UAF.
---
Hi!
So, after spending some time trying to find possible races throught code review and testing, I modified the fix according to Jeff's suggestion.
Cheers,
Luis
fs/ceph/caps.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
fs/ceph/caps.c +5 -5
@@ -1058,6 +1058,11 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release)dout ("__ceph_remove_cap %p from %p\n" , cap, &ci->vfs_inode); +/* remove from inode list */ +rb_erase (&cap->ci_node, &ci->i_caps); +if (ci->i_auth_cap == cap) + ci->i_auth_cap =NULL ; +/* remove from session list */ spin_lock (&session->s_cap_lock);if (session->s_cap_iterator == cap) {
@@ -1091,11 +1096,6 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release)spin_unlock (&session->s_cap_lock); -/* remove from inode list */ -rb_erase (&cap->ci_node, &ci->i_caps); -if (ci->i_auth_cap == cap) - ci->i_auth_cap =NULL ; -if (removed)ceph_put_cap (mdsc, cap);
Quoted 59 lines · Luis Henriques
KASAN reports a use-after-free when running xfstest generic/531, with the following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9This happens because
__ceph_remove_cap()may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list withrb_erase(&cap->ci_node, &ci->i_caps);And, when this finally happens, the use-after-free will occur.
This can be fixed by removing the cap from the inode list before being removed from the session list, and thus eliminating the risk of an UAF.
Signed-off-by Luis Henriques <[email protected]>---
Hi!So, after spending some time trying to find possible races throught code review and testing, I modified the fix according to Jeff's suggestion.
Cheers,
Luis
fs/ceph/caps.c| 10 +++++-----1 file changed, 5 insertions(+), 5 deletions(-)fs/ceph/caps.c +5
-0@@ -1058,6 +1058,11 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release)dout ("__ceph_remove_cap %p from %p\n" , cap, &ci->vfs_inode); +/* remove from inode list */ +rb_erase (&cap->ci_node, &ci->i_caps); +if (ci->i_auth_cap == cap) + ci->i_auth_cap =NULL ; +/* remove from session list */ spin_lock (&session->s_cap_lock);if (session->s_cap_iterator == cap) {@@ -1091,11 +1096,6 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release)
fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) · line 1101
Quoted 10 lines
fs/ceph/caps.c +0
-5spin_unlock (&session->s_cap_lock); -/* remove from inode list */ -rb_erase (&cap->ci_node, &ci->i_caps); -if (ci->i_auth_cap == cap) - ci->i_auth_cap =NULL ; -if (removed)ceph_put_cap (mdsc, cap);
Looks good. Merged with a slight modification to the comment:
+ /* remove from inode's cap rbtree, and clear auth cap */
Thanks!
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 73 lines · Jeff Layton
Quoted 59 lines · Luis Henriques
KASAN reports a use-after-free when running xfstest generic/531, with the following trace:
[ 293.903362] kasan_report+0xe/0x20 [ 293.903365] rb_erase+0x1f/0x790 [ 293.903370] __ceph_remove_cap+0x201/0x370 [ 293.903375] __ceph_remove_caps+0x4b/0x70 [ 293.903380] ceph_evict_inode+0x4e/0x360 [ 293.903386] evict+0x169/0x290 [ 293.903390] __dentry_kill+0x16f/0x250 [ 293.903394] dput+0x1c6/0x440 [ 293.903398] __fput+0x184/0x330 [ 293.903404] task_work_run+0xb9/0xe0 [ 293.903410] exit_to_usermode_loop+0xd3/0xe0 [ 293.903413] do_syscall_64+0x1a0/0x1c0 [ 293.903417] entry_SYSCALL_64_after_hwframe+0x44/0xa9This happens because
__ceph_remove_cap()may queue a cap release (__ceph_queue_cap_release) which can be scheduled before that cap is removed from the inode list withrb_erase(&cap->ci_node, &ci->i_caps);And, when this finally happens, the use-after-free will occur.
This can be fixed by removing the cap from the inode list before being removed from the session list, and thus eliminating the risk of an UAF.
Signed-off-by Luis Henriques <[email protected]>---
Hi!So, after spending some time trying to find possible races throught code review and testing, I modified the fix according to Jeff's suggestion.
Cheers,
Luis
fs/ceph/caps.c| 10 +++++-----1 file changed, 5 insertions(+), 5 deletions(-)fs/ceph/caps.c +5
-0@@ -1058,6 +1058,11 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release)dout ("__ceph_remove_cap %p from %p\n" , cap, &ci->vfs_inode); +/* remove from inode list */ +rb_erase (&cap->ci_node, &ci->i_caps); +if (ci->i_auth_cap == cap) + ci->i_auth_cap =NULL ; +/* remove from session list */ spin_lock (&session->s_cap_lock);if (session->s_cap_iterator == cap) {@@ -1091,11 +1096,6 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release)fs/ceph/caps.c void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release) · line 1101
Quoted 10 lines
fs/ceph/caps.c +0
-5spin_unlock (&session->s_cap_lock); -/* remove from inode list */ -rb_erase (&cap->ci_node, &ci->i_caps); -if (ci->i_auth_cap == cap) - ci->i_auth_cap =NULL ; -if (removed)ceph_put_cap (mdsc, cap);Looks good. Merged with a slight modification to the comment:
Quoted 2 lines
+ /* remove from inode's cap rbtree, and clear auth cap */
Awesome, thanks. Regarding CC: stable@, feel free to add that too. I guess it makes sense, even if this issue has been there for a while apparently without any reports. (Although UAFs may have non-obvious side effects, I guess...)
Cheers,
--
Luís
Quoted 4 lines
Thanks!
Signature · 1 line
Jeff Layton <[email protected]>