SUNRPC: fix refcounting problems with auth_gss messages.
Subsystems (3)
PATCH 1/1 SUNRPC: fix refcounting problems with auth_gss messages.
There are two problems with refcounting of auth_gss messages.
First, the reference on the pipe->pipe list (taken by a call to rpc_queue_upcall()) is not counted. It seems to be assumed that a message in pipe->pipe will always also be in pipe->in_downcall, where it is correctly reference counted.
However there is no guaranty of this. I have a report of a NULL dereferences in rpc_pipe_read() which suggests a msg that has been freed is still on the pipe->pipe list.
One way I imagine this might happen is:
- message is queued for uid=U and auth->service=S1
- rpc.gssd reads this message and starts processing.
This removes the message from pipe->pipe
- message is queued for uid=U and auth->service=S2
- rpc.gssd replies to the first message. gss_pipe_downcall() calls __gss_find_upcall(pipe, U, NULL) and it finds the *second* message, as new messages are placed at the head of ->in_downcall, and the service type is not checked.
- This second message is removed from ->in_downcall and freed by gss_release_msg() (even though it is still on pipe->pipe)
- rpc.gssd tries to read another message, and dereferences a pointer to this message that has just been freed.
I fix this by incrementing the reference count before calling rpc_queue_upcall(), and decrementing it if that fails, or normally in gss_pipe_destroy_msg().
It seems strange that the reply doesn't target the message more precisely, but I don't know all the details. In any case, I think the reference counting irregularity became a measureable bug when the extra arg was added to __gss_find_upcall(), hence the Fixes: line below.
The second problem is that if rpc_queue_upcall() fails, the new message is not freed. gss_alloc_msg() set the ->count to 1, gss_add_msg() increments this to 2, gss_unhash_msg() decrements to 1, then the pointer is discarded so the memory never gets freed.
--- net/sunrpc/auth_gss/auth_gss.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
net/sunrpc/auth_gss/auth_gss.c +6 -1
@@ -541,9 +541,13 @@ gss_setup_upcall(struct gss_auth *gss_auth, struct rpc_cred *cred)return gss_new; gss_msg =gss_add_msg (gss_new);if (gss_msg == gss_new) { -int res =rpc_queue_upcall (gss_new->pipe, &gss_new->msg); +int res; +atomic_inc (&gss_msg->count); + res =rpc_queue_upcall (gss_new->pipe, &gss_new->msg);if (res) {gss_unhash_msg (gss_new); +atomic_dec (&gss_msg->count); +gss_release_msg (gss_new); gss_msg =ERR_PTR (res); } }else
@@ -836,6 +840,7 @@ gss_pipe_destroy_msg(struct rpc_pipe_msg *msg)warn_gssd ();gss_release_msg (gss_msg); } +gss_release_msg (gss_msg); } static void gss_pipe_dentry_destroy(struct dentry *dir,
Signature · 1 line
2.10.2
Quoted 12 lines · NeilBrown
There are two problems with refcounting of
auth_gssmessages.First, the reference on the
pipe->pipelist (taken by a call torpc_queue_upcall()) is not counted. It seems to be assumed that a message inpipe->pipewill always also be inpipe->in_downcall, where it is correctly reference counted.However there is no guaranty of this. I have a report of a NULL dereferences in
rpc_pipe_read()which suggests a msg that has been freed is still on thepipe->pipelist.
Quoted 9 lines
One way I imagine this might happen is:
- message is queued for uid=U and auth->service=S1
- rpc.gssd reads this message and starts processing. This removes the message frompipe->pipe
- message is queued for uid=U and auth->service=S2
- rpc.gssd replies to the first message.gss_pipe_downcall()calls__gss_find_upcall(pipe, U, NULL)and it finds the *second* message, as new messages are placed at the head of ->in_downcall, and the service type is not checked.
Correct "service" was/is not a part of the protocol between the kernel and gssd. So that's why the kernel can't match what's coming from gssd to a particular waiting upcall.
Quoted 4 lines
- This second message is removed from ->in_downcall and freed by
gss_release_msg()(even though it is still onpipe->pipe)
- rpc.gssd tries to read another message, and dereferences a pointer to this message that has just been freed.
Agreed. This is a problem.
Doesn't the problem still exist even with this patch because gss_add_msg() adds the msg onto the in_downcall() list? So gssd in __gss_fin_upcall() can find the 2nd upcall even before the 2nd msg is added to the pipe->pipe()?
In the perfect world, the real solution (even for the initial problem) would have been changing the gssd-kernel protocol to include "service" as a part of the upcall. But that's not going to happen...
The solution I can thinking of is something where the message is not on "in_downcall" list until it's consumed by the rpc_pipe. I'd need to some time to figure out how to do that...
Quoted 52 lines
I fix this by incrementing the reference count before calling
rpc_queue_upcall(), and decrementing it if that fails, or normally ingss_pipe_destroy_msg().
It seems strange that the reply doesn't target the message more precisely, but I don't know all the details. In any case, I think the reference counting irregularity became a measureable bug when the extra arg was added to__gss_find_upcall(), hence the Fixes: line below.The second problem is that if
rpc_queue_upcall()fails, the new message is not freed.gss_alloc_msg()set the ->count to 1,gss_add_msg()increments this to 2,gss_unhash_msg()decrements to 1, then the pointer is discarded so the memory never gets freed.Fixes 9130b8dbc6ac ("SUNRPC: allow for upcalls for same uid but different gss service")Signed-off-by NeilBrown <[email protected]>--- net/sunrpc/auth_gss/auth_gss.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)net/sunrpc/auth_gss/auth_gss.c +6
-2@@ -541,9 +541,13 @@ gss_setup_upcall(struct gss_auth *gss_auth, struct rpc_cred *cred)return gss_new; gss_msg =gss_add_msg (gss_new);if (gss_msg == gss_new) { -int res =rpc_queue_upcall (gss_new->pipe, &gss_new->msg); +int res; +atomic_inc (&gss_msg->count); + res =rpc_queue_upcall (gss_new->pipe, &gss_new->msg);if (res) {gss_unhash_msg (gss_new); +atomic_dec (&gss_msg->count); +gss_release_msg (gss_new); gss_msg =ERR_PTR (res); } }else @@ -836,6 +840,7 @@ gss_pipe_destroy_msg(struct rpc_pipe_msg *msg)warn_gssd ();gss_release_msg (gss_msg); } +gss_release_msg (gss_msg); } static void gss_pipe_dentry_destroy(struct dentry *dir, --2.10.2
Quoted 32 lines · Olga Kornievskaia
Quoted 12 lines · NeilBrown
There are two problems with refcounting of
auth_gssmessages.First, the reference on the
pipe->pipelist (taken by a call torpc_queue_upcall()) is not counted. It seems to be assumed that a message inpipe->pipewill always also be inpipe->in_downcall, where it is correctly reference counted.However there is no guaranty of this. I have a report of a NULL dereferences in
rpc_pipe_read()which suggests a msg that has been freed is still on thepipe->pipelist.Quoted 9 lines
One way I imagine this might happen is:
- message is queued for uid=U and auth->service=S1
- rpc.gssd reads this message and starts processing. This removes the message frompipe->pipe
- message is queued for uid=U and auth->service=S2
- rpc.gssd replies to the first message.gss_pipe_downcall()calls__gss_find_upcall(pipe, U, NULL)and it finds the *second* message, as new messages are placed at the head of ->in_downcall, and the service type is not checked.Correct "service" was/is not a part of the protocol between the kernel and gssd. So that's why the kernel can't match what's coming from gssd to a particular waiting upcall.
Quoted 4 lines
- This second message is removed from ->in_downcall and freed by
gss_release_msg()(even though it is still onpipe->pipe)
- rpc.gssd tries to read another message, and dereferences a pointer to this message that has just been freed.
Quoted 6 lines
Agreed. This is a problem.
Doesn't the problem still exist even with this patch because
gss_add_msg()adds the msg onto thein_downcall()list? So gssd in__gss_fin_upcall()can find the 2nd upcall even before the 2nd msg is added to thepipe->pipe()?
The use-after-free problem is solved I think. It doesn't really make any difference if the down-call arrives before or after rpc_queue_upcall() is called. The msg will still not be freed before it is removed from both lists.
Thanks,
NeilBrown
Quoted 41 lines · NeilBrown
Quoted 32 lines · Olga Kornievskaia
Quoted 12 lines · NeilBrown
There are two problems with refcounting of
auth_gssmessages.First, the reference on the
pipe->pipelist (taken by a call torpc_queue_upcall()) is not counted. It seems to be assumed that a message inpipe->pipewill always also be inpipe->in_downcall, where it is correctly reference counted.However there is no guaranty of this. I have a report of a NULL dereferences in
rpc_pipe_read()which suggests a msg that has been freed is still on thepipe->pipelist.Quoted 9 lines
One way I imagine this might happen is:
- message is queued for uid=U and auth->service=S1
- rpc.gssd reads this message and starts processing. This removes the message frompipe->pipe
- message is queued for uid=U and auth->service=S2
- rpc.gssd replies to the first message.gss_pipe_downcall()calls__gss_find_upcall(pipe, U, NULL)and it finds the *second* message, as new messages are placed at the head of ->in_downcall, and the service type is not checked.Correct "service" was/is not a part of the protocol between the kernel and gssd. So that's why the kernel can't match what's coming from gssd to a particular waiting upcall.
Quoted 4 lines
- This second message is removed from ->in_downcall and freed by
gss_release_msg()(even though it is still onpipe->pipe)
- rpc.gssd tries to read another message, and dereferences a pointer to this message that has just been freed.Quoted 6 lines
Agreed. This is a problem.
Doesn't the problem still exist even with this patch because
gss_add_msg()adds the msg onto thein_downcall()list? So gssd in__gss_fin_upcall()can find the 2nd upcall even before the 2nd msg is added to thepipe->pipe()?
Quoted 5 lines
The use-after-free problem is solved I think. It doesn't really make any difference if the down-call arrives before or after
rpc_queue_upcall()is called. The msg will still not be freed before it is removed from both lists.
Sorry I don't see it.
Thread 1 adds an upcall and it's getting processed by gssd. Thread 2 executes gss_add_msg() which puts the message on the in_downcall list. Context switch (before the atomic_inc()!). Upcall comes back from the gssd, finds msg from Thread2 in_downcall list. gss_release_msg() will dec the counter to 0 and will remove the msg.
Quoted 8 lines · Olga Kornievskaia
Quoted 8 lines
Quoted 7 lines
Agreed. This is a problem.
Doesn't the problem still exist even with this patch because
gss_add_msg()adds the msg onto thein_downcall()list? So gssd in__gss_fin_upcall()can find the 2nd upcall even before the 2nd msg is added to thepipe->pipe()?
Quoted 7 lines
Quoted 5 lines
The use-after-free problem is solved I think. It doesn't really make any difference if the down-call arrives before or after
rpc_queue_upcall()is called. The msg will still not be freed before it is removed from both lists.Sorry I don't see it.
Maybe we are looking at different code?
Quoted 4 lines
Thread 1 adds an upcall and it's getting processed by gssd. Thread 2 executes
gss_add_msg()which puts the message on thein_downcalllist. Context switch (before theatomic_inc()!).
gss_add_msg(), as of 4.9-rc8, is
spin_lock(&pipe->lock);
old = __gss_find_upcall(pipe, gss_msg->uid, gss_msg->auth);
if (old == NULL) {
atomic_inc(&gss_msg->count);
list_add(&gss_msg->list, &pipe->in_downcall);
} else
gss_msg = old;
spin_unlock(&pipe->lock);so the gss_msg is added to in_downcall *after* the atomic_inc(), and the whole is protected by pipe->lock anyway so even if the atomic_inc() were delayed by the CPU reordering things, there would be no risk of gss_pipe_downcall() finding a gss_msg which didn't have the ->count elevated.
NeilBrown
Quoted 3 lines
Upcall comes back from the gssd, finds msg from Thread2
in_downcalllist.gss_release_msg()will dec the counter to 0 and will remove the msg.
Quoted 34 lines · NeilBrown
Quoted 8 lines · Olga Kornievskaia
Quoted 8 lines
Quoted 7 lines
Agreed. This is a problem.
Doesn't the problem still exist even with this patch because
gss_add_msg()adds the msg onto thein_downcall()list? So gssd in__gss_fin_upcall()can find the 2nd upcall even before the 2nd msg is added to thepipe->pipe()?Quoted 7 lines
Quoted 5 lines
The use-after-free problem is solved I think. It doesn't really make any difference if the down-call arrives before or after
rpc_queue_upcall()is called. The msg will still not be freed before it is removed from both lists.Sorry I don't see it.
Maybe we are looking at different code?
Quoted 4 lines
Thread 1 adds an upcall and it's getting processed by gssd. Thread 2 executes
gss_add_msg()which puts the message on thein_downcalllist. Context switch (before theatomic_inc()!).
gss_add_msg(), as of 4.9-rc8, isspin_lock(&pipe->lock); old = __gss_find_upcall(pipe, gss_msg->uid, gss_msg->auth); if (old == NULL) { atomic_inc(&gss_msg->count); list_add(&gss_msg->list, &pipe->in_downcall); } else gss_msg = old; spin_unlock(&pipe->lock);
Quoted 6 lines
so the
gss_msgis added toin_downcall*after* theatomic_inc(), and the whole is protected bypipe->lockanyway so even if theatomic_inc()were delayed by the CPU reordering things, there would be no risk ofgss_pipe_downcall()finding agss_msgwhich didn't have the ->count elevated.
Ah, I missed the atomic_inc() in the gss_add_msg(). I was only looking at your patch that did another atomic_inc(). Makes sense now.
Quoted 5 lines
NeilBrown
Quoted 3 lines
Upcall comes back from the gssd, finds msg from Thread2
in_downcalllist.gss_release_msg()will dec the counter to 0 and will remove the msg.