Skip to contentSkip to conversation

net: 9p: fix possible refcount leak in p9_read_work() and recv_done()

PATCH In review Hangyu Hua 2 participants 5 messages Jul 11, 2022 – Jul 12, 2022

Subsystems (3)
lore
Share & tools
lore

Hangyu Hua

author
#0 lore raw

PATCH 1/1 net: 9p: fix possible refcount leak in p9_read_work() and recv_done()

A ref got in p9_tag_lookup needs to be put when functions enter the error path.

Fix this by adding p9_req_put in error path.

Fixes 728356dedeff ("9p: Add refcount to p9_req_t")
---
 net/9p/trans_fd.c   | 3 +++
 net/9p/trans_rdma.c | 1 +
 2 files changed, 4 insertions(+)
net/9p/trans_fd.c +3 -0
index 8f8f95e39b03..c4ccb7b9e1bf 100644
@@ -343,6 +343,7 @@ static void p9_read_work(struct work_struct *work)
 			p9_debug(P9_DEBUG_ERROR,
 				 "No recv fcall for tag %d (req %p), disconnecting!\n",
 				 m->rc.tag, m->rreq);
+			p9_req_put(m->rreq);
 			m->rreq = NULL;
 			err = -EIO;
 			goto error;
@@ -372,6 +373,8 @@ static void p9_read_work(struct work_struct *work)
 				 "Request tag %d errored out while we were reading the reply\n",
 				 m->rc.tag);
 			err = -EIO;
+			p9_req_put(m->rreq);
+			m->rreq = NULL;
 			goto error;
 		}
 		spin_unlock(&m->client->lock);
net/9p/trans_rdma.c +1 -0
index 88e563826674..82b5d6894ee2 100644
@@ -317,6 +317,7 @@ recv_done(struct ib_cq *cq, struct ib_wc *wc)
 	/* Check that we have not yet received a reply for this request.
 	 */
 	if (unlikely(req->rc.sdata)) {
+		p9_req_put(req);
 		pr_err("Duplicate reply for request %d", tag);
 		goto err_out;
 	}
Signature · 1 line
2.25.1
#1 lore raw

Hangyu Hua wrote on Mon, Jul 11, 2022 at 02:59:07PM +0800:

Quoted 4 lines

A ref got in p9_tag_lookup needs to be put when functions enter the error path.

Fix this by adding p9_req_put in error path.

I wish it was that simple.

Did you actually observe a leak?

Quoted 13 lines
net/9p/trans_fd.c +1 -0
index 8f8f95e39b03..c4ccb7b9e1bf 100644
@@ -343,6 +343,7 @@ static void p9_read_work(struct work_struct *work)
 			p9_debug(P9_DEBUG_ERROR,
 				 "No recv fcall for tag %d (req %p), disconnecting!\n",
 				 m->rc.tag, m->rreq);
+			p9_req_put(m->rreq);
 			m->rreq = NULL;
 			err = -EIO;
 			goto error;

net/9p/trans_fd.c static void p9_read_work(struct work_struct *work) · line 380

Quoted 8 lines
net/9p/trans_fd.c +2 -0
@@ -372,6 +373,8 @@ static void p9_read_work(struct work_struct *work)
 				 "Request tag %d errored out while we were reading the reply\n",
 				 m->rc.tag);
 			err = -EIO;
+			p9_req_put(m->rreq);
+			m->rreq = NULL;
 			goto error;
 		}
 		spin_unlock(&m->client->lock);

for tcp, we still have that request in m->req_list, so we should be calling p9_client_cb which will do the p9_req_put in p9_conn_cancel.

If you do it here, you'll get a refcount overflow and use after free.

net/9p/trans_rdma.c recv_done(struct ib_cq *cq, struct ib_wc *wc) · line 323

Quoted 12 lines
net/9p/trans_rdma.c +1 -0
index 88e563826674..82b5d6894ee2 100644
@@ -317,6 +317,7 @@ recv_done(struct ib_cq *cq, struct ib_wc *wc)
 	/* Check that we have not yet received a reply for this request.
 	 */
 	if (unlikely(req->rc.sdata)) {
+		p9_req_put(req);
 		pr_err("Duplicate reply for request %d", tag);
 		goto err_out;
 	}

This one isn't as clear cut, I see that they put the client in a FLUSHING state but nothing seems to acton on it... But if this happens we're already in the use after free realm -- it means rc.sdata was already set so the other thread could be calling p9_client_cb anytime if it already hasn't, and yet another thread will then do the final ref put and free this.
We shouldn't free this here as that would also be an overflow. The best possible thing to do at this point is just to stop using that pointer.

If you actually run into a problem with these refcounts (should get a warning on umount that something didn't get freed) then by all mean let's look further into it, but please don't send such patches without testing the error paths you're "fixing" -- I'm pretty sure a reproducer to hit these paths would bark errors in dmesg as refcount has an overflow check.

--
Dominique

author
#2 lore raw
Quoted 35 lines · On 2022/7/11 15:39, [email protected] wrote:

Hangyu Hua wrote on Mon, Jul 11, 2022 at 02:59:07PM +0800:

Quoted 4 lines

A ref got in p9_tag_lookup needs to be put when functions enter the error path.

Fix this by adding p9_req_put in error path.

I wish it was that simple.

Did you actually observe a leak? >

Quoted 13 lines
net/9p/trans_fd.c +1 -0
index 8f8f95e39b03..c4ccb7b9e1bf 100644
@@ -343,6 +343,7 @@ static void p9_read_work(struct work_struct *work)
  			p9_debug(P9_DEBUG_ERROR,
  				 "No recv fcall for tag %d (req %p), disconnecting!\n",
  				 m->rc.tag, m->rreq);
+			p9_req_put(m->rreq);
  			m->rreq = NULL;
  			err = -EIO;
  			goto error;

net/9p/trans_fd.c static void p9_read_work(struct work_struct *work) · line 380

Quoted 8 lines
net/9p/trans_fd.c +2 -0
@@ -372,6 +373,8 @@ static void p9_read_work(struct work_struct *work)
  				 "Request tag %d errored out while we were reading the reply\n",
  				 m->rc.tag);
  			err = -EIO;
+			p9_req_put(m->rreq);
+			m->rreq = NULL;
  			goto error;
  		}
  		spin_unlock(&m->client->lock);

for tcp, we still have that request in m->req_list, so we should be calling p9_client_cb which will do the p9_req_put in p9_conn_cancel.

Quoted 2 lines

If you do it here, you'll get a refcount overflow and use after free.

That's a little weird. If you are right, the three return paths of this function are inconsistent with the handling of refcount.

static void p9_read_work(struct work_struct *work) {
...

	if ((m->rreq) && (m->rc.offset == m->rc.capacity)) {
		p9_debug(P9_DEBUG_TRANS, "got new packet\n");
		m->rreq->rc.size = m->rc.offset;
		spin_lock(&m->client->lock);
		if (m->rreq->status == REQ_STATUS_SENT) {
			list_del(&m->rreq->req_list);
			p9_client_cb(m->client, m->rreq, REQ_STATUS_RCVD);	<---- [1]
		} else if (m->rreq->status == REQ_STATUS_FLSHD) {
			/* Ignore replies associated with a cancelled request. */
			p9_debug(P9_DEBUG_TRANS,
				 "Ignore replies associated with a cancelled request\n");	<---- [2]
		} else {
			spin_unlock(&m->client->lock);
			p9_debug(P9_DEBUG_ERROR,
				 "Request tag %d errored out while we were reading the reply\n",
				 m->rc.tag);
			err = -EIO;
			goto error;	<---- [3]
		}
		spin_unlock(&m->client->lock);
		m->rc.sdata = NULL;
		m->rc.offset = 0;
		m->rc.capacity = 0;
		p9_req_put(m->rreq);	<---- [4]
		m->rreq = NULL;
	}
...
error:
	p9_conn_cancel(m, err);		<---- [5]
	clear_bit(Rworksched, &m->wsched);
}

There are three return paths here, [1] and [2] and [3].

[1]: m->rreq will be put twice in [1] and [4]. And m->rreq will be deleted from the m->req_list in [1].

[2]: m->rreq will be put in [4]. And m->rreq will not be deleted from m->req_list.

[3]: m->rreq will be put in [5]. And m->rreq will be deleted from the m->req_list in [5].

If p9_tag_lookup keep the refcount of req which is in m->req_list. There will be a double put in return path [1] and a potential UAF in return path [2]. And this also means a req in m->req_list without getting refcount before p9_tag_lookup.

static void p9_write_work(struct work_struct *work) {
...

		list_move_tail(&req->req_list, &m->req_list);

		m->wbuf = req->tc.sdata;
		m->wsize = req->tc.size;
		m->wpos = 0;
		p9_req_get(req);

...
}

But if you check out p9_write_work, a refcount already get after list_move_tail. We don't need to rely on p9_tag_lookup to keep a list's refcount. Whatsmore, code comments in p9_tag_alloc also proves that the refcount get by p9_tag_lookup is a temporary refcount.

So i still think there may be a refcount leak.

Quoted 13 lines

net/9p/trans_rdma.c recv_done(struct ib_cq *cq, struct ib_wc *wc) · line 323

Quoted 12 lines
net/9p/trans_rdma.c +1 -0
index 88e563826674..82b5d6894ee2 100644
@@ -317,6 +317,7 @@ recv_done(struct ib_cq *cq, struct ib_wc *wc)
  	/* Check that we have not yet received a reply for this request.
  	 */
  	if (unlikely(req->rc.sdata)) {
+		p9_req_put(req);
  		pr_err("Duplicate reply for request %d", tag);
  		goto err_out;
  	}
Quoted 9 lines

This one isn't as clear cut, I see that they put the client in a FLUSHING state but nothing seems to acton on it... But if this happens we're already in the use after free realm -- it means rc.sdata was already set so the other thread could be calling p9_client_cb anytime if it already hasn't, and yet another thread will then do the final ref put and free this.
We shouldn't free this here as that would also be an overflow. The best possible thing to do at this point is just to stop using that pointer.

But p9_tag_lookup have a lock inside. Doesn't this mean p9_tag_lookup won't return a freed req? Otherwise we should fix the lock to avoid falling into the use after free realm.

Thanks,
Hangyu

Quoted 10 lines

If you actually run into a problem with these refcounts (should get a warning on umount that something didn't get freed) then by all mean let's look further into it, but please don't send such patches without testing the error paths you're "fixing" -- I'm pretty sure a reproducer to hit these paths would bark errors in dmesg as refcount has an overflow check.

--
Dominique

#3 lore raw

Hangyu Hua wrote on Tue, Jul 12, 2022 at 11:24:36AM +0800:

Quoted 38 lines

That's a little weird. If you are right, the three return paths of this function are inconsistent with the handling of refcount.

static void p9_read_work(struct work_struct *work) {
...

	if ((m->rreq) && (m->rc.offset == m->rc.capacity)) {
		p9_debug(P9_DEBUG_TRANS, "got new packet\n");
		m->rreq->rc.size = m->rc.offset;
		spin_lock(&m->client->lock);
		if (m->rreq->status == REQ_STATUS_SENT) {
			list_del(&m->rreq->req_list);
			p9_client_cb(m->client, m->rreq, REQ_STATUS_RCVD);	<---- [1]
		} else if (m->rreq->status == REQ_STATUS_FLSHD) {
			/* Ignore replies associated with a cancelled request. */
			p9_debug(P9_DEBUG_TRANS,
				 "Ignore replies associated with a cancelled request\n");	<---- [2]
		} else {
			spin_unlock(&m->client->lock);
			p9_debug(P9_DEBUG_ERROR,
				 "Request tag %d errored out while we were reading the reply\n",
				 m->rc.tag);
			err = -EIO;
			goto error;	<---- [3]
		}
		spin_unlock(&m->client->lock);
		m->rc.sdata = NULL;
		m->rc.offset = 0;
		m->rc.capacity = 0;
		p9_req_put(m->rreq);	<---- [4]
		m->rreq = NULL;
	}
...
error:
	p9_conn_cancel(m, err);		<---- [5]
	clear_bit(Rworksched, &m->wsched);
}
Quoted 6 lines

There are three return paths here, [1] and [2] and [3].
[1]: m->rreq will be put twice in [1] and [4]. And m->rreq will be deleted from the m->req_list in [1].

[2]: m->rreq will be put in [4]. And m->rreq will not be deleted from m->req_list.

when req status got put to FLUSHD the req was dropped from the list already and put in p9_fd_cancel, so we shouldn't put it here.

Quoted 2 lines

[3]: m->rreq will be put in [5]. And m->rreq will be deleted from the m->req_list in [5].

On this error case I really can't say anything: it depends on how the req got in this state in the first place -- more precisely is it still in req_list or not?

But even if it is and we leak it here, we return an error here, so the connection will be marked as disconnected and won't be usable anymore. The memory will be freed when the user umounts after that.

If we took the time to re-init the rreq->req_list everytime we could check if it's empty (don't think we can rely on it being poisoned), but I just don't think it's worth it: it's better to consume a bit more memory until umount than to risk a UAF.

(note: while writing this I noticed p9_tag_cleanup() in p9_client_destroy() only tracks requests still in the idr, so doesn't work for requests that went through p9_tag_remove().
We don't need p9_tag_remove() anymore so I've just gotten rid of it and we will catch these now)

Quoted 4 lines

If p9_tag_lookup keep the refcount of req which is in m->req_list. There will be a double put in return path [1] and a potential UAF in return path [2]. And this also means a req in m->req_list without getting refcount before p9_tag_lookup.

That is the nominal path, we'd notice immediately if there are too many puts there.
A request is initialized with two refs so that we can have one for the transport ((a), for fd, "is the request tracked in a list?") and one for the main thread ((b), p9_client_rpc which will put it at the end) Then you get a third ref from p9_tag_lookup that I was forgetting about, (c).

Going through [1] removes it from the list, and removes the associated ref (a), then through p9_client_cb which removes ref (c) and wakes up p9_client_rpc which takes the last ref (b), freeing the request.

Now you were correct on one of these error paths not described in your last mail: we -are- missing a p9_req_ut in the "No recv fcall for tag %d" error path shortly after p9_tag_lookup, for the ref obtained from p9_tag_lookup itself -- feel free to resend a patch with just that one. But once again the connection is now unusable and it'll be caught on umount so it's not the end of the world...

(I'd appreciate if you base the new patch on top of
https://github.com/martinetd/linux/commits/9p-next )

Quoted 10 lines

static void p9_write_work(struct work_struct *work) {
...

		list_move_tail(&req->req_list, &m->req_list);

		m->wbuf = req->tc.sdata;
		m->wsize = req->tc.size;
		m->wpos = 0;
		p9_req_get(req);
Quoted 6 lines

...
}

But if you check out p9_write_work, a refcount already get after list_move_tail. We don't need to rely on p9_tag_lookup to keep a list's refcount.

This refcount is because we are keeping a ref in m->wreq, and is freed when m->wreq is set back to null when the packet is done writing a few lines below (but possibly in another call of the function).

refs don't have to come from p9_tag_lookup, they're managing pointers
lifecycle: we're making a copy of the pointer, so we should increment the refcount so another thread can't free the req under us. In this case the p9_req_get() is under the trans fd m->client->lock where we got the req from the list, so req can't be freed between its obtention from the list and then; once the lock is dropped the req is protected by the ref.

Quoted 2 lines

Whatsmore, code comments in p9_tag_alloc also proves that the refcount get by p9_tag_lookup is a temporary refcount.

comments don't prove anything, but yes I forgot p9_tag_alloc takes a ref when I commented earlier, sorry.

Quoted 10 lines
Quoted 9 lines

This one isn't as clear cut, I see that they put the client in a FLUSHING state but nothing seems to acton on it... But if this happens we're already in the use after free realm -- it means rc.sdata was already set so the other thread could be calling p9_client_cb anytime if it already hasn't, and yet another thread will then do the final ref put and free this.
We shouldn't free this here as that would also be an overflow. The best possible thing to do at this point is just to stop using that pointer.

Quoted 3 lines

But p9_tag_lookup have a lock inside. Doesn't this mean p9_tag_lookup won't return a freed req? Otherwise we should fix the lock to avoid falling into the use after free realm.

Right, that falls into the p9_tag_lookup ref, I had implemented this better than I thought I did...

I agree that one is also more correct to add, although I'd really want to make some rdma setup and trigger a few errors to test.

--
Dominique

author
#4 lore raw
Quoted 85 lines · On 2022/7/12 14:06, [email protected] wrote:

Hangyu Hua wrote on Tue, Jul 12, 2022 at 11:24:36AM +0800:

Quoted 38 lines

That's a little weird. If you are right, the three return paths of this function are inconsistent with the handling of refcount.

static void p9_read_work(struct work_struct *work) {
...

     if ((m->rreq) && (m->rc.offset == m->rc.capacity)) {
             p9_debug(P9_DEBUG_TRANS, "got new packet\n");
             m->rreq->rc.size = m->rc.offset;
             spin_lock(&m->client->lock);
             if (m->rreq->status == REQ_STATUS_SENT) {
                     list_del(&m->rreq->req_list);
                     p9_client_cb(m->client, m->rreq, REQ_STATUS_RCVD);      <---- [1]
             } else if (m->rreq->status == REQ_STATUS_FLSHD) {
                     /* Ignore replies associated with a cancelled request. */
                     p9_debug(P9_DEBUG_TRANS,
                              "Ignore replies associated with a cancelled request\n");       <---- [2]
             } else {
                     spin_unlock(&m->client->lock);
                     p9_debug(P9_DEBUG_ERROR,
                              "Request tag %d errored out while we were reading the reply\n",
                              m->rc.tag);
                     err = -EIO;
                     goto error;     <---- [3]
             }
             spin_unlock(&m->client->lock);
             m->rc.sdata = NULL;
             m->rc.offset = 0;
             m->rc.capacity = 0;
             p9_req_put(m->rreq);    <---- [4]
             m->rreq = NULL;
     }
...
error:
     p9_conn_cancel(m, err);         <---- [5]
     clear_bit(Rworksched, &m->wsched);
}
Quoted 6 lines

There are three return paths here, [1] and [2] and [3].
[1]: m->rreq will be put twice in [1] and [4]. And m->rreq will be deleted from the m->req_list in [1].

[2]: m->rreq will be put in [4]. And m->rreq will not be deleted from m->req_list.

when req status got put to FLUSHD the req was dropped from the list already and put in p9_fd_cancel, so we shouldn't put it here.

Quoted 2 lines

[3]: m->rreq will be put in [5]. And m->rreq will be deleted from the m->req_list in [5].

On this error case I really can't say anything: it depends on how the req got in this state in the first place -- more precisely is it still in req_list or not?

But even if it is and we leak it here, we return an error here, so the connection will be marked as disconnected and won't be usable anymore. The memory will be freed when the user umounts after that.

If we took the time to re-init the rreq->req_list everytime we could check if it's empty (don't think we can rely on it being poisoned), but I just don't think it's worth it: it's better to consume a bit more memory until umount than to risk a UAF.

(note: while writing this I noticed p9_tag_cleanup() in p9_client_destroy() only tracks requests still in the idr, so doesn't work for requests that went through p9_tag_remove().
We don't need p9_tag_remove() anymore so I've just gotten rid of it and we will catch these now)

Quoted 4 lines

If p9_tag_lookup keep the refcount of req which is in m->req_list. There will be a double put in return path [1] and a potential UAF in return path [2]. And this also means a req in m->req_list without getting refcount before p9_tag_lookup.

That is the nominal path, we'd notice immediately if there are too many puts there.
A request is initialized with two refs so that we can have one for the transport ((a), for fd, "is the request tracked in a list?") and one for the main thread ((b), p9_client_rpc which will put it at the end) Then you get a third ref from p9_tag_lookup that I was forgetting about, (c).

Quoted 4 lines

Going through [1] removes it from the list, and removes the associated ref (a), then through p9_client_cb which removes ref (c) and wakes up p9_client_rpc which takes the last ref (b), freeing the request.

I think the normal path is right beacuase p9_tag_lookup and [4] keep the balance of refcount. This just proves that error path may have refcount leak. Beacause error path only put refcount once. In general, either [4] of the normal path is redundant(like you said, this is easy to catch), or the error path may have a refcount leak.

But you are right, it'll be caught on umount.

Quoted 11 lines

Now you were correct on one of these error paths not described in your last mail: we -are- missing a p9_req_ut in the "No recv fcall for tag %d" error path shortly after p9_tag_lookup, for the ref obtained from p9_tag_lookup itself -- feel free to resend a patch with just that one. But once again the connection is now unusable and it'll be caught on umount so it's not the end of the world...

(I'd appreciate if you base the new patch on top of
https://github.com/martinetd/linux/commits/9p-next )

I see. I will make a new patch later.

Quoted 21 lines
Quoted 10 lines

static void p9_write_work(struct work_struct *work) {
...

             list_move_tail(&req->req_list, &m->req_list);

             m->wbuf = req->tc.sdata;
             m->wsize = req->tc.size;
             m->wpos = 0;
             p9_req_get(req);
Quoted 6 lines

...
}

But if you check out p9_write_work, a refcount already get after list_move_tail. We don't need to rely on p9_tag_lookup to keep a list's refcount.

This refcount is because we are keeping a ref in m->wreq, and is freed when m->wreq is set back to null when the packet is done writing a few lines below (but possibly in another call of the function).

Quoted 8 lines

refs don't have to come from p9_tag_lookup, they're managing pointers
lifecycle: we're making a copy of the pointer, so we should increment the refcount so another thread can't free the req under us. In this case the p9_req_get() is under the trans fd m->client->lock where we got the req from the list, so req can't be freed between its obtention from the list and then; once the lock is dropped the req is protected by the ref.

My fault. I misunderstood here.

Thanks,
Hangyu

Quoted 27 lines
Quoted 2 lines

Whatsmore, code comments in p9_tag_alloc also proves that the refcount get by p9_tag_lookup is a temporary refcount.

comments don't prove anything, but yes I forgot p9_tag_alloc takes a ref when I commented earlier, sorry.

Quoted 10 lines
Quoted 9 lines

This one isn't as clear cut, I see that they put the client in a FLUSHING state but nothing seems to acton on it... But if this happens we're already in the use after free realm -- it means rc.sdata was already set so the other thread could be calling p9_client_cb anytime if it already hasn't, and yet another thread will then do the final ref put and free this.
We shouldn't free this here as that would also be an overflow. The best possible thing to do at this point is just to stop using that pointer.

Quoted 3 lines

But p9_tag_lookup have a lock inside. Doesn't this mean p9_tag_lookup won't return a freed req? Otherwise we should fix the lock to avoid falling into the use after free realm.

Right, that falls into the p9_tag_lookup ref, I had implemented this better than I thought I did...

I agree that one is also more correct to add, although I'd really want to make some rdma setup and trigger a few errors to test.

Quoted 1 line

--

Dominique