Skip to contentSkip to conversation

sctp: fix refcount bug in sctp_wfree

PATCH Superseded Qiujun Huang 3 participants 9 messages Mar 20, 2020 – Mar 21, 2020

Subsystems (3)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 sctp: fix refcount bug in sctp_wfree

Do accounting for skb's real sk.
In some case skb->sk != asoc->base.sk:

for the trouble SKB, it was in outq->transmitted queue

sctp_outq_sack

	sctp_check_transmitted
		SKB was moved to outq->sack
	then throw away the sack queue
		SKB was deleted from outq->sack

(but the datamsg held SKB at sctp_datamsg_to_asoc
So, sctp_wfree was not called to destroy SKB)

then migrate happened

	sctp_for_each_tx_datachunk(
	sctp_clear_owner_w);
	sctp_assoc_migrate();
	sctp_for_each_tx_datachunk(
	sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

finally

__sctp_outq_teardown
sctp_chunk_put (for another skb)
sctp_datamsg_put
__kfree_skb(msg->frag_list)
sctp_wfree (for SKB)
this case in sctp_wfree SKB->sk was oldsk.

It looks only trouble here so handling it in sctp_wfree is enough.

Reported-and-tested-by [email protected]
---
 net/sctp/socket.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/sctp/socket.c +2 -2
index 1b56fc440606..5f5c28b30e25 100644
@@ -9080,7 +9080,7 @@ static void sctp_wfree(struct sk_buff *skb)
 {
 	struct sctp_chunk *chunk = skb_shinfo(skb)->destructor_arg;
 	struct sctp_association *asoc = chunk->asoc;
-	struct sock *sk = asoc->base.sk;
+	struct sock *sk = skb->sk;
 
 	sk_mem_uncharge(sk, skb->truesize);
 	sk->sk_wmem_queued -= skb->truesize + sizeof(struct sctp_chunk);
@@ -9109,7 +9109,7 @@ static void sctp_wfree(struct sk_buff *skb)
 	}
 
 	sock_wfree(skb);
-	sctp_wake_up_waiters(sk, asoc);
+	sctp_wake_up_waiters(asoc->base.sk, asoc);
 
 	sctp_association_put(asoc);
 }
Signature · 1 line
2.17.1
#1 lore raw
Quoted 54 lines · Qiujun Huang

Do accounting for skb's real sk.
In some case skb->sk != asoc->base.sk:

for the trouble SKB, it was in outq->transmitted queue

sctp_outq_sack

	sctp_check_transmitted
		SKB was moved to outq->sack
	then throw away the sack queue
		SKB was deleted from outq->sack

(but the datamsg held SKB at sctp_datamsg_to_asoc
So, sctp_wfree was not called to destroy SKB)

then migrate happened

	sctp_for_each_tx_datachunk(
	sctp_clear_owner_w);
	sctp_assoc_migrate();
	sctp_for_each_tx_datachunk(
	sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

finally

__sctp_outq_teardown
sctp_chunk_put (for another skb)
sctp_datamsg_put
__kfree_skb(msg->frag_list)
sctp_wfree (for SKB)
this case in sctp_wfree SKB->sk was oldsk.

It looks only trouble here so handling it in sctp_wfree is enough.

Reported-and-tested-by [email protected]
---
 net/sctp/socket.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/sctp/socket.c +1 -1
index 1b56fc440606..5f5c28b30e25 100644
@@ -9080,7 +9080,7 @@ static void sctp_wfree(struct sk_buff *skb)
 {
 	struct sctp_chunk *chunk = skb_shinfo(skb)->destructor_arg;
 	struct sctp_association *asoc = chunk->asoc;
-	struct sock *sk = asoc->base.sk;
+	struct sock *sk = skb->sk;
 
 	sk_mem_uncharge(sk, skb->truesize);
 	sk->sk_wmem_queued -= skb->truesize + sizeof(struct sctp_chunk);

net/sctp/socket.c static void sctp_wfree(struct sk_buff *skb) · line 9115

Quoted 8 lines
net/sctp/socket.c +1 -1
@@ -9109,7 +9109,7 @@ static void sctp_wfree(struct sk_buff *skb)
 	}
 
 	sock_wfree(skb);
-	sctp_wake_up_waiters(sk, asoc);
+	sctp_wake_up_waiters(asoc->base.sk, asoc);
 
 	sctp_association_put(asoc);
 }

This does not really solve the issue.

Even if the particular syzbot repro is now fine.

Really, having anything _after_ the sock_wfree(skb) is the bug, since the current thread no longer own a reference on a socket.

#2 lore raw
Quoted 10 lines · Eric Dumazet

This does not really solve the issue.

Even if the particular syzbot repro is now fine.

Really, having anything _after_ the sock_wfree(skb) is the bug, since the current thread no longer own a reference on a socket.

I get it, thanks.

Quoted 4 lines
#3 lore raw
In reply to Qiujun Huang
Quoted 8 lines · Qiujun Huang

Do accounting for skb's real sk.
In some case skb->sk != asoc->base.sk:

for the trouble SKB, it was in outq->transmitted queue

sctp_outq_sack

	sctp_check_transmitted
		SKB was moved to outq->sack

There is no outq->sack. You mean outq->sacked, I assume.

Quoted 1 line

then throw away the sack queue

Where? How?
If you mean:

        /* Throw away stuff rotting on the sack queue.  */
        list_for_each_safe(lchunk, temp, &q->sacked) {
                tchunk = list_entry(lchunk, struct sctp_chunk,
                                    transmitted_list);
                tsn = ntohl(tchunk->subh.data_hdr->tsn);
                if (TSN_lte(tsn, ctsn)) {
                        list_del_init(&tchunk->transmitted_list);
                        if (asoc->peer.prsctp_capable &&
                            SCTP_PR_PRIO_ENABLED(chunk->sinfo.sinfo_flags))
                                asoc->sent_cnt_removable--;
                        sctp_chunk_free(tchunk);

Then sctp_chunk_free is supposed to free the datamsg as well for chunks that were cumulative-sacked.
For those not cumulative-sacked, sctp_for_each_tx_datachunk() will handle q->sacked queue as well:

        list_for_each_entry(chunk, &q->sacked, transmitted_list)
                cb(chunk);

So I don't see how skbs can be overlooked here.

Quoted 2 lines
		SKB was deleted from outq->sack

(but the datamsg held SKB at sctp_datamsg_to_asoc

You mean sctp_datamsg_from_user ? If so, isn't it the other way around? sctp_datamsg_assign() will hold the datamsg, not the skb.

Quoted 10 lines

So, sctp_wfree was not called to destroy SKB)

then migrate happened

	sctp_for_each_tx_datachunk(
	sctp_clear_owner_w);
	sctp_assoc_migrate();
	sctp_for_each_tx_datachunk(
	sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.

The 2nd sendto() call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.

Thanks,
Marcelo

#4 lore raw
Quoted 5 lines · Marcelo Ricardo Leitner
Quoted 3 lines · Qiujun Huang

Do accounting for skb's real sk.
In some case skb->sk != asoc->base.sk:

Quoted 7 lines
Quoted 5 lines

for the trouble SKB, it was in outq->transmitted queue

sctp_outq_sack

      sctp_check_transmitted
              SKB was moved to outq->sack

There is no outq->sack. You mean outq->sacked, I assume.

Yes, my typo.

Quoted 5 lines
Quoted 2 lines

then throw away the sack queue

Where? How?
If you mean:

Quoted 11 lines
        /* Throw away stuff rotting on the sack queue.  */
        list_for_each_safe(lchunk, temp, &q->sacked) {
                tchunk = list_entry(lchunk, struct sctp_chunk,
                                    transmitted_list);
                tsn = ntohl(tchunk->subh.data_hdr->tsn);
                if (TSN_lte(tsn, ctsn)) {
                        list_del_init(&tchunk->transmitted_list);
                        if (asoc->peer.prsctp_capable &&
                            SCTP_PR_PRIO_ENABLED(chunk->sinfo.sinfo_flags))
                                asoc->sent_cnt_removable--;
                        sctp_chunk_free(tchunk);

Yes, it was delected here.

Quoted 3 lines

Then sctp_chunk_free is supposed to free the datamsg as well for chunks that were cumulative-sacked.

Datamsg should be freed until all his chunks had been freed.

sctp_datamsg_from_user->sctp_datamsg_assign
every chunks holds datamsg.

Quoted 5 lines

For those not cumulative-sacked, sctp_for_each_tx_datachunk() will handle q->sacked queue as well:

        list_for_each_entry(chunk, &q->sacked, transmitted_list)
                cb(chunk);
Quoted 7 lines

So I don't see how skbs can be overlooked here.

Quoted 2 lines
              SKB was deleted from outq->sack

(but the datamsg held SKB at sctp_datamsg_to_asoc

You mean sctp_datamsg_from_user ? If so, isn't it the other way around? sctp_datamsg_assign() will hold the datamsg, not the skb.

yeah.

Quoted 23 lines
Quoted 11 lines

So, sctp_wfree was not called to destroy SKB)

then migrate happened

      sctp_for_each_tx_datachunk(
      sctp_clear_owner_w);
      sctp_assoc_migrate();
      sctp_for_each_tx_datachunk(
      sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.

The 2nd sendto() call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.

Thanks,
Marcelo

#5 lore raw
In reply to Marcelo Ricardo Leitner
Quoted 55 lines · Marcelo Ricardo Leitner
Quoted 8 lines · Qiujun Huang

Do accounting for skb's real sk.
In some case skb->sk != asoc->base.sk:

for the trouble SKB, it was in outq->transmitted queue

sctp_outq_sack

      sctp_check_transmitted
              SKB was moved to outq->sack

There is no outq->sack. You mean outq->sacked, I assume.

Quoted 1 line

then throw away the sack queue

Where? How?
If you mean:

        /* Throw away stuff rotting on the sack queue.  */
        list_for_each_safe(lchunk, temp, &q->sacked) {
                tchunk = list_entry(lchunk, struct sctp_chunk,
                                    transmitted_list);
                tsn = ntohl(tchunk->subh.data_hdr->tsn);
                if (TSN_lte(tsn, ctsn)) {
                        list_del_init(&tchunk->transmitted_list);
                        if (asoc->peer.prsctp_capable &&
                            SCTP_PR_PRIO_ENABLED(chunk->sinfo.sinfo_flags))
                                asoc->sent_cnt_removable--;
                        sctp_chunk_free(tchunk);

Then sctp_chunk_free is supposed to free the datamsg as well for chunks that were cumulative-sacked.
For those not cumulative-sacked, sctp_for_each_tx_datachunk() will handle q->sacked queue as well:

        list_for_each_entry(chunk, &q->sacked, transmitted_list)
                cb(chunk);

So I don't see how skbs can be overlooked here.

Quoted 2 lines
              SKB was deleted from outq->sack

(but the datamsg held SKB at sctp_datamsg_to_asoc

You mean sctp_datamsg_from_user ? If so, isn't it the other way around? sctp_datamsg_assign() will hold the datamsg, not the skb.

Quoted 10 lines

So, sctp_wfree was not called to destroy SKB)

then migrate happened

      sctp_for_each_tx_datachunk(
      sctp_clear_owner_w);
      sctp_assoc_migrate();
      sctp_for_each_tx_datachunk(
      sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

Quoted 8 lines

The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.

The 2nd sendto() call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.

in sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freed

list_for_each_entry(chunk, &datamsg->chunks, frag_list) { sctp_chunk_hold(chunk);
sctp_set_owner_w(chunk);
chunk->transport = transport;
}

any ideas to handle it?

Quoted 3 lines

Thanks,
Marcelo

#6 lore raw

On Sat, Mar 21, 2020 at 07:53:29AM +0800, Qiujun Huang wrote:
...

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines

So, sctp_wfree was not called to destroy SKB)

then migrate happened

      sctp_for_each_tx_datachunk(
      sctp_clear_owner_w);
      sctp_assoc_migrate();
      sctp_for_each_tx_datachunk(
      sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.

Quoted 8 lines
Quoted 5 lines

The 2nd sendto() call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.

in sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freed

Right! Now I see it, thanks.
In the end, it's not a locking race condition. It's just not iterating on the lists properly.

Quoted 8 lines

list_for_each_entry(chunk, &datamsg->chunks, frag_list) { sctp_chunk_hold(chunk);
sctp_set_owner_w(chunk);
chunk->transport = transport;
}

any ideas to handle it?

sctp_for_each_tx_datachunk() needs to be aware of this situation. Instead of iterating directly/only over the chunk list, it should iterate over the datamsgs instead. Something like the below (just compile tested).

Then, the old socket will be free to die regardless of the new one. Otherwise, if this association gets stuck on retransmissions or so, the old socket would not be freed till then.

net/sctp/socket.c +10 -3
index fed26a1e9518..85c742310d26 100644
@@ -151,9 +151,10 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,
 				       void (*cb)(struct sctp_chunk *))
 
 {
+	struct sctp_datamsg *msg, *prev_msg = NULL;
 	struct sctp_outq *q = &asoc->outqueue;
 	struct sctp_transport *t;
-	struct sctp_chunk *chunk;
+	struct sctp_chunk *chunk, *c;
 
 	list_for_each_entry(t, &asoc->peer.transport_addr_list, transports)
 		list_for_each_entry(chunk, &t->transmitted, transmitted_list)
@@ -162,8 +163,14 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,
 	list_for_each_entry(chunk, &q->retransmit, transmitted_list)
 		cb(chunk);
 
-	list_for_each_entry(chunk, &q->sacked, transmitted_list)
-		cb(chunk);
+	list_for_each_entry(chunk, &q->sacked, transmitted_list) {
+		msg = chunk->msg;
+		if (msg = prev_msg)
+			continue;
+		list_for_each_entry(c, &msg->chunks, frag_list)
+			cb(c);
+		prev_msg = msg;
+	}
 
 	list_for_each_entry(chunk, &q->abandoned, transmitted_list)
 		cb(chunk);
#7 lore raw
Quoted 68 lines · Marcelo Ricardo Leitner

On Sat, Mar 21, 2020 at 07:53:29AM +0800, Qiujun Huang wrote:
...

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines

So, sctp_wfree was not called to destroy SKB)

then migrate happened

      sctp_for_each_tx_datachunk(
      sctp_clear_owner_w);
      sctp_assoc_migrate();
      sctp_for_each_tx_datachunk(
      sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.

Quoted 8 lines
Quoted 5 lines

The 2nd sendto() call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.

in sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freed

Right! Now I see it, thanks.
In the end, it's not a locking race condition. It's just not iterating on the lists properly.

Quoted 8 lines

list_for_each_entry(chunk, &datamsg->chunks, frag_list) { sctp_chunk_hold(chunk);
sctp_set_owner_w(chunk);
chunk->transport = transport;
}

any ideas to handle it?

sctp_for_each_tx_datachunk() needs to be aware of this situation. Instead of iterating directly/only over the chunk list, it should iterate over the datamsgs instead. Something like the below (just compile tested).

Then, the old socket will be free to die regardless of the new one. Otherwise, if this association gets stuck on retransmissions or so, the old socket would not be freed till then.

net/sctp/socket.c +2 -1
index fed26a1e9518..85c742310d26 100644
@@ -151,9 +151,10 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,
                                       void (*cb)(struct sctp_chunk *))
 
 {
+       struct sctp_datamsg *msg, *prev_msg = NULL;
        struct sctp_outq *q = &asoc->outqueue;
        struct sctp_transport *t;
-       struct sctp_chunk *chunk;
+       struct sctp_chunk *chunk, *c;
 
        list_for_each_entry(t, &asoc->peer.transport_addr_list, transports)
                list_for_each_entry(chunk, &t->transmitted, transmitted_list)
@@ -162,8 +163,14 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,
        list_for_each_entry(chunk, &q->retransmit, transmitted_list)
                cb(chunk);
 

net/sctp/socket.c static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, · line 173

Quoted 10 lines
net/sctp/socket.c +8 -2
-       list_for_each_entry(chunk, &q->sacked, transmitted_list)
-               cb(chunk);
+       list_for_each_entry(chunk, &q->sacked, transmitted_list) {
+               msg = chunk->msg;
+               if (msg = prev_msg)
+                       continue;
+               list_for_each_entry(c, &msg->chunks, frag_list)
+                       cb(c);
+               prev_msg = msg;
+       }

great. I'll trigger a syzbot test. Thanks.

Quoted 3 lines
net/sctp/socket.c +0 -0
        list_for_each_entry(chunk, &q->abandoned, transmitted_list)
                cb(chunk);
#8 lore raw
Quoted 57 lines · Qiujun Huang
Quoted 55 lines · Marcelo Ricardo Leitner

On Sat, Mar 21, 2020 at 07:53:29AM +0800, Qiujun Huang wrote:
...

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines

So, sctp_wfree was not called to destroy SKB)

then migrate happened

      sctp_for_each_tx_datachunk(
      sctp_clear_owner_w);
      sctp_assoc_migrate();
      sctp_for_each_tx_datachunk(
      sctp_set_owner_w);

SKB was not in the outq, and was not changed to newsk

The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.

Quoted 8 lines
Quoted 5 lines

The 2nd sendto() call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.

in sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freed

Right! Now I see it, thanks.
In the end, it's not a locking race condition. It's just not iterating on the lists properly.

Quoted 8 lines

list_for_each_entry(chunk, &datamsg->chunks, frag_list) { sctp_chunk_hold(chunk);
sctp_set_owner_w(chunk);
chunk->transport = transport;
}

any ideas to handle it?

sctp_for_each_tx_datachunk() needs to be aware of this situation. Instead of iterating directly/only over the chunk list, it should iterate over the datamsgs instead. Something like the below (just compile tested).

Then, the old socket will be free to die regardless of the new one. Otherwise, if this association gets stuck on retransmissions or so, the old socket would not be freed till then.

net/sctp/socket.c +0 -0
index fed26a1e9518..85c742310d26 100644
@@ -151,9 +151,10 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,
                                       void (*cb)(struct sctp_chunk *))
 
Quoted 6 lines

net/sctp/socket.c static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, · line 157

Quoted 6 lines
net/sctp/socket.c +2 -1
 {
+       struct sctp_datamsg *msg, *prev_msg = NULL;
        struct sctp_outq *q = &asoc->outqueue;
        struct sctp_transport *t;
-       struct sctp_chunk *chunk;
+       struct sctp_chunk *chunk, *c;

I missed to swap some lines here, for reverse christmass-tree style, btw.

Quoted 7 lines
Quoted 7 lines
net/sctp/socket.c +0 -0
        list_for_each_entry(t, &asoc->peer.transport_addr_list, transports)
                list_for_each_entry(chunk, &t->transmitted, transmitted_list)
@@ -162,8 +163,14 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,
        list_for_each_entry(chunk, &q->retransmit, transmitted_list)
                cb(chunk);
 
Quoted 12 lines

net/sctp/socket.c static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, · line 173

Quoted 10 lines
net/sctp/socket.c +8 -2
-       list_for_each_entry(chunk, &q->sacked, transmitted_list)
-               cb(chunk);
+       list_for_each_entry(chunk, &q->sacked, transmitted_list) {
+               msg = chunk->msg;
+               if (msg = prev_msg)
+                       continue;
+               list_for_each_entry(c, &msg->chunks, frag_list)
+                       cb(c);
+               prev_msg = msg;
+       }

great. I'll trigger a syzbot test. Thanks.

Mind that it may need to handled on the other lists as well. I didn't check them :]

Quoted 4 lines
Quoted 4 lines
net/sctp/socket.c +0 -0
        list_for_each_entry(chunk, &q->abandoned, transmitted_list)
                cb(chunk);