sctp: fix refcount bug in sctp_wfree
Subsystems (3)
PATCH v3 1/1 sctp: fix refcount bug in sctp_wfree
Do accounting for skb's real sk.
In some case skb->sk != asoc->base.sk:
for the trouble SKB, it was in outq->transmitted queue
sctp_outq_sack
sctp_check_transmitted SKB was moved to outq->sack then throw away the sack queue SKB was deleted from outq->sack
(but the datamsg held SKB at sctp_datamsg_to_asoc
So, sctp_wfree was not called to destroy SKB)
then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);
SKB was not in the outq, and was not changed to newsk
finally
__sctp_outq_teardown
sctp_chunk_put (for another skb)
sctp_datamsg_put
__kfree_skb(msg->frag_list)
sctp_wfree (for SKB)
this case in sctp_wfree SKB->sk was oldsk.
It looks only trouble here so handling it in sctp_wfree is enough.
--- net/sctp/socket.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/sctp/socket.c +2 -2
@@ -9080,7 +9080,7 @@ static void sctp_wfree(struct sk_buff *skb) {struct sctp_chunk *chunk =skb_shinfo (skb)->destructor_arg;struct sctp_association *asoc = chunk->asoc; -struct sock *sk = asoc->base.sk; +struct sock *sk = skb->sk;sk_mem_uncharge (sk, skb->truesize); sk->sk_wmem_queued -= skb->truesize +sizeof (struct sctp_chunk );
@@ -9109,7 +9109,7 @@ static void sctp_wfree(struct sk_buff *skb) }sock_wfree (skb); -sctp_wake_up_waiters (sk, asoc); +sctp_wake_up_waiters (asoc->base.sk, asoc);sctp_association_put (asoc); }
Signature · 1 line
2.17.1
Quoted 54 lines · Qiujun Huang
Do accounting for skb's real sk.
In some caseskb->sk!=asoc->base.sk:for the trouble SKB, it was in
outq->transmittedqueue
sctp_outq_sacksctp_check_transmitted SKB was moved to outq->sack then throw away the sack queue SKB was deleted from outq->sack(but the datamsg held SKB at
sctp_datamsg_to_asoc
So,sctp_wfreewas not called to destroy SKB)then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);SKB was not in the outq, and was not changed to newsk
finally
__sctp_outq_teardown
sctp_chunk_put(for another skb)
sctp_datamsg_put
__kfree_skb(msg->frag_list)
sctp_wfree(for SKB)
this case insctp_wfreeSKB->skwas oldsk.It looks only trouble here so handling it in
sctp_wfreeis enough.Reported-and-tested-by [email protected]Signed-off-by Qiujun Huang <[email protected]>--- net/sctp/socket.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)net/sctp/socket.c +1
-1@@ -9080,7 +9080,7 @@ static void sctp_wfree(struct sk_buff *skb) {struct sctp_chunk *chunk =skb_shinfo (skb)->destructor_arg;struct sctp_association *asoc = chunk->asoc; -struct sock *sk = asoc->base.sk; +struct sock *sk = skb->sk;sk_mem_uncharge (sk, skb->truesize); sk->sk_wmem_queued -= skb->truesize +sizeof (struct sctp_chunk );
net/sctp/socket.c static void sctp_wfree(struct sk_buff *skb) · line 9115
Quoted 8 lines
net/sctp/socket.c +1
-1@@ -9109,7 +9109,7 @@ static void sctp_wfree(struct sk_buff *skb) }sock_wfree (skb); -sctp_wake_up_waiters (sk, asoc); +sctp_wake_up_waiters (asoc->base.sk, asoc);sctp_association_put (asoc); }
This does not really solve the issue.
Even if the particular syzbot repro is now fine.
Really, having anything _after_ the sock_wfree(skb) is the bug, since the current thread no longer own a reference on a socket.
Marcelo Ricardo Leitner
maintainerQuoted 8 lines · Qiujun Huang
Do accounting for skb's real sk.
In some caseskb->sk!=asoc->base.sk:for the trouble SKB, it was in
outq->transmittedqueue
sctp_outq_sacksctp_check_transmitted SKB was moved to outq->sack
There is no outq->sack. You mean outq->sacked, I assume.
Quoted 1 line
then throw away the sack queue
Where? How?
If you mean:
/* Throw away stuff rotting on the sack queue. */
list_for_each_safe(lchunk, temp, &q->sacked) {
tchunk = list_entry(lchunk, struct sctp_chunk,
transmitted_list);
tsn = ntohl(tchunk->subh.data_hdr->tsn);
if (TSN_lte(tsn, ctsn)) {
list_del_init(&tchunk->transmitted_list);
if (asoc->peer.prsctp_capable &&
SCTP_PR_PRIO_ENABLED(chunk->sinfo.sinfo_flags))
asoc->sent_cnt_removable--;
sctp_chunk_free(tchunk);Then sctp_chunk_free is supposed to free the datamsg as well for chunks that were cumulative-sacked.
For those not cumulative-sacked, sctp_for_each_tx_datachunk() will handle q->sacked queue as well:
list_for_each_entry(chunk, &q->sacked, transmitted_list)
cb(chunk);So I don't see how skbs can be overlooked here.
Quoted 2 lines
SKB was deleted from outq->sack(but the datamsg held SKB at
sctp_datamsg_to_asoc
You mean sctp_datamsg_from_user ? If so, isn't it the other way around? sctp_datamsg_assign() will hold the datamsg, not the skb.
Quoted 10 lines
So,
sctp_wfreewas not called to destroy SKB)then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);SKB was not in the outq, and was not changed to newsk
The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.
The 2nd sendto() call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.
Thanks,
Marcelo
Quoted 10 lines · Eric Dumazet
This does not really solve the issue.
Even if the particular syzbot repro is now fine.
Really, having anything _after_ the
sock_wfree(skb)is the bug, since the current thread no longer own a reference on a socket.
I get it, thanks.
Quoted 4 lines
Quoted 5 lines · Marcelo Ricardo Leitner
Quoted 3 lines · Qiujun Huang
Do accounting for skb's real sk.
In some caseskb->sk!=asoc->base.sk:
Quoted 7 lines
Quoted 5 lines
for the trouble SKB, it was in
outq->transmittedqueue
sctp_outq_sacksctp_check_transmitted SKB was moved to outq->sackThere is no
outq->sack. You meanoutq->sacked, I assume.
Yes, my typo.
Quoted 5 lines
Quoted 2 lines
then throw away the sack queue
Where? How?
If you mean:
Quoted 11 lines
/* Throw away stuff rotting on the sack queue. */
list_for_each_safe(lchunk, temp, &q->sacked) {
tchunk = list_entry(lchunk, struct sctp_chunk,
transmitted_list);
tsn = ntohl(tchunk->subh.data_hdr->tsn);
if (TSN_lte(tsn, ctsn)) {
list_del_init(&tchunk->transmitted_list);
if (asoc->peer.prsctp_capable &&
SCTP_PR_PRIO_ENABLED(chunk->sinfo.sinfo_flags))
asoc->sent_cnt_removable--;
sctp_chunk_free(tchunk);Yes, it was delected here.
Quoted 3 lines
Then
sctp_chunk_freeis supposed to free the datamsg as well for chunks that were cumulative-sacked.
Datamsg should be freed until all his chunks had been freed.
sctp_datamsg_from_user->sctp_datamsg_assign
every chunks holds datamsg.
Quoted 5 lines
For those not cumulative-sacked,
sctp_for_each_tx_datachunk()will handleq->sackedqueue as well:list_for_each_entry(chunk, &q->sacked, transmitted_list) cb(chunk);
Quoted 7 lines
So I don't see how skbs can be overlooked here.
Quoted 2 lines
SKB was deleted from outq->sack(but the datamsg held SKB at
sctp_datamsg_to_asocYou mean
sctp_datamsg_from_user? If so, isn't it the other way around?sctp_datamsg_assign()will hold the datamsg, not the skb.
yeah.
Quoted 23 lines
Quoted 11 lines
So,
sctp_wfreewas not called to destroy SKB)then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);SKB was not in the outq, and was not changed to newsk
The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.
The 2nd
sendto()call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.Thanks,
Marcelo
Quoted 55 lines · Marcelo Ricardo Leitner
Quoted 8 lines · Qiujun Huang
Do accounting for skb's real sk.
In some caseskb->sk!=asoc->base.sk:for the trouble SKB, it was in
outq->transmittedqueue
sctp_outq_sacksctp_check_transmitted SKB was moved to outq->sackThere is no
outq->sack. You meanoutq->sacked, I assume.Quoted 1 line
then throw away the sack queue
Where? How?
If you mean:/* Throw away stuff rotting on the sack queue. */ list_for_each_safe(lchunk, temp, &q->sacked) { tchunk = list_entry(lchunk, struct sctp_chunk, transmitted_list); tsn = ntohl(tchunk->subh.data_hdr->tsn); if (TSN_lte(tsn, ctsn)) { list_del_init(&tchunk->transmitted_list); if (asoc->peer.prsctp_capable && SCTP_PR_PRIO_ENABLED(chunk->sinfo.sinfo_flags)) asoc->sent_cnt_removable--; sctp_chunk_free(tchunk);Then
sctp_chunk_freeis supposed to free the datamsg as well for chunks that were cumulative-sacked.
For those not cumulative-sacked,sctp_for_each_tx_datachunk()will handleq->sackedqueue as well:list_for_each_entry(chunk, &q->sacked, transmitted_list) cb(chunk);So I don't see how skbs can be overlooked here.
Quoted 2 lines
SKB was deleted from outq->sack(but the datamsg held SKB at
sctp_datamsg_to_asocYou mean
sctp_datamsg_from_user? If so, isn't it the other way around?sctp_datamsg_assign()will hold the datamsg, not the skb.Quoted 10 lines
So,
sctp_wfreewas not called to destroy SKB)then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);SKB was not in the outq, and was not changed to newsk
Quoted 8 lines
The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.
The 2nd
sendto()call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.
in sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freed
list_for_each_entry(chunk, &datamsg->chunks, frag_list) { sctp_chunk_hold(chunk);sctp_set_owner_w(chunk);chunk->transport = transport;
}
any ideas to handle it?
Quoted 3 lines
Thanks,
Marcelo
Marcelo Ricardo Leitner
maintainerOn Sat, Mar 21, 2020 at 07:53:29AM +0800, Qiujun Huang wrote:
...
Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
So,
sctp_wfreewas not called to destroy SKB)then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);SKB was not in the outq, and was not changed to newsk
The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.
Quoted 8 lines
Quoted 5 lines
The 2nd
sendto()call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.in
sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freed
Right! Now I see it, thanks.
In the end, it's not a locking race condition. It's just not iterating on the lists properly.
Quoted 8 lines
list_for_each_entry(chunk, &datamsg->chunks, frag_list){sctp_chunk_hold(chunk);sctp_set_owner_w(chunk);chunk->transport= transport;
}any ideas to handle it?
sctp_for_each_tx_datachunk() needs to be aware of this situation. Instead of iterating directly/only over the chunk list, it should iterate over the datamsgs instead. Something like the below (just compile tested).
Then, the old socket will be free to die regardless of the new one. Otherwise, if this association gets stuck on retransmissions or so, the old socket would not be freed till then.
net/sctp/socket.c +10 -3
@@ -151,9 +151,10 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, void (*cb)(struct sctp_chunk *)) { +struct sctp_datamsg *msg, *prev_msg =NULL ;struct sctp_outq *q = &asoc->outqueue;struct sctp_transport *t; -struct sctp_chunk *chunk; +struct sctp_chunk *chunk, *c;list_for_each_entry (t, &asoc->peer.transport_addr_list, transports)list_for_each_entry (chunk, &t->transmitted, transmitted_list)
@@ -162,8 +163,14 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,list_for_each_entry (chunk, &q->retransmit, transmitted_list)cb (chunk); -list_for_each_entry (chunk, &q->sacked, transmitted_list) -cb (chunk); +list_for_each_entry (chunk, &q->sacked, transmitted_list) { + msg = chunk->msg; +if (msg = prev_msg) +continue ; +list_for_each_entry (c, &msg->chunks, frag_list) +cb (c); + prev_msg = msg; + }list_for_each_entry (chunk, &q->abandoned, transmitted_list)cb (chunk);
Quoted 68 lines · Marcelo Ricardo Leitner
On Sat, Mar 21, 2020 at 07:53:29AM +0800, Qiujun Huang wrote:
...Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
So,
sctp_wfreewas not called to destroy SKB)then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);SKB was not in the outq, and was not changed to newsk
The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.
Quoted 8 lines
Quoted 5 lines
The 2nd
sendto()call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.in
sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freedRight! Now I see it, thanks.
In the end, it's not a locking race condition. It's just not iterating on the lists properly.Quoted 8 lines
list_for_each_entry(chunk, &datamsg->chunks, frag_list){sctp_chunk_hold(chunk);sctp_set_owner_w(chunk);chunk->transport= transport;
}any ideas to handle it?
sctp_for_each_tx_datachunk()needs to be aware of this situation. Instead of iterating directly/only over the chunk list, it should iterate over the datamsgs instead. Something like the below (just compile tested).Then, the old socket will be free to die regardless of the new one. Otherwise, if this association gets stuck on retransmissions or so, the old socket would not be freed till then.
net/sctp/socket.c +2
-1@@ -151,9 +151,10 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, void (*cb)(struct sctp_chunk *)) { +struct sctp_datamsg *msg, *prev_msg =NULL ;struct sctp_outq *q = &asoc->outqueue;struct sctp_transport *t; -struct sctp_chunk *chunk; +struct sctp_chunk *chunk, *c;list_for_each_entry (t, &asoc->peer.transport_addr_list, transports)list_for_each_entry (chunk, &t->transmitted, transmitted_list)@@ -162,8 +163,14 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,list_for_each_entry (chunk, &q->retransmit, transmitted_list)cb (chunk);
net/sctp/socket.c static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, · line 173
Quoted 10 lines
net/sctp/socket.c +8
-2-list_for_each_entry (chunk, &q->sacked, transmitted_list) -cb (chunk); +list_for_each_entry (chunk, &q->sacked, transmitted_list) { + msg = chunk->msg; +if (msg = prev_msg) +continue ; +list_for_each_entry (c, &msg->chunks, frag_list) +cb (c); + prev_msg = msg; + }
great. I'll trigger a syzbot test. Thanks.
Quoted 3 lines
net/sctp/socket.c +0
-0list_for_each_entry (chunk, &q->abandoned, transmitted_list)cb (chunk);
Marcelo Ricardo Leitner
maintainerQuoted 57 lines · Qiujun Huang
Quoted 55 lines · Marcelo Ricardo Leitner
On Sat, Mar 21, 2020 at 07:53:29AM +0800, Qiujun Huang wrote:
...Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
So,
sctp_wfreewas not called to destroy SKB)then migrate happened
sctp_for_each_tx_datachunk( sctp_clear_owner_w); sctp_assoc_migrate(); sctp_for_each_tx_datachunk( sctp_set_owner_w);SKB was not in the outq, and was not changed to newsk
The real fix is to fix the migration to the new socket, though the situation on which it is happening is still not clear.
Quoted 8 lines
Quoted 5 lines
The 2nd
sendto()call on the reproducer is sending 212992 bytes on a single call. That's usually the whole sndbuf size, and will cause fragmentation to happen. That means the datamsg will contain several skbs. But still, the sacked chunks should be freed if needed while the remaining ones will be left on the queues that they are.in
sctp_sendmsg_to_asoc
datamsg holds his chunk result in that the sacked chunks can't be freedRight! Now I see it, thanks.
In the end, it's not a locking race condition. It's just not iterating on the lists properly.Quoted 8 lines
list_for_each_entry(chunk, &datamsg->chunks, frag_list){sctp_chunk_hold(chunk);sctp_set_owner_w(chunk);chunk->transport= transport;
}any ideas to handle it?
sctp_for_each_tx_datachunk()needs to be aware of this situation. Instead of iterating directly/only over the chunk list, it should iterate over the datamsgs instead. Something like the below (just compile tested).Then, the old socket will be free to die regardless of the new one. Otherwise, if this association gets stuck on retransmissions or so, the old socket would not be freed till then.
net/sctp/socket.c +0
-0@@ -151,9 +151,10 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, void (*cb)(struct sctp_chunk *))
Quoted 6 lines
net/sctp/socket.c static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, · line 157
Quoted 6 lines
net/sctp/socket.c +2
-1{ +struct sctp_datamsg *msg, *prev_msg =NULL ;struct sctp_outq *q = &asoc->outqueue;struct sctp_transport *t; -struct sctp_chunk *chunk; +struct sctp_chunk *chunk, *c;
I missed to swap some lines here, for reverse christmass-tree style, btw.
Quoted 7 lines
Quoted 7 lines
net/sctp/socket.c +0
-0list_for_each_entry (t, &asoc->peer.transport_addr_list, transports)list_for_each_entry (chunk, &t->transmitted, transmitted_list)@@ -162,8 +163,14 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,list_for_each_entry (chunk, &q->retransmit, transmitted_list)cb (chunk);
Quoted 12 lines
net/sctp/socket.c static void sctp_for_each_tx_datachunk(struct sctp_association *asoc, · line 173
Quoted 10 lines
net/sctp/socket.c +8
-2-list_for_each_entry (chunk, &q->sacked, transmitted_list) -cb (chunk); +list_for_each_entry (chunk, &q->sacked, transmitted_list) { + msg = chunk->msg; +if (msg = prev_msg) +continue ; +list_for_each_entry (c, &msg->chunks, frag_list) +cb (c); + prev_msg = msg; + }great. I'll trigger a syzbot test. Thanks.
Mind that it may need to handled on the other lists as well. I didn't check them :]
Quoted 4 lines
Quoted 4 lines
net/sctp/socket.c +0
-0list_for_each_entry (chunk, &q->abandoned, transmitted_list)cb (chunk);