tipc: fix broadcast sender hang and leak on last acker departure
Subsystems (4)
PATCH 1/1 tipc: fix broadcast sender hang and leak on last acker departure
TIPC group broadcast senders are blocked in poll()/write() until all members have acked the replicast. When a departing member held the last outstanding ack, tipc_group_delete_member() decrements grp->bc_ackers to zero but never restores *grp->open nor wakes the blocked senders; write() then sleeps until the send timeout and poll() is never woken at all. Commit 99cc2a62e07a ("tipc: reject invalid and unexpected GRP_ACK_MSG to prevent bc_ackers underflow") noted this as a pre-existing issue.
Restore the open state exactly on the 1->0 acker transition in the delete path, mirroring the GRP_ACK_MSG completion path, and invoke sk->sk_write_space() from tipc_sk_filter_rcv() when the group just became writable again.
This vulnerability was discovered by Tencent CodeBuddy Security.
--- net/tipc/group.c | 9 ++++++--- net/tipc/socket.c | 12 +++++++++++- 2 files changed, 17 insertions(+), 4 deletions(-)
net/tipc/group.c +6 -3
@@ -340,9 +340,12 @@ static void tipc_group_delete_member(struct tipc_group *grp,rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member held the last outstanding replicast ack, +* restore the open state so blocked broadcast senders can proceed. +*/ +if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 ) && + !--grp->bc_ackers) + *grp->open = true;list_del_init (&m->list);list_del_init (&m->small_win);
net/tipc/socket.c +11 -1
@@ -2356,8 +2356,18 @@ static void tipc_sk_filter_rcv(struct sock *sk, struct sk_buff *skb,if (unlikely (!msg_isdata (hdr)))tipc_sk_proto_rcv (sk, &inputq, xmitq); -if (unlikely (grp)) +if (unlikely (grp)) { +bool was_open = tsk->group_is_open; +tipc_group_filter_msg (grp, &inputq, xmitq); +/* A departing member may have held the last outstanding +* replicast ack, which can restore the open state inside +* tipc_group_filter_msg(); wake up senders blocked in +* poll()/write() if the group just became writable. +*/ +if (!was_open && tsk->group_is_open) + sk->sk_write_space (sk); + }if (unlikely (!grp) && mtyp ==TIPC_MCAST_MSG )tipc_mcast_filter_msg (net, &tsk->mc_method.deferredq, &inputq);
--
2.43.7
[email protected]
AI reviewThis is an automated message. This series looks like a fix, but its commit messages seem to be missing some information:
Hi!
This is an automated message. This series looks like a fix, but its commit messages seem to be missing some information:
- Whether the issue was actually triggered, or is only theoretical (e.g. found by code inspection). If it was triggered please include the symptoms, like the stack trace or error messages.
Please do not repost the series just to address the above. Instead, reply to this email with the missing information, so that reviewers can take it into account. If the series needs another revision for other reasons, please include the information in the commit messages then.
The evaluation is done by an LLM so it may be wrong, if you think that is the case please reply and explain.
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 (1) · Low: 1 (1)
If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough).
- [Medium] The new open-restore in tipc_group_delete_member() (net/tipc/group.c:346-348) depends on less(m->bc_acked,…
- [Low] The commit message does not match the patch.
Pre-existing issues:
- [Medium] This problem wasn't introduced by this patch, but a node-down withdrawal event can get stuck behind a deferred unicast, so the…
- [Low] This problem wasn't introduced by this patch, but tsk->group_is_open (aliased as *grp->open) is written with plain…
--
tipc: fix broadcast sender hang and leak on last acker departure
This patch makes tipc_group_delete_member() restore *grp->open when grp->bc_ackers drops from 1 to 0. It also makes tipc_sk_filter_rcv() call sk->sk_write_space() when tipc_group_filter_msg() switches the group from closed to open. The aim is to unblock group broadcast senders when the last outstanding acker leaves.
Quoted 1 line
tipc: fix broadcast sender hang and leak on last acker departure
[Severity: Low]
Which leak does this patch fix? The diff only adds the reopen and the wakeup. The kfree(m) and the bc_ackers decrement in tipc_group_delete_member() were already there.
This patch carries Cc: stable and a Fixes: tag. Could "leak" be dropped from the subject so it doesn't mislead backporters?
Quoted 2 lines
blocked senders;
write()then sleeps until the send timeout andpoll()is never woken at all.
Is the write() part accurate? tipc_wait_for_cond() sleeps in wait_woken() with woken_wake_function(), and that function doesn't filter on the poll key:
net/tipc/socket.c:tipc_wait_for_cond() {
...
add_wait_queue(sk_sleep(sk_), &wait_);
release_sock(sk_);
*(timeo_) = wait_woken(&wait_, TASK_INTERRUPTIBLE, *(timeo_));
...
}So any tipc_data_ready() on the socket wakes the writer and re-runs tipc_group_bc_cong(). One example is delivery of the TIPC_WITHDRAWN member event when TIPC_GROUP_MEMBER_EVTS is set.
If no such wakeup comes, the default blocking sk_sndtimeo is MAX_SCHEDULE_TIMEOUT, so there is no bounded send timeout either.
The poll() part looks correct, since tipc_poll() reports EPOLLOUT only when tsk->group_is_open is set.
Quoted 7 lines
net/tipc/group.c +0
-0@@ -340,9 +340,12 @@ static void tipc_group_delete_member(struct tipc_group *grp,rb_erase (&m->tree_node, &grp->members); grp->member_cnt--;
net/tipc/group.c static void tipc_group_delete_member(struct tipc_group *grp, · line 348
Quoted 10 lines
net/tipc/group.c +6
-3-/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member held the last outstanding replicast ack, +* restore the open state so blocked broadcast senders can proceed. +*/ +if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 ) && + !--grp->bc_ackers) + *grp->open = true;
[Severity: Medium]
Can this reopen the group while a counted acker is still outstanding?
The less(m->bc_acked, grp->bc_snd_nxt - 1) check also matches members that were never counted in bc_ackers. tipc_group_update_bc_members() only counts a member, and refreshes its bc_acked, when tipc_group_is_receiver() is true:
net/tipc/group.c:tipc_group_update_bc_members() {
...
if (tipc_group_is_receiver(m)) {
tipc_group_update_member(m, len);
m->bc_acked = prev;
ackers++;
}
...
}tipc_group_is_receiver() excludes MBR_JOINING and MBR_LEAVING. A member in either state keeps a stale bc_acked.
For example, say members X and Y are both JOINED:
- A non-ack broadcast at seqno 10 sets X.bc_acked = Y.bc_acked = 9 and
bc_snd_nxt = 11.
- X becomes MBR_LEAVING. This can happen when TIPC_WITHDRAWN arrives
with the node still up and before GRP_LEAVE_MSG, or when the leave
event is still waiting in m->deferredq.
- The local socket sends a unicast. tipc_send_group_anycast()/unicast
then sets method->rcast and method->mandatory, so the next broadcast
goes out as an ack-requesting replicast at seqno 11. Only Y is
counted: bc_ackers = 1, Y.bc_acked = 10, bc_snd_nxt = 12. - X's leave event is delivered, and tipc_group_filter_msg() calls tipc_group_delete_member(X).
At that point less(9, 11) is true. So --bc_ackers reaches 0 and *grp->open is set to true, even though Y hasn't acked yet.
The new check in tipc_sk_filter_rcv() then sees this:
if (!was_open && tsk->group_is_open) sk->sk_write_space(sk);
The writer blocked in tipc_send_group_bcast() wakes up and tipc_group_bc_cong() returns false. The next broadcast can then go out, possibly over the broadcast link, before Y has acked the replicast.
Doesn't that break the unicast/broadcast ordering that commit 2f487712b893 added the bc_ackers gate for? Y's later GRP_ACK_MSG is also dropped by the !grp->bc_ackers check in tipc_group_proto_rcv().
The miscount itself predates this patch. Before it, a false zero only took effect on the next send or the next unrelated wakeup. With this patch it causes an immediate wakeup and EPOLLOUT. The new comment ("If this member held the last outstanding replicast ack") also doesn't hold in this case.
Would it be better to record whether the member was counted when the replicast was sent, for example with a per-member pending-ack flag, instead of comparing a possibly stale bc_acked?
[Severity: Medium]
This is a pre-existing issue and not one introduced by this patch. Can the departure path that leads here get stuck behind a deferred unicast, so that this reopen never runs?
A member's group unicast carries msg_grp_bc_seqno = N+1. It can arrive before that member's broadcast N, which travels on a different link. The unicast is then deferred in m->deferredq with m->bc_rcv_nxt == N.
If the peer node fails before broadcast N arrives, tipc_group_member_evt() creates the withdrawal event with seqno m->bc_rcv_nxt:
net/tipc/group.c:tipc_group_member_evt() {
...
if (!tipc_node_is_up(net, node))
tipc_group_create_event(grp, m, TIPC_WITHDRAWN,
m->bc_rcv_nxt, inputq);
...
}tipc_group_sort_msg() only sorts BCAST/MCAST messages, so the event is appended behind the deferred unicast:
/* Unicasts are never bypassed, - always add to tail */ __skb_queue_tail(defq, skb);
tipc_group_filter_msg() then peeks the unicast at the head of the queue and stops:
if (more(msg_grp_bc_seqno(hdr), m->bc_rcv_nxt)) break;
Broadcast N is lost along with the node, so tipc_group_delete_member() is never called for this member, and the stale member stays in the tree. If that member was the last outstanding acker, bc_ackers stays non-zero and group broadcasts on this socket stay blocked.
Should the node-down case be handled before the event is queued behind messages that can no longer be delivered?
[Severity: Low]
This isn't a bug introduced by this patch, but tsk->group_is_open (which *grp->open points to) is written with plain stores under the socket lock. tipc_poll() reads it without the lock:
net/tipc/socket.c:tipc_poll() {
...
case TIPC_OPEN:
if (tsk->group_is_open && !tsk->cong_link_cnt)
revents |= EPOLLOUT;
...
}This new store is one more writer of the same kind as the ones in tipc_group_create(), tipc_group_cong(), tipc_group_bc_cong(), tipc_group_open() and the GRP_ACK_MSG handler. The new was_open read in tipc_sk_filter_rcv() runs under the socket lock, so it is fine.
Could KCSAN report this as a data race? If so, READ_ONCE()/WRITE_ONCE() annotations on group_is_open would cover all of these accesses.
[ ... ]
Signature · 1 line
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261009055601.2304158-1-bsdhenrymartin%40gmail.com
Tung Quang Nguyen
maintainerQuoted 38 lines
Subject: Re: [PATCH] tipc: fix broadcast sender hang and leak on last acker departure
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 (1) · Low: 1 (1)
If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough).- [Medium] The new open-restore in
tipc_group_delete_member()(net/tipc/group.c:346-348) depends on less(m->bc_acked,…
- [Low] The commit message does not match the patch.Pre-existing issues:
- [Medium] This problem wasn't introduced by this patch, but a node-down withdrawal event can get stuck behind a deferred unicast, so the…
- [Low] This problem wasn't introduced by this patch, buttsk->group_is_open(aliased as*grp->open) is written with plain…--
tipc: fix broadcast sender hang and leak on last acker departure
This patch makes
tipc_group_delete_member()restore*grp->openwhengrp->bc_ackersdrops from 1 to 0. It also makestipc_sk_filter_rcv()grp->callsk->sk_write_space()whentipc_group_filter_msg()switches the groupsk->from
closed to open. The aim is to unblock group broadcast senders when the last outstanding acker leaves.Quoted 1 line
tipc: fix broadcast sender hang and leak on last acker departure
Quoted 8 lines
[Severity: Low]
Which leak does this patch fix? The diff only adds the reopen and the wakeup. The kfree(m) and thebc_ackersdecrement intipc_group_delete_member()were already there.This patch carries Cc: stable and a Fixes: tag. Could "leak" be dropped from the subject so it doesn't mislead backporters?
Agree. Please rewrite the changelog.
Quoted 15 lines
Quoted 2 lines
blocked senders;
write()then sleeps until the send timeout andpoll()is never woken at all.Is the
write()part accurate?tipc_wait_for_cond()sleeps inwait_woken()withwoken_wake_function(), and that function doesn't filter on the poll key:net/tipc/socket.c:tipc_wait_for_cond() { ... add_wait_queue(sk_sleep(sk_), &wait_); release_sock(sk_); *(timeo_) = wait_woken(&wait_, TASK_INTERRUPTIBLE, *(timeo_)); ... }
Quoted 9 lines
So any
tipc_data_ready()on the socket wakes the writer and re-runstipc_group_bc_cong(). One example is delivery of theTIPC_WITHDRAWNmember event whenTIPC_GROUP_MEMBER_EVTSis set.If no such wakeup comes, the default blocking
sk_sndtimeoisMAX_SCHEDULE_TIMEOUT, so there is no bounded send timeout either.The
poll()part looks correct, sincetipc_poll()reports EPOLLOUT only when tsk-Quoted 1 line
group_is_openis set.
Agree. Please rewrite the changelog.
Quoted 72 lines
net/tipc/group.c static void tipc_group_delete_member(struct
Quoted 6 lines
net/tipc/group.c index renamed from net/tipc/group.c +0
-0
74f6d3dac0784..e20eb612f82ce100644net/tipc/group.c +0
-0@@ -340,9 +340,12 @@ static void tipc_group_delete_member(struct
tipc_group*grp,net/tipc/group.c static void tipc_group_delete_member(struct tipc_group *grp, · line 348
Quoted 12 lines
net/tipc/group.c +6
-3rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member held the last outstanding replicast ack, +* restore the open state so blocked broadcast senders can proceed. +*/ +if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 ) && + !--grp->bc_ackers) + *grp->open = true;[Severity: Medium]
Can this reopen the group while a counted acker is still outstanding?The
less(m->bc_acked, grp->bc_snd_nxt - 1)check also matches members that were never counted inbc_ackers.tipc_group_update_bc_members()only counts a member, and refreshes itsbc_acked, whentipc_group_is_receiver()is true:net/tipc/group.c:tipc_group_update_bc_members() { ... if (tipc_group_is_receiver(m)) { tipc_group_update_member(m, len); m->bc_acked = prev; ackers++; } ... }
tipc_group_is_receiver()excludesMBR_JOININGandMBR_LEAVING. A member in either state keeps a stalebc_acked.For example, say members X and Y are both JOINED:
- A non-ack broadcast at seqno 10 sets X.bc_acked = Y.bc_acked = 9 and
bc_snd_nxt= 11.
- X becomesMBR_LEAVING. This can happen whenTIPC_WITHDRAWN
arrives
with the node still up and beforeGRP_LEAVE_MSG, or when the leave event is still waiting inm->deferredq.
- The local socket sends a unicast. tipc_send_group_anycast()/unicast then setsmethod->rcastandmethod->mandatory, so the next broadcast goes out as an ack-requesting replicast at seqno 11. Only Y is
counted:bc_ackers= 1, Y.bc_acked = 10,bc_snd_nxt= 12.
- X's leave event is delivered, andtipc_group_filter_msg()callstipc_group_delete_member(X).At that point
less(9, 11)is true. So --bc_ackers reaches 0 and*grp->openis set to true, even though Y hasn't acked yet.The new check in
tipc_sk_filter_rcv()then sees this:if (!was_open && tsk->group_is_open) sk->sk_write_space(sk);The writer blocked in
tipc_send_group_bcast()wakes up andtipc_group_bc_cong()returns false. The next broadcast can then go out, possibly over the broadcast link, before Y has acked the replicast.Doesn't that break the unicast/broadcast ordering that commit
2f487712b893added thebc_ackersgate for? Y's laterGRP_ACK_MSGis also dropped by the !grp->bc_ackers check intipc_group_proto_rcv().
Quoted 9 lines
The miscount itself predates this patch. Before it, a false zero only took effect on the next send or the next unrelated wakeup. With this patch it causes an immediate wakeup and EPOLLOUT. The new comment ("If this member held the last outstanding replicast ack") also doesn't hold in this case.
Would it be better to record whether the member was counted when the replicast was sent, for example with a per-member pending-ack flag, instead of comparing a possibly stale
bc_acked?
Agree. Please address the incomplete check.
--
pw-bot: cr
PATCH v2 1/1 tipc: fix broadcast sender hang on last acker departure
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks: poll(EPOLLOUT) only fires when tsk->group_is_open, and writers keep re-sleeping on every unrelated data-ready wakeup because tipc_group_bc_cong() keeps seeing *grp->open == false (the default sk_sndtimeo is MAX_SCHEDULE_TIMEOUT, so there is no bounded send timeout either).
When the last outstanding acker departs before acking, tipc_group_delete_member() decrements grp->bc_ackers but never restores *grp->open, and the blocked senders are never woken. Commit 99cc2a62e07a ("tipc: reject invalid and unexpected GRP_ACK_MSG to prevent bc_ackers underflow") already calls this out in its changelog as a pre-existing issue.
Reproduced on a two-node TIPC group cluster (v7.3-rc5, two QEMU guests over a UDP bearer): with one member's ack outstanding and that member then departing, poll(POLLOUT) on the sender never fired and a blocking sendmsg() never returned; with the fix applied both are released when the leave event is delivered (log below the fold).
Restore the open state exactly on the 1->0 transition in the delete path, mirroring the GRP_ACK_MSG completion path, and invoke sk->sk_write_space() from tipc_sk_filter_rcv() when the group just became writable again.
While at it, make the accounting precise: the delete path used to test less(m->bc_acked, grp->bc_snd_nxt - 1), which also matches members that were never counted for the current replicast (JOINING/LEAVING members keep a stale bc_acked). Zeroing bc_ackers on such a member could silently void a still-outstanding ack from a real acker. Track whether a member owes an ack for the current replicast in a new per-member bc_pending flag, set when the member is counted and cleared when its ack arrives, and test that in the delete path instead. Suggested by Tung Quang Nguyen.
This issue was discovered by Tencent CodeBuddy Security.
---
v2: Subject no longer claims a leak; describe the actual wakeup
mechanics and add the reproduction; the delete path now discriminates members that were actually counted for the current replicast via a new per-member bc_pending flag instead of comparing a possibly stale bc_acked (Tung Quang Nguyen).
Reproduction (two QEMU guests, TIPC over UDP bearer; the acking peer withheld GRP_ACK_MSG via a test-only knob to emulate the lossy peer, the sender's kernel ran unmodified TIPC):
UNPATCHED (#88/#90, fix absent):
sender: NO POLLOUT for 30s after member departed -> STALL
sender2: R2 NEVER returned in 32s (blocked writer not woken)
PATCHED (#89, this patch):
sender: POLLOUT at t=18.5s, when the leave event was delivered
sender2: R2 returned at t=18.6s, EHOSTUNREACH on now-empty group
net/tipc/group.c | 13 ++++++++++--- net/tipc/socket.c | 12 +++++++++++- 2 files changed, 21 insertions(+), 4 deletions(-)
net/tipc/group.c +10 -3
@@ -75,6 +75,7 @@ struct tipc_member {u16 bc_rcv_nxt;u16 bc_syncpt;u16 bc_acked; +bool bc_pending; };struct tipc_group {
@@ -340,9 +341,12 @@ static void tipc_group_delete_member(struct tipc_group *grp,rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member owed an ack for the current replicast and it +* was the last one, restore the open state so blocked broadcast +* senders can proceed. +*/ +if (grp->bc_ackers && m->bc_pending && !--grp->bc_ackers) + *grp->open = true;list_del_init (&m->list);list_del_init (&m->small_win);
@@ -406,6 +410,8 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack) m->bc_acked = prev; ackers++; } +if (ack) + m->bc_pending =tipc_group_is_receiver (m); }/* Mark number of acknowledges to expect, if any */
@@ -803,6 +809,7 @@ void tipc_group_proto_rcv(struct tipc_group *grp, bool *usr_wakeup,if (acked != grp->bc_snd_nxt || m->bc_acked == acked)return ; m->bc_acked = acked; + m->bc_pending = false;if (--grp->bc_ackers)return ;list_del_init (&m->small_win);
net/tipc/socket.c +11 -1
@@ -2356,8 +2356,18 @@ static void tipc_sk_filter_rcv(struct sock *sk, struct sk_buff *skb,if (unlikely (!msg_isdata (hdr)))tipc_sk_proto_rcv (sk, &inputq, xmitq); -if (unlikely (grp)) +if (unlikely (grp)) { +bool was_open = tsk->group_is_open; +tipc_group_filter_msg (grp, &inputq, xmitq); +/* A departing member may have held the last outstanding +* replicast ack, which can restore the open state inside +* tipc_group_filter_msg(); wake up senders blocked in +* poll()/write() if the group just became writable. +*/ +if (!was_open && tsk->group_is_open) + sk->sk_write_space (sk); + }if (unlikely (!grp) && mtyp ==TIPC_MCAST_MSG )tipc_mcast_filter_msg (net, &tsk->mc_method.deferredq, &inputq);
Signature · 1 line
2.43.7
[email protected]
AI reviewRe: [PATCH net v2] tipc: fix broadcast sender hang on last acker departure
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 1
If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough).
- [Medium] The patch adds m->bc_pending so that only members counted for the current replicast can reduce grp->bc_ackers, but it checks the…
- [Low] The commit message gives the wrong reason why blocked writers stay asleep, and it overstates when the blocking sendmsg() hang…
--
tipc: fix broadcast sender hang on last acker departure
This patch restores *grp->open when the last member that owes a replicast ack is deleted. It also calls sk->sk_write_space() from tipc_sk_filter_rcv() when the group becomes writable again. A new per-member bc_pending flag limits the bc_ackers decrement in the delete path to members that were counted for the current replicast.
Quoted 6 lines
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks: poll(EPOLLOUT) only fires when
tsk->group_is_open, and writers keep re-sleeping on every unrelated data-ready wakeup becausetipc_group_bc_cong()keeps seeing*grp->open== false (the defaultsk_sndtimeoisMAX_SCHEDULE_TIMEOUT, so there is no bounded send timeout either).
[Severity: Low]
Is this description of the mechanism accurate? tipc_group_bc_cong() only writes *grp->open and never reads it:
net/tipc/group.c:tipc_group_bc_cong() {
if (grp->bc_ackers) {
*grp->open = false;
return true;
}
if (list_empty(&grp->small_win))
return false;
...
}Its return value depends only on grp->bc_ackers and the head of grp->small_win.
Quoted 3 lines
When the last outstanding acker departs before acking,
tipc_group_delete_member()decrementsgrp->bc_ackersbut never restores*grp->open, and the blocked senders are never woken.
The old delete path already brought bc_ackers to 0 when the last owing member left. That member has bc_acked == S-1 and bc_snd_nxt == S+1, so the less() test matches.
tipc_wait_for_cond() waits with woken_wake_function, which ignores the poll key, and tipc_data_ready() wakes every sleeper on sk_wq.
So with group member events enabled, the sk_data_ready() after the leave event is delivered already woke the writer in tipc_send_group_bcast(). tipc_group_bc_cong() then returned false and sendmsg() went on, returning EHOSTUNREACH if the group was now empty.
The poll(EPOLLOUT) hang does look real in every case, because tipc_poll() only sets EPOLLOUT when tsk->group_is_open is true.
The blocking sendmsg() hang seems to happen only when nothing wakes the writer after the deletion. One example is group member events being disabled. In that case TIPC_GRP_MEMBER_EVT is freed with deliver = false and sk_data_ready() is never called.
The code change itself looks correct. Since this is tagged for stable, could the changelog describe the actual wakeup path and say when the sendmsg() hang occurs?
Quoted 4 lines
net/tipc/group.c +0
-0
[ ... ]
Quoted 26 lines
net/tipc/group.c +8
-3@@ -340,9 +341,12 @@ static void tipc_group_delete_member(struct tipc_group *grp,rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member owed an ack for the current replicast and it +* was the last one, restore the open state so blocked broadcast +* senders can proceed. +*/ +if (grp->bc_ackers && m->bc_pending && !--grp->bc_ackers) + *grp->open = true;list_del_init (&m->list);list_del_init (&m->small_win);@@ -406,6 +410,8 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack) m->bc_acked = prev; ackers++; } +if (ack) + m->bc_pending =tipc_group_is_receiver (m); }/* Mark number of acknowledges to expect, if any */
net/tipc/group.c void tipc_group_proto_rcv(struct tipc_group *grp, bool *usr_wakeup, · line 815
Quoted 7 lines
net/tipc/group.c +1
-0@@ -803,6 +809,7 @@ void tipc_group_proto_rcv(struct tipc_group *grp, bool *usr_wakeup,if (acked != grp->bc_snd_nxt || m->bc_acked == acked)return ; m->bc_acked = acked; + m->bc_pending = false;if (--grp->bc_ackers)return ;list_del_init (&m->small_win);
[Severity: Medium]
Should the GRP_ACK_MSG handler also check m->bc_pending before it decrements grp->bc_ackers?
The new test in tipc_group_delete_member() assumes bc_ackers equals the number of members with bc_pending set.
This handler still decrements bc_ackers for any member that passes the acked == grp->bc_snd_nxt and m->bc_acked != acked checks.
Two kinds of member can pass those checks without having been counted:
- Members that were MBR_JOINING or MBR_LEAVING when tipc_group_update_bc_members() ran keep a stale bc_acked and get bc_pending = false.
- Members created after the replicast have bc_pending = false and bc_acked = bc_snd_nxt - 1 from tipc_group_create_member().
For both, a GRP_ACK_MSG carrying acked == bc_snd_nxt passes every check here.
For example, with bc_ackers == 2 for counted members A and B:
ack from uncounted member X bc_ackers 2 -> 1
B acks, or B is deleted bc_ackers 1 -> 0, *grp->open = true
A still has bc_pending == true and has never acked.
Isn't this the case the commit message gives as the reason for adding bc_pending, where a non-counted member "could silently void a still-outstanding ack from a real acker"?
A cluster peer can trigger this. It sends GRP_JOIN_MSG, which makes tipc_group_proto_rcv() create an MBR_JOINING member for any port. It then sends GRP_ACK_MSG with acked == bc_snd_nxt. I could not find a benign protocol sequence that ends up here.
Would something like this close the gap?
if (acked != grp->bc_snd_nxt || !m->bc_pending) return;
Signature · 1 line
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261012153357.1863316-1-bsdhenrymartin%40gmail.com
Tung Quang Nguyen
maintainerRE: [PATCH net v2] tipc: fix broadcast sender hang on last acker departure
Quoted 72 lines
Subject: Re: [PATCH net v2] tipc: fix broadcast sender hang on last acker departure
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 1
If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough).- [Medium] The patch adds
m->bc_pendingso that only members counted for the current replicast can reducegrp->bc_ackers, but it checks the…
- [Low] The commit message gives the wrong reason why blocked writers stay asleep, and it overstates when the blockingsendmsg()hang…--
tipc: fix broadcast sender hang on last acker departure
This patch restores
*grp->openwhen the last member that owes a replicast ack is deleted. It also callssk->sk_write_space()fromtipc_sk_filter_rcv()when the group becomes writable again. A new per-memberbc_pendingflag limits thebc_ackersdecrement in the delete path to members that were counted for the current replicast.Quoted 6 lines
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks: poll(EPOLLOUT) only fires when
tsk->group_is_open, and writers keep re-sleeping on every unrelated data-ready wakeup becausetipc_group_bc_cong()keeps seeing*grp->open== false (the defaultsk_sndtimeoisMAX_SCHEDULE_TIMEOUT, so there is no bounded send timeout either).[Severity: Low]
Is this description of the mechanism accurate?tipc_group_bc_cong()only writes*grp->openand never reads it:net/tipc/group.c:tipc_group_bc_cong() { if (grp->bc_ackers) { *grp->open = false; return true; } if (list_empty(&grp->small_win)) return false; ... }Its return value depends only on
grp->bc_ackersand the head ofgrp->small_win.Quoted 3 lines
When the last outstanding acker departs before acking,
tipc_group_delete_member()decrementsgrp->bc_ackersbut never restores*grp->open, and the blocked senders are never woken.The old delete path already brought
bc_ackersto 0 when the last owing member left. That member hasbc_acked== S-1 andbc_snd_nxt== S+1, so theless()test matches.
tipc_wait_for_cond()waits withwoken_wake_function, which ignores the poll key, andtipc_data_ready()wakes every sleeper onsk_wq.So with group member events enabled, the
sk_data_ready()after the leave event is delivered already woke the writer intipc_send_group_bcast().tipc_group_bc_cong()then returned false andsendmsg()went on, returning EHOSTUNREACH if the group was now empty.The poll(EPOLLOUT) hang does look real in every case, because
tipc_poll()only sets EPOLLOUT whentsk->group_is_openis true.
Quoted 8 lines
The blocking
sendmsg()hang seems to happen only when nothing wakes the writer after the deletion. One example is group member events being disabled. In that caseTIPC_GRP_MEMBER_EVTis freed with deliver = false andsk_data_ready()is never called.The code change itself looks correct. Since this is tagged for stable, could the changelog describe the actual wakeup path and say when the
sendmsg()hang occurs?
Please rewrite the changelog.
Quoted 82 lines
Quoted 5 lines
net/tipc/group.c index renamed from net/tipc/group.c +0
-0
74f6d3dac0784..e9c77f12c9819100644net/tipc/group.c +0
-0[ ... ]
net/tipc/group.c static void tipc_group_delete_member(struct
Quoted 1 line
net/tipc/group.c +0
-0@@ -340,9 +341,12 @@ static void tipc_group_delete_member(struct
tipc_group*grp,net/tipc/group.c void tipc_group_update_bc_members(struct
Quoted 16 lines
net/tipc/group.c +6
-3rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member owed an ack for the current replicast and it +* was the last one, restore the open state so blocked broadcast +* senders can proceed. +*/ +if (grp->bc_ackers && m->bc_pending && !--grp->bc_ackers) + *grp->open = true;list_del_init (&m->list);list_del_init (&m->small_win);@@ -406,6 +410,8 @@ void tipc_group_update_bc_members(struct
tipc_group*grp, int len, bool ack)net/tipc/group.c void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack)
Quoted 8 lines
net/tipc/group.c +2
-0m->bc_acked = prev; ackers++; } +if (ack) + m->bc_pending =tipc_group_is_receiver (m); } /* Mark number of acknowledges to expect, if any */ @@ -803,6+809,7
net/tipc/group.c void tipc_group_proto_rcv(struct tipc_group *grp, bool *usr_wakeup, · line 815
Quoted 8 lines
@@ void tipc_group_proto_rcv(struct
tipc_group*grp, bool*usr_wakeup,net/tipc/group.c +1
-0if (acked != grp->bc_snd_nxt || m->bc_acked == acked)return ; m->bc_acked = acked; + m->bc_pending = false;if (--grp->bc_ackers)return ;list_del_init (&m->small_win);[Severity: Medium]
Should theGRP_ACK_MSGhandler also checkm->bc_pendingbefore it decrementsgrp->bc_ackers?The new test in
tipc_group_delete_member()assumesbc_ackersequals the number of members withbc_pendingset.This handler still decrements
bc_ackersfor any member that passes the acked ==grp->bc_snd_nxtandm->bc_acked!= acked checks.Two kinds of member can pass those checks without having been counted:
- Members that were
MBR_JOININGorMBR_LEAVINGwhentipc_group_update_bc_members()ran keep a stalebc_ackedand getbc_pending= false.
- Members created after the replicast havebc_pending= false andbc_acked=bc_snd_nxt- 1 fromtipc_group_create_member().For both, a
GRP_ACK_MSGcarrying acked ==bc_snd_nxtpasses every check here.For example, with
bc_ackers== 2 for counted members A and B:ack from uncounted member X
bc_ackers2 -> 1
B acks, or B is deletedbc_ackers1 -> 0,*grp->open= trueA still has
bc_pending== true and has never acked.Isn't this the case the commit message gives as the reason for adding
bc_pending, where a non-counted member "could silently void a still- outstanding ack from a real acker"?A cluster peer can trigger this. It sends
GRP_JOIN_MSG, which makestipc_group_proto_rcv()create anMBR_JOININGmember for any port. It then sendsGRP_ACK_MSGwith acked ==bc_snd_nxt. I could not find a benign protocol sequence that ends up here.
Quoted 5 lines
Would something like this close the gap?
if (acked != grp->bc_snd_nxt || !m->bc_pending) return;
Looks right. Please use this approach (Perhaps, bc_acked field of tipc_member needs to be removed).
PATCH v3 1/1 tipc: fix broadcast sender hang on last acker departure
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks: tipc_group_bc_cong() returns true as long as grp->bc_ackers is non-zero, and clears *grp->open, which is what tipc_poll() uses to report EPOLLOUT (via tsk->group_is_open).
When the last outstanding acker departs before acking, tipc_group_delete_member() decrements grp->bc_ackers to zero but never restores *grp->open. As a result poll(EPOLLOUT) never fires again. A writer blocked in sendmsg() is only released if some other wakeup happens after the deletion: with TIPC_GROUP_MEMBER_EVTS the leave event delivery calls sk_data_ready() and wakes it, but without member events nothing does and it sleeps indefinitely, since the default send timeout is MAX_SCHEDULE_TIMEOUT.
Commit 99cc2a62e07a ("tipc: reject invalid and unexpected GRP_ACK_MSG to prevent bc_ackers underflow") already calls this out in its changelog as a pre-existing issue.
Restore the open state exactly on the 1->0 transition in the delete path, mirroring the GRP_ACK_MSG completion path, and invoke sk->sk_write_space() from tipc_sk_filter_rcv() when the group just became writable again.
Also make the ack accounting precise. The delete path used to test less(m->bc_acked, grp->bc_snd_nxt - 1), which also matches members that were never counted for the current replicast (JOINING/LEAVING members keep a stale bc_acked), and the GRP_ACK_MSG handler accepted an ack from any member whose bc_acked differed from the expected value. Either could zero bc_ackers on behalf of a member that owes nothing and silently void an ack still owed by a real acker. Track whether a member owes an ack for the current replicast in a new per-member bc_pending flag, set when the member is counted and cleared when its ack arrives, and use it both to accept an ack and to decide on deletion. bc_acked has no remaining user and is removed. Suggested by Tung Quang Nguyen.
This issue was discovered by Tencent CodeBuddy Security.
---
v3: Correct the changelog: tipc_group_bc_cong() only writes *grp->open,
and a blocked sendmsg() hangs only when nothing wakes it after the deletion (e.g. member events disabled); also require bc_pending in the GRP_ACK_MSG handler so an uncounted member cannot decrement bc_ackers, and drop the now unused bc_acked field.
v2: Subject no longer claims a leak; add the reproduction; the delete
path now discriminates members that were actually counted for the current replicast via a new per-member bc_pending flag instead of comparing a possibly stale bc_acked (Tung Quang Nguyen).
Reproduction (two QEMU guests, TIPC over UDP bearer, groups joined without member events; the acking peer withheld GRP_ACK_MSG via a test-only knob to emulate the lossy peer, the sender's kernel ran unmodified TIPC):
UNPATCHED (#88/#90, fix absent):
sender: NO POLLOUT for 30s after member departed -> STALL
sender2: R2 NEVER returned in 32s (blocked writer not woken)
PATCHED (#89, v1 of this patch):
sender: POLLOUT at t=18.5s, when the leave event was delivered
sender2: R2 returned at t=18.6s, EHOSTUNREACH on now-empty group
net/tipc/group.c | 20 +++++++++++--------- net/tipc/socket.c | 12 +++++++++++- 2 files changed, 22 insertions(+), 10 deletions(-)
net/tipc/group.c +11 -9
@@ -74,7 +74,7 @@ struct tipc_member {u16 window;u16 bc_rcv_nxt;u16 bc_syncpt; -u16 bc_acked; +bool bc_pending; };struct tipc_group {
@@ -316,7 +316,6 @@ static struct tipc_member *tipc_group_create_member(struct tipc_group *grp, m->node = node; m->port = port; m->instance = instance; - m->bc_acked = grp->bc_snd_nxt -1 ; ret =tipc_group_add_to_tree (grp, m);if (ret <0 ) {kfree (m);
@@ -340,9 +339,12 @@ static void tipc_group_delete_member(struct tipc_group *grp,rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member owed an ack for the current replicast and it +* was the last one, restore the open state so blocked broadcast +* senders can proceed. +*/ +if (grp->bc_ackers && m->bc_pending && !--grp->bc_ackers) + *grp->open = true;list_del_init (&m->list);list_del_init (&m->small_win);
@@ -394,7 +396,6 @@ void tipc_group_update_member(struct tipc_member *m, int len)void tipc_group_update_bc_members (struct tipc_group *grp,int len,bool ack) { -u16 prev = grp->bc_snd_nxt -1 ;struct tipc_member *m;struct rb_node *n;u16 ackers =0 ;
@@ -403,9 +404,10 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack) m = container_of(n, struct tipc_member, tree_node);if (tipc_group_is_receiver (m)) {tipc_group_update_member (m, len); - m->bc_acked = prev; ackers++; } +if (ack) + m->bc_pending =tipc_group_is_receiver (m); }/* Mark number of acknowledges to expect, if any */
@@ -800,9 +802,9 @@ void tipc_group_proto_rcv(struct tipc_group *grp, bool *usr_wakeup,if (!m || !grp->bc_ackers)return ; acked =msg_grp_bc_acked (hdr); -if (acked != grp->bc_snd_nxt || m->bc_acked == acked) +if (acked != grp->bc_snd_nxt || !m->bc_pending)return ; - m->bc_acked = acked; + m->bc_pending = false;if (--grp->bc_ackers)return ;list_del_init (&m->small_win);
net/tipc/socket.c +11 -1
@@ -2356,8 +2356,18 @@ static void tipc_sk_filter_rcv(struct sock *sk, struct sk_buff *skb,if (unlikely (!msg_isdata (hdr)))tipc_sk_proto_rcv (sk, &inputq, xmitq); -if (unlikely (grp)) +if (unlikely (grp)) { +bool was_open = tsk->group_is_open; +tipc_group_filter_msg (grp, &inputq, xmitq); +/* A departing member may have held the last outstanding +* replicast ack, which can restore the open state inside +* tipc_group_filter_msg(); wake up senders blocked in +* poll()/write() if the group just became writable. +*/ +if (!was_open && tsk->group_is_open) + sk->sk_write_space (sk); + }if (unlikely (!grp) && mtyp ==TIPC_MCAST_MSG )tipc_mcast_filter_msg (net, &tsk->mc_method.deferredq, &inputq);
Signature · 1 line
2.43.7
Tung Quang Nguyen
maintainerRE: [PATCH net v3] tipc: fix broadcast sender hang on last acker departure
Quoted 120 lines
Subject: [PATCH net v3] tipc: fix broadcast sender hang on last acker departure
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks:
tipc_group_bc_cong()returns true as long asgrp->bc_ackersis non-zero, and clears*grp->open, which is whattipc_poll()uses to report EPOLLOUT (viatsk->group_is_open).When the last outstanding acker departs before acking,
tipc_group_delete_member()decrementsgrp->bc_ackersto zero but never restores*grp->open. As a result poll(EPOLLOUT) never fires again. A writer blocked insendmsg()is only released if some other wakeup happens after the
deletion: withTIPC_GROUP_MEMBER_EVTSthe leave event delivery callssk_data_ready()and wakes it, but without member events nothing does and it sleeps indefinitely, since the default send timeout isMAX_SCHEDULE_TIMEOUT.Commit
99cc2a62e07a("tipc: reject invalid and unexpectedGRP_ACK_MSGto preventbc_ackersunderflow") already calls this out in its changelog as a pre- existing issue.Restore the open state exactly on the 1->0 transition in the delete path, mirroring the
GRP_ACK_MSGcompletion path, and invokesk->sk_write_space()fromtipc_sk_filter_rcv()when the group just became writable again.Also make the ack accounting precise. The delete path used to test less(m-
Quoted 1 line
bc_acked,grp->bc_snd_nxt- 1), which also matches members that werenever counted for the current replicast (JOINING/LEAVING members keep a stale
bc_acked), and theGRP_ACK_MSGhandler accepted an ack from any member whosebc_ackeddiffered from the expected value. Either could zerobc_ackerson behalf of a member that owes nothing and silently void an ack still owed by a real acker. Track whether a member owes an ack for the current replicast in a new per-memberbc_pendingflag, set when the member is counted and cleared when its ack arrives, and use it both to accept an ack and to decide on deletion.bc_ackedhas no remaining user and is removed. Suggested by Tung Quang Nguyen.This issue was discovered by Tencent CodeBuddy Security.
Fixes 2f487712b893 ("tipc: guarantee that group broadcast doesn't bypassgroup unicast")
Signed-off-by Henry Martin <[email protected]>---
v3: Correct the changelog:
tipc_group_bc_cong()only writes*grp->open,
and a blockedsendmsg()hangs only when nothing wakes it after the deletion (e.g. member events disabled); also requirebc_pendingin theGRP_ACK_MSGhandler so an uncounted member cannot decrementbc_ackers, and drop the now unusedbc_ackedfield.v2: Subject no longer claims a leak; add the reproduction; the delete
path now discriminates members that were actually counted for the current replicast via a new per-memberbc_pendingflag instead of comparing a possibly stalebc_acked(Tung Quang Nguyen).Reproduction (two QEMU guests, TIPC over UDP bearer, groups joined without member events; the acking peer withheld
GRP_ACK_MSGvia a test-only knob to emulate the lossy peer, the sender's kernel ran unmodified TIPC):UNPATCHED (#88/#90, fix absent):
sender: NO POLLOUT for 30s after member departed -> STALL
sender2: R2 NEVER returned in 32s (blocked writer not woken)
PATCHED (#89, v1 of this patch):
sender: POLLOUT at t=18.5s, when the leave event was delivered
sender2: R2 returned at t=18.6s, EHOSTUNREACH on now-empty group
net/tipc/group.c| 20 +++++++++++---------net/tipc/socket.c| 12
+++++++++++-2 files changed, 22 insertions(+), 10 deletions(-)net/tipc/group.c index renamed from net/tipc/group.c +0
-0
74f6d3dac0784..b76edd5001ffa100644net/tipc/group.c +1
-2@@ -74,7 +74,7 @@ struct tipc_member {u16 window;u16 bc_rcv_nxt;u16 bc_syncpt; -u16 bc_acked; +bool bc_pending; };struct tipc_group {@@ -316,7 +316,6 @@ static struct tipc_member *tipc_group_create_member(struct tipc_group *grp, m->node = node; m->port = port; m->instance = instance; - m->bc_acked = grp->bc_snd_nxt -1 ; ret =tipc_group_add_to_tree (grp, m);if (ret <0 ) {kfree(m);net/tipc/group.c +6
-3@@ -340,9 +339,12 @@ static void tipc_group_delete_member(struct tipc_group *grp,rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member owed an ack for the current replicast and it +* was the last one, restore the open state so blocked broadcast +* senders can proceed. +*/ +if (grp->bc_ackers && m->bc_pending && !--grp->bc_ackers) + *grp->open = true;list_del_init (&m->list);list_del_init(&m->small_win);net/tipc/group.c +0
-1@@ -394,7 +396,6 @@ void tipc_group_update_member(struct tipc_member *m, int len)void tipc_group_update_bc_members (struct tipc_group *grp,int len,bool ack) { -u16 prev = grp->bc_snd_nxt -1 ;struct tipc_member *m;struct rb_node *n;u16 ackers = 0;
net/tipc/group.c void tipc_group_update_bc_members(struct · line 412
Quoted 10 lines
net/tipc/group.c +2
-1@@ -403,9 +404,10 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack) m = container_of(n, struct tipc_member, tree_node);if (tipc_group_is_receiver (m)) {tipc_group_update_member (m, len); - m->bc_acked = prev; ackers++; } +if (ack) + m->bc_pending =tipc_group_is_receiver (m); }
Not sure if you read my comment for your v2:
" This needs to be done in a simple way:
@ -405,6 +405,7 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack)
tipc_group_update_member(m, len);
m->bc_acked = prev;
ackers++;
+ m->bc_pending = ack;
}
}Quoted 1 line
"
--
pw-bot: cr
PATCH v4 1/1 tipc: fix broadcast sender hang on last acker departure
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks: tipc_group_bc_cong() returns true as long as grp->bc_ackers is non-zero, and clears *grp->open, which is what tipc_poll() uses to report EPOLLOUT (via tsk->group_is_open).
When the last outstanding acker departs before acking, tipc_group_delete_member() decrements grp->bc_ackers to zero but never restores *grp->open. As a result poll(EPOLLOUT) never fires again. A writer blocked in sendmsg() is only released if some other wakeup happens after the deletion: with TIPC_GROUP_MEMBER_EVTS the leave event delivery calls sk_data_ready() and wakes it, but without member events nothing does and it sleeps indefinitely, since the default send timeout is MAX_SCHEDULE_TIMEOUT.
Commit 99cc2a62e07a ("tipc: reject invalid and unexpected GRP_ACK_MSG to prevent bc_ackers underflow") already calls this out in its changelog as a pre-existing issue.
Restore the open state exactly on the 1->0 transition in the delete path, mirroring the GRP_ACK_MSG completion path, and invoke sk->sk_write_space() from tipc_sk_filter_rcv() when the group just became writable again.
Also make the ack accounting precise. The delete path used to test less(m->bc_acked, grp->bc_snd_nxt - 1), which also matches members that were never counted for the current replicast (JOINING/LEAVING members keep a stale bc_acked), and the GRP_ACK_MSG handler accepted an ack from any member whose bc_acked differed from the expected value. Either could zero bc_ackers on behalf of a member that owes nothing and silently void an ack still owed by a real acker. Track whether a member owes an ack for the current replicast in a new per-member bc_pending flag, set when the member is counted and cleared when its ack arrives, and use it both to accept an ack and to decide on deletion. bc_acked has no remaining user and is removed. Suggested by Tung Quang Nguyen.
This issue was discovered by Tencent CodeBuddy Security.
---
v4: Assign bc_pending inside the receiver loop body in the simple
form requested by Jon Maloy (m->bc_pending = ack); bc_ackers==0 implies no member has bc_pending set (a pending flag is cleared either by the member's ack or by its deletion), so skipping non-receivers cannot leave stale state.
v3: Correct the changelog: tipc_group_bc_cong() only writes *grp->open,
and a blocked sendmsg() hangs only when nothing wakes it after the deletion (e.g. member events disabled); also require bc_pending in the GRP_ACK_MSG handler so an uncounted member cannot decrement bc_ackers, and drop the now unused bc_acked field.
v2: Subject no longer claims a leak; add the reproduction; the delete
path now discriminates members that were actually counted for the current replicast via a new per-member bc_pending flag instead of comparing a possibly stale bc_acked (Tung Quang Nguyen).
Reproduction (two QEMU guests, TIPC over UDP bearer, groups joined without member events; the acking peer withheld GRP_ACK_MSG via a test-only knob to emulate the lossy peer, the sender's kernel ran unmodified TIPC):
UNPATCHED (#88/#90, fix absent):
sender: NO POLLOUT for 30s after member departed -> STALL
sender2: R2 NEVER returned in 32s (blocked writer not woken)
PATCHED (#89, v1 of this patch):
sender: POLLOUT at t=18.5s, when the leave event was delivered
sender2: R2 returned at t=18.6s, EHOSTUNREACH on now-empty group
net/tipc/group.c | 19 ++++++++++--------- net/tipc/socket.c | 12 +++++++++++- 2 files changed, 21 insertions(+), 10 deletions(-)
net/tipc/group.c +10 -9
@@ -74,7 +74,7 @@ struct tipc_member {u16 window;u16 bc_rcv_nxt;u16 bc_syncpt; -u16 bc_acked; +bool bc_pending; };struct tipc_group {
@@ -316,7 +316,6 @@ static struct tipc_member *tipc_group_create_member(struct tipc_group *grp, m->node = node; m->port = port; m->instance = instance; - m->bc_acked = grp->bc_snd_nxt -1 ; ret =tipc_group_add_to_tree (grp, m);if (ret <0 ) {kfree (m);
@@ -340,9 +339,12 @@ static void tipc_group_delete_member(struct tipc_group *grp,rb_erase (&m->tree_node, &grp->members); grp->member_cnt--; -/* Check if we were waiting for replicast ack from this member */ -if (grp->bc_ackers &&less (m->bc_acked, grp->bc_snd_nxt -1 )) - grp->bc_ackers--; +/* If this member owed an ack for the current replicast and it +* was the last one, restore the open state so blocked broadcast +* senders can proceed. +*/ +if (grp->bc_ackers && m->bc_pending && !--grp->bc_ackers) + *grp->open = true;list_del_init (&m->list);list_del_init (&m->small_win);
@@ -394,7 +396,6 @@ void tipc_group_update_member(struct tipc_member *m, int len)void tipc_group_update_bc_members (struct tipc_group *grp,int len,bool ack) { -u16 prev = grp->bc_snd_nxt -1 ;struct tipc_member *m;struct rb_node *n;u16 ackers =0 ;
@@ -403,8 +404,8 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack) m = container_of(n, struct tipc_member, tree_node);if (tipc_group_is_receiver (m)) {tipc_group_update_member (m, len); - m->bc_acked = prev; ackers++; + m->bc_pending = ack; } }
@@ -800,9 +801,9 @@ void tipc_group_proto_rcv(struct tipc_group *grp, bool *usr_wakeup,if (!m || !grp->bc_ackers)return ; acked =msg_grp_bc_acked (hdr); -if (acked != grp->bc_snd_nxt || m->bc_acked == acked) +if (acked != grp->bc_snd_nxt || !m->bc_pending)return ; - m->bc_acked = acked; + m->bc_pending = false;if (--grp->bc_ackers)return ;list_del_init (&m->small_win);
net/tipc/socket.c +11 -1
@@ -2356,8 +2356,18 @@ static void tipc_sk_filter_rcv(struct sock *sk, struct sk_buff *skb,if (unlikely (!msg_isdata (hdr)))tipc_sk_proto_rcv (sk, &inputq, xmitq); -if (unlikely (grp)) +if (unlikely (grp)) { +bool was_open = tsk->group_is_open; +tipc_group_filter_msg (grp, &inputq, xmitq); +/* A departing member may have held the last outstanding +* replicast ack, which can restore the open state inside +* tipc_group_filter_msg(); wake up senders blocked in +* poll()/write() if the group just became writable. +*/ +if (!was_open && tsk->group_is_open) + sk->sk_write_space (sk); + }if (unlikely (!grp) && mtyp ==TIPC_MCAST_MSG )tipc_mcast_filter_msg (net, &tsk->mc_method.deferredq, &inputq);
Signature · 1 line
2.43.7
RE: [PATCH net v4] tipc: fix broadcast sender hang on last acker departure
Quoted 37 lines
Subject: [PATCH net v4] tipc: fix broadcast sender hang on last acker departure
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks:
tipc_group_bc_cong()returns true as long asgrp->bc_ackersis non-zero, and clears*grp->open, which is whattipc_poll()uses to report EPOLLOUT (viatsk->group_is_open).When the last outstanding acker departs before acking,
tipc_group_delete_member()decrementsgrp->bc_ackersto zero but never restores*grp->open. As a result poll(EPOLLOUT) never fires again. A writer blocked insendmsg()is only released if some other wakeup happens after the
deletion: withTIPC_GROUP_MEMBER_EVTSthe leave event delivery callssk_data_ready()and wakes it, but without member events nothing does and it sleeps indefinitely, since the default send timeout isMAX_SCHEDULE_TIMEOUT.Commit
99cc2a62e07a("tipc: reject invalid and unexpectedGRP_ACK_MSGto preventbc_ackersunderflow") already calls this out in its changelog as a pre- existing issue.Restore the open state exactly on the 1->0 transition in the delete path, mirroring the
GRP_ACK_MSGcompletion path, and invokesk->sk_write_space()fromtipc_sk_filter_rcv()when the group just became writable again.Also make the ack accounting precise. The delete path used to test less(m-
Quoted 1 line
bc_acked,grp->bc_snd_nxt- 1), which also matches members that werenever counted for the current replicast (JOINING/LEAVING members keep a stale
bc_acked), and theGRP_ACK_MSGhandler accepted an ack from any member whosebc_ackeddiffered from the expected value. Either could zerobc_ackerson behalf of a member that owes nothing and silently void an ack still owed by a real acker. Track whether a member owes an ack for the current replicast in a new per-memberbc_pendingflag, set when the member is counted and cleared when its ack arrives, and use it both to accept an ack and to decide on deletion.bc_ackedhas no remaining user and is removed. Suggested by Tung Quang Nguyen.
Quoted 6 lines
This issue was discovered by Tencent CodeBuddy Security.
Fixes 2f487712b893 ("tipc: guarantee that group broadcast doesn't bypassgroup unicast")
Signed-off-by Henry Martin <[email protected]>
Re: [PATCH net v4] tipc: fix broadcast sender hang on last acker departure
In reply to Henry MartinThis patch was applied to netdev/net.git (main) by Paolo Abeni <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Paolo Abeni <[email protected]>:
Quoted 14 lines · On Mon, 19 Oct 2026 19:58:59 +0800 you wrote:
TIPC group broadcast senders are blocked while a replicast still has outstanding member acks:
tipc_group_bc_cong()returns true as long asgrp->bc_ackersis non-zero, and clears*grp->open, which is whattipc_poll()uses to report EPOLLOUT (viatsk->group_is_open).When the last outstanding acker departs before acking,
tipc_group_delete_member()decrementsgrp->bc_ackersto zero but never restores*grp->open. As a result poll(EPOLLOUT) never fires again. A writer blocked insendmsg()is only released if some other wakeup happens after the deletion: withTIPC_GROUP_MEMBER_EVTSthe leave event delivery callssk_data_ready()and wakes it, but without member events nothing does and it sleeps indefinitely, since the default send timeout isMAX_SCHEDULE_TIMEOUT.
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] tipc: fix broadcast sender hang on last acker departure
https://git.kernel.org/netdev/net/c/f4377bf85887
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Tung Quang Nguyen
maintainerRe: [PATCH net v2] tipc: fix broadcast sender hang on last acker departure
In reply to Henry Martinnet/tipc/group.c void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack) · line 415
Quoted 7 lines
net/tipc/group.c +2
-0@@ -406,6 +410,8 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack) m->bc_acked = prev; ackers++; } +if (ack) + m->bc_pending =tipc_group_is_receiver (m); }
This needs to be done in a simple way:
@ -405,6 +405,7 @@ void tipc_group_update_bc_members(struct tipc_group *grp, int len, bool ack)
tipc_group_update_member(m, len);
m->bc_acked = prev;
ackers++;
+ m->bc_pending = ack;
}
}