net/smc: drop the abort_work reference when the work is cancelled
PATCH 1/1 net/smc: drop the abort_work reference when the work is cancelled
The schedulers of conn->abort_work hand a socket reference to the work item and rely on it to give the reference back:
sock_hold(&smc->sk); /* sock_put in abort_work */
if (!queue_work(smc_close_wq, &conn->abort_work))
sock_put(&smc->sk);The queue_work() failure case is handled, but the cancellation case is not. smc_conn_free() cancels a still-pending abort_work:
if (current_work() != &conn->abort_work)
cancel_work_sync(&conn->abort_work);and discards the return value. When cancel_work_sync() returns true the work was queued but had not started, so smc_conn_abort_work() never runs and its sock_put() never happens. The reference is lost.
As in smc_switch_conns(), a leaked sk_refcnt means the smc_sock is never
destroyed: its buffers stay allocated and the network namespace reference a user socket holds is never released, so the netns cannot be torn down.
smc_cdc_msg_validate() queues abort_work from the receive tasklet when a peer sends a CDC message with an out-of-order sequence number, so a remote peer combined with a concurrent local close is enough to reach it.
Drop the reference when the work is cancelled, matching the pattern smc_close_cancel_work() already uses for conn->close_work:
if (cancel_work_sync(&smc->conn.close_work))
sock_put(sk);The sock_put() is safe here: every caller of smc_conn_free() passes the connection of a socket it holds a reference to, so this cannot release the last one.
--- net/smc/smc_core.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
net/smc/smc_core.c +8 -2
@@ -1254,6 +1254,7 @@ static void smc_buf_unuse(struct smc_connection *conn,/* remove a finished connection from its link group */ void smc_conn_free (struct smc_connection *conn) { + struct smc_sock *smc = container_of(conn, struct smc_sock, conn);struct smc_link_group *lgr = conn->lgr;if (!lgr || conn->freed)
@@ -1277,8 +1278,13 @@ void smc_conn_free(struct smc_connection *conn)tasklet_kill (&conn->rx_tsklet); }else {smc_cdc_wait_pend_tx_wr (conn); -if (current_work () != &conn->abort_work) -cancel_work_sync (&conn->abort_work); +/* If the work was pending (cancel returns true) it never ran, +* so the sock_hold taken by its scheduler was never released. +*/ +if (current_work () != &conn->abort_work) { +if (cancel_work_sync (&conn->abort_work)) +sock_put (&smc->sk); + } }if (!list_empty (&lgr->list)) {smc_buf_unuse (conn, lgr);/* allow buffer reuse */
base-commit: 2b1c2bc2355fd59cc75045e42d9dc5470ef5fa9a
Signature · 1 line
2.52.0
Quoted 35 lines · Hidayath Khan
The schedulers of
conn->abort_workhand a socket reference to the work item and rely on it to give the reference back:sock_hold(&smc->sk); /* sock_put in abort_work */ if (!queue_work(smc_close_wq, &conn->abort_work)) sock_put(&smc->sk);The
queue_work()failure case is handled, but the cancellation case is not.smc_conn_free()cancels a still-pendingabort_work:if (current_work() != &conn->abort_work) cancel_work_sync(&conn->abort_work);and discards the return value. When
cancel_work_sync()returns true the work was queued but had not started, sosmc_conn_abort_work()never runs and itssock_put()never happens. The reference is lost.As in
smc_switch_conns(), a leakedsk_refcntmeans thesmc_sockis never
destroyed: its buffers stay allocated and the network namespace reference a user socket holds is never released, so the netns cannot be torn down.
smc_cdc_msg_validate()queuesabort_workfrom the receive tasklet when a peer sends a CDC message with an out-of-order sequence number, so a remote peer combined with a concurrent local close is enough to reach it.Drop the reference when the work is cancelled, matching the pattern
smc_close_cancel_work()already uses forconn->close_work:if (cancel_work_sync(&smc->conn.close_work)) sock_put(sk);The
sock_put()is safe here: every caller ofsmc_conn_free()passes the connection of a socket it holds a reference to, so this cannot release the last one.
Quoted 5 lines
Fixes b286a0651e44 ("net/smc: handle incoming CDC validation message")Reviewed-by Mahanta Jambigi <[email protected]>Reviewed-by Sidraya Jayagond <[email protected]>Signed-off-by Hidayath Khan <[email protected]>
Best regards,
Dust
Quoted 36 lines
---
net/smc/smc_core.c| 10 ++++++++--1 file changed, 8 insertions(+), 2 deletions(-)net/smc/smc_core.c +8
-2@@ -1254,6 +1254,7 @@ static void smc_buf_unuse(struct smc_connection *conn,/* remove a finished connection from its link group */ void smc_conn_free (struct smc_connection *conn) { + struct smc_sock *smc = container_of(conn, struct smc_sock, conn);struct smc_link_group *lgr = conn->lgr;if (!lgr || conn->freed)@@ -1277,8 +1278,13 @@ void smc_conn_free(struct smc_connection *conn)tasklet_kill (&conn->rx_tsklet); }else {smc_cdc_wait_pend_tx_wr (conn); -if (current_work () != &conn->abort_work) -cancel_work_sync (&conn->abort_work); +/* If the work was pending (cancel returns true) it never ran, +* so the sock_hold taken by its scheduler was never released. +*/ +if (current_work () != &conn->abort_work) { +if (cancel_work_sync (&conn->abort_work)) +sock_put (&smc->sk); + } }if (!list_empty (&lgr->list)) {smc_buf_unuse (conn, lgr);/* allow buffer reuse */ base-commit:
2b1c2bc2355fd59cc75045e42d9dc5470ef5fa9aSignature · 1 line
2.52.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] AB-BA deadlock between smc_conn_free() and smc_conn_abort_work() caused by calling cancel_work_sync() while holding the socket lock. --
--- Patch [1]: [PATCH net] net/smc: drop the abort_work reference when the work is cancelled --- commit 1b6cdab59cd811b97297f9b5597cac1aa3e420d9
Author: Hidayath Khan <[email protected]>
net/smc: drop the abort_work reference when the work is cancelled
This commit fixes a socket reference leak that occurs when an abort_work item is cancelled before it runs. It drops the socket reference if cancel_work_sync() returns true, ensuring the socket can be properly destroyed.
Quoted 4 lines
net/smc/smc_core.c +0
-0
[ ... ]
net/smc/smc_core.c void smc_conn_free(struct smc_connection *conn) · line 1284
Quoted 12 lines
net/smc/smc_core.c +6
-2@@ -1275,8 +1276,13 @@ void smc_conn_free(struct smc_connection *conn)tasklet_kill (&conn->rx_tsklet); }else {smc_cdc_wait_pend_tx_wr (conn); -if (current_work () != &conn->abort_work) -cancel_work_sync (&conn->abort_work); +/* If the work was pending (cancel returns true) it never ran, +* so the sock_hold taken by its scheduler was never released. +*/ +if (current_work () != &conn->abort_work) { +if (cancel_work_sync (&conn->abort_work)) +sock_put (&smc->sk);
[Severity: High]
This isn't a bug introduced by this patch, but can calling cancel_work_sync() here cause an AB-BA deadlock with the workqueue thread?
If a thread initiates a socket close, such as via smc_release() or smc_close_passive_work(), it acquires the socket lock:
smc_release()
lock_sock(&smc->sk);
...
smc_conn_free()
cancel_work_sync(&conn->abort_work);
If a remote peer concurrently sends an out-of-order CDC message, abort_work is queued. The workqueue thread starts executing smc_conn_abort_work() and attempts to acquire the same socket lock:
smc_conn_abort_work()
lock_sock(&smc->sk);
smc_conn_kill(conn, true);
release_sock(&smc->sk);
Could this result in the workqueue thread blocking on the socket lock, while the socket release thread blocks indefinitely in cancel_work_sync() waiting for the workqueue thread to finish?
Quoted 4 lines
net/smc/smc_core.c +1
-0+ } }if (!list_empty (&lgr->list)) {smc_buf_unuse (conn, lgr);/* allow buffer reuse */
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Jakub Kicinski
maintainerThis is an AI-generated review of your patch. The human sending this email says "*shiko finds more issues with the work termination here. Can you please TAL and fix these all in one series?"
---
net/smc: drop the abort_work reference when the work is cancelled
The patch makes smc_conn_free() drop the socket reference handed to conn->abort_work when cancel_work_sync() reports that the work was still pending, so the sock_hold() taken by the scheduler of the work is not leaked. It follows the pattern smc_close_cancel_work() already uses for conn->close_work.
A few questions about the surrounding cancel logic follow.
Quoted 15 lines
net/smc/smc_core.c +1
-0@@ -1252,6 +1252,7 @@ static void smc_buf_unuse(struct smc_connection *conn,/* remove a finished connection from its link group */ void smc_conn_free (struct smc_connection *conn) { + struct smc_sock *smc = container_of(conn, struct smc_sock, conn);struct smc_link_group *lgr = conn->lgr;if (!lgr || conn->freed)@@ -1275,8 +1276,13 @@ static void smc_buf_unuse(struct smc_connection *conn,tasklet_kill (&conn->rx_tsklet); }else {
net/smc/smc_core.c static void smc_buf_unuse(struct smc_connection *conn, · line 1285
Quoted 10 lines
net/smc/smc_core.c +7
-2smc_cdc_wait_pend_tx_wr (conn); -if (current_work () != &conn->abort_work) -cancel_work_sync (&conn->abort_work); +/* If the work was pending (cancel returns true) it never ran, +* so the sock_hold taken by its scheduler was never released. +*/ +if (current_work () != &conn->abort_work) { +if (cancel_work_sync (&conn->abort_work)) +sock_put (&smc->sk); + }
[Severity: High]
This isn't a bug introduced by this patch, but can this cancel_work_sync() deadlock against a already-running abort_work?
The worker takes the socket lock as its first action:
net/smc/smc_core.c:smc_conn_abort_work() {
lock_sock(&smc->sk);
smc_conn_kill(conn, true);
release_sock(&smc->sk);
sock_put(&smc->sk); /* sock_hold done by schedulers of abort_work */
}and the callers of smc_conn_free() reach it while holding that same lock:
net/smc/af_smc.c:__smc_release()
lock_sock(sk); ... if (!smc->use_fallback) smc_conn_free(&smc->conn);
net/smc/smc_close.c:smc_close_passive_work() (inside lock_sock(sk))
if ((sk->sk_state == SMC_CLOSED) &&
(sock_flag(sk, SOCK_DEAD) || !sk->sk_socket)) {
smc_conn_free(conn);net/smc/smc_close.c:smc_close_active_abort() (SMC_PEERCLOSEWAIT1/2, SMC_PEERFINCLOSEWAIT, SMC_PROCESSABORT, SMC_APPFINCLOSEWAIT)
smc_conn_free(&smc->conn);
So if abort_work has already started on another CPU and is blocked in lock_sock(), the closing task holds the socket lock and waits in cancel_work_sync() for that worker, which cannot proceed. The current_work() != &conn->abort_work test only covers the case where the same task is running abort_work itself.
Note both abort_work and close_work are queued on smc_close_wq, so close_work running smc_close_passive_work() can be the blocking canceller.
The commit message cites smc_close_cancel_work() as the model:
if (cancel_work_sync(&smc->conn.close_work)) sock_put(sk);
but that helper drops the socket lock around the cancel:
net/smc/smc_close.c:smc_close_cancel_work() {
release_sock(sk);
if (cancel_work_sync(&smc->conn.close_work))
sock_put(sk);
cancel_delayed_work_sync(&smc->conn.tx_work);
lock_sock(sk);
}Should the abort_work cancel do the same, and would that change where the new sock_put() has to live?
[Severity: High]
This is a pre-existing ordering issue rather than something the patch adds, but does cancel_work_sync() here actually keep abort_work from being re-armed?
The cancel happens before smc_lgr_unregister_conn(conn), so the receive path can still find the connection:
net/smc/smc_cdc.c:smc_cdc_rx_handler() {
read_lock_bh(&lgr->conns_lock);
conn = smc_lgr_find_conn(ntohl(cdc->token), lgr);
if (!conn || conn->out_of_sync) {
read_unlock_bh(&lgr->conns_lock);
return;
}
smc = container_of(conn, struct smc_sock, conn);
sock_hold(&smc->sk);
read_unlock_bh(&lgr->conns_lock);and then queue the work again:
net/smc/smc_cdc.c:smc_cdc_msg_validate() {
sock_hold(&smc->sk); /* sock_put in abort_work */
if (!queue_work(smc_close_wq, &conn->abort_work))
sock_put(&smc->sk);cancel_work_sync() passes cflags 0, so it re-enables the work before returning:
kernel/workqueue.c:__cancel_work_sync() {
ret = __cancel_work(work, cflags | WORK_CANCEL_DISABLE);
...
if (wq_online)
__flush_work(work, true);
if (!(cflags & WORK_CANCEL_DISABLE))
enable_work(work);which means a queue_work() issued after the cancel returns succeeds. The freshly queued worker then runs smc_conn_abort_work() -> smc_conn_kill() on a connection with conn->freed == 1 whose lgr reference smc_conn_free() already dropped, and dereferences conn->lgr:
net/smc/smc_core.c:smc_conn_kill() {
if (conn->lgr->is_smcd && conn->lgr->peer_shutdown)Would unregistering the connection from lgr->conns_all before cancelling, or gating the queueing on conn->freed under conns_lock, close that window?
net/smc/smc_core.c void smc_conn_free(struct smc_connection *conn) · line 1290
Quoted 3 lines
net/smc/smc_core.c +0
-0}if (!list_empty (&lgr->list)) {smc_buf_unuse (conn, lgr);/* allow buffer reuse */
[Severity: High]
This is also a pre-existing problem and not something introduced here, but the patch documents smc_conn_free() as the place that disposes of a pending abort_work, and one exit path of the function skips that disposal:
net/smc/smc_core.c:smc_conn_free() {
if (!smc_conn_lgr_valid(conn))
/* Connection has already unregistered from
* link group.
*/
goto lgr_put;
...
lgr_put:
if (!lgr->is_smcd)
smcr_link_put(conn->lnk); /* link_hold in smc_conn_create() */
smc_lgr_put(lgr); /* lgr_hold in smc_conn_create() */
}On that path neither the cancel nor the new sock_put() runs, yet the connection's link and link-group references are still dropped, and conn->lgr is not reset. A pending abort_work then dereferences it:
net/smc/smc_core.c:smc_conn_kill() {
if (conn->lgr->is_smcd && conn->lgr->peer_shutdown)Concretely, a peer sends a CDC message with prod_flags.failover_validation and a stale seqno, so smc_cdc_msg_validate() queues abort_work. Meanwhile __smc_lgr_terminate() calls smc_conn_kill(), which does smc_lgr_unregister_conn() (clearing alert_token_local) and then smc_close_active_abort() -> smc_conn_free(); smc_conn_lgr_valid(conn) is now false, so the goto lgr_put path is taken. __smc_lgr_terminate() then calls smc_lgr_free(), whose smc_lgr_put() drops the last reference and kfree()s the lgr while abort_work is still queued.
Can the queued abort_work read a freed struct smc_link_group here, and should this exit path also cancel the work and return its reference?
Thanks for the review.
All three issues are real, and none of them is fixed by anything currently in flight. I will include this patch in a series against net that fixes all four.
Quoted 34 lines · Jakub Kicinski
This is an AI-generated review of your patch. The human sending this email says "*shiko finds more issues with the work termination here. Can you please TAL and fix these all in one series?"
---
net/smc: drop theabort_workreference when the work is cancelledThe patch makes
smc_conn_free()drop the socket reference handed toconn->abort_workwhencancel_work_sync()reports that the work was still pending, so thesock_hold()taken by the scheduler of the work is not leaked. It follows the patternsmc_close_cancel_work()already uses forconn->close_work.A few questions about the surrounding cancel logic follow.
Quoted 20 lines
net/smc/smc_core.c +3
-2@@ -1252,6 +1252,7 @@ static void smc_buf_unuse(struct smc_connection *conn,/* remove a finished connection from its link group */ void smc_conn_free (struct smc_connection *conn) { + struct smc_sock *smc = container_of(conn, struct smc_sock, conn);struct smc_link_group *lgr = conn->lgr;if (!lgr || conn->freed)@@ -1275,8 +1276,13 @@ static void smc_buf_unuse(struct smc_connection *conn,tasklet_kill (&conn->rx_tsklet); }else {smc_cdc_wait_pend_tx_wr (conn); -if (current_work () != &conn->abort_work) -cancel_work_sync (&conn->abort_work); +/* If the work was pending (cancel returns true) it never ran, +* so the sock_hold taken by its scheduler was never released.
Quoted 8 lines
net/smc/smc_core.c void smc_conn_free(struct smc_connection *conn) · line 1287
Quoted 5 lines
net/smc/smc_core.c +5
-0+*/ +if (current_work () != &conn->abort_work) { +if (cancel_work_sync (&conn->abort_work)) +sock_put (&smc->sk); + }[Severity: High]
This isn't a bug introduced by this patch, but can thiscancel_work_sync()deadlock against a already-runningabort_work?
Yes, smc_conn_free() holds the socket lock and smc_conn_abort_work() takes the same lock, so once the work has started the cancel waits for the work and the work waits for the caller. The current_work() test only stops the work cancelling itself, so it does not help when the two are on different CPUs. I will drop the lock around the cancel, the way smc_close_cancel_work() already does for close_work.
Quoted 51 lines
The worker takes the socket lock as its first action:
net/smc/smc_core.c:smc_conn_abort_work() { lock_sock(&smc->sk); smc_conn_kill(conn, true); release_sock(&smc->sk); sock_put(&smc->sk); /* sock_hold done by schedulers of abort_work */ }and the callers of
smc_conn_free()reach it while holding that same lock:net/smc/af_smc.c:__smc_release()
lock_sock(sk); ... if (!smc->use_fallback) smc_conn_free(&smc->conn);net/smc/smc_close.c:smc_close_passive_work() (inside
lock_sock(sk))if ((sk->sk_state == SMC_CLOSED) && (sock_flag(sk, SOCK_DEAD) || !sk->sk_socket)) { smc_conn_free(conn);net/smc/smc_close.c:smc_close_active_abort() (SMC_PEERCLOSEWAIT1/2,
SMC_PEERFINCLOSEWAIT,SMC_PROCESSABORT,SMC_APPFINCLOSEWAIT)smc_conn_free(&smc->conn);So if
abort_workhas already started on another CPU and is blocked inlock_sock(), the closing task holds the socket lock and waits incancel_work_sync()for that worker, which cannot proceed. Thecurrent_work()!=&conn->abort_worktest only covers the case where the same task is runningabort_workitself.Note both
abort_workandclose_workare queued onsmc_close_wq, soclose_workrunningsmc_close_passive_work()can be the blocking canceller.The commit message cites
smc_close_cancel_work()as the model:if (cancel_work_sync(&smc->conn.close_work)) sock_put(sk);but that helper drops the socket lock around the cancel:
net/smc/smc_close.c:smc_close_cancel_work() { release_sock(sk); if (cancel_work_sync(&smc->conn.close_work)) sock_put(sk); cancel_delayed_work_sync(&smc->conn.tx_work); lock_sock(sk); }
Quoted 2 lines
Should the
abort_workcancel do the same, and would that change where the newsock_put()has to live?
Yes to both. I will move the cancel into a small helper that drops the socket lock first, and the sock_put() goes inside that helper, after the lock is dropped, exactly as in smc_close_cancel_work().
Quoted 44 lines
[Severity: High]
This is a pre-existing ordering issue rather than something the patch adds, but doescancel_work_sync()here actually keepabort_workfrom being re-armed?The cancel happens before
smc_lgr_unregister_conn(conn), so the receive path can still find the connection:net/smc/smc_cdc.c:smc_cdc_rx_handler() { read_lock_bh(&lgr->conns_lock); conn = smc_lgr_find_conn(ntohl(cdc->token), lgr); if (!conn || conn->out_of_sync) { read_unlock_bh(&lgr->conns_lock); return; } smc = container_of(conn, struct smc_sock, conn); sock_hold(&smc->sk); read_unlock_bh(&lgr->conns_lock);and then queue the work again:
net/smc/smc_cdc.c:smc_cdc_msg_validate() { sock_hold(&smc->sk); /* sock_put in abort_work */ if (!queue_work(smc_close_wq, &conn->abort_work)) sock_put(&smc->sk);
cancel_work_sync()passes cflags 0, so it re-enables the work before returning:kernel/workqueue.c:__cancel_work_sync() { ret = __cancel_work(work, cflags | WORK_CANCEL_DISABLE); ... if (wq_online) __flush_work(work, true); if (!(cflags & WORK_CANCEL_DISABLE)) enable_work(work);which means a
queue_work()issued after the cancel returns succeeds. The freshly queued worker then runssmc_conn_abort_work()->smc_conn_kill()on a connection withconn->freed== 1 whose lgr referencesmc_conn_free()already dropped, and dereferencesconn->lgr:
Quoted 5 lines
net/smc/smc_core.c:smc_conn_kill() { if (conn->lgr->is_smcd && conn->lgr->peer_shutdown)Would unregistering the connection from
lgr->conns_allbefore cancelling, or gating the queueing onconn->freedunderconns_lock, close that window?
The first one. Gating the queue site would need smc_cdc_msg_validate() to take conns_lock, which it does not do today, and it would not help with the buffers.
So I will unregister first, then cancel. The buffer release has to move as
well: cancel_work_sync() waits for a work item that may still start during the wait, and that item would then run against buffers smc_buf_unuse() had already returned. The order becomes unregister, cancel, release.
Quoted 38 lines
net/smc/smc_core.c void smc_conn_free(struct smc_connection *conn) · line 1290
Quoted 4 lines
net/smc/smc_core.c +0
-0}if (!list_empty (&lgr->list)) {smc_buf_unuse (conn, lgr);/* allow buffer reuse */ [Severity: High]
This is also a pre-existing problem and not something introduced here, but the patch documentssmc_conn_free()as the place that disposes of a pendingabort_work, and one exit path of the function skips that disposal:net/smc/smc_core.c:smc_conn_free() { if (!smc_conn_lgr_valid(conn)) /* Connection has already unregistered from * link group. */ goto lgr_put; ... lgr_put: if (!lgr->is_smcd) smcr_link_put(conn->lnk); /* link_hold in smc_conn_create() */ smc_lgr_put(lgr); /* lgr_hold in smc_conn_create() */ }On that path neither the cancel nor the new
sock_put()runs, yet the connection's link and link-group references are still dropped, andconn->lgris not reset. A pendingabort_workthen dereferences it:net/smc/smc_core.c:smc_conn_kill() { if (conn->lgr->is_smcd && conn->lgr->peer_shutdown)Concretely, a peer sends a CDC message with prod_flags.failover_validation and a stale seqno, so
smc_cdc_msg_validate()queuesabort_work. Meanwhile__smc_lgr_terminate()callssmc_conn_kill(), which doessmc_lgr_unregister_conn()(clearingalert_token_local) and thensmc_close_active_abort()->smc_conn_free();smc_conn_lgr_valid(conn)is now false, so the gotolgr_putpath is taken.__smc_lgr_terminate()then callssmc_lgr_free(), whosesmc_lgr_put()drops the last reference and kfree()s the lgr whileabort_workis still queued.
Quoted 2 lines
Can the queued
abort_workread a freed structsmc_link_grouphere, and should this exit path also cancel the work and return its reference?
Yes to both. smc_conn_kill() clears alert_token_local through smc_lgr_unregister_conn(), so smc_conn_lgr_valid() is already false when smc_close_active_abort() reaches smc_conn_free(), and the goto lgr_put path skips both the cancel and the reference. __smc_lgr_terminate() then frees the link group underneath the queued work. I will cancel on that path as well.
I will send the series shortly.