net/rds: make connection lifetime reference-counted
Subsystems (4)
Allison Henderson
maintainerv12 cover net/rds: make connection lifetime reference-counted
Hi all,
This is v12 of the connection-lifetime set (v1 at [1], v2 at [2], v3 at [3], v4 at [6], v5 at [7], v6 at [8], v7 at [9], v8 at [10], v9 at [11], v10 at [13], v11 at [14]), following
"net/rds: own the fastpath locks across connection teardown", which is in net-next. It is targeted at net-next: although the series fixes real use-after-frees (one syzbot report and one report from Chengfeng Ye below), it does so by reworking connection lifetime rather than patching the individual crash sites, and that rework is too invasive for net.
rds_conn_destroy() frees the connection, its paths and its workqueues on the spot, relying on the documented assumption that "no one else is referencing the connection". That assumption stopped being true a long time ago: connections are destroyed on netns teardown and on a protocol-version mismatch as well as on rmmod, while pointers to them still live in socket rs_conn caches, congestion-map conn lists, CM callbacks and workers, and - for as long as an application leaves data unread - in every rds_incoming sitting on a receive queue. No single
Fixes: commit covers the rot, so the series carries Reported-by tags where there are concrete reports instead.
Patches 5, 8, 7 and 12 are ports of the connection kref work Sharath Srinivasan did for Oracle UEK, adapted to the upstream code; patch 2 mirrors UEK's device-removal gate in rds_ib_add_conn().
The series has been validated with the RDS selftests over loopback-TCP and RXE-RDMA, plus churn tests that delete network namespaces and unload the modules under live rds-stress traffic.
Patch 1 makes the passive-connection exits of __rds_conn_create() undo conn_alloc() the same way the lost-race exit does, through one helper (a refactor: the passive twin only exists for IB, so nothing leaked there). It comes first because the reference-counted teardown calls conn_free() from more places.
Patch 2 has rds_ib_dev_shutdown() mark a device as shutting down under the lock rds_ib_add_conn() attaches under, and makes rds_ib_add_conn() refuse a marked device: a connect that was past rds_ib_get_client_data() when the device's connections were dropped could attach to it afterwards and, on module unload, never be torn down.
Based on UEK commit:
8d0639e6ba2c ("net/rds: Clean up ib_nodev_conns list handling")
https://github.com/oracle/linux-uek/commit/8d0639e6ba2c
Patch 3 guards the five work-arming sites that never tested rds_destroy_pending(): two IB completion paths, the IB recv refill, the TCP accept kick and the multipath reconnect in sendmsg.
Patch 4 gives the connection its own destroy-in-progress marker. One destroy caller does escape the old predicate - unloading the core module runs the loopback pernet exit for every live namespace, where check_net() is still true and the loop transport's unloading flag is not yet set - though nothing can queue work at that point today. The later patches need the precise answer: the passive-twin creation must refuse a parent whose destroy has begun, and sendmsg must drop a cached connection whose destroy has.
Based on UEK commits:
e2f5005adf63 net/rds: Add krefs to struct rds_connection
https://github.com/oracle/linux-uek/commit/e2f5005adf63
6c53ef92f46e net/rds: Merge uses of conn->c_destroy_in_prog & RDS_DESTROY_PENDING
https://github.com/oracle/linux-uek/commit/6c53ef92f46e Patch 5 splits rds_conn_destroy() into a synchronous quiesce and a kref-governed free.
Based on UEK commits:
2c8569e4c880 ("net/rds: Add krefs to struct rds_connection").
https://github.com/oracle/linux-uek/commit/2c8569e4c880Patch 6 unlinks each transport node before its destroy, so a free deferred past the teardown loop cannot write into the loop's stack-local list head. IB marks a gathered node as claimed by the sweep, since its connect and shutdown paths move the node too; for IB this also closes an existing corruption of the nodev list on module unload, through the stale spliced head.
Patch 7 makes each transport's exit path wait for its connections to actually be freed before the module goes away. It comes after the unlink patch now, so that the IB resweep only ever finds claimed nodes. The wait is
unbounded and warns every ten seconds: a connection reference can be held for an application-controlled time (unread data), so a timeout would only move the use-after-free from freed memory to unloaded module text. rmmod blocking while data is queued and unread is the historical RDS contract. For IB the wait re-sweeps the nodev list, since device connections migrate to it asynchronously.
Based on UEK commits:
ece4b4e39afa ("net/rds: wait_event_timeout until zero connections during rmmod")
https://github.com/oracle/linux-uek/commit/ece4b4e39afa
905ec90e6166 ("net/rds: Each RDS transport should keep its own connection count")
https://github.com/oracle/linux-uek/commit/905ec90e6166
Patch 8 makes struct rds_incoming hold a reference on i_conn, the
fix for the KASAN use-after-free Chengfeng Ye reported [4].
Based on UEK commits:
99b9a3715419 ("net/rds: fix crash by expanding kref coverage to rds_incoming.i_conn").
https://github.com/oracle/linux-uek/commit/99b9a3715419
Patch 9 has the quiesce purge cp_send_queue under cp_lock, as every adder to that queue holds it, and clear RDS_MSG_ON_CONN as it does so, the way rds_send_drop_to() expects. No sender can be in flight at any of today's destroy triggers (netns teardown, module unload), so this is hygiene rather than a race fix; the old BUG_ON() on a socket-owned message becomes a WARN_ON_ONCE().
Patch 10 hands out real references everywhere a connection pointer previously escaped bare, RCU-annotates parent->c_passive, refuses to revive a passive connection whose destroy has begun, and serializes the SIOCRDSSETTOS check with the rs_conn cache.
Based on UEK commits:
2c8569e4c880 ("net/rds: Add krefs to struct rds_connection")
https://github.com/oracle/linux-uek/commit/2c8569e4c880
0e9e3a72b7f7 ("net/rds: rds_sendmsg must use rs_conn only when not being destroyed").
https://github.com/oracle/linux-uek/commit/0e9e3a72b7f7 Patch 11 gives cm_id->context a reference on its connection, taken when the id is created or installed and dropped after rdma_destroy_id(), and, on both the passive and the active side, declines to set up a QP for a connection whose destroy has already begun.
Patch 12 drops the now-unused global rds_conn_count.
Based on UEK commits:
905ec90e6166 ("net/rds: Each RDS transport should keep its own connection count").
https://github.com/oracle/linux-uek/commit/905ec90e6166
Changes since v11 [14]:
- Rebased onto current net-next (no net/rds changes in between).
- Patch 4: the commit message's history of RDS_DESTROY_PENDING was wrong. It now follows c14b0366813a -> c90ecbfaf50d2 -> ebeeb1ad9b8a -> cdc306a5c9cd3 and says that the patch brings back a per-connection flag mainline once had.
- Patch 11: cm_id->context now owns a reference on the connection, taken in rds_ib_conn_path_connect() and rds_ib_cm_handle_connect() and dropped after rdma_destroy_id(), in place of the handler pinning the connection per event - a pin that could not have caught a connection freed before the handler ran. Retitled.
- Patch 8 is unchanged: the unload wait on unread datagrams stays, see the v11 thread.
Changes since v10 [13]:
- Rebased onto current net-next.
- The v10 patch 1 (rds_ib_conn_free() irqsave) is dropped: Nguyen Ngoc Thang's fix for the same bug [12] is in net-next as f6fa1d60ec3c. Every later patch moves up by one.
- Patch 6: the rds_ib_conn_free() comment merges with the one that fix added.
- Patch 10: the sendmsg cache-hit test keeps the c_dev_if comparison from "net/rds: include the scope id in the sendmsg connection cache check", now in net-next.
Changes since v9 [11]:
- Rebased onto net-next with the RDMA-CM handler fixes (492ba6ac1b1f, 98d446b7c013); patch 12 no longer has the transport assignment lines to remove.
- Patch 3: a connect refused by the gate can hold the device's last reference and drop it after the only rds_wq flush on unload, so rds_ib_exit() flushes rds_wq again after the sweep here; patch 8 then moves that flush after the wait. The changelog says which half of the in-flight-connect race this closes and that the unload wait closes the other.
- Patches 7 and 8 keep their Fixes: tags and gain a
"Cc: <[email protected]>" naming the dependency on the reference counting, so stable tooling skips them; patch 9 gains a
Fixes: for the original i_conn with the same Cc, and patch 11 a
Fixes: for the sendmsg cache with the same Cc.
- Patch 7: garbled paragraph fixed; the loop conn_free() description corrected.
- Patch 11: authored by Sharath Srinivasan, matching its SoB chain and bracket note like patches 6 and 8.
- Patch 1 carries the syzbot Reported-by/Closes.
Changes since v8 [10]:
- Patch 3: changelog names unloading rds_rdma with a connect in flight as the trigger; a hot removal is caught by the rdma_cm's own DEVICE_REMOVAL event.
- Patch 4: blank line before the trailers.
- Patch 7: rds_ib_add_conn() refuses a node a sweep has claimed, as it refuses a device that is shutting down, so the "dropped or refused" invariant holds by construction; changelog describes the nodev-list corruption on unload the IB part fixes, with
stack list.
- Patch 8: Fixes: 4716af3897e9 for the two unload holes the resweep and the rds_wq flush close; opening paragraph no longer calls the wait a no-op at that point in the series.
- Patch 9: the reachable trigger is a cross-transport unload (a socket bound with one transport holding incs from another's connection); the deliberate absence of a Fixes: tag is stated; the stale XXX comment in rds_inc_on_sock_queue() is removed.
- Patch 11: the eviction comments say the same sendmsg call creates the replacement, and the rs_conn comment lists the eviction.
- Patch 12: the initiate_connect and handle_connect checks are described as defensive, like the pin; neither path is reachable today.
- Rebased onto current net-next.
- The scope-id comparison the review asked for in the sendmsg cache test is a separate fix in flight for net ("net/rds: scope-aware sendmsg connection cache").
Changes since v7 [9]:
- New patch 3: rds_ib_add_conn() refuses a device that rds_ib_dev_shutdown() has marked, so a connect racing device removal or module unload cannot attach a connection nothing tears down; with the unload wait that race would have become an rmmod that never returns.
- Patches 6 and 7 swapped (unlink/claim before the unload wait), so the resweep never runs over a list an in-flight connect can move a node out of; the nodev-list assertion goes with the other two in the unlink patch.
- Patches 4 and 5 lose their Fixes: tags: the sites patch 4 guards cannot fire at a destroy with today's predicate, and the loopback pernet-exit window patch 5 describes has no queuer at module exit (every socket pins the module). Both are described as preparation for the per-connection predicate the series needs.
- Patch 5: the c_destroy_in_prog comment and the changelog describe the destroy == true rds_conn_path_drop() from IB device removal as flushed by the module exit's later destroy, not by the removal.
- Patch 6: the first-caller guard is documented as the contract, not as closing a double destroy - no sweep hands a connection to rds_conn_destroy() twice now that every sweep unlinks or claims the node first; the "at this point in the series" wording is gone.
- Patch 7: the IB unlink happens off the claimed stack list without the transport lock, and the changelog says so.
- Patch 8: changelog names the two existing rmmod holes the resweep and the rds_wq flush close (a late-detaching connection outlived the module; rds_ib_dev_free() could run after unload).
- Patch 10: the BUG_ON() on a socket-owned message is a WARN_ON_ONCE() rather than gone, and the changelog describes what the lock and the RDS_MSG_ON_CONN clear each do.
- Patch 11: SIOCRDSSETTOS stores rs_tos with WRITE_ONCE() to pair with the lockless READ_ONCE() in sendmsg; the rs_lock comment places rs_tos correctly and names that exception; the dead rds_destroy_pending(passive) branch under rds_conn_lock is removed (a passive still installed cannot have its destroy begun); the c_destroy_in_prog comment says the c_passive checks use the predicate; the changelog says the functional change today is the rs_lock serialization.
- Patch 12: changelog notes that an unload beginning after the initiate_connect check is covered by patch 3.
- Patch 13: names the patch that introduced t_conn_count instead of "the previous patch".
- Rebased onto net-next with the rdma_cm id restriction (c7fca8aae6fe).
- The pre-existing rdma_resolve_route() failure return in the ADDR_RESOLVED case, and the active-side non-IB device case, are fixed separately for net (v4 of "net: rds: fix uninitialized trans dereference in CM event handler").
Changes since v6 [8]:
- Rebased onto net-next with ba3d1f480c7a ("net/rds: size a connection's path set by the transport it ends up with"), which closes the pre-existing per-path workqueue leak the v5 review found under patch 8, and 5ccac2330deb ("net/rds: replace tasklets with workqueue").
- Patch order: the rds_incoming reference patch (v6 patch 12) moves ahead of the purge and the reference-holder patches, and the purge precedes the reference holders, so no bisect point has a queued message or a cached connection without the reference that protects it, nor a pinned connection over an unlocked purge.
- Changelogs and comments no longer describe a destroy under a live socket as reachable: no trigger in this tree (netns teardown, module unload) runs while a socket is sending, and IB device removal drops rather than destroys. Patches 7, 8, 10, 11 and 12 are reworded accordingly; patch 7 no longer cites the version-mismatch destroy.
- Patch 4 names the destroy == true rds_conn_path_drop() among the arming sites the predicate does not cover.
- Patch 5 states the first-caller-wins semantics of the destroy guard, in the changelog and the function comment, and the c_destroy_in_prog comment records that it is set under rds_conn_lock as a test-and-set.
- Patch 10: the rds_tcp_accept_one() comment credits rds_tcp_kill_sock() for the clear-then-flush ordering; the c_destroy_in_prog comment covers the direct reads at the c_passive sites.
- Patch 11: the pin's rationale says what it guards - the holders outside the handler - and that it is defensive today.
- Patch 4 is a fix after all: the loopback pernet exit at core-module unload destroys connections with the old predicate false. Changelog rewritten around that path, Fixes: c809195f5523; the c_destroy_in_prog comment names the two exempt arming sites.
- Patch 5: stale version-mismatch example removed; the first-caller guard documented as inert until the re-sweep needs it.
- Patch 6: the nodev-list BUG_ON in rds_ib_add_conn() goes here, not two patches later, so no bisect point empties that list under it; changelog no longer describes holders that arrive later.
- Patch 7: rds_ib_conn_free() comment describes the claimed state.
- Patch 8: a cached connection found with its destroy begun is dropped from the cache on the spot, instead of staying pinned until close; rs_lock comment names rs_conn and rs_tos.
- Patch 9: the purge clears RDS_MSG_ON_CONN under cp_lock, so rds_send_drop_to() can neither double-put nor unlink from the purge list, and the BUG_ON on a socket-linked message goes; send.c is no longer touched (v6 had accidentally dropped the READ_ONCE() on rs_tos there).
- Patch 10: rds_ib_cm_initiate_connect() gets the same rds_destroy_pending() check as the passive side, so a connect that completes after the destroy's quiesce cannot leave a QP and a device reference nothing tears down.
Changes since v5 [7]:
- Rebased; the version-mismatch destroy is now an rds_conn_drop() in net-next (f97d8c7bab78), so patch 4's Fixes: tag is gone and its changelog describes what still needs the per-connection flag, and the CM-callback deadlock reports against earlier versions no longer apply.
- Patch 2: described as the refactor it is (the passive twin is IB-only, single path); Fixes: tag dropped.
- Patch 9: reduced to the cp_lock purge; the send-side refusals and the negative *queued signalling are dropped, since no sender can be in flight at a destroy trigger.
- Patch 6: wait comment describes the initial-sweep-plus-resweep contract; changelog notes the guarantee is complete only once incs hold references.
- Patch 8: c_destroy_in_prog read through rds_destroy_pending() in __rds_conn_create(); the c_passive puts documented as possibly the last; READ_ONCE() on the unlocked rs_tos sample; rs_conn comment notes the rds_release() exception.
- Patch 10: stale forward-reference comment fixed.
Changes since v4 [6]:
- Patch 7: the IB sweep marks each gathered node with a new i_ib_node_detached flag instead of relying on list emptiness. A node parked on the sweep's stack list is not empty, so a concurrent rds_ib_add_conn() would have unlinked it from under the sweep's lockless walk; rds_ib_add_conn(), rds_ib_remove_conn() and rds_ib_conn_free() now leave a claimed node alone.
Changes since v3 [3]:
- Rebased; the version-mismatch destroy that patch 10 also had to cope with is now an rds_conn_drop() in net (f97d8c7bab78), and patch 10's changelog says what remains for it to cover.
- The uninitialised transport pointer in the CM event handler that the second v2 review pass raised turned out to be the bug Aohan Mei already posted a fix for (v2 at [5], stalled after review); it is carried forward separately for net rather than added here.
- Patch 7: the teardown walks take a reference on each gathered connection, so a connection destroyed earlier and freed by a pending holder cannot vanish under the iterator, and the two IB list movers tolerate a node the sweep already unlinked instead of BUG_ON()ing; the nodev sweep gathers entry by entry, so the resweep from the unload wait no longer relies on list_splice_init().
- Patch 8: SIOCRDSSETTOS check-then-act closed - the install in rds_sendmsg() re-checks the socket's ToS under rs_lock; rs_conn documented as a referenced, rs_lock-serialised cache; contract comment restored to rds_conn_lookup().
- Patch 9: rds_send_probe() gets the same rds_destroy_pending() test under cp_lock as rds_send_queue_rm() (a probe queued after the purge pinned the connection for good).
- v3 patch 10 (rds_tcp_accept_one() destroy check) dropped: the check was not serialised against the destroy, and the race it aimed at is not reachable - every TCP destroy path stops the listener, flushing the accept work, first. A comment now records that ordering.
- Changelog and comment corrections from the second v2 review pass (self-requeue exemption in patch 3, c_refcount comment in patch 5, the uninterruptible wait and the transport-text wake in patch 6, "lock-free" wording in patch 11, cross-netns comment in patch 12).
Changes since v2 [2]:
- New patches 1 and 2: pre-existing rds_ib_conn_free() interrupt state clobber and passive-path transport data leak, surfaced by review of the teardown changes.
- Patch 6: rds_ib_destroy_nodev_conns() uses list_splice_init(), so the resweep from the unload wait cannot splice a stale list head.
- Patch 8: SIOCRDSGETTOS reads rs_tos under rs_lock like SETTOS.
- New patch 9: rds_send_queue_rm() refuses a connection whose destroy has begun, under cp_lock, and the quiesce purges cp_send_queue under cp_lock (list corruption against an in-flight sender, and a message that would pin the connection forever).
- New patch 10: rds_tcp_accept_one() does not install a socket on a connection whose destroy has begun (socket left pointing at a freed path).
Changes since v1 [1]:
- New patch 1: guard the five work-arming sites that never tested rds_destroy_pending(); patch 2's changelog and comments narrowed to what it actually newly covers.
- Patch 4 moved ahead of the reference holders, so no bisect point has references without the unload wait; wait made unbounded with a periodic warning instead of a 10 s timeout; IB exit re-sweeps the nodev list for connections that detach from their device late.
- New patch 5: transport nodes unlinked before destroy (stack list head use-after-free from a deferred conn_free).
- Patch 6: c_passive RCU-annotated; a destroyed passive child is refused by __rds_conn_create() and clears the parent's pointer itself; SIOCRDSSETTOS uses rs_lock; lookup comment reworded; explicit not-for-stable note.
- New patch 7: reference across the CM event handler, and a destroy-pending re-check in rds_ib_cm_handle_connect().
- Patch 9: changelog states what a lingering inc keeps alive and that it blocks module unload.
- Changelog corrections throughout (netns teardown paths named as the non-rmmod destroyers, stale rds_conn_path_destroy() reference).
[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/netdev/[email protected]/
[4] https://lore.kernel.org/netdev/[email protected]/
[5] https://lore.kernel.org/netdev/[email protected]/
[6] https://lore.kernel.org/netdev/[email protected]/
[7] https://lore.kernel.org/netdev/[email protected]/
[8] https://lore.kernel.org/netdev/[email protected]/
[9] https://lore.kernel.org/netdev/[email protected]/
[10] https://lore.kernel.org/netdev/[email protected]/
[11] https://lore.kernel.org/netdev/[email protected]/
[12] https://lore.kernel.org/netdev/[email protected]/
[13] https://lore.kernel.org/netdev/[email protected]/
[14] https://lore.kernel.org/netdev/[email protected]/
Allison
Allison Henderson (9):
net/rds: undo conn_alloc() the same way on every __rds_conn_create()
exit
net/rds: ib: refuse to attach a connection to a device being removed net/rds: guard every work-requeueing site with rds_destroy_pending() net/rds: make rds_destroy_pending() report a connection's own destroy net/rds: unlink transport nodes before a possibly deferred connection
free
net/rds: hold a connection reference from struct rds_incoming net/rds: take cp_lock to purge cp_send_queue in the quiesce net/rds: give cm_id->context a reference on its connection net/rds: drop rds_conn_count in favor of t_conn_count
Sharath Srinivasan (3):
net/rds: split connection destroy into quiesce and kref-governed free net/rds: wait for connections to be freed on transport unload net/rds: hold connection references in lookup, sockets and c_passive
net/rds/af_rds.c | 26 ++- net/rds/connection.c | 346 ++++++++++++++++++++++++++++++++++----- net/rds/ib.c | 27 ++- net/rds/ib.h | 11 +- net/rds/ib_cm.c | 59 ++++++- net/rds/ib_rdma.c | 81 +++++++-- net/rds/ib_recv.c | 6 +- net/rds/ib_send.c | 18 +- net/rds/loop.c | 61 +++++-- net/rds/message.c | 16 +- net/rds/rdma_transport.c | 7 + net/rds/rds.h | 64 +++++++- net/rds/recv.c | 27 ++- net/rds/send.c | 82 ++++++++-- net/rds/tcp.c | 53 +++++- net/rds/tcp_listen.c | 22 ++- 16 files changed, 781 insertions(+), 125 deletions(-)
base-commit: c88fd6989aea9237a60a4859139a265bfb7ddc0b
Signature · 1 line
2.50.1
Allison Henderson
maintainerPATCH v12 1/12 net/rds: undo conn_alloc() the same way on every __rds_conn_create() exit
trans->conn_alloc() may allocate transport data for every path of a multipath connection - rds_tcp_conn_alloc() does - which is why the lost-creation-race exit of __rds_conn_create() loops over all npaths when it frees the connection it just built. The passive-connection exit right above it frees only path 0.
That is not a leak today: a passive twin is only created for an IB loopback connection (an incoming TCP connect to a local address is refused with -EOPNOTSUPP before it gets here), and the IB transport is not multipath, so npaths is 1 on that exit. But the two exits express the same "undo conn_alloc()" step in two different ways, and the following patches add another exit of the same kind. Move the loop into a helper and use it everywhere, so that the step cannot silently diverge if a multipath transport ever grows a passive twin.
--- net/rds/connection.c | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-)
net/rds/connection.c +18 -13
@@ -161,6 +161,22 @@ static void __rds_conn_path_init(struct rds_connection *conn, cp->cp_flags =0 ; } +/* Undo trans->conn_alloc(): it may have allocated transport data for +* every path of a multipath connection, not just for path 0. +*/ +static void rds_conn_free_transport_data (struct rds_connection *conn, +int npaths) +{ +struct rds_conn_path *cp; +int i; + +for (i =0 ; i < npaths; i++) { + cp = &conn->c_path[i]; +if (cp->cp_transport_data) + conn->c_trans->conn_free (cp->cp_transport_data); + } +} +/* * There is only every one 'conn' for a given pair of addresses in the * system at a time. They contain messages to be retransmitted and so
@@ -322,7 +338,7 @@ static struct rds_connection *__rds_conn_create(struct net *net,if (parent) {/* Creating passive conn */ if (parent->c_passive) { - trans->conn_free (conn->c_path[0 ].cp_transport_data); +rds_conn_free_transport_data (conn, npaths); free_cp = conn->c_path;kmem_cache_free (rds_conn_slab, conn); conn = parent->c_passive;
@@ -338,18 +354,7 @@ static struct rds_connection *__rds_conn_create(struct net *net, found =rds_conn_lookup (net, head, laddr, faddr, trans, tos, dev_if);if (found) { -struct rds_conn_path *cp; -int i; - -for (i =0 ; i < npaths; i++) { - cp = &conn->c_path[i]; -/* The ->conn_alloc invocation may have -* allocated resource for all paths, so all -* of them may have to be freed here. -*/ -if (cp->cp_transport_data) - trans->conn_free (cp->cp_transport_data); - } +rds_conn_free_transport_data (conn, npaths); free_cp = conn->c_path;kmem_cache_free (rds_conn_slab, conn); conn = found;
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 01/12] net/rds: undo conn_alloc() the same way on every __rds_conn_create() exit
Sashiko has reviewed this patch and found no issues.
Quoted 15 lines
trans->conn_alloc()may allocate transport data for every path of a multipath connection -rds_tcp_conn_alloc()does - which is why the lost-creation-race exit of__rds_conn_create()loops over all npaths when it frees the connection it just built. The passive-connection exit right above it frees only path 0.That is not a leak today: a passive twin is only created for an IB loopback connection (an incoming TCP connect to a local address is refused with
-EOPNOTSUPPbefore it gets here), and the IB transport is not multipath, so npaths is 1 on that exit. But the two exits express the same "undoconn_alloc()" step in two different ways, and the following patches add another exit of the same kind. Move the loop into a helper and use it everywhere, so that the step cannot silently diverge if a multipath transport ever grows a passive twin.
Quoted 2 lines
Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Allison Henderson
maintainerPATCH v12 2/12 net/rds: ib: refuse to attach a connection to a device being removed
In reply to Allison Hendersonrds_ib_remove_one() drops every connection on the device's conn_list in rds_ib_dev_shutdown(), then clears the client data so that no new connect can find the device. A connect that is already past rds_ib_get_client_data() when the walk runs is not covered by either:rds_ib_setup_qp() goes on to rds_ib_add_conn(), which moves the connection onto the conn_list the walk has just finished with, and builds a QP on a device that is on its way out.
On a hot removal the rdma_cm still catches that: its own client removal runs after RDS's and delivers RDMA_CM_EVENT_DEVICE_REMOVAL to the connection's id, whose drop shuts the connection down and returns it to ib_nodev_conns. On module unload there is no such event - the device stays registered - and rds_destroy_pending() is already true, so every drop short of a destroy returns early. Nothing drops that connection then: the device's shutdown walk is over, and the connection never returns to ib_nodev_conns, which is the only list the transport exit sweeps. So it outlives the module, QP, device reference and all. The trigger is unloading rds_rdma while a connect is in flight.
Make rds_ib_dev_shutdown() mark the device as shutting down under rds_ibdev->spinlock before it walks conn_list, and have rds_ib_add_conn() refuse, under the same lock, to attach a connection to a device so marked. Every connection is then either on the list when the walk drops it, or refused: the connect fails, the connection stays on ib_nodev_conns, and either its own drop or the exit sweep tears it down. rds_ib_setup_qp() has not taken anything from the device at that point, so the failure needs no unwinding beyond the client-data reference it already releases - but on the unload path that release can now be the device's last: rds_ib_remove_one() has dropped its own references by then, and the refused connect, holding c_cm_lock, releases its reference only when the exit sweep's rds_conn_destroy() of that connection waits for the lock, after rds_ib_unregister_client()'s flush of rds_wq. rds_ib_dev_free() would then run on rds_wq with no flush left before the module text goes away, so rds_ib_exit() flushes rds_wq once more after the sweep.
This covers the connect that attaches after rds_ib_dev_shutdown()'s walk. Its twin - a connection still on conn_list when the exit sweep runs, because its shutdown work has not yet moved it back to ib_nodev_conns - is left for the unload wait later in the series, which re-sweeps that list until every connection is gone.
This mirrors the UEK gate on the device removal flag in rds_ib_add_conn().
--- net/rds/ib.c | 12 ++++++++++++ net/rds/ib.h | 7 ++++++- net/rds/ib_cm.c | 4 +++- net/rds/ib_rdma.c | 17 +++++++++++++++-- 4 files changed, 36 insertions(+), 4 deletions(-)
net/rds/ib.c +12 -0
@@ -86,6 +86,11 @@ static void rds_ib_dev_shutdown(struct rds_ib_device *rds_ibdev)unsigned long flags;spin_lock_irqsave (&rds_ibdev->spinlock, flags); +/* Close the device to new connections under the same lock that +* rds_ib_add_conn() attaches them under, so that every +* connection is either dropped by the walk below or refused. +*/ + rds_ibdev->shutting_down = true;list_for_each_entry (ic, &rds_ibdev->conn_list, ib_node)rds_conn_path_drop (&ic->conn->c_path[0 ], true);spin_unlock_irqrestore (&rds_ibdev->spinlock, flags);
@@ -539,6 +544,13 @@ void rds_ib_exit(void)#endif rds_ib_unregister_client ();rds_ib_destroy_nodev_conns (); +/* A connect that was refused above may have held the last +* reference on its device, and dropped it - deferring +* rds_ib_dev_free() to rds_wq - only once the sweep had destroyed +* its connection, after rds_ib_unregister_client()'s flush. +* Drain rds_wq again before this module's text goes away. +*/ +flush_workqueue (rds_wq);rds_ib_sysctl_exit ();rds_ib_recv_exit ();rds_trans_unregister (&rds_ib_transport);
net/rds/ib.h +6 -1
@@ -258,6 +258,10 @@ struct rds_ib_device {unsigned int max_initiator_depth;unsigned int max_responder_resources;spinlock_t spinlock;/* protect the above */ +/* set under spinlock by rds_ib_dev_shutdown(): the device is +* going away and no connection may attach to it any more +*/ +bool shutting_down;refcount_t refcount;struct work_struct free_work;int *vector_load;
@@ -384,7 +388,8 @@ void rds_ib_cm_connect_complete(struct rds_connection *conn,struct rds_ib_device *rds_ib_get_device (__be32 ipaddr);int rds_ib_update_ipaddr (struct rds_ib_device *rds_ibdev,struct in6_addr *ipaddr); -void rds_ib_add_conn (struct rds_ib_device *rds_ibdev,struct rds_connection *conn); +int rds_ib_add_conn (struct rds_ib_device *rds_ibdev, +struct rds_connection *conn);void rds_ib_remove_conn (struct rds_ib_device *rds_ibdev,struct rds_connection *conn);void rds_ib_destroy_nodev_conns (void );void rds_ib_mr_cqe_handler (struct rds_ib_connection *ic,struct ib_wc *wc);
net/rds/ib_cm.c +3 -1
@@ -524,7 +524,9 @@ static int rds_ib_setup_qp(struct rds_connection *conn) fr_queue_space =RDS_IB_DEFAULT_FR_WR ;/* add the conn now so that connection establishment has the dev */ -rds_ib_add_conn (rds_ibdev, conn); + ret =rds_ib_add_conn (rds_ibdev, conn); +if (ret) + goto out; max_wrs = rds_ibdev->max_wrs < rds_ib_sysctl_max_send_wr +1 ? rds_ibdev->max_wrs -1 : rds_ib_sysctl_max_send_wr;
net/rds/ib_rdma.c +15 -2
@@ -119,7 +119,8 @@ int rds_ib_update_ipaddr(struct rds_ib_device *rds_ibdev,return 0 ; } -void rds_ib_add_conn (struct rds_ib_device *rds_ibdev,struct rds_connection *conn) +int rds_ib_add_conn (struct rds_ib_device *rds_ibdev, +struct rds_connection *conn) {struct rds_ib_connection *ic = conn->c_transport_data;
@@ -127,15 +128,27 @@ void rds_ib_add_conn(struct rds_ib_device *rds_ibdev, struct rds_connection *conspin_lock_irq (&ib_nodev_conns_lock);BUG_ON (list_empty (&ib_nodev_conns));BUG_ON (list_empty (&ic->ib_node)); -list_del (&ic->ib_node);spin_lock (&rds_ibdev->spinlock); +/* rds_ib_dev_shutdown() has walked conn_list, or is about to +* with this lock held: a connection attached now would never be +* dropped by it, so leave the connection on the nodev list for +* the caller to fail and the transport exit to find. +*/ +if (rds_ibdev->shutting_down) { +spin_unlock (&rds_ibdev->spinlock); +spin_unlock_irq (&ib_nodev_conns_lock); +return -ENODEV ; + } +list_del (&ic->ib_node);list_add_tail (&ic->ib_node, &rds_ibdev->conn_list);spin_unlock (&rds_ibdev->spinlock);spin_unlock_irq (&ib_nodev_conns_lock); ic->rds_ibdev = rds_ibdev;refcount_inc (&rds_ibdev->refcount); + +return 0 ; }void rds_ib_remove_conn (struct rds_ib_device *rds_ibdev,struct rds_connection *conn)
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 02/12] net/rds: ib: refuse to attach a connection to a device being removed
Sashiko has reviewed this patch and found no issues.
Quoted 18 lines
rds_ib_remove_one()drops every connection on the device'sconn_listinrds_ib_dev_shutdown(), then clears the client data so that no new connect can find the device. A connect that is already pastrds_ib_get_client_data()when the walk runs is not covered by either:rds_ib_setup_qp()goes on tords_ib_add_conn(), which moves the connection onto theconn_listthe walk has just finished with, and builds a QP on a device that is on its way out.On a hot removal the
rdma_cmstill catches that: its own client removal runs after RDS's and deliversRDMA_CM_EVENT_DEVICE_REMOVALto the connection's id, whose drop shuts the connection down and returns it toib_nodev_conns. On module unload there is no such event - the device stays registered - andrds_destroy_pending()is already true, so every drop short of a destroy returns early. Nothing drops that connection then: the device's shutdown walk is over, and the [ ... ]rds_ib_add_conn().
Quoted 3 lines
Fixes fc19de38be92 ("RDS/IB: disconnect when IB devices are removed")Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Allison Henderson
maintainerPATCH v12 3/12 net/rds: guard every work-requeueing site with rds_destroy_pending()
In reply to Allison Hendersonrds_conn_destroy() cancels the path works and then destroys the per-path workqueue. The sites that can re-arm those works are supposed to test rds_destroy_pending() under rcu_read_lock() first, paired with the synchronize_rcu() in the destroy path, so that no new work can be queued once the cancellation has begun.
Five arming sites never got that guard:
- rds_ib_send_cqe_handler() and rds_ib_send_add_credits() re-arm cp_send_w when a send completion or a credit update clears RDS_LL_SEND_FULL,
- rds_ib_recv_refill() re-arms cp_recv_w when the recv ring runs low,
- rds_tcp_accept_one() kicks cp_recv_w on the freshly accepted socket, and
- rds_sendmsg() arms cp_conn_w for a multipath connection whose path 0 is not up yet.
A completion landing in the window between the cancel and destroy_workqueue() in rds_conn_path_destroy() would re-arm a work on a workqueue that is about to be destroyed: with delay 0 the work is queued directly on the freed workqueue, and with delay 1 the timer survives destroy_workqueue() unseen and fires afterwards, queueing from a timer_list that lives in the freed c_path array. Today's destroy triggers keep these sites out of that window - an IB connection reaches rds_conn_destroy() only from the nodev list, with its QP and CQs already gone, the TCP accept work is flushed before any connection is destroyed, and a sending socket pins its netns and its transport module - so this is preparation: the following patch gives the predicate a per-connection term, and these are the sites that would otherwise miss it.
Wrap all five sites in the same rcu_read_lock() + rds_destroy_pending() pattern the other arming sites already use.
The four self-requeues in rds_send_worker() and rds_recv_worker() are left alone on purpose: they run from inside the work item itself, and cancel_delayed_work_sync() disables the work for the duration of the cancel, so a requeue issued by the still-running callback is dropped and none can follow once the cancel has returned.
With the predicate as it stands the guards cover the netns teardown and module unload cases; the following patch extends it to the destroy of a single connection.
--- net/rds/ib_recv.c | 6 +++++- net/rds/ib_send.c | 18 ++++++++++++++---- net/rds/send.c | 11 ++++++++--- net/rds/tcp_listen.c | 10 +++++++--- 4 files changed, 34 insertions(+), 11 deletions(-)
net/rds/ib_recv.c +5 -1
@@ -458,7 +458,11 @@ void rds_ib_recv_refill(struct rds_connection *conn, int prefill, gfp_t gfp) (must_wake || (can_wait && rds_ib_ring_low(&ic->i_recv_ring)) || rds_ib_ring_empty(&ic->i_recv_ring))) { -queue_delayed_work (conn->c_path->cp_wq, &conn->c_recv_w,1 ); +rcu_read_lock (); +if (!rds_destroy_pending (conn)) +queue_delayed_work (conn->c_path->cp_wq, + &conn->c_recv_w,1 ); +rcu_read_unlock (); }if (can_wait)cond_resched ();
net/rds/ib_send.c +14 -4
@@ -298,8 +298,13 @@ void rds_ib_send_cqe_handler(struct rds_ib_connection *ic, struct ib_wc *wc)rds_ib_sub_signaled (ic, nr_sig);if (test_and_clear_bit (RDS_LL_SEND_FULL , &conn->c_flags) || -test_bit (0 , &conn->c_map_queued)) -queue_delayed_work (conn->c_path->cp_wq, &conn->c_send_w,0 ); +test_bit (0 , &conn->c_map_queued)) { +rcu_read_lock (); +if (!rds_destroy_pending (conn)) +queue_delayed_work (conn->c_path->cp_wq, + &conn->c_send_w,0 ); +rcu_read_unlock (); + }/* We expect errors as the qp is drained during shutdown */ if (wc->status !=IB_WC_SUCCESS &&rds_conn_up (conn)) {
@@ -420,8 +425,13 @@ void rds_ib_send_add_credits(struct rds_connection *conn, unsigned int credits) test_bit(RDS_LL_SEND_FULL, &conn->c_flags) ? ", ll_send_full" : "");atomic_add (IB_SET_SEND_CREDITS (credits), &ic->i_credits); -if (test_and_clear_bit (RDS_LL_SEND_FULL , &conn->c_flags)) -queue_delayed_work (conn->c_path->cp_wq, &conn->c_send_w,0 ); +if (test_and_clear_bit (RDS_LL_SEND_FULL , &conn->c_flags)) { +rcu_read_lock (); +if (!rds_destroy_pending (conn)) +queue_delayed_work (conn->c_path->cp_wq, + &conn->c_send_w,0 ); +rcu_read_unlock (); + }WARN_ON (IB_GET_SEND_CREDITS (credits) >=16384 );
net/rds/send.c +8 -3
@@ -1383,9 +1383,14 @@ int rds_sendmsg(struct socket *sock, struct msghdr *msg, size_t payload_len)* outstanding. */ if (!test_and_set_bit (RDS_RECONNECT_PENDING , - &conn->c_path[0 ].cp_flags)) -queue_delayed_work (conn->c_path[0 ].cp_wq, - &conn->c_path[0 ].cp_conn_w,0 ); + &conn->c_path[0 ].cp_flags)) { +rcu_read_lock (); +if (!rds_destroy_pending (conn)) +queue_delayed_work (conn->c_path[0 ].cp_wq, + &conn->c_path[0 ].cp_conn_w, +0 ); +rcu_read_unlock (); + }rds_send_ping (conn,0 ); }
net/rds/tcp_listen.c +7 -3
@@ -316,10 +316,14 @@ int rds_tcp_accept_one(struct rds_tcp_net *rtn)*/ if (READ_ONCE (sk->sk_state) ==TCP_CLOSE_WAIT ||READ_ONCE (sk->sk_state) ==TCP_LAST_ACK || -READ_ONCE (sk->sk_state) ==TCP_CLOSE ) +READ_ONCE (sk->sk_state) ==TCP_CLOSE ) {rds_conn_path_drop (cp,0 ); -else -queue_delayed_work (cp->cp_wq, &cp->cp_recv_w,0 ); + }else { +rcu_read_lock (); +if (!rds_destroy_pending (cp->cp_conn)) +queue_delayed_work (cp->cp_wq, &cp->cp_recv_w,0 ); +rcu_read_unlock (); + }sock_put (sk);
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 03/12] net/rds: guard every work-requeueing site with rds_destroy_pending()
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
rds_conn_destroy()cancels the path works and then destroys the per-path workqueue. The sites that can re-arm those works are supposed to testrds_destroy_pending()underrcu_read_lock()first, paired with thesynchronize_rcu()in the destroy path, so that no new work can be queued once the cancellation has begun.Five arming sites never got that guard:
-
rds_ib_send_cqe_handler()andrds_ib_send_add_credits()re-armcp_send_wwhen a send completion or a credit update clearsRDS_LL_SEND_FULL,
-rds_ib_recv_refill()re-armscp_recv_wwhen the recv ring runs low,
-rds_tcp_accept_one()kickscp_recv_won the freshly accepted socket, and
[ ... ]
module unload cases; the following patch extends it to the destroy of a single connection.
Quoted 2 lines
Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Allison Henderson
maintainerPATCH v12 4/12 net/rds: make rds_destroy_pending() report a connection's own destroy
In reply to Allison Hendersonrds_conn_destroy() cancels the path works and then destroys the per-path workqueue. However, nothing currently stops the work-requeueing sites from queueing new work on the connection while that happens. The existing code would suggest that this protection is supposed to come from rds_destroy_pending(), since all of those sites - apart from the workers' own self-requeues, which the sync cancel in the destroy path already rejects, and the destroy == true rds_conn_path_drop(), which the destroy itself issues and flushes and IB device removal issues ahead of the module exit that destroys the connection - guard the queueing withrds_destroy_pending() under rcu_read_lock() (the last stragglers were converted by the previous patch), and rds_conn_destroy() already issues a synchronize_rcu() after unhashing the connection. But the predicate only tests for the two global teardown cases (netns destruction via check_net(), module unload via ->t_unloading). Because the conn itself lacks any indication that a destroy is in progress, the predicate does not cover the destruction of a single connection outside these two cases.
One caller escapes both terms. When the core rds module unloads, rds_conn_exit() runs rds_loop_net_exit() first, and unregistering the pernet operations invokes rds_loop_exit_net() -> rds_loop_kill_conns() -> rds_conn_destroy() for the loopback connections of every network namespace that is still alive: check_net() is true for all of them, and the loop transport's unloading flag is only set afterwards, by rds_loop_exit(). For the duration of those destroys the predicate is false, so a concurrent rds_cong_queue_updates() can still find the connection on the congestion map's m_conn_list (the conn is only removed from it after the paths are torn down) and call queue_delayed_work() on a cp_wq that destroy_workqueue() has already freed. Nothing can exploit that today: every socket pins the rds module, so none exists by the time rds_exit() runs, and the works a connection's own ordered workqueue re-arms are drained by destroy_workqueue(). But the predicate is wrong for those destroys, and the following patches need it to be right.
The predicate is also imprecise even where it is true: it answers "is this connection's world going away", not "is this connection being destroyed", and the following patches need the second answer. Once the free is deferred to the last reference, a connection can be handed to rds_conn_destroy() more than once (the IB unload path re-sweeps its list until every connection is gone) and must recognise its own destroy in progress, and the passive-twin creation must refuse a parent whose destroy has begun.
Mainline once had this. Commit c14b0366813a ("rds: tcp: set linger to 1 when unloading a rds-tcp") added conn->c_destroy_in_prog, set in rds_conn_destroy(); commit c90ecbfaf50d2 ("rds: Use atomic flag to track connections being destroyed") replaced it with the per-path RDS_DESTROY_PENDING bit, set at the top of rds_conn_path_destroy() for every transport; and
commit ebeeb1ad9b8a ("rds: tcp: use rds_destroy_pending() to synchronize netns/module teardown and rds connection/workq management") removed that set_bit and the synchronize_rcu() with it, folding the test into the global rds_destroy_pending() predicate above and leaving the bit set only on the IB protocol-version mismatch path, which commit cdc306a5c9cd3 ("rds: make v3.1 as compat version") then removed. The global predicate stopped being sufficient once
commit c809195f5523 ("rds: clean up loopback rds_connections on netns deletion") had rds_conn_exit() destroy the loopback connections of live namespaces, through rds_loop_net_exit(), ahead of rds_loop_exit(). This patch brings the per-connection flag back and partly reverses that consolidation.
Record the destroy on the connection itself, where every caller is
covered: setconn->c_destroy_in_prog before the unhash + synchronize_rcu() sequence in rds_conn_destroy() and test it first in rds_destroy_pending(). The existing rcu_read_lock() around every check-and-queue site pairs with that synchronize_rcu(): once it returns, every new reader observes the flag and refuses to queue, and anything queued before it is flushed or cancelled by the existing teardown. Drop the now-unreferenced RDS_DESTROY_PENDING bit and its dead test.
In the Oracle UEK kernel the equivalent conn->c_destroy_in_prog flag is part of the larger connection refcounting rework ("net/rds: Add krefs to struct rds_connection"), including ("net/rds: Merge uses of conn->c_destroy_in_prog & RDS_DESTROY_PENDING"). This ports the missing pieces of the requeue guard, which stand on their own.
--- net/rds/connection.c | 9 +++++++++ net/rds/ib.c | 5 +---- net/rds/rds.h | 17 +++++++++++++++-- 3 files changed, 25 insertions(+), 6 deletions(-)
net/rds/connection.c +9 -0
@@ -585,6 +585,15 @@ void rds_conn_destroy(struct rds_connection *conn)"%pI4\n" , conn, &conn->c_laddr, &conn->c_faddr); +/* Make rds_destroy_pending() true for this conn. Together with +* the synchronize_rcu() below this stops the work-requeueing +* sites (which test rds_destroy_pending() under rcu_read_lock(), +* bar the exemptions noted at c_destroy_in_prog) from queueing +* new work on the path workqueues once we start cancelling and +* destroying them. +*/ +WRITE_ONCE (conn->c_destroy_in_prog, true); +/* Ensure conn will not be scheduled for reconnect */ spin_lock_irq (&rds_conn_lock);hlist_del_init_rcu (&conn->c_hash_node);
net/rds/ib.c +1 -4
@@ -528,10 +528,7 @@ static void rds_ib_set_unloading(void)static bool rds_ib_is_unloading (struct rds_connection *conn) { -struct rds_conn_path *cp = &conn->c_path[0 ]; - -return (test_bit (RDS_DESTROY_PENDING , &cp->cp_flags) || -atomic_read (&rds_ib_unloading) !=0 ); +return atomic_read (&rds_ib_unloading) !=0 ; }void rds_ib_exit (void )
net/rds/rds.h +15 -2
@@ -89,7 +89,6 @@ enum {#define RDS_RECONNECT_PENDING 1#define RDS_IN_XMIT 2#define RDS_RECV_REFILL 3 -#define RDS_DESTROY_PENDING 4/* Max number of multipaths per RDS connection. Must be a power of 2 */ #define RDS_MPATH_WORKERS 8
@@ -148,6 +147,19 @@ struct rds_connection { c_pad_to_32:29 ;int c_npaths;bool c_with_sport_idx; +/* Set once, by rds_conn_destroy(), before it cancels the path +* works; read through rds_destroy_pending(). A site that arms +* a path work must test the predicate and queue the work inside +* one rcu_read_lock() section: the synchronize_rcu() that +* follows the store is what keeps a queue issued after the +* cancellation from landing on a destroyed workqueue. Two kinds +* of site are exempt: the workers' own self-requeues, which the +* sync cancel in the destroy path rejects, and the destroy == true +* rds_conn_path_drop(): the destroy issues and flushes it itself, +* and IB device removal issues it ahead of the module exit, which +* destroys - and so flushes - that connection afterwards. +*/ +bool c_destroy_in_prog;struct rds_connection *c_passive;struct rds_transport *c_trans;
@@ -996,7 +1008,8 @@ void __rds_put_mr_final(struct kref *kref);static inline bool rds_destroy_pending (struct rds_connection *conn) { -return !check_net (rds_conn_net (conn)) || +return READ_ONCE (conn->c_destroy_in_prog) || + !check_net (rds_conn_net (conn)) || (conn->c_trans->t_unloading && conn->c_trans->t_unloading (conn)); }
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 04/12] net/rds: make rds_destroy_pending() report a connection's own destroy
Sashiko has reviewed this patch and found no issues.
Quoted 18 lines
rds_conn_destroy()cancels the path works and then destroys the per-path workqueue. However, nothing currently stops the work-requeueing sites from queueing new work on the connection while that happens. The existing code would suggest that this protection is supposed to come fromrds_destroy_pending(), since all of those sites - apart from the workers' own self-requeues, which the sync cancel in the destroy path already rejects, and the destroy == truerds_conn_path_drop(), which the destroy itself issues and flushes and IB device removal issues ahead of the module exit that destroys the connection - guard the queueing withrds_destroy_pending()underrcu_read_lock()(the last stragglers were converted by the previous patch), andrds_conn_destroy()already issues asynchronize_rcu()after unhashing the connection. But the predicate only tests for the two global teardown cases (netns destruction viacheck_net(), module unload via ->t_unloading). Because the conn [ ... ]
missing pieces of the requeue guard, which stand on their own.
Quoted 3 lines
Suggested-by Sharath Srinivasan <[email protected]>Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Allison Henderson
maintainerPATCH v12 5/12 net/rds: split connection destroy into quiesce and kref-governed free
In reply to Allison HendersonFrom: Sharath Srinivasan <[email protected]>
rds_conn_destroy() tears down the transport state and immediately frees the connection, along with its paths and its workqueues. This relies on the assumption (documented in the rds_conn_destroy() comments) that "no one else is referencing the connection", which "we can only ensure ... in the rmmod path". However, the callers stopped honoring that long ago. Today, connections are also destroyed on network namespace teardown (rds_tcp_kill_sock() and rds_loop_kill_conns()), and the following patches deliberately let a connection outlive the call.
This leaves loose ends since references to these destroyed connections still exist. Sockets cache their connections in rs_conn, and congestion updates will walk the maps' m_conn_list. The CM callbacks and workers may also still hold the pointer.
Prepare to close those holes by making the connection refcounted:
- kref_init() the connection in __rds_conn_create(); the initial reference belongs to whoever is responsible for destroying the connection.
- rds_conn_destroy() still quiesces synchronously exactly as before (workers cancelled, paths dropped and shut down, queued messages purged, congestion list removal), but the frees - the transport's conn_free, the path workqueues, the c_path array and the connection slab object - move to rds_conn_destroy_fini(), which runs when the last reference is dropped via rds_conn_put().
- Export rds_conn_get()/rds_conn_put(), plus an inline rds_conn_get_unless_zero() for holders that may find a connection already on its way out, for the reference holders introduced in the following patches.
rds_conn_destroy() also gains a first-caller-wins guard on the new c_destroy_in_prog flag. No caller in this tree hands a connection to rds_conn_destroy() twice - every teardown sweep unlinks or claims its node before the destroy - so the guard documents the contract rather than closing a path: a second caller returns at once, without quiescing anything and without waiting for the first. With the initial reference the only one, it is inert here either way.
With no additional reference holders yet, this only sets up the refcounting framework and is functionally equivalent to the current code (the initial reference is the only one), so every free still completes inside rds_conn_destroy(). The next two patches make a deferred free safe - the transport unload wait and the unlinking of the transport nodes ahead of the destroy - and only then do the patches that follow take references at the places that today rely on bare pointers.
Based on the Oracle UEK commit "net/rds: Add krefs to struct rds_connection".
[achender: substantial reimplementation for net-next: UEK's
rds_conn_destroy_init()/_fini() split redone against upstream's rds_conn_destroy()/rds_conn_path_destroy() (no heartbeat/reap/trace infrastructure, no rds_net, single conn hash); destroy keeps its one-call external interface; holder coverage split out into follow-up patches; rewrite commit message]
--- net/rds/connection.c | 99 ++++++++++++++++++++++++++++++++++++-------- net/rds/rds.h | 19 ++++++++- 2 files changed, 98 insertions(+), 20 deletions(-)
net/rds/connection.c +81 -18
@@ -231,6 +231,7 @@ static struct rds_connection *__rds_conn_create(struct net *net, goto out; } +kref_init (&conn->c_refcount);INIT_HLIST_NODE (&conn->c_hash_node); conn->c_laddr = *laddr; conn->c_isv6 = !ipv6_addr_v4mapped (laddr);
@@ -477,9 +478,10 @@ void rds_conn_shutdown(struct rds_conn_path *cp)* Quiesce the reconnect timer before bailing * out, though. When a pending destroy did * suppress the queue, no later pass runs, and -* rds_conn_path_destroy() is about to flush -* cp_down_w and free the path: it must not -* find cp_conn_w still armed. A successor +* rds_conn_path_quiesce() is about to flush +* cp_down_w, ahead of the path's deferred +* free: it must not find cp_conn_w still +* armed. A successor * pass, when there is one, re-arms the * reconnect from its own tail. */
@@ -526,10 +528,12 @@ void rds_conn_shutdown(struct rds_conn_path *cp) conn->c_trans->conn_slots_available (conn, false); } -/* destroy a single rds_conn_path. rds_conn_destroy() iterates over -* all paths using rds_conn_path_destroy() +/* quiesce a single rds_conn_path: shut it down and tear down any +* queued messages. rds_conn_destroy() iterates over all paths using +* rds_conn_path_quiesce(); the transport state and the workqueue are +* freed later, from rds_conn_path_free(). */ -static void rds_conn_path_destroy (struct rds_conn_path *cp) +static void rds_conn_path_quiesce (struct rds_conn_path *cp) {struct rds_message *rm, *rtmp;
@@ -558,6 +562,16 @@ static void rds_conn_path_destroy(struct rds_conn_path *cp)WARN_ON (delayed_work_pending (&cp->cp_recv_w));WARN_ON (delayed_work_pending (&cp->cp_conn_w));WARN_ON (work_pending (&cp->cp_down_w)); +} + +/* free a quiesced rds_conn_path's transport state and workqueue; runs +* from rds_conn_destroy_fini() once the last connection reference is +* dropped. +*/ +static void rds_conn_path_free (struct rds_conn_path *cp) +{ +if (!cp->cp_transport_data) +return ;if (cp->cp_wq != rds_wq) {destroy_workqueue (cp->cp_wq);
@@ -567,16 +581,55 @@ static void rds_conn_path_destroy(struct rds_conn_path *cp) cp->cp_conn->c_trans->conn_free (cp->cp_transport_data); } +/* Free a connection. This runs from rds_conn_put() when the last +* reference is dropped, after rds_conn_destroy() has quiesced the +* connection and dropped the initial reference. +*/ +static void rds_conn_destroy_fini (struct kref *kref) +{ + struct rds_connection *conn = container_of(kref, struct rds_connection, + c_refcount); +int npaths = (conn->c_trans->t_mp_capable ?RDS_MPATH_WORKERS :1 ); +unsigned long flags; +int i; + +for (i =0 ; i < npaths; i++) +rds_conn_path_free (&conn->c_path[i]); + +kfree (conn->c_path); +kmem_cache_free (rds_conn_slab, conn); + +spin_lock_irqsave (&rds_conn_lock, flags); + rds_conn_count--; +spin_unlock_irqrestore (&rds_conn_lock, flags); +} + +void rds_conn_get (struct rds_connection *conn) +{ +kref_get (&conn->c_refcount); +} +EXPORT_SYMBOL_GPL (rds_conn_get); + +void rds_conn_put (struct rds_connection *conn) +{ +kref_put (&conn->c_refcount, rds_conn_destroy_fini); +} +EXPORT_SYMBOL_GPL (rds_conn_put); +/* * Stop and free a connection. * -* This can only be used in very limited circumstances. It assumes that once -* the conn has been shutdown that no one else is referencing the connection. -* We can only ensure this in the rmmod path in the current code. +* Quiesces the connection synchronously (workers cancelled, transport +* connections shut down, queued messages dropped) and drops the +* initial reference. Only the first call for a connection does any +* of that: a later one finds c_destroy_in_prog already set and +* returns at once, without waiting for the first. The memory - +* including the transport's per-connection state and the path +* workqueues - is freed once the last rds_conn_put() runs, which may +* be after this returns. */ void rds_conn_destroy (struct rds_connection *conn) { -unsigned long flags;int i;struct rds_conn_path *cp;int npaths = (conn->c_trans->t_mp_capable ?RDS_MPATH_WORKERS :1 );
@@ -591,11 +644,23 @@ void rds_conn_destroy(struct rds_connection *conn)* bar the exemptions noted at c_destroy_in_prog) from queueing * new work on the path workqueues once we start cancelling and * destroying them. +* +* Only the first caller proceeds. No caller in this tree hands +* a connection to rds_conn_destroy() twice - every teardown +* sweep unlinks or claims its node before the destroy - so this +* documents the contract rather than closing a path: a second +* caller returns at once, without quiescing or waiting. The +* unhash also happens under rds_conn_lock, so a looked-up conn +* can never be quiesced twice. */ +spin_lock_irq (&rds_conn_lock); +if (conn->c_destroy_in_prog) { +spin_unlock_irq (&rds_conn_lock); +return ; + }WRITE_ONCE (conn->c_destroy_in_prog, true);/* Ensure conn will not be scheduled for reconnect */ -spin_lock_irq (&rds_conn_lock);hlist_del_init_rcu (&conn->c_hash_node);spin_unlock_irq (&rds_conn_lock);synchronize_rcu ();
@@ -603,7 +668,7 @@ void rds_conn_destroy(struct rds_connection *conn)/* shut the connection down */ for (i =0 ; i < npaths; i++) { cp = &conn->c_path[i]; -rds_conn_path_destroy (cp); +rds_conn_path_quiesce (cp);BUG_ON (!list_empty (&cp->cp_retrans)); }
@@ -614,12 +679,10 @@ void rds_conn_destroy(struct rds_connection *conn)*/ rds_cong_remove_conn (conn); -kfree (conn->c_path); -kmem_cache_free (rds_conn_slab, conn); - -spin_lock_irqsave (&rds_conn_lock, flags); - rds_conn_count--; -spin_unlock_irqrestore (&rds_conn_lock, flags); +/* drop the initial reference; the connection is freed from +* rds_conn_destroy_fini() once every holder has dropped theirs +*/ +rds_conn_put (conn); }EXPORT_SYMBOL_GPL (rds_conn_destroy);
net/rds/rds.h +17 -2
@@ -137,6 +137,12 @@ struct rds_conn_path {/* One rds_connection per RDS address pair */ struct rds_connection {struct hlist_node c_hash_node; +/* rds_conn_destroy() quiesces the connection synchronously; +* freeing it - the connection memory, the path workqueues and +* the transport's per-connection state - is deferred until the +* last reference is dropped via rds_conn_put(). +*/ +struct kref c_refcount;struct in6_addr c_laddr;struct in6_addr c_faddr;int c_dev_if;/* ifindex used for this conn */
@@ -147,8 +153,10 @@ struct rds_connection { c_pad_to_32:29 ;int c_npaths;bool c_with_sport_idx; -/* Set once, by rds_conn_destroy(), before it cancels the path -* works; read through rds_destroy_pending(). A site that arms +/* Set once, by rds_conn_destroy() under rds_conn_lock - a +* test-and-set, so a second destroy of the same connection +* returns at once - before it cancels the path works. Read +* through rds_destroy_pending(). A site that arms * a path work must test the predicate and queue the work inside * one rcu_read_lock() section: the synchronize_rcu() that * follows the store is what keeps a queue issued after the
@@ -833,6 +841,13 @@ struct rds_connection *rds_conn_create_outgoing(struct net *net, u8 tos, gfp_t gfp, int dev_if);void rds_conn_shutdown (struct rds_conn_path *cpath);void rds_conn_destroy (struct rds_connection *conn); +void rds_conn_get (struct rds_connection *conn); +void rds_conn_put (struct rds_connection *conn); +/* take a reference unless the connection is already being freed */ +static inline bool rds_conn_get_unless_zero (struct rds_connection *conn) +{ +return kref_get_unless_zero (&conn->c_refcount); +}void rds_conn_drop (struct rds_connection *conn);void rds_conn_path_drop (struct rds_conn_path *cpath,bool destroy);void rds_conn_connect_if_down (struct rds_connection *conn);
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 05/12] net/rds: split connection destroy into quiesce and kref-governed free
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
From: Sharath Srinivasan <[email protected]>
rds_conn_destroy()tears down the transport state and immediately frees the connection, along with its paths and its workqueues. This relies on the assumption (documented in therds_conn_destroy()comments) that "no one else is referencing the connection", which "we can only ensure ... in the rmmod path". However, the callers stopped honoring that long ago. Today, connections are also destroyed on network namespace teardown (rds_tcp_kill_sock()andrds_loop_kill_conns()), and the following patches deliberately let a connection outlive the call.This leaves loose ends since references to these destroyed connections still exist. Sockets cache their connections in
rs_conn, and congestion updates will walk the maps'm_conn_list. The CM [ ... ]
infrastructure, nords_net, single conn hash); destroy keeps its one-call external interface; holder coverage split out into follow-up patches; rewrite commit message]
Quoted 2 lines
Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Allison Henderson
maintainerPATCH v12 6/12 net/rds: unlink transport nodes before a possibly deferred connection free
In reply to Allison HendersonThe transport teardown helpers - rds_tcp_destroy_conns(), rds_tcp_kill_sock(), rds_ib_destroy_nodev_conns(), rds_loop_exit() and rds_loop_kill_conns() - gather the per-connection transport nodes onto a list head on their own stack and call rds_conn_destroy() for each. The node is unlinked much later, by the transport's conn_free():rds_tcp_conn_free() skips a node that rds_tcp_kill_sock() marked detached, while rds_loop_conn_free() and rds_ib_conn_free() list_del() it unconditionally.
That is fine for as long as rds_conn_destroy() frees the connection before it returns, which is still the case at this point in the
series: the initial reference is the only one. The following patches hand out references that outlive the teardown loop - a socket's cached rs_conn, an inc parked on a receive queue - and with those, a conn_free() deferred until after the helper has returned would list_del() the node from a stack frame that no longer exists. Make the helpers ready for that first.
Unlink each node right before its rds_conn_destroy() - under the transport lock for TCP and loopback, and off the claimed stack list for IB - so that nothing is left on the stack list for a later free to touch. TCP marks the node detached, as rds_tcp_kill_sock() already does for the secondary paths of a multipath connection; loopback uses list_del_init() and has its conn_free() skip a node that is already empty.
IB needs an explicit flag, i_ib_node_detached, because its node has other movers: a connect worker moves it from the nodev list to a device's list in rds_ib_add_conn(), and a shutdown moves it back in rds_ib_remove_conn(). Either can run while the sweep holds the node on its stack list, and "the node is linked" cannot tell that list from the nodev list - an add_conn() that went by list emptiness would unlink the node from under the sweep's lockless walk. So the sweep sets the flag when it gathers the node, under ib_nodev_conns_lock, and from then on add_conn(), remove_conn() and conn_free() leave the node alone; the node belongs to the sweep, and its walk needs no lock. Those movers used to assert that the node is linked (and rds_ib_add_conn() that the nodev list is non-empty); a connect or shutdown worker can still be running for a connection the sweep has claimed, and such a connection is about to be destroyed anyway, so the assertions go, and rds_ib_add_conn() refuses a claimed node the way it refuses a device that is shutting down, rather than attach a connection the sweep is destroying.
For IB this also closes a race that exists today. rds_ib_destroy_nodev_conns() spliced the nodev list onto its stack without re-initialising the global head, and since commit 4716af3897e9 ("net/rds: Give each connection path its own workqueue") the shutdown works that rds_ib_dev_shutdown() queues run on the per-path cp_wq, which the flush_workqueue(rds_wq) in rds_ib_unregister_client() no longer waits for. A shutdown finishing during the sweep links its node through the stale head, which points into the stack list, so the last entry of that list now points at the global head and the sweep's lockless walk runs off the end of its list into it; a connect finishing then could likewise unlink a node from under the walk. Gathering entry by entry under ib_nodev_conns_lock and claiming each node closes both.
The walk itself must not lose the entries either. Once the following patches hand out references, a connection on one of these lists can be kept alive only by a reference that is dropped at any point during the walk, freeing the transport node the iterator is about to read; nothing in this tree destroys a connection behind the sweep's back today, so that part is preparation for those patches, while the IB list handling above is a fix in its own right. So the gather takes a reference on each connection it moves
onto the stack list, under the transport lock, and drops it after the destroy; a connection whose free is already running gets no reference and is left where it is, since that free unlinks the node itself once the lock is released. The tmp_list gathering itself remains: it is what keeps rds_conn_destroy() from being called with the transport lock held.
--- net/rds/ib.h | 4 +++ net/rds/ib_cm.c | 13 +++++--- net/rds/ib_rdma.c | 78 +++++++++++++++++++++++++++++++++++------------ net/rds/loop.c | 59 +++++++++++++++++++++++++++-------- net/rds/tcp.c | 52 +++++++++++++++++++++++++++---- 5 files changed, 164 insertions(+), 42 deletions(-)
net/rds/ib.h +4 -0
@@ -143,6 +143,10 @@ struct rds_ib_device;struct rds_ib_connection {struct list_head ib_node; +/* set under ib_nodev_conns_lock once a transport teardown has +* claimed ib_node; from then on only the teardown touches it +*/ +bool i_ib_node_detached;struct rds_ib_device *rds_ibdev;struct rds_connection *conn;
net/rds/ib_cm.c +9 -4
@@ -1291,15 +1291,20 @@ void rds_ib_conn_free(void *arg)rdsdebug ("ic %p\n" , ic);/* -* Conn is either on a dev's list or on the nodev list. -* A race with shutdown() or connect() would cause problems -* (since rds_ibdev would change) but that should never happen. +* Conn is on a dev's list or on the nodev list - or, once a +* transport teardown has claimed it (i_ib_node_detached), on +* neither, in which case the lock chosen here only guards the +* test below. A connect or shutdown still running for a +* claimed conn leaves the node alone, see rds_ib_add_conn() and +* rds_ib_remove_conn(). */ lock_ptr = ic->rds_ibdev ? &ic->rds_ibdev->spinlock : &ib_nodev_conns_lock;/* May be called with IRQs already disabled (see __rds_conn_create()) */ spin_lock_irqsave (lock_ptr, flags); -list_del (&ic->ib_node); +/* a transport teardown that gathered us first owns the node */ +if (!ic->i_ib_node_detached) +list_del (&ic->ib_node);spin_unlock_irqrestore (lock_ptr, flags);rds_ib_recv_free_caches (ic);
net/rds/ib_rdma.c +58 -20
113 lines not on this page · raw message
net/rds/loop.c +47 -12
88 lines not on this page · raw message
net/rds/tcp.c +46 -6
79 lines not on this page · raw message
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 06/12] net/rds: unlink transport nodes before a possibly deferred connection free
Sashiko has reviewed this patch and found no issues.
Quoted 17 lines
The transport teardown helpers -
rds_tcp_destroy_conns(),rds_tcp_kill_sock(),rds_ib_destroy_nodev_conns(),rds_loop_exit()andrds_loop_kill_conns()- gather the per-connection transport nodes onto a list head on their own stack and callrds_conn_destroy()for each. The node is unlinked much later, by the transport'sconn_free():rds_tcp_conn_free()skips a node thatrds_tcp_kill_sock()marked detached, whilerds_loop_conn_free()andrds_ib_conn_free()list_del()it unconditionally.That is fine for as long as
rds_conn_destroy()frees the connection before it returns, which is still the case at this point in the
series: the initial reference is the only one. The following patches hand out references that outlive the teardown loop - a socket's cachedrs_conn, an inc parked on a receive queue - and with those, aconn_free()deferred until after the helper has returned would [ ... ]
Quoted 4 lines
Fixes 4716af3897e9 ("net/rds: Give each connection path its own workqueue")Cc <[email protected]> # depends on the connection reference counting earlier in this seriesAssisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Allison Henderson
maintainerPATCH v12 7/12 net/rds: wait for connections to be freed on transport unload
In reply to Allison HendersonFrom: Sharath Srinivasan <[email protected]>
Since connection free became asynchronous, rds_conn_destroy() only quiesces the connection; the actual free - including the transport's conn_free, which lives in the transport module - runs when the last reference is dropped. The transports' exit paths destroy all of their connections and then proceed to unload, so a free that is still pending - once the following patches hand out references to lookups, sockets and incs - would execute transport module code after that module's text is gone. At this point in the series the initial reference is the only one, so the wait only has work to do for IB, where it also catches the late arrivals described below; the guarantee becomes load-bearing with those patches.
Count each transport's live connections in t_conn_count (incremented when a connection is published in __rds_conn_create(), decremented as the last step of rds_conn_destroy_fini()) and make the transport exit paths - rds_ib_exit(), rds_tcp_exit() and rds_loop_exit() - wait for the count to drop to zero after destroying their connections.
A bound socket holds a module reference on its own transport (rds_trans_get_preferred()), but that does not bound the wait: once the following patches make incs hold a connection reference, an unread datagram pins its connection for as long as the application leaves it on the receive queue, and rds_find_bound() does not filter on transport, so that socket may well belong to a different transport than the connection and pin nothing that stops this unload. Proceeding after a timeout would turn a leak into a use-after-free in this module's text, so the wait is unbounded: it polls the count, warns every ten seconds naming the transport and the number of connections outstanding, and returns only when the count reaches zero. rmmod therefore blocks while data is queued and unread, which is the historical RDS contract - teardown does not discard queued data.
The wait is deliberately not interruptible: the exit function has already passed the point of no return, and a killed rmmod would leave the module half torn down and unloadable, which is worse than a blocked one. The polling wakes every 100 ms, so the hung task detector does not fire.
The wake at the end of rds_conn_destroy_fini() can run from a thread executing transport module text, but never as that thread's last use of it: the CM event handler's reference (added later in the series) is always dropped before the rdma_destroy_id() in the connection's own shutdown returns, and rds_rdma_exit() stops the listener - waiting for any handler running on it - before rds_ib_exit() starts to wait.
rds_ib_exit() has one more wrinkle, and an existing hole. rds_ib_dev_shutdown() only drops the connections still attached to a device, and each of them moves itself to ib_nodev_conns from its own shutdown work, asynchronously; the flush_workqueue(rds_wq) in rds_ib_unregister_client() does not wait for those per-connection workqueues. A connection that had not migrated by the time rds_ib_destroy_nodev_conns() made its single sweep was never
destroyed: it outlived the module, still pointing at rds_ib_transport. With the count, it would hold the count up for good instead. So the wait takes a resweep callback, which rds_ib_exit() points at rds_ib_destroy_nodev_conns(): late arrivals get destroyed on the next poll instead of being waited on. The sweep claims and unlinks every node it gathers (see the previous patch), so a resweep never hands a connection to rds_conn_destroy() twice.
In rds_ib_exit(), tearing down the last connection can also drop the final reference on a device, which defers rds_ib_dev_free() - again this module's text - to rds_wq. The flush of rds_wq that the device gate patch added after the sweep moves after the wait, so that it covers the connections the resweep destroys as well; rds_ib_dev_free() queues nothing further on rds_wq, so a single pass drains it.
Based on the Oracle UEK commits "net/rds: wait_event_timeout until zero connections during rmmod" and "net/rds:
Each RDS transport should keep its own connection count".
[achender: reimplementation for net-next: t_conn_count did not exist
upstream and is introduced here; single global waitqueue instead of per-transport (the loop transport never goes through rds_trans_register()); unbounded wait with a periodic warning in place of UEK's wait_event_timeout() + WARN_ON(), plus the resweep for IB's asynchronous device detach; also cover rds_loop_exit(); rewrite commit message]
--- net/rds/connection.c | 52 ++++++++++++++++++++++++++++++++++++++++++++ net/rds/ib.c | 20 ++++++++++++----- net/rds/loop.c | 2 ++ net/rds/rds.h | 14 ++++++++++++ net/rds/tcp.c | 1 + 5 files changed, 84 insertions(+), 5 deletions(-)
net/rds/connection.c +52 -0
@@ -48,6 +48,8 @@/* converting this to RCU is a chore for another day.. */ static DEFINE_SPINLOCK (rds_conn_lock);static unsigned long rds_conn_count; +/* woken whenever a transport's t_conn_count drops to zero */ +static DECLARE_WAIT_QUEUE_HEAD (rds_conn_freed_waitq);static struct hlist_head rds_conn_hash[RDS_CONNECTION_HASH_ENTRIES ];static struct kmem_cache *rds_conn_slab;
@@ -347,6 +349,7 @@ static struct rds_connection *__rds_conn_create(struct net *net, parent->c_passive = conn;rds_cong_add_conn (conn); rds_conn_count++; +atomic_inc (&conn->c_trans->t_conn_count); } }else {/* Creating normal conn */
@@ -365,6 +368,7 @@ static struct rds_connection *__rds_conn_create(struct net *net,hlist_add_head_rcu (&conn->c_hash_node, head);rds_cong_add_conn (conn); rds_conn_count++; +atomic_inc (&conn->c_trans->t_conn_count); } }spin_unlock_irqrestore (&rds_conn_lock, flags);
@@ -590,6 +594,7 @@ static void rds_conn_destroy_fini(struct kref *kref) struct rds_connection *conn = container_of(kref, struct rds_connection, c_refcount);int npaths = (conn->c_trans->t_mp_capable ?RDS_MPATH_WORKERS :1 ); +struct rds_transport *trans = conn->c_trans;unsigned long flags;int i;
@@ -602,7 +607,54 @@ static void rds_conn_destroy_fini(struct kref *kref)spin_lock_irqsave (&rds_conn_lock, flags); rds_conn_count--;spin_unlock_irqrestore (&rds_conn_lock, flags); + +/* only after everything the transport module owns has been +* freed above may its unload proceed +*/ +if (!atomic_dec_return (&trans->t_conn_count)) +wake_up_all (&rds_conn_freed_waitq); +} + +/* Wait for all of @trans's connections to be freed; the free runs +* asynchronously once rds_conn_destroy() has quiesced a connection. +* Called on transport module unload after an initial sweep has +* destroyed the transport's connections; @resweep, when given, is +* called on every poll to destroy connections that were still +* detaching from a device when the sweep ran (IB). A connection +* reference can be held for an application-controlled time - once +* incs hold one, an unread datagram pins the inc that carries it, and +* thus the connection - so the wait is unbounded: the frees that run +* after unload call into this module's text (conn_free, inc_free) and +* free into its slabs, so proceeding while any remain would be a +* use-after-free, not a leak. Warn periodically so a stuck count is +* diagnosable, but never stop waiting. This matches the historical +* RDS contract that teardown does not discard queued data. +*/ +void rds_conn_wait_conns_freed (struct rds_transport *trans, +void (*resweep)(void )) +{ +unsigned long warn_interval = +msecs_to_jiffies (RDS_CONN_FREE_WARN_INTERVAL_MS ); +unsigned long warn_at = jiffies + warn_interval; + +while (!wait_event_timeout (rds_conn_freed_waitq, + !atomic_read (&trans->t_conn_count), +msecs_to_jiffies (RDS_CONN_FREE_POLL_MS ))) { +/* A transport whose teardown is asynchronous (IB moves a +* connection off its device from the shutdown work) gives +* us a resweep to destroy what has arrived since. +*/ +if (resweep) +resweep (); +if (time_after_eq (jiffies, warn_at)) { +pr_warn ("RDS/%s: still waiting for %d connection(s) to be freed before unload\n" , + trans->t_name, +atomic_read (&trans->t_conn_count)); + warn_at = jiffies + warn_interval; + } + } } +EXPORT_SYMBOL_GPL (rds_conn_wait_conns_freed);void rds_conn_get (struct rds_connection *conn) {
net/rds/ib.c +15 -5
@@ -540,14 +540,24 @@ void rds_ib_exit(void)rds_info_deregister_func (RDS6_INFO_IB_CONNECTIONS , rds6_ib_ic_info);#endif rds_ib_unregister_client (); + +/* rds_ib_dev_shutdown() only dropped the connections still +* attached to a device; each moves itself to ib_nodev_conns +* from its shutdown work. Destroy what is there now and keep +* sweeping the list while the wait sees connections outstanding, +* so a late arrival is destroyed rather than waited on forever. +*/ rds_ib_destroy_nodev_conns (); -/* A connect that was refused above may have held the last -* reference on its device, and dropped it - deferring -* rds_ib_dev_free() to rds_wq - only once the sweep had destroyed -* its connection, after rds_ib_unregister_client()'s flush. -* Drain rds_wq again before this module's text goes away. +rds_conn_wait_conns_freed (&rds_ib_transport, + rds_ib_destroy_nodev_conns); + +/* Tearing down the last connection may have dropped the final +* reference on a device, deferring rds_ib_dev_free() to rds_wq. +* Drain it before the module goes away; it queues nothing +* further on rds_wq. */ flush_workqueue (rds_wq); +rds_ib_sysctl_exit ();rds_ib_recv_exit ();rds_trans_unregister (&rds_ib_transport);
net/rds/loop.c +2 -0
@@ -233,6 +233,8 @@ void rds_loop_exit(void)spin_unlock_irq (&loop_conns_lock);rds_loop_destroy_gathered_conns (&tmp_list); + +rds_conn_wait_conns_freed (&rds_loop_transport,NULL ); }static void rds_loop_kill_conns (struct net *net)
net/rds/rds.h +14 -0
@@ -564,6 +564,12 @@ struct rds_transport { unsigned int t_prefer_loopback:1, t_mp_capable:1;unsigned int t_type; +/* Connections of this transport not yet freed; freeing runs +* asynchronously once rds_conn_destroy() has quiesced a +* connection, so transport module unload has to wait for this +* to reach zero (rds_conn_wait_conns_freed()). +*/ +atomic_t t_conn_count;int (*laddr_check)(struct net *net,const struct in6_addr *addr,__u32 scope_id);
@@ -848,6 +854,14 @@ static inline bool rds_conn_get_unless_zero(struct rds_connection *conn) {return kref_get_unless_zero (&conn->c_refcount); } + +/* transport unload waits for its connections to be freed, polling at +* the first interval and warning at the second +*/ +#define RDS_CONN_FREE_POLL_MS 100 +#define RDS_CONN_FREE_WARN_INTERVAL_MS 10000 +void rds_conn_wait_conns_freed (struct rds_transport *trans, +void (*resweep)(void ));void rds_conn_drop (struct rds_connection *conn);void rds_conn_path_drop (struct rds_conn_path *cpath,bool destroy);void rds_conn_connect_if_down (struct rds_connection *conn);
net/rds/tcp.c +1 -0
@@ -845,6 +845,7 @@ static void rds_tcp_exit(void)#endif unregister_pernet_device (&rds_tcp_net_ops);rds_tcp_destroy_conns (); +rds_conn_wait_conns_freed (&rds_tcp_transport,NULL );rds_trans_unregister (&rds_tcp_transport);rds_tcp_recv_exit ();kmem_cache_destroy (rds_tcp_conn_slab);
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 07/12] net/rds: wait for connections to be freed on transport unload
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines
From: Sharath Srinivasan <[email protected]>
Since connection free became asynchronous,
rds_conn_destroy()only quiesces the connection; the actual free - including the transport'sconn_free, which lives in the transport module - runs when the last reference is dropped. The transports' exit paths destroy all of their connections and then proceed to unload, so a free that is still pending - once the following patches hand out references to lookups, sockets and incs - would execute transport module code after that module's text is gone. At this point in the series the initial reference is the only one, so the wait only has work to do for IB, where it also catches the late arrivals described below; the guarantee becomes load-bearing with those patches.
Quoted 7 lines
Count each transport's live connections in
t_conn_count(incremented [ ... ]
place of UEK'swait_event_timeout()+WARN_ON(), plus the resweep for IB's asynchronous device detach; also coverrds_loop_exit(); rewrite commit message]Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Allison Henderson
maintainerPATCH v12 8/12 net/rds: hold a connection reference from struct rds_incoming
In reply to Allison Hendersonstruct rds_incoming->i_conn stores a pointer to the connection a message it belongs to, for both received messages and messages the socket sends. But without taking a reference, nothing keeps that connection alive. Embedded as the messages m_inc, an inc routinely outlives the connection it points at, by sitting in the socket's receive queue until the application reads it, while the connection is destroyed under it - and every dereference of i_conn after that point touches freed memory. Netns teardown and unloading rds.ko cannot do that to a live socket, which pins both, but a transport unload can: rds_find_bound() does not check the transport, so a socket bound with one transport (SO_RDS_TRANSPORT) can hold incs that arrived over another transport's connection, and unloading that other transport destroys the connection while the socket stays open.
Chengfeng Ye reported one way to reach it, where the socket info callbacks walk a receive queue after rmmod freed the connections:
BUG: KASAN: slab-use-after-free in rds6_inc_info_copy+0x459/0x530 [rds] Read of size 1 at addr ffff888106031c50 by task poc/101 Call Trace: rds6_inc_info_copy+0x459/0x530 [rds] rds6_sock_inc_info+0x2b9/0x3c0 [rds] rds_info_getsockopt+0x19d/0x380 [rds] do_sock_getsockopt+0x2ac/0x480 __sys_getsockopt+0x128/0x210 Freed by task 102: kmem_cache_free+0x1b5/0x3d0 rds_conn_destroy+0x484/0x600 [rds] rds_loop_exit_net+0x32/0x50 [rds] unregister_pernet_device+0x2c/0x50 rds_conn_exit+0x13/0xa0 [rds] rds_exit+0x1a/0xc40 [rds] __do_sys_delete_module+0x30a/0x4d0
That report has rds.ko itself unloading with an RDS socket still open, which the socket's module reference would normally prevent, so the reproduction appears to rely on a forced unload; the cross-transport case above needs none. Closing the socket gets there too, with no reader of i_conn other than RDS itself: freeing an IB inc dereferences i_conn to hand the inc and its fragments back to the connection's recycle cache, so draining the receive queue of a socket whose connection is gone crashes in the transport:
panic
...
rds_ib_recv_cache_put (net/rds/ib_recv.c:703)
rds_ib_inc_free (net/rds/ib_recv.c:207)
rds_clear_recv_queue (net/rds/recv.c:909)
rds_release (net/rds/af_rds.c:212)
__sock_release (net/socket.c:649)
sock_close (net/socket.c:1336)
with the freed connection confirmed by its now-zero reference count:
-trace[9]["inc"].i_conn.c_refcount
(struct kref){
.refcount = (refcount_t){
.refs = (atomic_t){
.counter = (int)0,
},
},
}Now that connections are reference counted, make every holder of i_conn own a reference. The pointer is assigned in six places - rds_inc_init() and rds_inc_path_init() for received messages, rds_recv_incoming() when it re-points an inc at the connection it arrived on, and rds_send_queue_rm(), rds_send_probe() and the congestion-map path of rds_send_xmit() for m_inc - and each of them now takes a reference. The references are dropped from rds_inc_put() and rds_message_put(), which are the points where the last user of the pointer goes away. rds_recv_incoming() takes the new reference before dropping the old one, so re-pointing an inc at the connection it already refers to cannot free it. rds_inc_put() drops its reference through a local copy, since inc_free() may free the memory the inc lives in.
This keeps a connection allocated for as long as messages that arrived over it are queued on sockets, which is longer than before. What lingers is the quiesced connection, its per-path workqueues (idle, every work cancelled) and the transport's per-connection state, including an IB connection's receive caches: rds_conn_destroy() still quiesces the connection synchronously, so nothing runs on any of it. It also means the transport cannot unload while such a datagram is unread - rds_conn_wait_conns_freed() waits, warning every ten seconds, until the application reads or closes. That is the intended
behaviour: teardown does not discard queued data, and unloading over a live reference would be a use-after-free in the transport's text.
The final rds_conn_put() runs the free path, which destroys the per-path workqueues and therefore may sleep, so the last reference has to be dropped from process context. It always is. While a connection is alive its hash-table entry holds the initial reference, so a put from a completion handler or tasklet can never be the last one; that reference is dropped by rds_conn_destroy(), from process context, after the connection has been quiesced and its queued messages freed. The references that survive that point are the ones held by incs on socket receive queues and by messages on socket send queues, and those are dropped from recvmsg and from close - process context in both cases.
This is not a stable candidate, and is marked for the stable tooling to skip so that it is not picked up without the patches it depends on:
reaching the use-after-free requires freeing a connection out from under a live socket, which needs CAP_SYS_MODULE, and the fix depends on the connection reference counting introduced earlier in this series.
Based on the Oracle UEK commit "net/rds: fix crash by expanding kref coverage to rds_incoming.i_conn".
[achender: port to net-next: same six assignment sites, but upstream
splits the message free across rds_message_unpin_worker(), so the m_inc reference is dropped from a shared rds_message_free() helper that both paths call; rds_recv_incoming() takes the new reference before dropping the old; rewrite commit message]
--- net/rds/message.c | 16 ++++++++++++++-- net/rds/recv.c | 27 +++++++++++++++++++++++---- net/rds/send.c | 4 ++++ 3 files changed, 41 insertions(+), 6 deletions(-)
net/rds/message.c +14 -2
@@ -182,6 +182,18 @@ static void rds_message_purge(struct rds_message *rm)kref_put (&rm->atomic.op_rdma_mr->r_kref, __rds_put_mr_final); } +static void rds_message_free (struct rds_message *rm) +{ +/* get in rds_send_queue_rm(), rds_send_probe() or the congestion +* map path of rds_send_xmit(). Messages that were never queued on +* a connection have no reference to drop. +*/ +if (rm->m_inc.i_conn) +rds_conn_put (rm->m_inc.i_conn); + +kfree (rm); +} +static void rds_message_unpin_worker (struct work_struct *work) { struct rds_message *rm = container_of(work, struct rds_message,
@@ -192,7 +204,7 @@ static void rds_message_unpin_worker(struct work_struct *work)if (rm->atomic.op_unpin_deferred)rds_atomic_op_unpin_page (&rm->atomic); -kfree (rm); +rds_message_free (rm); }void rds_message_put (struct rds_message *rm)
@@ -217,7 +229,7 @@ void rds_message_put(struct rds_message *rm)return ; } -kfree (rm); +rds_message_free (rm); } }EXPORT_SYMBOL_GPL (rds_message_put);
net/rds/recv.c +23 -4
@@ -46,6 +46,7 @@ void rds_inc_init(struct rds_incoming *inc, struct rds_connection *conn, {refcount_set (&inc->i_refcount,1 );INIT_LIST_HEAD (&inc->i_item); +rds_conn_get (conn);/* put in rds_inc_put() */ inc->i_conn = conn; inc->i_conn_path =NULL ; inc->i_saddr = *saddr;
@@ -61,6 +62,7 @@ void rds_inc_path_init(struct rds_incoming *inc, struct rds_conn_path *cp, {refcount_set (&inc->i_refcount,1 );INIT_LIST_HEAD (&inc->i_item); +rds_conn_get (cp->cp_conn);/* put in rds_inc_put() */ inc->i_conn = cp->cp_conn; inc->i_conn_path = cp; inc->i_saddr = *saddr;
@@ -81,9 +83,19 @@ void rds_inc_put(struct rds_incoming *inc) {rdsdebug ("put inc %p ref %d\n" , inc,refcount_read (&inc->i_refcount));if (refcount_dec_and_test (&inc->i_refcount)) { +struct rds_connection *conn = inc->i_conn; +BUG_ON (!list_empty (&inc->i_item)); - inc->i_conn->c_trans->inc_free (inc); +/* inc_free() can free the memory @inc lives in, so the +* connection reference has to be dropped through the +* copy taken above. +*/ + conn->c_trans->inc_free (inc); +/* get in rds_inc_init(), rds_inc_path_init() or +* rds_recv_incoming() +*/ +rds_conn_put (conn); } }EXPORT_SYMBOL_GPL (rds_inc_put);
@@ -325,6 +337,13 @@ void rds_recv_incoming(struct rds_connection *conn, struct in6_addr *saddr,unsigned long flags;struct rds_conn_path *cp; +/* every caller initialized @inc with rds_inc_init() or +* rds_inc_path_init() first, so i_conn already holds a reference. +* Take the new one before dropping the old, so that re-pointing an +* inc at the connection it already refers to cannot free it. +*/ +rds_conn_get (conn); +rds_conn_put (inc->i_conn); inc->i_conn = conn; inc->i_rx_jiffies = jiffies;if (conn->c_trans->t_mp_capable)
@@ -408,8 +427,9 @@ void rds_recv_incoming(struct rds_connection *conn, struct in6_addr *saddr,* rds_find_bound() uses a global (netns-agnostic) hash table. * An RDS connection created in netns A can match a socket bound * in the init netns, delivering inc cross-netns with inc->i_conn -* pointing into netns A. When cleanup_net() then frees that conn, -* any subsequent dereference of inc->i_conn is a use-after-free. +* pointing into netns A. Cross-netns delivery is wrong on its +* own, and the inc's connection reference would keep a +* connection of a dead netns around, with a stale c_net. * Drop the inc if the receiving socket lives in a different netns. */ if (!net_eq (sock_net (rds_rs_to_sk (rs)),rds_conn_net (conn))) {
@@ -484,7 +504,6 @@ static int rds_still_queued(struct rds_sock *rs, struct rds_incoming *inc,if (!list_empty (&inc->i_item)) { ret =1 ;if (drop) { -/* XXX make sure this i_conn is reliable */ rds_recv_rcvbuf_delta (rs, sk, inc->i_conn->c_lcong, -be32_to_cpu (inc->i_hdr.h_len), inc->i_hdr.h_dport);
net/rds/send.c +4 -0
@@ -290,6 +290,8 @@ int rds_send_xmit(struct rds_conn_path *cp) } rm->data.op_active =1 ; rm->m_inc.i_conn_path = cp; +/* put in rds_message_put() */ +rds_conn_get (cp->cp_conn); rm->m_inc.i_conn = cp->cp_conn; cp->cp_xmit_rm = rm;
@@ -947,6 +949,7 @@ static int rds_send_queue_rm(struct rds_sock *rs, struct rds_connection *conn,/* The code ordering is a little weird, but we're trying to minimize the time we hold c_lock */ rds_message_populate_header (&rm->m_inc.i_hdr, sport, dport,0 ); +rds_conn_get (conn);/* put in rds_message_put() */ rm->m_inc.i_conn = conn; rm->m_inc.i_conn_path = cp;rds_message_addref (rm);
@@ -1532,6 +1535,7 @@ rds_send_probe(struct rds_conn_path *cp, __be16 sport,list_add_tail (&rm->m_conn_item, &cp->cp_send_queue);set_bit (RDS_MSG_ON_CONN , &rm->m_flags);rds_message_addref (rm); +rds_conn_get (cp->cp_conn);/* put in rds_message_put() */ rm->m_inc.i_conn = cp->cp_conn; rm->m_inc.i_conn_path = cp;
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 08/12] net/rds: hold a connection reference from struct rds_incoming
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines
struct
rds_incoming->i_connstores a pointer to the connection a message it belongs to, for both received messages and messages the socket sends. But without taking a reference, nothing keeps that connection alive. Embedded as the messagesm_inc, an inc routinely outlives the connection it points at, by sitting in the socket's receive queue until the application reads it, while the connection is destroyed under it - and every dereference ofi_connafter that point touches freed memory. Netns teardown and unloading rds.ko cannot do that to a live socket, which pins both, but a transport unload can:rds_find_bound()does not check the transport, so a socket bound with one transport (SO_RDS_TRANSPORT) can hold incs that arrived over another transport's connection, and unloading that other transport destroys the connection while the socket stays open.
Quoted 7 lines
Chengfeng Ye reported one way to reach it, where the socket info [ ... ]
m_increference is dropped from a sharedrds_message_free()helper that both paths call;rds_recv_incoming()takes the new reference before dropping the old; rewrite commit message]Assisted-by Claude-Code:claude-opus-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Allison Henderson
maintainerPATCH v12 9/12 net/rds: take cp_lock to purge cp_send_queue in the quiesce
In reply to Allison Hendersonrds_conn_path_quiesce() empties cp_send_queue by walking it with no lock held, while every path that adds to that queue - rds_send_queue_rm(), rds_send_probe(), the retransmit requeue in rds_send_path_reset() - does so under cp_lock. The unlocked walk was justified by the destroy running with nothing else alive: at every destroy trigger there is - netns teardown and module unload - no socket can still be sending on the connection, since a bound socket pins its transport module and a namespace closes its sockets before its RDS connections are torn down.
That argument still holds, but it is an argument about the callers, not a property of the code. Splice the queue away under cp_lock and drop the message references outside it, so that the one walker of the list follows the same lock discipline as its adders. This does not by itself make a sender that is still running safe - nothing here tells such a sender that the queue is closed - and it does not need to, since no such sender exists at any destroy trigger.
While at it, make the purge coherent with rds_send_drop_to(), the other path that removes messages from a connection queue. drop_to decides whether it owns the queue's reference by test_and_clear on RDS_MSG_ON_CONN, and unlinks m_conn_item from whatever list the message is on. The purge used to BUG_ON() a message that a socket still had queued, and that assertion is also what kept a drop_to racing it from putting the queue's reference a second time or unlinking from the purge list; the lock above is what closes the stale-next hazard of the old walk, not the assertion. Clear the bit under cp_lock as part of the splice, so that drop_to leaves a purged message alone, and turn the BUG_ON() into a WARN_ON_ONCE(): a socket with messages queued at destroy is still a condition worth reporting - no sender or closer can be running at any destroy trigger today - but not one worth a panic, since the socket side keeps its own reference and retires the message on close.
--- net/rds/connection.c | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-)
net/rds/connection.c +24 -5
@@ -540,6 +540,8 @@ void rds_conn_shutdown(struct rds_conn_path *cp)static void rds_conn_path_quiesce (struct rds_conn_path *cp) {struct rds_message *rm, *rtmp; +unsigned long flags; +LIST_HEAD (purge);if (!cp->cp_transport_data)return ;
@@ -551,12 +553,29 @@ static void rds_conn_path_quiesce(struct rds_conn_path *cp)rds_conn_path_drop (cp, true);flush_work (&cp->cp_down_w); -/* tear down queued messages */ -list_for_each_entry_safe (rm, rtmp, - &cp->cp_send_queue, - m_conn_item) { +/* Tear down queued messages. Every path that adds to +* cp_send_queue does so under cp_lock; take it here too. No +* sender can still be running at any destroy trigger, so this is +* lock discipline rather than a race fix: nothing here tells a +* sender that the queue is closed. +*/ +spin_lock_irqsave (&cp->cp_lock, flags); +list_splice_init (&cp->cp_send_queue, &purge); +/* Give up the queue's claim on each message while still under +* the lock, so that rds_send_drop_to(), which decides ownership +* of the connection-queue reference by this bit, neither drops +* it a second time nor unlinks the message from our list. +*/ +list_for_each_entry (rm, &purge, m_conn_item) +clear_bit (RDS_MSG_ON_CONN , &rm->m_flags); +spin_unlock_irqrestore (&cp->cp_lock, flags); +list_for_each_entry_safe (rm, rtmp, &purge, m_conn_item) { +/* No socket can still have messages queued on a connection +* at any destroy trigger; say so if one does, since the +* socket side then retires the message on its own. +*/ +WARN_ON_ONCE (!list_empty (&rm->m_sock_item));list_del_init (&rm->m_conn_item); -BUG_ON (!list_empty (&rm->m_sock_item));rds_message_put (rm); }if (cp->cp_xmit_rm)
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 09/12] net/rds: take cp_lock to purge cp_send_queue in the quiesce
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
rds_conn_path_quiesce()emptiescp_send_queueby walking it with no lock held, while every path that adds to that queue -rds_send_queue_rm(),rds_send_probe(), the retransmit requeue inrds_send_path_reset()- does so undercp_lock. The unlocked walk was justified by the destroy running with nothing else alive: at every destroy trigger there is - netns teardown and module unload - no socket can still be sending on the connection, since a bound socket pins its transport module and a namespace closes its sockets before its RDS connections are torn down.That argument still holds, but it is an argument about the callers, not a property of the code. Splice the queue away under
cp_lockand drop the message references outside it, so that the one walker of the list follows the same lock discipline as its adders. This does not by itself make a sender that is still running safe - nothing here [ ... ]
not one worth a panic, since the socket side keeps its own reference and retires the message on close.
Quoted 2 lines
Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Allison Henderson
maintainerPATCH v12 10/12 net/rds: hold connection references in lookup, sockets and c_passive
In reply to Allison HendersonFrom: Sharath Srinivasan <[email protected]>
Hand out real references everywhere a struct rds_connection pointer previously escaped bare:
- rds_conn_lookup() takes a reference on the connection it returns (kref_get_unless_zero(), so that it only ever hands out a live reference), and
__rds_conn_create() returns the connection with a reference held for the caller on every path: lookup hit, fresh creation, lost creation race, and the passive-loopback lookup, which now also holds the parent while it dereferences parent->c_passive.
- The rs->rs_conn sendmsg cache owns a reference, which is dropped when the cache is replaced or the socket is released. rds_sendmsg() itself holds a reference for the duration of the call, during which reads and updates of rs_conn are serialized by rs_lock. So neither a concurrent rds_conn_destroy() nor another sender replacing the cache can free the connection under a sender. No destroy trigger in this tree - netns teardown, module unload - can run while a socket is still sending, since a socket holds its netns and pins its transport module; today the functional change is the rs_lock serialization that closes the data race below, and the reference is the discipline the later patches rely on. A cached connection whose destruction has begun is no longer reused, which likewise no trigger reaches today.
Instead, sendmsg evicts it and looks up or creates a live one in the same call, so a socket cannot get stuck returning -EAGAIN forever against a quiesced connection; the cache's own reference on it is dropped right there, so a socket that never sends again does not pin a quiesced connection until it is closed. Both ToS ioctls use the same lock now (they used the unrelated global rds_sock_lock before), and since the create in rds_sendmsg() samples rs_tos without the lock, the install re-checks under it that the new connection's ToS still matches the socket's and returns -EAGAIN if a SIOCRDSSETTOS slipped in between, rather than sending on, and caching, a connection with the old ToS.
- parent->c_passive owns a reference, dropped when the parent is destroyed. The pointer is read under rcu_read_lock() and written under rds_conn_lock, so it is RCU-annotated and accessed through rcu_dereference()/rcu_assign_pointer(). A passive connection whose own destroy has begun is neither handed out nor left dangling in the parent: __rds_conn_create() refuses it, and the child's destroy clears the parent's pointer and drops that reference itself, so a quiesced passive conn cannot be revived by a later connect request.
Serializing the rs_conn cache under rs_lock also resolves a syzbot-reported KCSAN data race between concurrent rds_sendmsg() calls on the same socket, each installing the connection it created into rs->rs_conn with a plain store:
BUG: KCSAN: data-race in rds_sendmsg / rds_sendmsg
write to 0xffff888101dec818 of 8 bytes by task 30904 on cpu 0:
rds_sendmsg+0xc1f/0x1580 net/rds/send.c:1332
write to 0xffff888101dec818 of 8 bytes by task 30905 on cpu 1:
rds_sendmsg+0xc1f/0x1580 net/rds/send.c:1332
value changed: 0x0000000000000000 -> 0xffff88811b61faf0
cm_id->context still carries no reference of its own after this patch; the following patch pins the connection for the duration of the CM event handler.
rds_tcp_accept_one() needs no destroy check of its own even though it now gets a referenced connection back from rds_conn_create(): TCP connections are only destroyed on netns teardown and module unload, and both go through rds_tcp_kill_sock(), which clears the listen socket the accept tests first and then flushes the accept work in rds_tcp_listen_stop(), before destroying anything.
This is not a stable candidate on its own: it depends on the connection reference counting introduced by the preceding patches, and the data race it closes has no consequence beyond the report while no destroy can run under a live socket, which is the case in this tree.
Based on the Oracle UEK commits "net/rds: Add krefs to struct rds_connection" and "net/rds: rds_sendmsg must use rs_conn only when not being destroyed".
[achender: substantial reimplementation for net-next: upstream has no
conn reaper, per-conn workers hold no references (destroy cancels them synchronously before the final put), and the sendmsg cache is serialized with rs_lock instead of UEK's socket flag; rewrite commit message]
--- net/rds/af_rds.c | 26 ++++++--- net/rds/connection.c | 125 +++++++++++++++++++++++++++++++++++++++++-- net/rds/ib_cm.c | 9 +++- net/rds/loop.c | 2 +- net/rds/rds.h | 20 +++++-- net/rds/send.c | 67 ++++++++++++++++++++--- net/rds/tcp_listen.c | 12 ++++- 7 files changed, 236 insertions(+), 25 deletions(-)
net/rds/af_rds.c +20 -6
@@ -80,6 +80,14 @@ static int rds_release(struct socket *sock)rds_notify_queue_get (rs,NULL );rds_notify_msg_zcopy_purge (&rs->rs_zcookie_queue); +/* drop the cached connection reference; no sendmsg can race +* with us here, the socket is going away +*/ +if (rs->rs_conn) { +rds_conn_put (rs->rs_conn); + rs->rs_conn =NULL ; + } +spin_lock_bh (&rds_sock_lock);list_del_init (&rs->rs_item);spin_unlock_bh (&rds_sock_lock);
@@ -254,6 +262,7 @@ static int rds_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) {struct rds_sock *rs =rds_sk_to_rs (sock->sk);rds_tos_t utos, tos =0 ; +unsigned long flags;switch (cmd) {case SIOCRDSSETTOS :
@@ -266,18 +275,23 @@ static int rds_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg)else return -ENOIOCTLCMD ; -spin_lock_bh (&rds_sock_lock); +/* rs_conn is serialized by rs_lock (see rds_sendmsg()); +* hold it across the "no connection yet" check and the +* rs_tos store so a racing sendmsg cannot cache a conn +* whose c_tos then disagrees with rs_tos. +*/ +spin_lock_irqsave (&rs->rs_lock, flags);if (rs->rs_tos || rs->rs_conn) { -spin_unlock_bh (&rds_sock_lock); +spin_unlock_irqrestore (&rs->rs_lock, flags);return -EINVAL ; } - rs->rs_tos = tos; -spin_unlock_bh (&rds_sock_lock); +WRITE_ONCE (rs->rs_tos, tos); +spin_unlock_irqrestore (&rs->rs_lock, flags);break ;case SIOCRDSGETTOS : -spin_lock_bh (&rds_sock_lock); +spin_lock_irqsave (&rs->rs_lock, flags); tos = rs->rs_tos; -spin_unlock_bh (&rds_sock_lock); +spin_unlock_irqrestore (&rs->rs_lock, flags);if (put_user (tos, (rds_tos_t __user *)arg))return -EFAULT ;break ;
net/rds/connection.c +120 -5
190 lines not on this page · raw message
net/rds/ib_cm.c +8 -1
@@ -926,8 +926,15 @@ int rds_ib_cm_handle_connect(struct rdma_cm_id *cm_id,rds_ib_conn_error (conn,"rdma_accept failed\n" ); out: -if (conn) +if (conn) {mutex_unlock (&conn->c_cm_lock); +/* Drop the reference rds_conn_create() handed us. The +* conn stays reachable through cm_id->context without a +* reference of its own for now; the CM event handler is +* given one of its own by a following patch. +*/ +rds_conn_put (conn); + }if (err)rdma_reject (cm_id, &err,sizeof (int ),IB_CM_REJ_CONSUMER_DEFINED );
net/rds/loop.c +1 -1
@@ -170,7 +170,7 @@ static void rds_loop_destroy_gathered_conns(struct list_head *tmp_list)list_for_each_entry_safe (lc, _lc, tmp_list, loop_node) { conn = lc->conn; -WARN_ON (conn->c_passive); +WARN_ON (rcu_access_pointer (conn->c_passive));spin_lock_irq (&loop_conns_lock);list_del_init (&lc->loop_node);
net/rds/rds.h +16 -4
@@ -156,7 +156,10 @@ struct rds_connection {/* Set once, by rds_conn_destroy() under rds_conn_lock - a * test-and-set, so a second destroy of the same connection * returns at once - before it cancels the path works. Read -* through rds_destroy_pending(). A site that arms +* through rds_destroy_pending(), which also reports netns +* teardown and module unload; the c_passive handling in +* __rds_conn_create() uses the same predicate, since those rule +* a passive connection out just as well. A site that arms * a path work must test the predicate and queue the work inside * one rcu_read_lock() section: the synchronize_rcu() that * follows the store is what keeps a queue issued after the
@@ -168,7 +171,7 @@ struct rds_connection {* destroys - and so flushes - that connection afterwards. */ bool c_destroy_in_prog; -struct rds_connection *c_passive; +struct rds_connection __rcu *c_passive;struct rds_transport *c_trans;struct rds_cong_map *c_lcong;
@@ -678,7 +681,12 @@ struct rds_sock {/* * rds_sendmsg caches the conn it used the last time around. -* This helps avoid costly lookups. +* This helps avoid costly lookups. The cache owns a connection +* reference, dropped when it is replaced, when rds_sendmsg() +* evicts a connection whose destroy has begun, or when the +* socket is released, and is read and written under rs_lock - +* except by +* rds_release(), which runs once no one else can reach the socket. */ struct rds_connection *rs_conn;
@@ -687,7 +695,11 @@ struct rds_sock {/* seen congestion (ENOBUFS) when sending? */ int rs_seen_congestion; -/* rs_lock protects all these adjacent members before the newline */ +/* rs_lock protects all these adjacent members before the newline, +* as well as rs_conn above and rs_tos at the end of the struct - +* except that rds_sendmsg() samples rs_tos locklessly, with +* READ_ONCE(), for the create, and re-checks it under the lock. +*/ spinlock_t rs_lock;struct list_head rs_send_queue;u32 rs_snd_bytes;
net/rds/send.c +60 -7
107 lines not on this page · raw message
net/rds/tcp_listen.c +11 -1
@@ -153,7 +153,7 @@ int rds_tcp_accept_one(struct rds_tcp_net *rtn) {struct socket *listen_sock = rtn->rds_tcp_listen_sock;struct socket *new_sock =NULL ; -struct rds_connection *conn; +struct rds_connection *conn =NULL ;int ret;struct inet_sock *inet;struct rds_tcp_connection *rs_tcp =NULL ;
@@ -229,6 +229,7 @@ int rds_tcp_accept_one(struct rds_tcp_net *rtn)if (IS_ERR (conn)) { ret =PTR_ERR (conn); + conn =NULL ; goto out; }/* An incoming SYN request came in, and TCP just accepted it.
@@ -277,6 +278,13 @@ int rds_tcp_accept_one(struct rds_tcp_net *rtn) cp = rs_tcp->t_cpath; conn_state =rds_conn_path_state (cp);WARN_ON (conn_state ==RDS_CONN_UP ); +/* A connection whose destroy has begun cannot be found here: +* TCP connections are only destroyed on netns teardown and on +* module unload, and both go through rds_tcp_kill_sock() - which +* clears the listen socket that the top of this function tests +* and then flushes this work in rds_tcp_listen_stop() - before +* any connection is destroyed. +*/ if (conn_state !=RDS_CONN_CONNECTING && conn_state !=RDS_CONN_ERROR ) {rds_conn_path_drop (cp,0 ); goto rst_nsk;
@@ -347,6 +355,8 @@ int rds_tcp_accept_one(struct rds_tcp_net *rtn)mutex_unlock (&rs_tcp->t_conn_path_lock);if (new_sock)sock_release (new_sock); +if (conn) +rds_conn_put (conn);mutex_unlock (&rtn->rds_tcp_accept_lock);
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 10/12] net/rds: hold connection references in lookup, sockets and c_passive
Sashiko has reviewed this patch and found no issues.
Quoted 13 lines
From: Sharath Srinivasan <[email protected]>
Hand out real references everywhere a struct
rds_connectionpointer previously escaped bare:-
rds_conn_lookup()takes a reference on the connection it returns (kref_get_unless_zero(), so that it only ever hands out a live reference), and
__rds_conn_create()returns the connection with a reference held for the caller on every path: lookup hit, fresh creation, lost creation race, and the passive-loopback lookup, which now also holds the parent while it dereferencesparent->c_passive.
Quoted 8 lines
- The
rs->rs_connsendmsg cache owns a reference, which is dropped when the cache is replaced or the socket is released.
[ ... ]
them synchronously before the final put), and the sendmsg cache is serialized withrs_lockinstead of UEK's socket flag; rewrite commit message]Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Allison Henderson
maintainerPATCH v12 11/12 net/rds: give cm_id->context a reference on its connection
In reply to Allison Hendersonrds_rdma_cm_event_handler_cmn() picks the connection up from cm_id->context and holds c_cm_lock - a mutex that lives in the connection's path array - across the transport callbacks, but the context pointer owns no reference. What keeps the connection alive under the handler is ordering: rds_ib_conn_path_shutdown() destroys the id, and rdma_destroy_id() waits for a running handler, before the destroy drops the connection's initial reference, so no event is handled on a connection that is being freed. Before this series that ordering was broken from inside the handler, since rds_conn_destroy() freed synchronously and the one callback that destroyed its connection, rds_ib_cm_connect_complete() on a protocol version below 3.1, left the handler's mutex_unlock() running on freed memory; commit f97d8c7bab78 ("rds: ib: use rds_conn_drop() on protocol version mismatch") has since switched that path to rds_conn_drop(), which also removed the deadlock that destroy took on c_cm_lock.
Now that connections are reference counted, give the id a reference of its own rather than rely on that ordering. rds_ib_conn_path_connect() takes it once rdma_create_id() has installed the connection as the context, rds_ib_cm_handle_connect() takes it when it installs the id on the passive side, and each is dropped after rdma_destroy_id() has
returned: in rds_ib_conn_path_shutdown(), and on the two failure paths of rds_ib_conn_path_connect() that destroy the id straight away. With that the handler cannot find its connection freed under it, whatever the callbacks do to the other holders - the destroy's initial reference, a socket's cache, a parent's c_passive, an inc. None of the new puts can be the last reference: the shutdown runs on behalf of a destroy, or of a drop on a connection still in the hash, and the connect worker runs on a connection still in the hash, and each of those holds a reference of its own.
rds_ib_cm_initiate_connect() and rds_ib_cm_handle_connect() get a defensive check of the same kind. A destroy that began while the address and route were resolving either waits on c_cm_lock behind the ROUTE_RESOLVED handler - and its shutdown pass then tears down whatever the handler set up, QP, cm_id and device reference alike - or has already destroyed the id, in which case no event for it reaches the handler. On the passive side a quiesced connection is unhashed, so rds_conn_create() cannot hand it to a connect request, and the listeners are stopped before any IB connection is destroyed at unload. Neither path is reachable today, then; both functions nevertheless decline to set up a QP, or install a new cm_id, on a connection whose destroy has begun, so that the contract does not rest on that ordering. rds_ib_cm_initiate_connect() returns success when it
declines: the id is still ic->i_cm_id, for the shutdown to destroy, and a non-zero return would have the rdma_cm destroy it instead. An unload that begins after the check has passed is covered by rds_ib_add_conn() refusing a device that rds_ib_dev_shutdown() has marked.
--- net/rds/ib_cm.c | 41 ++++++++++++++++++++++++++++++++++++---- net/rds/rdma_transport.c | 7 +++++++ 2 files changed, 44 insertions(+), 4 deletions(-)
net/rds/ib_cm.c +37 -4
@@ -876,6 +876,15 @@ int rds_ib_cm_handle_connect(struct rdma_cm_id *cm_id,* see the comment above rds_queue_reconnect() */ mutex_lock (&conn->c_cm_lock); +/* Defensive: a destroy that has quiesced this conn also unhashed +* it, so rds_conn_create() cannot have returned it, and the +* listeners are stopped before any IB connection is destroyed at +* unload. Should a request reach a conn in RDS_CONN_DOWN with +* its destroy begun all the same, do not install a new cm_id and +* QP that nothing would tear down. +*/ +if (rds_destroy_pending (conn)) + goto out;if (!rds_conn_transition (conn,RDS_CONN_DOWN ,RDS_CONN_CONNECTING )) {if (rds_conn_state (conn) ==RDS_CONN_UP ) {rdsdebug ("incoming connect while connecting\n" );
@@ -904,6 +913,10 @@ int rds_ib_cm_handle_connect(struct rdma_cm_id *cm_id,BUG_ON (ic->i_cm_id); ic->i_cm_id = cm_id; +/* The id owns a reference on the connection it points at, put +* after rdma_destroy_id() in rds_ib_conn_path_shutdown(). +*/ +rds_conn_get (conn); cm_id->context = conn;/* We got halfway through setting up the ib_connection, if we
@@ -928,10 +941,8 @@ int rds_ib_cm_handle_connect(struct rdma_cm_id *cm_id, out:if (conn) {mutex_unlock (&conn->c_cm_lock); -/* Drop the reference rds_conn_create() handed us. The -* conn stays reachable through cm_id->context without a -* reference of its own for now; the CM event handler is -* given one of its own by a following patch. +/* Drop the reference rds_conn_create() handed us; the +* cm_id holds its own from here on. */ rds_conn_put (conn); }
@@ -950,6 +961,18 @@ int rds_ib_cm_initiate_connect(struct rdma_cm_id *cm_id, bool isv6)union rds_ib_conn_priv dp;int ret; +/* Defensive: a destroy that began while the address and route +* were being resolved either waits on c_cm_lock behind this +* handler, and its shutdown pass then tears down whatever is +* set up here, or has already destroyed the id this event +* arrived on, in which case the event never gets here. Decline +* to set up a QP all the same, and return success: the id is +* still ic->i_cm_id, for that shutdown to destroy, and a +* non-zero return would have the rdma_cm destroy it instead. +*/ +if (rds_destroy_pending (conn)) +return 0 ; +/* If the peer doesn't do protocol negotiation, we must * default to RDSv3.0 */ rds_ib_set_protocol (conn,RDS_PROTOCOL_4_1 );
@@ -1007,6 +1030,10 @@ int rds_ib_conn_path_connect(struct rds_conn_path *cp)rdsdebug ("rdma_create_id() failed: %d\n" , ret); goto out; } +/* The id owns a reference on the connection it points at, put +* after rdma_destroy_id() wherever the id is destroyed. +*/ +rds_conn_get (conn);/* rds_ib_laddr_check() only vouched for the local address being * on an IB device; the address resolution below picks the device
@@ -1017,6 +1044,7 @@ int rds_ib_conn_path_connect(struct rds_conn_path *cp)rdsdebug ("rdma_restrict_node_type() failed: %d\n" , ret);rdma_destroy_id (ic->i_cm_id); ic->i_cm_id =NULL ; +rds_conn_put (conn); goto out; }
@@ -1058,6 +1086,7 @@ int rds_ib_conn_path_connect(struct rds_conn_path *cp) ret);rdma_destroy_id (ic->i_cm_id); ic->i_cm_id =NULL ; +rds_conn_put (conn); } out:
@@ -1188,6 +1217,10 @@ void rds_ib_conn_path_shutdown(struct rds_conn_path *cp)rds_ib_recv_clear_ring (ic);rdma_destroy_id (ic->i_cm_id); +/* The id's reference on the connection. Never the last +* one: whoever runs this shutdown holds its own. +*/ +rds_conn_put (conn);/* * Move connection back to the nodev list.
net/rds/rdma_transport.c +7 -0
@@ -60,6 +60,13 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id,rdsdebug ("conn %p id %p handling event %u (%s)\n" , conn, cm_id, event->event,rdma_event_msg (event->event)); +/* cm_id->context owns a reference on the connection, dropped +* only after rdma_destroy_id() has returned, and that waits for +* a running handler. The connection, and the mutex released at +* out: that lives in its path array, therefore outlive this +* handler whatever the callbacks below do to the other holders. +*/ +/* Prevent shutdown from tearing down the connection * while we're executing. */ if (conn) {
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 11/12] net/rds: give cm_id->context a reference on its connection
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
rds_rdma_cm_event_handler_cmn()picks the connection up fromcm_id->contextand holdsc_cm_lock- a mutex that lives in the connection's path array - across the transport callbacks, but the context pointer owns no reference. What keeps the connection alive under the handler is ordering:rds_ib_conn_path_shutdown()destroys the id, andrdma_destroy_id()waits for a running handler, before the destroy drops the connection's initial reference, so no event is handled on a connection that is being freed. Before this series that ordering was broken from inside the handler, sincerds_conn_destroy()freed synchronously and the one callback that destroyed its connection,rds_ib_cm_connect_complete()on a protocol version below 3.1, left the handler'smutex_unlock()running on freed memory; commitf97d8c7bab78("rds: ib: userds_conn_drop()on protocol version mismatch") has since switched that path tords_conn_drop(), which also removed the deadlock that destroy took onc_cm_lock.
[ ... ]rds_ib_add_conn()refusing a device thatrds_ib_dev_shutdown()has marked.
Quoted 2 lines
Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Allison Henderson
maintainerPATCH v12 12/12 net/rds: drop rds_conn_count in favor of t_conn_count
In reply to Allison HendersonThe patch "net/rds: wait for connections to be freed on transport unload", earlier in this series, gave each transport its own connection count in t_conn_count, incremented and decremented at exactly the points where the global rds_conn_count is. That leaves rds_conn_count with a single remaining consumer: the seed of the per-path workqueue names in __rds_conn_create().
Switch the name seed to t_conn_count, as UEK does, and remove rds_conn_count. The numbering becomes per-transport instead of global, so connections of different transports can now receive the same seed; workqueue names carry no uniqueness requirement, and the seed was already reused as the count rose and fell. Removing the counter also removes the rds_conn_lock round-trip that rds_conn_destroy_fini() took solely to decrement it. (The free path still sleeps in destroy_workqueue(), so the last reference has to be dropped from process context as before; only that one lock goes.)
Based on the Oracle UEK commit "net/rds: Each RDS transport should keep its own connection count".
--- net/rds/connection.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-)
net/rds/connection.c +3 -10
@@ -47,7 +47,6 @@/* converting this to RCU is a chore for another day.. */ static DEFINE_SPINLOCK (rds_conn_lock); -static unsigned long rds_conn_count;/* woken whenever a transport's t_conn_count drops to zero */ static DECLARE_WAIT_QUEUE_HEAD (rds_conn_freed_waitq);static struct hlist_head rds_conn_hash[RDS_CONNECTION_HASH_ENTRIES ];
@@ -338,12 +337,13 @@ static struct rds_connection *__rds_conn_create(struct net *net,init_waitqueue_head (&conn->c_hs_waitq);for (i =0 ; i < npaths; i++) { +int seq =atomic_read (&trans->t_conn_count); +__rds_conn_path_init (conn, &conn->c_path[i], is_outgoing); conn->c_path[i].cp_index = i; conn->c_path[i].cp_wq = -alloc_ordered_workqueue ("krds_cp_wq#%lu/%d" ,0 , - rds_conn_count, i); +alloc_ordered_workqueue ("krds_cp_wq#%d/%d" ,0 , seq, i);if (!conn->c_path[i].cp_wq) conn->c_path[i].cp_wq = rds_wq; }
@@ -412,7 +412,6 @@ static struct rds_connection *__rds_conn_create(struct net *net,rds_conn_get (conn);/* caller */ rcu_assign_pointer (parent->c_passive, conn);rds_cong_add_conn (conn); - rds_conn_count++;atomic_inc (&conn->c_trans->t_conn_count); } }else {
@@ -435,7 +434,6 @@ static struct rds_connection *__rds_conn_create(struct net *net,rds_conn_get (conn);hlist_add_head_rcu (&conn->c_hash_node, head);rds_cong_add_conn (conn); - rds_conn_count++;atomic_inc (&conn->c_trans->t_conn_count); } }
@@ -684,7 +682,6 @@ static void rds_conn_destroy_fini(struct kref *kref) c_refcount);int npaths = (conn->c_trans->t_mp_capable ?RDS_MPATH_WORKERS :1 );struct rds_transport *trans = conn->c_trans; -unsigned long flags;int i;for (i =0 ; i < npaths; i++)
@@ -693,10 +690,6 @@ static void rds_conn_destroy_fini(struct kref *kref)kfree (conn->c_path);kmem_cache_free (rds_conn_slab, conn); -spin_lock_irqsave (&rds_conn_lock, flags); - rds_conn_count--; -spin_unlock_irqrestore (&rds_conn_lock, flags); -/* only after everything the transport module owns has been * freed above may its unload proceed */
Signature · 1 line
2.50.1
[email protected]
AI reviewRe: [PATCH net-next v12 12/12] net/rds: drop rds_conn_count in favor of t_conn_count
Sashiko has reviewed this patch and found no issues.
Quoted 17 lines
The patch "net/rds: wait for connections to be freed on transport unload", earlier in this series, gave each transport its own connection count in
t_conn_count, incremented and decremented at exactly the points where the globalrds_conn_countis. That leavesrds_conn_countwith a single remaining consumer: the seed of the per-path workqueue names in__rds_conn_create().Switch the name seed to
t_conn_count, as UEK does, and removerds_conn_count. The numbering becomes per-transport instead of global, so connections of different transports can now receive the same seed; workqueue names carry no uniqueness requirement, and the seed was already reused as the count rose and fell. Removing the counter also removes therds_conn_lockround-trip thatrds_conn_destroy_fini()took solely to decrement it. (The free path still sleeps indestroy_workqueue(), so the last reference has to be dropped from process context as before; only that one lock goes.)
Quoted 5 lines
Based on the Oracle UEK commit "net/rds: Each RDS transport should keep its own connection count".
Assisted-by Claude-Code:claude-fable-5Signed-off-by Allison Henderson <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 12 lines · On Thu, 12 Nov 2026 21:43:43 -0700 you wrote:
Hi all,
This is v12 of the connection-lifetime set (v1 at [1], v2 at [2], v3 at [3], v4 at [6], v5 at [7], v6 at [8], v7 at [9], v8 at [10], v9 at [11], v10 at [13], v11 at [14]), following
"net/rds: own the fastpath locks across connection teardown", which is in net-next. It is targeted at net-next: although the series fixes real use-after-frees (one syzbot report and one report from Chengfeng Ye below), it does so by reworking connection lifetime rather than patching the individual crash sites, and that rework is too invasive for net.
Quoted 1 line
[...]
Here is the summary with links:
- [net-next,v12,01/12] net/rds: undo conn_alloc() the same way on every __rds_conn_create() exit
https://git.kernel.org/netdev/net-next/c/5483e92733cb - [net-next,v12,02/12] net/rds: ib: refuse to attach a connection to a device being removed https://git.kernel.org/netdev/net-next/c/725ebc948965 - [net-next,v12,03/12] net/rds: guard every work-requeueing site with rds_destroy_pending() https://git.kernel.org/netdev/net-next/c/306da594a7c3 - [net-next,v12,04/12] net/rds: make rds_destroy_pending() report a connection's own destroy https://git.kernel.org/netdev/net-next/c/8e0bf84ecbdb - [net-next,v12,05/12] net/rds: split connection destroy into quiesce and kref-governed free https://git.kernel.org/netdev/net-next/c/cfa3110a44e3 - [net-next,v12,06/12] net/rds: unlink transport nodes before a possibly deferred connection free https://git.kernel.org/netdev/net-next/c/9f000d125055 - [net-next,v12,07/12] net/rds: wait for connections to be freed on transport unload https://git.kernel.org/netdev/net-next/c/32252168aff5 - [net-next,v12,08/12] net/rds: hold a connection reference from struct rds_incoming https://git.kernel.org/netdev/net-next/c/217fac53504e - [net-next,v12,09/12] net/rds: take cp_lock to purge cp_send_queue in the quiesce https://git.kernel.org/netdev/net-next/c/d3c3a41fd3a3 - [net-next,v12,10/12] net/rds: hold connection references in lookup, sockets and c_passive https://git.kernel.org/netdev/net-next/c/51ac0ecd2f81 - [net-next,v12,11/12] net/rds: give cm_id->context a reference on its connection https://git.kernel.org/netdev/net-next/c/5b61d9aed8f6 - [net-next,v12,12/12] net/rds: drop rds_conn_count in favor of t_conn_count https://git.kernel.org/netdev/net-next/c/e7a282d9858a
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html