net/smc: bound the peer producer cursor on SMC-D and SMC-R CDC receive
Subsystems (5)
PATCH 1/1 net/smc: bound the peer producer cursor on SMC-D and SMC-R CDC receive
The SMC CDC receive path copies the peer's producer cursor -- a wire-controlled value -- into the local connection cursor with no upper bound against the receive buffer (RMB). A malicious peer can advertise a producer cursor past rmb_desc->len, which is then used out of bounds:
- the urgent path uses the cursor count as a raw index:
smc_cdc_handle_urg_data_arrival() dereferences *(rmb_desc->cpu_addr + rx_off + urg_curs.count - 1);
- the receive path turns the cursor into a length:
smc_cdc_msg_recv_action() feeds it to smc_curs_diff() and atomic_add()s the result into conn->bytes_to_rcv. The differing-wrap branch returns (len - old.count) + new.count, which exceeds len for a forged cursor and accumulates across CDCs, so bytes_to_rcv grows past rmb_desc->len even when the cursor count itself is bounded; smc_rx_recvmsg() then copies the wrap-around second chunk past the RMB.
The RMB is a kernel allocation, so the reads disclose adjacent kernel memory, and a cursor pointing at an unmapped offset faults in the receive tasklet (softirq). Both transports are affected: SMC-D converts the cursor in smcd_cdc_msg_to_host() and SMC-R in smc_cdc_cursor_to_host(), and neither bounds the count.
Bound the producer cursor count to the RMB at both conversion boundaries (this closes the urgent index on SMC-D and SMC-R) and enforce the documented "0 <= bytes_to_rcv <= rmb_desc->len" invariant in smc_cdc_msg_recv_action() (this closes the receive length, which the cursor-count clamp alone cannot because of the differing-wrap diff and the cumulative atomic_add).
---
Reproduced under KASAN on 6.12.y. A forged producer cursor with prod.count = rmb_desc->len + 1 and prod_flags.urg_data_present set produces a 1-byte out-of-bounds read in smc_cdc_handle_urg_data_arrival() (slab-out-of-bounds Read of size 1); the cursor-count clamp makes the same input in-bounds. A wrap-flipped cursor drives bytes_to_rcv past rmb_desc->len across several CDCs so smc_rx_recvmsg() over-reads; the bytes_to_rcv invariant keeps it bounded. SMC-D was exercised over the in-kernel loopback-ism and the SMC-R converter (smc_cdc_cursor_to_host) over an emulated RDMA loopback. Clamping prod.count alone does not bound the recv length, hence the separate bytes_to_rcv hunk.
net/smc/smc_cdc.c | 2 ++ net/smc/smc_cdc.h | 12 ++++++++++++ 2 files changed, 14 insertions(+)
net/smc/smc_cdc.c +2 -0
@@ -382,6 +382,8 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc,smp_mb__before_atomic ();atomic_add (diff_prod, &conn->bytes_to_rcv);/* guarantee 0 <= bytes_to_rcv <= rmb_desc->len */ +if (atomic_read (&conn->bytes_to_rcv) > conn->rmb_desc->len) +atomic_set (&conn->bytes_to_rcv, conn->rmb_desc->len);smp_mb__after_atomic (); smc->sk.sk_data_ready (&smc->sk); }else {
net/smc/smc_cdc.h +12 -0
@@ -230,6 +230,12 @@ static inline void smc_cdc_cursor_to_host(union smc_host_cursor *local,smc_curs_copy_net (&net, peer, conn); temp.count =ntohl (net.count); temp.wrap =ntohs (net.wrap); +/* the peer producer cursor is wire-controlled; bound the SMC-R count to +* our RMB before it is used as a raw index by the urgent path, mirroring +* the SMC-D conversion in smcd_cdc_msg_to_host(). +*/ +if (temp.count > conn->rmb_desc->len) + temp.count = conn->rmb_desc->len;if ((old.wrap > temp.wrap) && temp.wrap)return ; if ((old.wrap == temp.wrap) &&
@@ -260,6 +266,12 @@ static inline void smcd_cdc_msg_to_host(struct smc_host_cdc_msg *local, temp.wrap = peer->prod.wrap; temp.count = peer->prod.count; +/* the peer producer cursor is wire-controlled; a count past our RMB is +* used as a raw index by the urgent path (smc_cdc_handle_urg_data_arrival) +* and as a length by the recv path. Bound it to the RMB. +*/ +if (temp.count > conn->rmb_desc->len) + temp.count = conn->rmb_desc->len;smc_curs_copy (&local->prod, &temp, conn); temp.wrap = peer->cons.wrap;
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
New issues:
- [High] Incorrect clamping of SMC-R consumer cursor permanently stalls connections with asymmetric buffer sizes.
- [High] A Time-of-Check to Time-of-Use (TOCTOU) race condition in clamping conn->bytes_to_rcv allows out-of-bounds reads and data corruption.
Pre-existing issues:
- [Critical] Unbounded consumer cursor in SMC-D leaves a kernel heap buffer overflow vulnerability unpatched in the send path. --
commit c64247d450ada1697a0e227038dd52be750039e2
Author: Bryam Vargas <[email protected]>
net/smc: bound the peer producer cursor on SMC-D and SMC-R CDC receive
This commit adds bounds checking to the peer producer cursor in the SMC-D and SMC-R receive paths to prevent out-of-bounds reads. It also clamps the bytes_to_rcv metric to the receive buffer length.
net/smc/smc_cdc.c static void smc_cdc_msg_recv_action(struct smc_sock *smc, · line 386
Quoted 10 lines
net/smc/smc_cdc.c +2
-0@@ -382,6 +382,8 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc,smp_mb__before_atomic ();atomic_add (diff_prod, &conn->bytes_to_rcv);/* guarantee 0 <= bytes_to_rcv <= rmb_desc->len */ +if (atomic_read (&conn->bytes_to_rcv) > conn->rmb_desc->len) +atomic_set (&conn->bytes_to_rcv, conn->rmb_desc->len);
[Severity: High]
Does this non-atomic clamping sequence introduce a race condition?
If a peer sends a forged producer cursor to inflate diff_prod, bytes_to_rcv will temporarily exceed rmb_desc->len during the window between the atomic_add() and the atomic_set().
If smc_rx_recvmsg() is running concurrently in process context on another CPU, could it read this transiently inflated value?
net/smc/smc_rx.c:smc_rx_recvmsg() {
...
copylen = min_t(size_t, read_remaining, readable);
...
chunk_len = copylen - chunk_len; /* remainder */
...
rc = memcpy_to_msg(msg, rcvbuf_base + chunk_off, chunk_len);
}If smc_rx_recvmsg() calculates a copylen larger than the buffer length based on the oversized bytes_to_rcv, would this lead to an out-of-bounds read on the second chunk?
Additionally, if smc_rx_recvmsg() executes an atomic_sub() during this exact window, wouldn't the subsequent atomic_set() overwrite the subtracted value, artificially inflating bytes_to_rcv again?
Quoted 2 lines
net/smc/smc_cdc.c +0
-0smp_mb__after_atomic (); smc->sk.sk_data_ready (&smc->sk);
[ ... ]
Quoted 5 lines
net/smc/smc_cdc.h +0
-0
net/smc/smc_cdc.h static inline void smc_cdc_cursor_to_host(union smc_host_cursor *local, · line 238
Quoted 9 lines
net/smc/smc_cdc.h +6
-0@@ -230,6 +230,12 @@ static inline void smc_cdc_cursor_to_host(union smc_host_cursor *local,smc_curs_copy_net (&net, peer, conn); temp.count =ntohl (net.count); temp.wrap =ntohs (net.wrap); +/* the peer producer cursor is wire-controlled; bound the SMC-R count to +* our RMB before it is used as a raw index by the urgent path, mirroring +* the SMC-D conversion in smcd_cdc_msg_to_host(). +*/ +if (temp.count > conn->rmb_desc->len) + temp.count = conn->rmb_desc->len;
[Severity: High]
Does this hardcoded clamp permanently stall connections that use asymmetric buffer sizes?
This function is used by smcr_cdc_msg_to_host() to parse both the producer and consumer cursors:
net/smc/smc_cdc.h:smcr_cdc_msg_to_host() {
smc_cdc_cursor_to_host(&local->prod, &peer->prod, conn);
smc_cdc_cursor_to_host(&local->cons, &peer->cons, conn);
}The consumer cursor represents the peer's read position within its own receive buffer, so its natural limit is conn->peer_rmbe_size. If a connection has a peer_rmbe_size larger than the local rmb_desc->len, legitimate consumer cursor updates will be truncated by this new clamp.
When smc_curs_diff() computes the consumed space using this truncated cursor, it yields zero. Does this prevent peer_rmbe_space from being replenished, causing the sender to falsely believe the peer's buffer is full and stalling the connection?
Quoted 2 lines
net/smc/smc_cdc.h +0
-0if ((old.wrap > temp.wrap) && temp.wrap)return ;
[ ... ]
net/smc/smc_cdc.h static inline void smcd_cdc_msg_to_host(struct smc_host_cdc_msg *local, · line 277
Quoted 13 lines
net/smc/smc_cdc.h +6
-0@@ -260,6 +266,12 @@ static inline void smcd_cdc_msg_to_host(struct smc_host_cdc_msg *local, temp.wrap = peer->prod.wrap; temp.count = peer->prod.count; +/* the peer producer cursor is wire-controlled; a count past our RMB is +* used as a raw index by the urgent path (smc_cdc_handle_urg_data_arrival) +* and as a length by the recv path. Bound it to the RMB. +*/ +if (temp.count > conn->rmb_desc->len) + temp.count = conn->rmb_desc->len;smc_curs_copy (&local->prod, &temp, conn); temp.wrap = peer->cons.wrap;
[Severity: Critical]
This is a pre-existing issue, but it appears the consumer cursor in smcd_cdc_msg_to_host() is still left unbounded.
While the producer cursor is clamped, the consumer cursor is copied directly:
net/smc/smc_cdc.h:smcd_cdc_msg_to_host() {
temp.wrap = peer->cons.wrap;
temp.count = peer->cons.count;
smc_curs_copy(&local->cons, &temp, conn);
}If a peer sends an SMC-D CDC message with an unexpectedly large consumer count, this unbounded cursor is passed to smc_curs_diff() in smc_cdc_msg_recv_action(), which generates an oversized diff_tx and inflates conn->sndbuf_space past conn->sndbuf_desc->len.
When the local sender transmits data, can this inflated sndbuf_space allow copylen to exceed the send buffer size in smc_tx_sendmsg()?
net/smc/smc_tx.c:smc_tx_sendmsg() {
...
chunk_len = copylen - chunk_len; /* remainder */
...
rc = memcpy_from_msg(sndbuf_base + chunk_off, msg, chunk_len);
}During wrap-around, wouldn't the second chunk's length exceed the allocated buffer bounds, resulting in an out-of-bounds memcpy_from_msg() into adjacent memory?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Jakub Kicinski
maintainerThis is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
---
net/smc: bound the peer producer cursor on SMC-D and SMC-R CDC receive
This patch attempts to close an out-of-bounds read in the SMC CDC receive path by clamping the peer producer cursor and the bytes_to_rcv accumulator against rmb_desc->len. A couple of questions on the approach below.
net/smc/smc_cdc.c static void smc_cdc_msg_recv_action(struct smc_sock *smc, · line 386
Quoted 10 lines
net/smc/smc_cdc.c +2
-0@@ -382,6 +382,8 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc,smp_mb__before_atomic ();atomic_add (diff_prod, &conn->bytes_to_rcv);/* guarantee 0 <= bytes_to_rcv <= rmb_desc->len */ +if (atomic_read (&conn->bytes_to_rcv) > conn->rmb_desc->len) +atomic_set (&conn->bytes_to_rcv, conn->rmb_desc->len);
[Critical]
Is this clamp safe against a concurrent smc_rx_recvmsg() on another CPU?
smc_cdc_msg_recv() runs the BH path under bh_lock_sock() (slock with BH disabled), while smc_rx_recvmsg() runs under lock_sock(), which sets sk_lock.owned but releases the slock for the body of the syscall. The body contains:
/* update cursors */
if (!(flags & MSG_PEEK)) {
smp_mb__before_atomic();
atomic_sub(copylen, &conn->bytes_to_rcv);
smp_mb__after_atomic();
...
}so the BH on CPU0 and the recvmsg body on CPU1 can run concurrently on the same connection.
The new clamp here is a non-atomic read-then-set:
atomic_add(diff_prod, &conn->bytes_to_rcv);
if (atomic_read(&conn->bytes_to_rcv) > conn->rmb_desc->len)
atomic_set(&conn->bytes_to_rcv, conn->rmb_desc->len);With L = rmb_desc->len, the commit message notes that the differing-wrap branch of smc_curs_diff() can return up to (L - old.count) + new.count, i.e. close to 2L, even after the count itself is clamped. Can the following interleaving happen?
CPU0 (BH) CPU1 (recvmsg)
atomic_add(diff_prod) -> X (~2L)
readable =
atomic_read(&bytes_to_rcv) = X
copylen = min(remaining, X) chunk_len = min(copylen,
L - cons.count)
second chunk = copylen - chunk_len memcpy_to_msg(rcvbuf_base + 0,
second_chunk_len)
atomic_read() -> X
atomic_set(&bytes_to_rcv, L)
With cons.count near L/2 and copylen near 2L, the second chunk reads roughly 3L/2 bytes starting at rcvbuf_base + 0, walking past the end of the RMB allocation. Is that not the same OOB read this patch is meant to close?
A second variant of the same race seems to silently corrupt the counter:
CPU0 (BH) CPU1 (recvmsg)
atomic_add(diff_prod) -> X > L
atomic_sub(Y) -> X - Y
atomic_read() -> X - Y
(X - Y still > L?)
atomic_set(L)
If X - Y is still above L, CPU0 overwrites the consumer's decrement and bytes_to_rcv ends up at L even though Y bytes were already consumed, leaving a persistent over-report that future recvmsg calls use as readable.
smp_mb__after_atomic() does not turn this read-then-set into an RMW; atomic_set() is a relaxed store. Would clamping diff_prod against (L - atomic_read(&bytes_to_rcv)) before the atomic_add (or using an atomic_cmpxchg loop) close both windows?
Quoted 8 lines
net/smc/smc_cdc.c +0
-0smp_mb__after_atomic (); smc->sk.sk_data_ready (&smc->sk); }else {net/smc/smc_cdc.h +0
-0
net/smc/smc_cdc.h static inline void smc_cdc_cursor_to_host(union smc_host_cursor *local, · line 238
Quoted 9 lines
net/smc/smc_cdc.h +6
-0@@ -230,6 +230,12 @@ static inline void smc_cdc_cursor_to_host(union smc_host_cursor *local,smc_curs_copy_net (&net, peer, conn); temp.count =ntohl (net.count); temp.wrap =ntohs (net.wrap); +/* the peer producer cursor is wire-controlled; bound the SMC-R count to +* our RMB before it is used as a raw index by the urgent path, mirroring +* the SMC-D conversion in smcd_cdc_msg_to_host(). +*/ +if (temp.count > conn->rmb_desc->len) + temp.count = conn->rmb_desc->len;
[High]
Should this clamp live in the shared helper, or only on the prod call site?
smc_cdc_cursor_to_host() is invoked twice from smcr_cdc_msg_to_host():
smc_cdc_cursor_to_host(&local->prod, &peer->prod, conn);
smc_cdc_cursor_to_host(&local->cons, &peer->cons, conn);The producer cursor is bounded by our local RMB (conn->rmb_desc->len), but the consumer cursor lives in the peer's RMB space and is bounded by conn->peer_rmbe_size, which is set independently in smcr_conn_save_peer_info() from the peer's CLC bufsize.
smc_cdc_msg_recv_action() confirms the asymmetry:
diff_cons = smc_curs_diff(conn->peer_rmbe_size, &cons_old,
&conn->local_rx_ctrl.cons);
...
atomic_add(diff_cons, &conn->peer_rmbe_space);so peer_rmbe_size (not rmb_desc->len) is the right bound for the cons cursor.
Whenever peer_rmbe_size > rmb_desc->len, legitimate peer cons.count values in (rmb_desc->len, peer_rmbe_size] are silently truncated to rmb_desc->len here, diff_cons is under-credited and peer_rmbe_space stays artificially low, which can stall TX flow control on perfectly well-behaved peers.
The SMC-D companion in this same patch is structured the way the prod/cons asymmetry suggests -- it clamps prod and copies cons unmodified:
net/smc/smc_cdc.h static inline void smcd_cdc_msg_to_host(struct smc_host_cdc_msg *local, · line 277
Quoted 13 lines
net/smc/smc_cdc.h +6
-0@@ -260,6 +266,12 @@ static inline void smcd_cdc_msg_to_host(struct smc_host_cdc_msg *local, temp.wrap = peer->prod.wrap; temp.count = peer->prod.count; +/* the peer producer cursor is wire-controlled; a count past our RMB is +* used as a raw index by the urgent path (smc_cdc_handle_urg_data_arrival) +* and as a length by the recv path. Bound it to the RMB. +*/ +if (temp.count > conn->rmb_desc->len) + temp.count = conn->rmb_desc->len;smc_curs_copy (&local->prod, &temp, conn); temp.wrap = peer->cons.wrap;
Should the SMC-R hunk be moved out of the shared smc_cdc_cursor_to_host() helper and onto the prod call site in smcr_cdc_msg_to_host(), to match the SMC-D shape?
Signature · 1 line
pw-bot: cr
Quoted 1 line · Jakub Kicinski
Is this clamp safe against a concurrent
smc_rx_recvmsg()on another CPU?
Confirmed -- the tasklet read-then-set is racy: a recvmsg()/sendmsg() on another CPU reads the inflated value in the window between the atomic_add() and the clamp (recvmsg() runs under lock_sock(), which leaves the slock free, so it is not serialized against the bh_lock_sock() CDC tasklet). Reworked as a v3 series:
https://lore.kernel.org/netdev/[email protected]/
The bound now lives at the consumer (smc_rx_recvmsg() / smc_tx_sendmsg()), where it is race-free; it also rejects a sign-overflowed (negative) accumulator (per the sashiko-bot review on the sndbuf_space patch); and the producer-cursor clamp is applied to the producer cursor only, so the consumer cursor stays bounded by peer_rmbe_size, not rmb_desc->len. The sndbuf_space fix is folded in as patch 3/3.
Thanks for the review.
Bryam