net: rds: fix uninitialized trans dereference in CM event handler
Subsystems (5)
PATCH v2 1/1 net: rds: fix uninitialized trans dereference in CM event handler
From: Aohan Mei <[email protected]>
rds_rdma_cm_event_handler_cmn() assigns trans only when the RDMA device is an InfiniBand CA (RDMA_NODE_IB_CA). On any other device type, e.g. an iWARP RNIC such as siw, trans stays uninitialized, but the event switch dereferences it: unconditionally in the RDMA_CM_EVENT_CONNECT_REQUEST case via trans->cm_handle_connect(), and (with a connection context) in the ROUTE_RESOLVED and ESTABLISHED cases.
An RDS listener on an iWARP device therefore crashes the kernel as soon as a connect request arrives: with CONFIG_INIT_STACK_ALL_ZERO the wild load becomes a NULL dereference at offset 0xa0 (&trans->cm_handle_connect) in the iw_cm_wq workqueue.
GCC masks the bug in default builds by folding the uninitialized load into &rds_ib_transport; Clang-built kernels take the real uninitialized path and oops.
The iWARP transport was dropped long ago and IB is the only transport left, so make that explicit: initialize trans to &rds_ib_transport at declaration, drop the conditional assignment, and reject events from non-IB devices before the event switch.
---
v2:
- Enforce the IB transport directly instead of NULL-guarding trans,
as suggested by Allison Henderson.
v1: https://lore.kernel.org/netdev/[email protected]
net/rds/rdma_transport.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
net/rds/rdma_transport.c +7 -4
@@ -52,7 +52,7 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, {/* this can be null in the listening path */ struct rds_connection *conn = cm_id->context; -struct rds_transport *trans; +struct rds_transport *trans = &rds_ib_transport;int ret =0 ;int *err;u8 len;
@@ -60,9 +60,6 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id,rdsdebug ("conn %p id %p handling event %u (%s)\n" , conn, cm_id, event->event,rdma_event_msg (event->event)); -if (cm_id->device->node_type ==RDMA_NODE_IB_CA ) - trans = &rds_ib_transport; -/* Prevent shutdown from tearing down the connection * while we're executing. */ if (conn) {
@@ -80,6 +77,12 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, } } +/* Only the IB transport is supported. */ +if (cm_id->device->node_type !=RDMA_NODE_IB_CA ) { + ret =1 ; + goto out; + } +switch (event->event) {case RDMA_CM_EVENT_CONNECT_REQUEST : ret = trans->cm_handle_connect (cm_id, event, isv6);
Signature · 1 line
2.43.7
Quoted 24 lines · Aohan Mei
From: Aohan Mei <[email protected]>
rds_rdma_cm_event_handler_cmn()assigns trans only when the RDMA device is an InfiniBand CA (RDMA_NODE_IB_CA). On any other device type, e.g. an iWARP RNIC such as siw, trans stays uninitialized, but the event switch dereferences it: unconditionally in theRDMA_CM_EVENT_CONNECT_REQUESTcase viatrans->cm_handle_connect(), and (with a connection context) in theROUTE_RESOLVEDand ESTABLISHED cases.An RDS listener on an iWARP device therefore crashes the kernel as soon as a connect request arrives: with
CONFIG_INIT_STACK_ALL_ZEROthe wild load becomes a NULL dereference at offset0xa0(&trans->cm_handle_connect) in theiw_cm_wqworkqueue.GCC masks the bug in default builds by folding the uninitialized load into
&rds_ib_transport; Clang-built kernels take the real uninitialized path and oops.The iWARP transport was dropped long ago and IB is the only transport left, so make that explicit: initialize trans to
&rds_ib_transportat declaration, drop the conditional assignment, and reject events from non-IB devices before the event switch.
Quoted 5 lines
Fixes dcdede0406d3 ("RDS: Drop stale iWARP RDMA transport")Reported-by TencentOS Corvus AI <[email protected]>Assisted-by CodeBuddy:Kimi-K3Signed-off-by Aohan Mei <[email protected]>
This looks good to me. Thanks for the quick response.
Quoted 44 lines
---
v2:
- Enforce the IB transport directly instead of NULL-guarding trans,
as suggested by Allison Henderson.
v1: https://lore.kernel.org/netdev/[email protected]net/rds/rdma_transport.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)net/rds/rdma_transport.c +7
-4@@ -52,7 +52,7 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, {/* this can be null in the listening path */ struct rds_connection *conn = cm_id->context; -struct rds_transport *trans; +struct rds_transport *trans = &rds_ib_transport;int ret =0 ;int *err;u8 len;@@ -60,9 +60,6 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id,rdsdebug ("conn %p id %p handling event %u (%s)\n" , conn, cm_id, event->event,rdma_event_msg (event->event)); -if (cm_id->device->node_type ==RDMA_NODE_IB_CA ) - trans = &rds_ib_transport; -/* Prevent shutdown from tearing down the connection * while we're executing. */ if (conn) {@@ -80,6 +77,12 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, } } +/* Only the IB transport is supported. */ +if (cm_id->device->node_type !=RDMA_NODE_IB_CA ) { + ret =1 ; + goto out; + } +switch (event->event) {case RDMA_CM_EVENT_CONNECT_REQUEST : ret = trans->cm_handle_connect (cm_id, event, isv6);
Jakub Kicinski
maintainerThis is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
---
net: rds: fix uninitialized trans dereference in CM event handler
This initializes trans to &rds_ib_transport at declaration in rds_rdma_cm_event_handler_cmn(), drops the conditional assignment based on cm_id->device->node_type, and adds an early bail-out that returns 1 for any event arriving on a device that is not RDMA_NODE_IB_CA.
Quoted 4 lines
net/rds/rdma_transport.c +0
-0
[ ... ]
net/rds/rdma_transport.c static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, · line 85
Quoted 10 lines
net/rds/rdma_transport.c +6
-0@@ -80,6 +77,12 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, } } +/* Only the IB transport is supported. */ +if (cm_id->device->node_type !=RDMA_NODE_IB_CA ) { + ret =1 ; + goto out; + } +
[Severity: High]
Should this non-zero return be limited to RDMA_CM_EVENT_CONNECT_REQUEST? As written it applies to every event type.
include/rdma/rdma_cm.h documents what a non-zero return means:
/**
* rdma_cm_event_handler - Callback used to report user events. *
* Notes: Users may not call rdma_destroy_id from this callback to destroy
* the passed in id, or a corresponding listen id. Returning a
* non-zero value from the callback will destroy the passed in id.
*/
The cma dispatchers implement exactly that, for example addr_handler() and cma_work_handler() in drivers/infiniband/core/cma.c:
if (cma_cm_event_handler(id_priv, &event)) {
destroy_id_handler_unlock(id_priv);
return;
}But RDS keeps ownership of connection cm_ids. rds_ib_conn_path_connect() stores it:
ic->i_cm_id = rdma_create_id(&init_net, handler, conn, RDMA_PS_TCP, IB_QPT_RC);
and rds_ib_conn_path_shutdown() later dereferences and destroys the same pointer, clearing it only afterwards:
if (ic->i_cm_id) {
rdsdebug("disconnecting cm %p\n", ic->i_cm_id);
err = rdma_disconnect(ic->i_cm_id);
...
rdma_destroy_id(ic->i_cm_id);
...
ic->i_cm_id = NULL;So if any event other than CONNECT_REQUEST is delivered with conn != NULL on a cm_id whose device node_type is not RDMA_NODE_IB_CA, does the core free the cm_id while ic->i_cm_id still points at it? Can the following conn drop, socket close or module unload then hit freed memory in rds_ib_conn_path_shutdown(), including a second rdma_destroy_id() on the same id?
A reachable path without any race: rds_ib_laddr_check() short-circuits via rds_ib_get_device(), which only walks the ipaddr_list of RDS's own IB devices:
net/rds/ib_rdma.c:rds_ib_get_device() {
list_for_each_entry_rcu(rds_ibdev, &rds_ib_devices, list) {
list_for_each_entry_rcu(i_ipaddr, &rds_ibdev->ipaddr_list, list) {
if (i_ipaddr->ipaddr == ipaddr) {
...
}while the device actually bound for the outgoing connection is picked later by cma_acquire_dev_by_src_ip(), which walks the global cma dev_list and can match a siw device on the same address. siw_device_create() sets:
base_dev->node_type = RDMA_NODE_RNIC;
In that configuration, rdma_resolve_addr() from rds_ib_conn_path_connect() binds a RDMA_NODE_RNIC device, and the resulting RDMA_CM_EVENT_ADDR_RESOLVED with conn != NULL now returns 1, so addr_handler() destroys the id while RDS still tracks it.
The same function a few lines above already documents the opposite convention:
/* If the connection is being shut down, bail out
* right away. We return 0 so cm_id doesn't get
* destroyed prematurely */
if (rds_conn_state(conn) == RDS_CONN_DISCONNECTING) {
/* Reject incoming connections while we're tearing
* down an existing one. */
if (event->event == RDMA_CM_EVENT_CONNECT_REQUEST)
ret = 1;
goto out;
}Separately, does the unconditional goto out also skip the rds_conn_drop() that the ADDR_ERROR, ROUTE_ERROR, CONNECT_ERROR, UNREACHABLE, DEVICE_REMOVAL, ADDR_CHANGE, DISCONNECTED and TIMEWAIT_EXIT cases perform, leaving the RDS connection stuck in CONNECTING or UP with no error propagated?
Would something along these lines match the existing convention, while still dropping the conn for the remaining events:
ret = (event->event == RDMA_CM_EVENT_CONNECT_REQUEST) ? 1 : 0;
or moving the node_type test into the RDMA_CM_EVENT_CONNECT_REQUEST case?
Allison Henderson
maintainerQuoted 116 lines · Jakub Kicinski
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
---
net: rds: fix uninitialized trans dereference in CM event handlerThis initializes trans to
&rds_ib_transportat declaration inrds_rdma_cm_event_handler_cmn(), drops the conditional assignment based oncm_id->device->node_type, and adds an early bail-out that returns 1 for any event arriving on a device that is notRDMA_NODE_IB_CA.Quoted 4 lines
net/rds/rdma_transport.c +0
-0[ ... ]
net/rds/rdma_transport.c static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, · line 85
Quoted 10 lines
net/rds/rdma_transport.c +6
-0@@ -80,6 +77,12 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, } } +/* Only the IB transport is supported. */ +if (cm_id->device->node_type !=RDMA_NODE_IB_CA ) { + ret =1 ; + goto out; + } +[Severity: High]
Should this non-zero return be limited toRDMA_CM_EVENT_CONNECT_REQUEST? As written it applies to every event type.
include/rdma/rdma_cm.hdocuments what a non-zero return means:/**
*rdma_cm_event_handler- Callback used to report user events. *
* Notes: Users may not callrdma_destroy_idfrom this callback to destroy
* the passed in id, or a corresponding listen id. Returning a
* non-zero value from the callback will destroy the passed in id.
*/The cma dispatchers implement exactly that, for example
addr_handler()andcma_work_handler()indrivers/infiniband/core/cma.c:if (cma_cm_event_handler(id_priv, &event)) { destroy_id_handler_unlock(id_priv); return; }But RDS keeps ownership of connection
cm_ids.rds_ib_conn_path_connect()stores it:ic->i_cm_id = rdma_create_id(&init_net, handler, conn, RDMA_PS_TCP, IB_QPT_RC);and
rds_ib_conn_path_shutdown()later dereferences and destroys the same pointer, clearing it only afterwards:if (ic->i_cm_id) { rdsdebug("disconnecting cm %p\n", ic->i_cm_id); err = rdma_disconnect(ic->i_cm_id); ... rdma_destroy_id(ic->i_cm_id); ... ic->i_cm_id = NULL;So if any event other than
CONNECT_REQUESTis delivered with conn != NULL on acm_idwhose devicenode_typeis notRDMA_NODE_IB_CA, does the core free thecm_idwhileic->i_cm_idstill points at it? Can the following conn drop, socket close or module unload then hit freed memory inrds_ib_conn_path_shutdown(), including a secondrdma_destroy_id()on the same id?A reachable path without any race:
rds_ib_laddr_check()short-circuits viards_ib_get_device(), which only walks theipaddr_listof RDS's own IB devices:net/rds/ib_rdma.c:rds_ib_get_device() { list_for_each_entry_rcu(rds_ibdev, &rds_ib_devices, list) { list_for_each_entry_rcu(i_ipaddr, &rds_ibdev->ipaddr_list, list) { if (i_ipaddr->ipaddr == ipaddr) { ... }while the device actually bound for the outgoing connection is picked later by
cma_acquire_dev_by_src_ip(), which walks the global cmadev_listand can match a siw device on the same address.siw_device_create()sets:base_dev->node_type = RDMA_NODE_RNIC;In that configuration,
rdma_resolve_addr()fromrds_ib_conn_path_connect()binds aRDMA_NODE_RNICdevice, and the resultingRDMA_CM_EVENT_ADDR_RESOLVEDwith conn != NULL now returns 1, soaddr_handler()destroys the id while RDS still tracks it.The same function a few lines above already documents the opposite convention:
/* If the connection is being shut down, bail out * right away. We return 0 so cm_id doesn't get * destroyed prematurely */ if (rds_conn_state(conn) == RDS_CONN_DISCONNECTING) { /* Reject incoming connections while we're tearing * down an existing one. */ if (event->event == RDMA_CM_EVENT_CONNECT_REQUEST) ret = 1; goto out; }Separately, does the unconditional goto out also skip the
rds_conn_drop()that theADDR_ERROR,ROUTE_ERROR,CONNECT_ERROR, UNREACHABLE,DEVICE_REMOVAL,ADDR_CHANGE, DISCONNECTED andTIMEWAIT_EXITcases perform, leaving the RDS connection stuck in CONNECTING or UP with no error propagated?
Quoted 6 lines
Would something along these lines match the existing convention, while still dropping the conn for the remaining events:
ret = (event->event == RDMA_CM_EVENT_CONNECT_REQUEST) ? 1 : 0;or moving the
node_typetest into theRDMA_CM_EVENT_CONNECT_REQUESTcase?
I think the cleaner solution here would be to hoist the check into the calling function. So in rds_rdma_cm_event_handler_cmn():
case RDMA_CM_EVENT_CONNECT_REQUEST:
/* Only the IB transport is supported. */
if (cm_id->device->node_type == RDMA_NODE_IB_CA)
ret = trans->cm_handle_connect(cm_id, event, isv6);
else
ret = 1;
break;Thanks for working on this!
Allison
Allison Henderson
maintainerQuoted 135 lines · Allison Henderson
Quoted 116 lines · Jakub Kicinski
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
---
net: rds: fix uninitialized trans dereference in CM event handlerThis initializes trans to
&rds_ib_transportat declaration inrds_rdma_cm_event_handler_cmn(), drops the conditional assignment based oncm_id->device->node_type, and adds an early bail-out that returns 1 for any event arriving on a device that is notRDMA_NODE_IB_CA.Quoted 4 lines
net/rds/rdma_transport.c +0
-0[ ... ]
net/rds/rdma_transport.c static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, · line 85
Quoted 10 lines
net/rds/rdma_transport.c +6
-0@@ -80,6 +77,12 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, } } +/* Only the IB transport is supported. */ +if (cm_id->device->node_type !=RDMA_NODE_IB_CA ) { + ret =1 ; + goto out; + } +[Severity: High]
Should this non-zero return be limited toRDMA_CM_EVENT_CONNECT_REQUEST? As written it applies to every event type.
include/rdma/rdma_cm.hdocuments what a non-zero return means:/**
*rdma_cm_event_handler- Callback used to report user events. *
* Notes: Users may not callrdma_destroy_idfrom this callback to destroy
* the passed in id, or a corresponding listen id. Returning a
* non-zero value from the callback will destroy the passed in id.
*/The cma dispatchers implement exactly that, for example
addr_handler()andcma_work_handler()indrivers/infiniband/core/cma.c:if (cma_cm_event_handler(id_priv, &event)) { destroy_id_handler_unlock(id_priv); return; }But RDS keeps ownership of connection
cm_ids.rds_ib_conn_path_connect()stores it:ic->i_cm_id = rdma_create_id(&init_net, handler, conn, RDMA_PS_TCP, IB_QPT_RC);and
rds_ib_conn_path_shutdown()later dereferences and destroys the same pointer, clearing it only afterwards:if (ic->i_cm_id) { rdsdebug("disconnecting cm %p\n", ic->i_cm_id); err = rdma_disconnect(ic->i_cm_id); ... rdma_destroy_id(ic->i_cm_id); ... ic->i_cm_id = NULL;So if any event other than
CONNECT_REQUESTis delivered with conn != NULL on acm_idwhose devicenode_typeis notRDMA_NODE_IB_CA, does the core free thecm_idwhileic->i_cm_idstill points at it? Can the following conn drop, socket close or module unload then hit freed memory inrds_ib_conn_path_shutdown(), including a secondrdma_destroy_id()on the same id?A reachable path without any race:
rds_ib_laddr_check()short-circuits viards_ib_get_device(), which only walks theipaddr_listof RDS's own IB devices:net/rds/ib_rdma.c:rds_ib_get_device() { list_for_each_entry_rcu(rds_ibdev, &rds_ib_devices, list) { list_for_each_entry_rcu(i_ipaddr, &rds_ibdev->ipaddr_list, list) { if (i_ipaddr->ipaddr == ipaddr) { ... }while the device actually bound for the outgoing connection is picked later by
cma_acquire_dev_by_src_ip(), which walks the global cmadev_listand can match a siw device on the same address.siw_device_create()sets:base_dev->node_type = RDMA_NODE_RNIC;In that configuration,
rdma_resolve_addr()fromrds_ib_conn_path_connect()binds aRDMA_NODE_RNICdevice, and the resultingRDMA_CM_EVENT_ADDR_RESOLVEDwith conn != NULL now returns 1, soaddr_handler()destroys the id while RDS still tracks it.The same function a few lines above already documents the opposite convention:
/* If the connection is being shut down, bail out * right away. We return 0 so cm_id doesn't get * destroyed prematurely */ if (rds_conn_state(conn) == RDS_CONN_DISCONNECTING) { /* Reject incoming connections while we're tearing * down an existing one. */ if (event->event == RDMA_CM_EVENT_CONNECT_REQUEST) ret = 1; goto out; }Separately, does the unconditional goto out also skip the
rds_conn_drop()that theADDR_ERROR,ROUTE_ERROR,CONNECT_ERROR, UNREACHABLE,DEVICE_REMOVAL,ADDR_CHANGE, DISCONNECTED andTIMEWAIT_EXITcases perform, leaving the RDS connection stuck in CONNECTING or UP with no error propagated?Quoted 6 lines
Would something along these lines match the existing convention, while still dropping the conn for the remaining events:
ret = (event->event == RDMA_CM_EVENT_CONNECT_REQUEST) ? 1 : 0;or moving the
node_typetest into theRDMA_CM_EVENT_CONNECT_REQUESTcase?I think the cleaner solution here would be to hoist the check into the calling function. So in
rds_rdma_cm_event_handler_cmn():case RDMA_CM_EVENT_CONNECT_REQUEST: /* Only the IB transport is supported. */ if (cm_id->device->node_type == RDMA_NODE_IB_CA) ret = trans->cm_handle_connect(cm_id, event, isv6); else ret = 1; break;
Quoted 3 lines
Thanks for working on this!
Allison
Hi Aohan,
I know this thread is a bit old, but this bug popped up as a pre-existing bug for another set review. Did you have an interest in sending a v3? I am fine to carry it forward if you're on to other things, but I wanted to check with you first. I'd preserve all your tags, sob, and reported-by. Just let me know, I'm fine either way.
Thanks!
Allison