RDMA/core: Reject CQE counts above max_cqe in ib_cq_pool_get()
PATCH 1/1 RDMA/core: Reject CQE counts above max_cqe in ib_cq_pool_get()
ib_cq_pool_get() does not validate nr_cqe against the device limit, while ib_alloc_cqs() caps the size passed to ib_alloc_cq() at dev->attrs.max_cqe:
nr_cqes = min(dev->attrs.max_cqe, max(nr_cqes, IB_MAX_SHARED_CQ_SZ));
If nr_cqe is larger than max_cqe, ib_alloc_cqs() cannot ask the device for that many entries. A device that reports the size it was asked for therefore returns a CQ smaller than nr_cqe, and the fit test skips every CQ in the pool:
if (cq->cqe_used + nr_cqe > cq->cqe) continue;
'found' remains NULL and each iteration calls ib_alloc_cqs() again, adding another batch of CQs to dev->cq_pools[]. The CQs stay in the pool, so each walk under cq_pools_lock gets longer. The loop ends only when an allocation fails, i.e. once the device or the system has run out of resources.
ib_srpt can reach this path when a privileged user configures srp_sq_size through configfs. ib_srpt accepts values up to 65535 and requests ch->rq_size + sq_size CQEs while establishing a connection. If that sum exceeds max_cqe, a valid connection request from a remote initiator can trigger the allocation loop.
Reproduced with ib_srpt over rxe, which reports max_cqe = 32767, after setting srp_sq_size to 65535. In a 1 GB guest, a login attempt requesting 65663 CQEs caused 115 allocation rounds in 185 ms, followed by:
Out of memory and no killable processes...
Kernel panic - not syncing: System is deadlocked on memory Workqueue: ib_cm cm_work_handler Call Trace: __vmalloc_node_range_noprof vmalloc_user_noprof rxe_queue_init rxe_cq_from_init rxe_create_cq __ib_alloc_cq ib_cq_pool_get srpt_cm_req_recv.cold srpt_rdma_cm_req_recv cma_cm_event_handler cma_ib_req_handler cm_process_work cm_work_handler
Reject oversized requests before entering the allocation loop. With the check in place, the same test allocates no CQs and ib_srpt rejects the login:
ib_srpt failed to create CQ cqe= 65663 ret= -EINVAL
Requests for max_cqe entries or fewer behave as before.
--- drivers/infiniband/core/cq.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/infiniband/core/cq.c +7 -0
@@ -449,6 +449,13 @@ struct ib_cq *ib_cq_pool_get(struct ib_device *dev, unsigned int nr_cqe,return ERR_PTR (-EINVAL ); } +/* +* ib_alloc_cqs() caps CQ size at max_cqe, so a larger request would +* keep allocating CQs that never fit until allocation fails. +*/ +if (nr_cqe > dev->attrs.max_cqe) +return ERR_PTR (-EINVAL ); + num_comp_vectors = min_t(unsigned int, dev->num_comp_vectors, num_online_cpus());/* Project the affinty to the device completion vector range */
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
Signature · 1 line
2.53.0
Quoted 49 lines · Serhat Kumral
ib_cq_pool_get()does not validatenr_cqeagainst the device limit, whileib_alloc_cqs()caps the size passed toib_alloc_cq()atdev->attrs.max_cqe:nr_cqes = min(dev->attrs.max_cqe, max(nr_cqes, IB_MAX_SHARED_CQ_SZ));If
nr_cqeis larger thanmax_cqe,ib_alloc_cqs()cannot ask the device for that many entries. A device that reports the size it was asked for therefore returns a CQ smaller thannr_cqe, and the fit test skips every CQ in the pool:if (cq->cqe_used + nr_cqe > cq->cqe) continue;'found' remains NULL and each iteration calls
ib_alloc_cqs()again, adding another batch of CQs to dev->cq_pools[]. The CQs stay in the pool, so each walk undercq_pools_lockgets longer. The loop ends only when an allocation fails, i.e. once the device or the system has run out of resources.
ib_srptcan reach this path when a privileged user configuressrp_sq_sizethrough configfs.ib_srptaccepts values up to 65535 and requestsch->rq_size+sq_sizeCQEs while establishing a connection. If that sum exceedsmax_cqe, a valid connection request from a remote initiator can trigger the allocation loop.Reproduced with
ib_srptover rxe, which reportsmax_cqe= 32767, after settingsrp_sq_sizeto 65535. In a 1 GB guest, a login attempt requesting 65663 CQEs caused 115 allocation rounds in 185 ms, followed by:Out of memory and no killable processes...
Kernel panic - not syncing: System is deadlocked on memory Workqueue: ib_cm cm_work_handler Call Trace: __vmalloc_node_range_noprof vmalloc_user_noprof rxe_queue_init rxe_cq_from_init rxe_create_cq __ib_alloc_cq ib_cq_pool_get srpt_cm_req_recv.cold srpt_rdma_cm_req_recv cma_cm_event_handler cma_ib_req_handler cm_process_work cm_work_handler
Quoted 7 lines
Reject oversized requests before entering the allocation loop. With the check in place, the same test allocates no CQs and
ib_srptrejects the login:
ib_srptfailed to create CQ cqe= 65663 ret=-EINVALRequests for
max_cqeentries or fewer behave as before.
Please fix ib_srpt to honor device capabilities.
Thanks
Quoted 30 lines
Fixes c7ff819aefea ("RDMA/core: Introduce shared CQ pool API")Signed-off-by Serhat Kumral <[email protected]>--- drivers/infiniband/core/cq.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/infiniband/core/cq.c +7
-0@@ -449,6 +449,13 @@ struct ib_cq *ib_cq_pool_get(struct ib_device *dev, unsigned int nr_cqe,return ERR_PTR (-EINVAL ); } +/* +* ib_alloc_cqs() caps CQ size at max_cqe, so a larger request would +* keep allocating CQs that never fit until allocation fails. +*/ +if (nr_cqe > dev->attrs.max_cqe) +return ERR_PTR (-EINVAL ); + num_comp_vectors = min_t(unsigned int, dev->num_comp_vectors, num_online_cpus());/* Project the affinty to the device completion vector range */ base-commit:
cee9395acd8043be0644b25c34bfa86623f2b935Signature · 1 line
2.53.0