net/smc: transition to RDMA core CQ pooling
Subsystems (5)
D. Wythe
maintainerPATCH v3 1/1 net/smc: transition to RDMA core CQ pooling
The current SMC-R implementation relies on global per-device CQs and manual polling within tasklets, which introduces severe scalability bottlenecks due to global lock contention and tasklet scheduling overhead, resulting in poor performance as concurrency increases.
Refactor the completion handling to utilize the ib_cqe API and standard RDMA core CQ pooling. This transition provides several key advantages:
1. Multi-CQ: Shift from a single shared per-device CQ to multiple link-specific CQs via the CQ pool. This allows completion processing to be parallelized across multiple CPU cores, effectively eliminating the global CQ bottleneck.
2. Leverage DIM: Utilizing the standard CQ pool with IB_POLL_SOFTIRQ enables Dynamic Interrupt Moderation from the RDMA core, optimizing interrupt frequency and reducing CPU load under high pressure.
3. O(1) Context Retrieval: Replaces the expensive wr_id based lookup logic (e.g., smc_wr_tx_find_pending_index) with direct context retrieval using container_of() on the embedded ib_cqe.
4. Code Simplification: This refactoring results in a reduction of ~150 lines of code. It removes redundant sequence tracking, complex lookup helpers, and manual CQ management, significantly improving maintainability.
Performance Test: redis-benchmark with max 32 connections per QP Data format: Requests Per Second (RPS), Percentage in brackets represents the gain/loss compared to TCP.
| Clients | TCP | SMC (original) | SMC (cq_pool) | |---------|----------|---------------------|---------------------| | c = 1 | 24449 | 31172 (+27%) | 34039 (+39%) | | c = 2 | 46420 | 53216 (+14%) | 64391 (+38%) | | c = 16 | 159673 | 83668 (-48%) <-- | 216947 (+36%) | | c = 32 | 164956 | 97631 (-41%) <-- | 249376 (+51%) | | c = 64 | 166322 | 118192 (-29%) <-- | 249488 (+50%) | | c = 128 | 167700 | 121497 (-27%) <-- | 249480 (+48%) | | c = 256 | 175021 | 146109 (-16%) <-- | 240384 (+37%) | | c = 512 | 168987 | 101479 (-40%) <-- | 226634 (+34%) |
The results demonstrate that this optimization effectively resolves the scalability bottleneck, with RPS increasing by over 110% at c=64 compared to the original implementation.
---
v3:
- Rebase to latest net-next tree.
- Remove a redundant blank line in smc_wr_alloc_link_mem().
v2:
- Fix a logic bug in smc_wr_tx_process_cqe() where a zeroed field was checked instead of the saved pnd_snd copy. (Jakub)
- Fix typo in comment: s/ib_draib_rq/ib_drain_rq/.
- Minor comment alignment fix in struct smc_link.
--- net/smc/smc_core.c | 9 +- net/smc/smc_core.h | 28 ++-- net/smc/smc_ib.c | 113 +++++----------- net/smc/smc_ib.h | 7 - net/smc/smc_tx.c | 1 - net/smc/smc_wr.c | 312 +++++++++++++++++++-------------------------- net/smc/smc_wr.h | 40 ++---- 7 files changed, 193 insertions(+), 317 deletions(-)
net/smc/smc_core.c +2 -7
@@ -815,17 +815,11 @@ int smcr_link_init(struct smc_link_group *lgr, struct smc_link *lnk, lnk->lgr = lgr;smc_lgr_hold (lgr);/* lgr_put in smcr_link_clear() */ lnk->link_idx = link_idx; - lnk->wr_rx_id_compl =0 ;smc_ibdev_cnt_inc (lnk);smcr_copy_dev_info_to_link (lnk);atomic_set (&lnk->conn_cnt,0 );smc_llc_link_set_uid (lnk);INIT_WORK (&lnk->link_down_wrk, smc_link_down_work); -if (!lnk->smcibdev->initialized) { - rc = (int )smc_ib_setup_per_ibdev (lnk->smcibdev); -if (rc) - goto out; - }get_random_bytes (rndvec,sizeof (rndvec)); lnk->psn_initial = rndvec[0 ] + (rndvec[1 ] <<8 ) + (rndvec[2 ] <<16 );
@@ -863,6 +857,7 @@ int smcr_link_init(struct smc_link_group *lgr, struct smc_link *lnk,if (rc) goto free_link_mem; lnk->state =SMC_LNK_ACTIVATING ; +smc_wr_init_cqes (lnk);return 0 ; free_link_mem:
@@ -1373,7 +1368,7 @@ void smcr_link_clear(struct smc_link *lnk, bool log)smc_llc_link_clear (lnk, log);smcr_buf_unmap_lgr (lnk);smcr_rtoken_clear_link (lnk); -smc_ib_modify_qp_error (lnk); +smc_wr_drain_rq (lnk);smc_wr_free_link (lnk);smc_ib_destroy_queue_pair (lnk);smc_ib_dealloc_protection_domain (lnk);
net/smc/smc_core.h +20 -8
@@ -89,8 +89,21 @@ struct smc_rdma_sges { /* sges per message send */struct smc_rdma_wr {/* work requests per message * send */ +struct ib_cqe cqe;struct ib_rdma_wr wr_tx_rdma[SMC_MAX_RDMA_WRITES ]; -}; +}____cacheline_aligned_in_smp ; + +struct smc_ib_recv_wr { +struct ib_cqe cqe; +struct ib_recv_wr wr; +int idx; +}____cacheline_aligned_in_smp ; + +struct smc_ib_send_wr { +struct ib_cqe cqe; +struct ib_send_wr wr; +int idx; +}____cacheline_aligned_in_smp ;#define SMC_LGR_ID_SIZE 4
@@ -100,23 +113,24 @@ struct smc_link {struct ib_pd *roce_pd;/* IB protection domain, * unique for every RoCE QP */ +unsigned int nr_cqe;/* number of CQ entries */ +struct ib_cq *ib_cq;/* IB completion queue */ struct ib_qp *roce_qp;/* IB queue pair */ struct ib_qp_attr qp_attr;/* IB queue pair attributes */ struct smc_wr_buf *wr_tx_bufs;/* WR send payload buffers */ -struct ib_send_wr *wr_tx_ibs;/* WR send meta data */ +struct smc_ib_send_wr *wr_tx_ibs;/* WR send meta data */ struct ib_sge *wr_tx_sges;/* WR send gather meta data */ struct smc_rdma_sges *wr_tx_rdma_sges;/*RDMA WRITE gather meta data*/ struct smc_rdma_wr *wr_tx_rdmas;/* WR RDMA WRITE */ struct smc_wr_tx_pend *wr_tx_pends;/* WR send waiting for CQE */ struct completion *wr_tx_compl;/* WR send CQE completion */ /* above four vectors have wr_tx_cnt elements and use the same index */ -struct ib_send_wr *wr_tx_v2_ib;/* WR send v2 meta data */ +struct smc_ib_send_wr *wr_tx_v2_ib;/* WR send v2 meta data */ struct ib_sge *wr_tx_v2_sge;/* WR send v2 gather meta data*/ struct smc_wr_tx_pend *wr_tx_v2_pend;/* WR send v2 waiting for CQE */ dma_addr_t wr_tx_dma_addr;/* DMA address of wr_tx_bufs */ dma_addr_t wr_tx_v2_dma_addr;/* DMA address of v2 tx buf*/ -atomic_long_t wr_tx_id;/* seq # of last sent WR */ unsigned long *wr_tx_mask;/* bit mask of used indexes */ u32 wr_tx_cnt;/* number of WR send buffers */ wait_queue_head_t wr_tx_wait;/* wait for free WR send buf */
@@ -126,7 +140,7 @@ struct smc_link {struct completion tx_ref_comp;u8 *wr_rx_bufs;/* WR recv payload buffers */ -struct ib_recv_wr *wr_rx_ibs;/* WR recv meta data */ +struct smc_ib_recv_wr *wr_rx_ibs;/* WR recv meta data */ struct ib_sge *wr_rx_sges;/* WR recv scatter meta data */ /* above three vectors have wr_rx_cnt elements and use the same index */ int wr_rx_sge_cnt;/* rx sge, V1 is 1, V2 is either 2 or 1 */
@@ -135,13 +149,11 @@ struct smc_link {*/ dma_addr_t wr_rx_dma_addr;/* DMA address of wr_rx_bufs */ dma_addr_t wr_rx_v2_dma_addr;/* DMA address of v2 rx buf*/ -u64 wr_rx_id;/* seq # of last recv WR */ -u64 wr_rx_id_compl;/* seq # of last completed WR */ u32 wr_rx_cnt;/* number of WR recv buffers */ unsigned long wr_rx_tstamp;/* jiffies when last buf rx */ -wait_queue_head_t wr_rx_empty_wait;/* wait for RQ empty */ struct ib_reg_wr wr_reg;/* WR register memory region */ +struct ib_cqe wr_reg_cqe;/* ib_cqe for wr_reg */ wait_queue_head_t wr_reg_wait;/* wait for wr_reg result */ struct {struct percpu_ref wr_reg_refs;
net/smc/smc_ib.c +32 -81
171 lines not on this page · raw message
net/smc/smc_ib.h +0 -7
@@ -37,17 +37,12 @@ struct smc_ib_device { /* ib-device infos for smc */struct ib_device *ibdev;struct ib_port_attr pattr[SMC_MAX_PORTS ];/* ib dev. port attrs */ struct ib_event_handler event_handler;/* global ib_event handler */ -struct ib_cq *roce_cq_send;/* send completion queue */ -struct ib_cq *roce_cq_recv;/* recv completion queue */ -struct tasklet_struct send_tasklet;/* called by send cq handler */ -struct tasklet_struct recv_tasklet;/* called by recv cq handler */ char mac[SMC_MAX_PORTS ][ETH_ALEN ];/* mac address per port*/ u8 pnetid[SMC_MAX_PORTS ][SMC_MAX_PNETID_LEN ];/* pnetid per port */ bool pnetid_by_user[SMC_MAX_PORTS ];/* pnetid defined by user? */ - u8 initialized : 1; /* ib dev CQ, evthdl done */struct work_struct port_event_work;unsigned long port_event_mask;DECLARE_BITMAP (ports_going_away,SMC_MAX_PORTS );
@@ -96,8 +91,6 @@ void smc_ib_destroy_queue_pair(struct smc_link *lnk);int smc_ib_create_queue_pair (struct smc_link *lnk);int smc_ib_ready_link (struct smc_link *lnk);int smc_ib_modify_qp_rts (struct smc_link *lnk); -int smc_ib_modify_qp_error (struct smc_link *lnk); -long smc_ib_setup_per_ibdev (struct smc_ib_device *smcibdev);int smc_ib_get_memory_region (struct ib_pd *pd,int access_flags,struct smc_buf_desc *buf_slot,u8 link_idx);void smc_ib_put_memory_region (struct ib_mr *mr);
net/smc/smc_tx.c +0 -1
@@ -321,7 +321,6 @@ static int smc_tx_rdma_write(struct smc_connection *conn, int peer_rmbe_offset,struct smc_link *link = conn->lnk;int rc; - rdma_wr->wr.wr_id =smc_wr_tx_get_next_wr_id (link); rdma_wr->wr.num_sge = num_sges; rdma_wr->remote_addr = lgr->rtokens[conn->rtoken_idx][link->link_idx].dma_addr +
net/smc/smc_wr.c +129 -183
490 lines not on this page · raw message
net/smc/smc_wr.h +10 -30
@@ -44,19 +44,6 @@ struct smc_wr_rx_handler {u8 type; }; -/* Only used by RDMA write WRs. -* All other WRs (CDC/LLC) use smc_wr_tx_send handling WR_ID implicitly -*/ -static inline long smc_wr_tx_get_next_wr_id (struct smc_link *link) -{ -return atomic_long_inc_return (&link->wr_tx_id); -} - -static inline void smc_wr_tx_set_wr_id (atomic_long_t *wr_tx_id,long val) -{ -atomic_long_set (wr_tx_id, val); -} -static inline bool smc_wr_tx_link_hold (struct smc_link *link) {if (!smc_link_sendable (link))
@@ -70,9 +57,10 @@ static inline void smc_wr_tx_link_put(struct smc_link *link)percpu_ref_put (&link->wr_tx_refs); } -static inline void smc_wr_drain_cq (struct smc_link *lnk) +static inline void smc_wr_drain_rq (struct smc_link *lnk) { -wait_event (lnk->wr_rx_empty_wait, lnk->wr_rx_id_compl == lnk->wr_rx_id); +if (lnk->qp_attr.cur_qp_state !=IB_QPS_RESET ) +ib_drain_rq (lnk->roce_qp); }static inline void smc_wr_wakeup_tx_wait (struct smc_link *lnk)
@@ -86,18 +74,12 @@ static inline void smc_wr_wakeup_reg_wait(struct smc_link *lnk) }/* post a new receive work request to fill a completed old work request entry */ -static inline int smc_wr_rx_post (struct smc_link *link) +static inline int smc_wr_rx_post (struct smc_link *link,struct ib_cqe *cqe) { -int rc; -u64 wr_id, temp_wr_id; -u32 index; - - wr_id = ++link->wr_rx_id;/* tasklet context, thus not atomic */ - temp_wr_id = wr_id; - index =do_div (temp_wr_id, link->wr_rx_cnt); - link->wr_rx_ibs[index].wr_id = wr_id; - rc =ib_post_recv (link->roce_qp, &link->wr_rx_ibs[index],NULL ); -return rc; +struct smc_ib_recv_wr *recv_wr; + + recv_wr = container_of(cqe, struct smc_ib_recv_wr, cqe); +return ib_post_recv (link->roce_qp, &recv_wr->wr,NULL ); }int smc_wr_create_link (struct smc_link *lnk);
@@ -107,8 +89,6 @@ void smc_wr_free_link(struct smc_link *lnk);void smc_wr_free_link_mem (struct smc_link *lnk);void smc_wr_free_lgr_mem (struct smc_link_group *lgr);void smc_wr_remember_qp_attr (struct smc_link *lnk); -void smc_wr_remove_dev (struct smc_ib_device *smcibdev); -void smc_wr_add_dev (struct smc_ib_device *smcibdev); int smc_wr_tx_get_free_slot(struct smc_link *link, smc_wr_tx_handler handler, struct smc_wr_buf **wr_buf,
@@ -126,12 +106,12 @@ int smc_wr_tx_v2_send(struct smc_link *link, struct smc_wr_tx_pend_priv *priv, int len);int smc_wr_tx_send_wait (struct smc_link *link,struct smc_wr_tx_pend_priv *priv,unsigned long timeout); -void smc_wr_tx_cq_handler (struct ib_cq *ib_cq,void *cq_context);void smc_wr_tx_wait_no_pending_sends (struct smc_link *link);int smc_wr_rx_register_handler (struct smc_wr_rx_handler *handler);int smc_wr_rx_post_init (struct smc_link *link); -void smc_wr_rx_cq_handler (struct ib_cq *ib_cq,void *cq_context);int smc_wr_reg_send (struct smc_link *link,struct ib_mr *mr); +void smc_wr_init_cqes (struct smc_link *lnk); +#endif /* SMC_WR_H */
Signature · 1 line
2.45.0
Quoted 57 lines · D. Wythe
The current SMC-R implementation relies on global per-device CQs and manual polling within tasklets, which introduces severe scalability bottlenecks due to global lock contention and tasklet scheduling overhead, resulting in poor performance as concurrency increases.
Refactor the completion handling to utilize the
ib_cqeAPI and standard RDMA core CQ pooling. This transition provides several key advantages:1. Multi-CQ: Shift from a single shared per-device CQ to multiple link-specific CQs via the CQ pool. This allows completion processing to be parallelized across multiple CPU cores, effectively eliminating the global CQ bottleneck.
2. Leverage DIM: Utilizing the standard CQ pool with
IB_POLL_SOFTIRQenables Dynamic Interrupt Moderation from the RDMA core, optimizing interrupt frequency and reducing CPU load under high pressure.3. O(1) Context Retrieval: Replaces the expensive
wr_idbased lookup logic (e.g.,smc_wr_tx_find_pending_index) with direct context retrieval usingcontainer_of()on the embeddedib_cqe.4. Code Simplification: This refactoring results in a reduction of ~150 lines of code. It removes redundant sequence tracking, complex lookup helpers, and manual CQ management, significantly improving maintainability.
Performance Test: redis-benchmark with max 32 connections per QP Data format: Requests Per Second (RPS), Percentage in brackets represents the gain/loss compared to TCP.
| Clients | TCP | SMC (original) | SMC (cq_pool) | |---------|----------|---------------------|---------------------| | c = 1 | 24449 | 31172 (+27%) | 34039 (+39%) | | c = 2 | 46420 | 53216 (+14%) | 64391 (+38%) | | c = 16 | 159673 | 83668 (-48%) <-- | 216947 (+36%) | | c = 32 | 164956 | 97631 (-41%) <-- | 249376 (+51%) | | c = 64 | 166322 | 118192 (-29%) <-- | 249488 (+50%) | | c = 128 | 167700 | 121497 (-27%) <-- | 249480 (+48%) | | c = 256 | 175021 | 146109 (-16%) <-- | 240384 (+37%) | | c = 512 | 168987 | 101479 (-40%) <-- | 226634 (+34%) |The results demonstrate that this optimization effectively resolves the scalability bottleneck, with RPS increasing by over 110% at c=64 compared to the original implementation.
Signed-off-by D. Wythe <[email protected]>---
v3:
- Rebase to latest net-next tree.
- Remove a redundant blank line insmc_wr_alloc_link_mem().v2:
- Fix a logic bug insmc_wr_tx_process_cqe()where a zeroed field was checked instead of the savedpnd_sndcopy. (Jakub)
- Fix typo in comment: s/ib_draib_rq/ib_drain_rq/.
- Minor comment alignment fix in structsmc_link.
Quoted 9 lines
--- net/smc/smc_core.c | 9 +- net/smc/smc_core.h | 28 ++-- net/smc/smc_ib.c | 113 +++++----------- net/smc/smc_ib.h | 7 - net/smc/smc_tx.c | 1 - net/smc/smc_wr.c | 312 +++++++++++++++++++-------------------------- net/smc/smc_wr.h | 40 ++---- 7 files changed, 193 insertions(+), 317 deletions(-)
Thanks a lot for this important conversion, for RDMA API usage:
Mahanta Jambigi
maintainerQuoted 42 lines · D. Wythe
The current SMC-R implementation relies on global per-device CQs and manual polling within tasklets, which introduces severe scalability bottlenecks due to global lock contention and tasklet scheduling overhead, resulting in poor performance as concurrency increases.
Refactor the completion handling to utilize the
ib_cqeAPI and standard RDMA core CQ pooling. This transition provides several key advantages:1. Multi-CQ: Shift from a single shared per-device CQ to multiple link-specific CQs via the CQ pool. This allows completion processing to be parallelized across multiple CPU cores, effectively eliminating the global CQ bottleneck.
2. Leverage DIM: Utilizing the standard CQ pool with
IB_POLL_SOFTIRQenables Dynamic Interrupt Moderation from the RDMA core, optimizing interrupt frequency and reducing CPU load under high pressure.3. O(1) Context Retrieval: Replaces the expensive
wr_idbased lookup logic (e.g.,smc_wr_tx_find_pending_index) with direct context retrieval usingcontainer_of()on the embeddedib_cqe.4. Code Simplification: This refactoring results in a reduction of ~150 lines of code. It removes redundant sequence tracking, complex lookup helpers, and manual CQ management, significantly improving maintainability.
Performance Test: redis-benchmark with max 32 connections per QP Data format: Requests Per Second (RPS), Percentage in brackets represents the gain/loss compared to TCP.
| Clients | TCP | SMC (original) | SMC (cq_pool) | |---------|----------|---------------------|---------------------| | c = 1 | 24449 | 31172 (+27%) | 34039 (+39%) | | c = 2 | 46420 | 53216 (+14%) | 64391 (+38%) | | c = 16 | 159673 | 83668 (-48%) <-- | 216947 (+36%) | | c = 32 | 164956 | 97631 (-41%) <-- | 249376 (+51%) | | c = 64 | 166322 | 118192 (-29%) <-- | 249488 (+50%) | | c = 128 | 167700 | 121497 (-27%) <-- | 249480 (+48%) | | c = 256 | 175021 | 146109 (-16%) <-- | 240384 (+37%) | | c = 512 | 168987 | 101479 (-40%) <-- | 226634 (+34%) |
Quoted 3 lines
The results demonstrate that this optimization effectively resolves the scalability bottleneck, with RPS increasing by over 110% at c=64 compared to the original implementation.
Since your performance results look really-really nice on x86 but ours show severe degradations on s390x, one way forward could be adding the cq_poll mechanism but also keeping the existing mechanism for now (because the things are right now it works better on s390x) and making it either runtime or compile time configurable which of the both is going to be used.
Alternatively, we could work together making the cq_poll mechanism does not introduce a regression to s390x (ideally improve performance for s390x as well). But it that case we would like to have this change deferred until we find a way to make the regression disappear.
I am aware that the first option, co-existence, would kill the simplification aspect of this and instead introduce added complexity. But we are talking about a major regression here on one end, and major improvements on the other end, so it might be still worth it. In any case, we are very motivated to eventually get rid of the old mechanism, provided significant performance regressions can be avoided.
D. Wythe
maintainerQuoted 61 lines · Mahanta Jambigi
Quoted 42 lines · D. Wythe
The current SMC-R implementation relies on global per-device CQs and manual polling within tasklets, which introduces severe scalability bottlenecks due to global lock contention and tasklet scheduling overhead, resulting in poor performance as concurrency increases.
Refactor the completion handling to utilize the
ib_cqeAPI and standard RDMA core CQ pooling. This transition provides several key advantages:1. Multi-CQ: Shift from a single shared per-device CQ to multiple link-specific CQs via the CQ pool. This allows completion processing to be parallelized across multiple CPU cores, effectively eliminating the global CQ bottleneck.
2. Leverage DIM: Utilizing the standard CQ pool with
IB_POLL_SOFTIRQenables Dynamic Interrupt Moderation from the RDMA core, optimizing interrupt frequency and reducing CPU load under high pressure.3. O(1) Context Retrieval: Replaces the expensive
wr_idbased lookup logic (e.g.,smc_wr_tx_find_pending_index) with direct context retrieval usingcontainer_of()on the embeddedib_cqe.4. Code Simplification: This refactoring results in a reduction of ~150 lines of code. It removes redundant sequence tracking, complex lookup helpers, and manual CQ management, significantly improving maintainability.
Performance Test: redis-benchmark with max 32 connections per QP Data format: Requests Per Second (RPS), Percentage in brackets represents the gain/loss compared to TCP.
| Clients | TCP | SMC (original) | SMC (cq_pool) | |---------|----------|---------------------|---------------------| | c = 1 | 24449 | 31172 (+27%) | 34039 (+39%) | | c = 2 | 46420 | 53216 (+14%) | 64391 (+38%) | | c = 16 | 159673 | 83668 (-48%) <-- | 216947 (+36%) | | c = 32 | 164956 | 97631 (-41%) <-- | 249376 (+51%) | | c = 64 | 166322 | 118192 (-29%) <-- | 249488 (+50%) | | c = 128 | 167700 | 121497 (-27%) <-- | 249480 (+48%) | | c = 256 | 175021 | 146109 (-16%) <-- | 240384 (+37%) | | c = 512 | 168987 | 101479 (-40%) <-- | 226634 (+34%) |Quoted 3 lines
The results demonstrate that this optimization effectively resolves the scalability bottleneck, with RPS increasing by over 110% at c=64 compared to the original implementation.
Since your performance results look really-really nice on x86 but ours show severe degradations on s390x, one way forward could be adding the
cq_pollmechanism but also keeping the existing mechanism for now (because the things are right now it works better on s390x) and making it either runtime or compile time configurable which of the both is going to be used.Alternatively, we could work together making the
cq_pollmechanism does not introduce a regression to s390x (ideally improve performance for s390x as well). But it that case we would like to have this change deferred until we find a way to make the regression disappear.
Quoted 6 lines
I am aware that the first option, co-existence, would kill the simplification aspect of this and instead introduce added complexity. But we are talking about a major regression here on one end, and major improvements on the other end, so it might be still worth it. In any case, we are very motivated to eventually get rid of the old mechanism, provided significant performance regressions can be avoided.
I'm in no rush to push this, since a significant performance degradation was observed on s390x, I'll withdraw this patch until the issue is resolved, and it would be great if you could investigate what specifically happened on s390x.
D. Wythe